Big Data and Cloud not a fit? Comments on Infoworld Article
Since Disqus seems to have completely eaten (bleh) my comment on @davidlinthicum’s very interesting InfoWorld post – Big data and the cloud: A far from perfect fit, I decided to just expand my comments and make a short blog post out of it. IMHO the problems that David is describing are more a reflection of problems with batch oriented technologies like Hadoop (more on my take on Hadoop here) in the cloud than a general problem for cloud based big data solutions.
Computing always has, and probably always will have, a bias towards creating batch focused technologies at the beginning of any large paradigm shift. But as new technologies are absorbed, understood, and move from early adopter to more mainstream use, the batch paradigm will inevitably start to shift to streaming and real-time. We have seen this again and again (from punch cards to touch sensitive tablets, downloaded media to streaming media, DOM to SAX parsers, HTML to Ajax, paper maps to real-time GPS). The reason this evolution almost always occurs is simple: humans live and think in real-time and when our tools do as well we are more productive and happier. So why do we have this bias for batch processing in our first generation computational technologies? Simply put, because batch processing is a lot easier.
See You at Strata West—Don’t Miss Our Panel on Big Data and Supply Chain Management!
While Terence and I and the rest of the PatternBuilders team (and they are all a wee bit tired) have been busy working on the launch of our new financial services application, FinancePBI, I had to take a break to talk about our upcoming Strata Conference panel. No, the panel is not about privacy and big data (good guess though and a golden opportunity for me to plug happily away). Instead, we decided to take a look at big data and supply chain management.
Did you know that manufacturers store more data than any other sector? That would be almost 2 exabytes in 2010. And that the data comes from all kinds of systems? Including process control, CAD, CAE, supply chain management, and all the way through the global supply chain to systems that monitor and capture data about the performance of products that have already been sold. Did you know that a Boeing 737 generates 240 terabytes of data on ONE cross-country flight? And all of this data is expected to grow exponentially. Take RFID for example: McKinsey predicts that the number of RFID tags will increase from 12 million in 2011 to 209 billion in 2021. (All stats courtesy of my dear friends at McKinsey in their report, “Big data: The next frontier for innovation, competition, and productivity” but there are also two great videos on this topic at The Tribulations of an Analytical Mind.) Yes folks, welcome to the big data world of SCM! (more…)
FinancePBI Begins its Shakeout Flight in the Cloud
I have been a little quiet on the blogging front recently as I and the rest of the PatternBuilders team have been focused on getting ready to launch our new financial services application: FinancePBI. It is the first cloud-based analytical platform for the Financial Services market. While this is our first public announcement of our entry into the market, behind the scenes we have been gearing the company up for a big splash for several months:
- Partnered with ActiveFinancial one of the premier real-time stock ticker vendors in the world. Look for more data partnerships shortly.
- We have added Doug Jeffrey to our board of advisors and board of directors. Doug is an executive with deep Wall Street and startup expertise who has already done outstanding things in the short time he has been with us.
- We have also partnered with the University of Sydney to use our technology to examine the influence of primary sources (NY Times, etc.) and secondary social media (Twitter, etc.) content on a company’s stock price over a 12 month period. This project will be done exclusively in the cloud and it’s our hope is that we will be able to convince our commercial partners to allow this PatternBuilders instance to be available to the general public. Of course, this would happen after the research is published. (more…)
On Privacy: The Supreme Court Finally Got It Right!
Today is a landmark day for those of us concerned about privacy as it applies to our government agencies. In United States v. Jones the Supreme Court unanimously ruled that police must have a search warrant before using GPS tracking devices to surveil criminal suspects. While all justices agreed that the tracking device placed on Jone’s jeep violated the Fourth Amendment’s unreasonable search and seizure protection, the justices were divided on how far the ruling should have gone. Justices Alito, Ginsburg, Breyer, and Kagan said that “the court should have gone further and dealt with GPS tracking of wireless devices, like mobile phones.”
While I agree that the ruling did not go far enough, it certainly is a sign that the court understands that technology capabilities aside, we (all citizens) do have a right to privacy within the vehicles we operate and that placing a GPS tracking device on a vehicle requires a search warrant. I must confess that I have been poring over the oral arguments and trying to figure out which way the court would rule, but felt that it was too close to call. So while the rest of the world was focused on SOPA and PIPA last week (my esteemed co-blogger included), I was anxiously awaiting the court’s ruling. And thankfully, as a card carrying member of the high tech community and American citizen, it was well worth the wait! (more…)
A quick thought on #BlackoutSopa day.
In our book Privacy & Big Data that was written pre-SOPA, Mary and I spent a fair amount of time looking at the ways that big media interests are pushing both technical and legislative solutions that were inimical to both privacy and free speech. On this day when the Internet is raising its collective voice against one of the most ill thought laws of the Internet age, I thought it would be a great time to quote from the conclusion of Chapter 4 – The Stakeholders.
“Powerful groups, like the MPAA and RIAA and their international counterparts, have borrowed from advertising’s playbook and extended it to every device we own. Today, it’s not just about tracking our online behavior; it’s about tracking what we do within the “four walls” of any device that we own and being able to remotely control them without our permission. These technologies and policies could end up delivering a mortal blow to privacy as well as cede to the government and IP holders unprecedented control over what media we are allowed to consume and share. However you look at this, it’s a pretty high price to pay to support an old business model that is unable to adapt to new technology.”
Tell your congressperson – SOPA/PIPA is bad for the Internet, bad for free speech and bad for due process and should be rejected! More info on the law here.
McKinsey Study: Location, Location, Location, Part 2
Greetings one and all and happy new year! As promised, part 2 of my post on McKinsey’s drill-down into the tremendous benefits location data offers to new businesses (and business models) as well as to all of us. If you need to refresh your memory (since the author was a wee bit late in meeting her stated publishing date), part 1 is available here. Certainly, the report, “Big data: The next frontier for innovation, competition, and productivity,” is chock full of illuminating ways that big data can be leveraged within specific industries, but personal location data is a somewhat different beast as it cuts across industries. For example, telecom, retail, and media (through location-based advertising) all stand to reap tremendous rewards.
Now, as I said in part 1 and will state again in part 2: I have a bit of angst around the collection and use of personal location data (see my many posts on privacy or our book on “Privacy and Big Data”). But that does not negate what can be gained if it is properly collected and used and with the appropriate regulations and guidance in place (my gosh—I am beginning to sound like one of the privacy policies I hate to read!). Put simply: all company’s data collection and usage policies should be clearly stated and always offered on an opt-in basis. Okay, privacy issues have been dealt with so let’s move on! (more…)
My Favorite Thing (Uh, Device) of 2011
Greetings one and all! This is my last post of 2011 and as promised, I will reveal (drum roll please) my favorite thing of 2011. But before I do (really, did you think that for once I would not bury the lead?), I must tell you a bit about the personal side of my life (and yes, I am breaking my very own rule of keeping my personal stuff off the Net).
Here it goes: I am an avid reader. I love books—all books, fiction, non-fiction, you name it and I’ve probably read it (or tried to anyway). My favorite fiction genres are: mystery, thrillers, police procedurals, action and adventure, historical, and of course, the classics. My favorite nonfiction categories are: marketing (duh), politics, privacy, business and technology (SCM, big data, BI, analytics), and history (World War I and II and anything to do with Hawaii—I am part-Hawaiian after all). (more…)
Confessions of a Privacy Junkie (and a list of my favorite privacy resources!)
The holiday season is upon us and we are being inundated (happily) with best and worst lists as well as predictions for 2012. Terence and I will reveal our own lists soon but I thought that I might take this post to chat about our book (Privacy and Big Data) and the evolution of a privacy junkie (me). But before I begin, a note to our regular readers: I know that I promised a part 2 in my series on McKinsey and location tracking and it’s coming… in early 2012 (yes, I have taken poetic license with scheduled dates!).
As you all are far more aware than you might want to be due to our incessant plugging, we (Terence and Mary) wrote a book on Privacy and Big Data this year. I have always been concerned about privacy issues and thought that I was well informed on the subject so why not co-author the book? How much work would it really entail? After all, this is a topic that Terence and I talk (and argue) about pretty much every day so all we needed to do was collate our combined knowledge and voilà, a book is born! Six months later, after reading through thousands and thousands of pages of research, books, articles, and posts on the topic of privacy, I can safely say that I am now a privacy junkie of the highest order (I suspect Terence is as well but I digress). (more…)
No, Hadoop Doesn’t Own Big Data Analytics!
A number of folks have asked me if I was concerned about Microsoft’s recent announcement that they would be partnering with HortonWorks and abandoning their own distributed processing technology for Hadoop. While I thought this was an unfortunate choice on Microsoft’s part (the Dryad project’s implementation of multi-server Linq was pretty compelling), since HPC is a small part of Microsoft’s business, it probably made sense from a business standpoint. In any case, we (as in all of us at PatternBuilders) are not concerned and just to be clear: we don’t believe that this announcement (or any other) means that the many Hadoop ecosystem players own the still forming big data analytics market.
That is not to say that the announcement isn’t proof of the strength of the Hadoop ecosystem. Hadoop is a nifty technology that offers one of the best distributed batch processing frameworks available, although there are other very good ones that don’t get nearly as much press, including Condor and Globus. All of these systems fit broadly into the High Performance, Parallel, or Grid computing categories and all have been or are currently used to perform analytics on large data sets (as well as other types of problems that can benefit from bringing the power of multiple computers to bear on a problem). The SETI project is probably the most well know (and IMHO, the coolest) application of these technologies outside of that little company in Mountain View indexing the Internet. (more…)
McKinsey Study: Location, Location, Location, Part 1
Yes, it’s that time again: a deep drill-down into a specific big data area, courtesy of McKinsey’s voluminous report on “Big data: The next frontier for innovation, competition, and productivity.” You may be wondering about the five month delay since my last foray into this particular study but, well, we have been just a bit busy with our book (Privacy and Big Data) while working on some very cool features for our Analytics Platform as well as handling all our other PatternBuilders responsibilities!
I also must confess to a bit of angst regarding location data, especially when it pertains to where we are located as opposed to where things (like shipping boxes) are located. From a privacy standpoint, this is a rather large (okay, huge) area of concern but it’s not the data itself that we should be worried about. As in most things surrounding the privacy debate, it is how the myriad of companies, organizations, and government agencies collect and use our personal location information without our knowledge or consent that we should be worried about. (more…)
