Big Data is the New Black… And it’s Here to Stay
February 9, 2011 at 7:50 pm Terence Craig
By Terence Craig
Some impressions from O’Reilly’s Strata Conference 2011.
We had a great time at Strata 2011. The O’Reilly folks did a great job of organizing a really well run show – hat tip to JB Wheatley for making life as easy as possible for speakers.
As a member of a company that has been carrying the big data, big analytics flag for a while, it was great to see a sold out show dedicated to the area that will provide one of the biggest challenges, if not the biggest challenge, for 21st century computing. Our presentation seemed to go well – we had a good crowd and a bunch of folks came up to us during the rest of the week to discuss/argue some of the points we raised. My statement about traditional databases/data warehouses not being up to the task definitely annoyed some people, but everybody that approached us was polite and thoughtful and we had some good discussions.
Since I hate going to presentations that are really sales pitches in disguise, I tried very hard to keep the presentation focused on the topic and not PatternBuilders. I may have gone overboard since a number of people that attended our session that I talked with later thought we were a retail services company. For the record: we sell real-time analytics solutions.
There was so much great information at the show that I am going take a few more days before I write about any conclusions, but here are a few highlights off the top of my head. If you can’t wait for a more comprehensive review, Ted Leung has a great one over at his blog.
- Everybody at the show, presenters and attendees alike, was aware of the huge potential for misuse of big data and analytics. The number of lectures and informal discussions around ethical use was both surprising and heartening. John Fritz’s With Big Data Comes Big Responsibility in particular was excellent.
- The energy level and excitement about both the commercial opportunities and power to have a positive impact on society was amazing. It reminded me of the early days of the Internet bubble but without the sleazy carpetbaggers who just wanted to make a buck. The folks that I talked to at the show, including some VCs, were focused on building real products and services that have a real impact, not just slideware that they can flip quickly.
- I was surprised at the mainstream media coverage at the event, including Forbes and the Wall Street Journal. I had a great conversation with Jennifer Valentino DeVries from the Journal about all things big data and was impressed by the amount of research and thought she had put into it and her understanding of the broader implications of the technology.
- Almost all the talk was about batch processing, not streaming (Jud Valeski, the Gnip CEO, mentioned that he noticed this as well in the Gnip blog). For the reasons I mentioned here, I think that streaming is one of the critical pieces for big data and big analytics. It would be a great topic for Strata East.
- During the Where is the Money in Big Data panel, Tim Guleri of Sierra Ventures got it exactly right when he described big data as a fundamental shift in the corporate IT stack that offers lots of “fissures” that new companies can take over and build great businesses in.
My final thoughts for now: I think a lot of vendors attending the show are making a classic mistake, confusing promising technologies with successful products. There was a lot of talk about the great successes that companies like Facebook and LinkedIn have had with technologies like Hadoop. That being said, the complexity barrier of Hadoop and similar technologies is still way too high for most companies to effectively use them for big analytics projects. We need systems that are a better fit for companies that don’t have, and don’t want, an army of programmers to support them. If we want big data to have a big impact, we have to avoid the situation in this famous cartoon – which I saw a couple of times at the show (edited to be work-friendly).
I think we will look back on Strata as the point where Big Data started to hit the mainstream. Can’t wait.
Entry filed under: Data, General Analytics. Tags: big data, Hadoop, LinkedIn, Sierra Ventures, Strata Conference, streaming analytics, Venture Capital.