Posts tagged ‘MongoDB’
I’ve been absent from the blog too long, but if you’ve been following my colleagues (Mary and Marilyn) postings, you’ll see it’s been a very busy and fruitful time at PatternBuilders. While I’m still overdue for the next segment of the architecture blog series, I thought I would take a break and talk a bit about some of the things we learned as we moved our product and business model to Microsoft Azure.
As someone who has worked with Microsoft technology and partnered with them off and on over the last two decades (even flirting with going to work for them a couple of times), the most surprising discovery was how serious Microsoft has become about the cloud, open source, and being an active and supportive partner for startups. As many of you who have been around as long as I have will no doubt remember, this is a very different, some would say revolutionary, move for the world’s most powerful proprietary software company. We had some concerns when we became members of Microsoft’s Azure Startup program BizSpark Plus and subsequently the more exclusive BizSpark One, but it has turned out to be a great experience for us on both the business and technical level. (more…)
It has been a while since I’ve done posts that focus on our technology (and big data tech in general). We are now about 2 months out from the launch of the Azure version
But before I start exercising my inner geek, it probably makes sense to take a look at the development philosophy and history that forms the basis of our upcoming release. Historically, we delivered our products in one of two ways:
- As a framework which morphed (as of release 2.0) into AnalyticsPBI, our general analytics application designed for business users, quants, and analysts across industries.
- As vertical applications (customized on top of AnalyticsPBI) for specific industries (like FinancePBI and our original Retail Analytics application) which we sold directly to companies in those industries.
There are times when Terence and I look at each other and say, “What on earth were we thinking?” And this is one of those times! PatternBuilders is crazy busy right now putting out release 3.0 of our Analytics Platform (the secret sauce for our analytics applications that we like to call data-science-in-a-box), ramping up on a funding round, working with partners on a University of Sydney research project on the impact of social media on a company’s stock price (a really fun project and a post about it is in the works), and, of course, supporting customers and prospects on their big data initiatives. So… since we did not have enough to do (sarcasm on), we decided it was time to update our book, participate in a pre-Strata East webcast, speak at the Strata Conference and the MongoDB User Group (that is collocated with Strata) in New York City! In the words of the immortal Bette Davis in All About Eve (and ever so slightly revised):
“Fasten your seat belts, it’s going to be a bumpy night ride!”
Really, what were we thinking????? (more…)
Big Data Tools Need to Get Out of the Stone Age: Business Users and Data Scientists Need Applications, Not Technology Stacks
Things have been crazy at PatternBuilders recently. The excitement and positive reactions to FinancePBI, our Financial Services big data analytics solution, from media, analysts, venture folks, cloud infrastructure partners, and users has been amazing. Our new cross industry graphical big data correlation mashups are generating a lot of excitement as well—we like to call this feature Google Correlate on steroids. Check out how our newest partner analytics consultancy, InsightVoices, has used it to find relationships between stock prices and traffic sensor data.
Mary’s recent post on Strata West 2012 provides a great overview of how hot the hype cycle around big data has become (while managing to work in a plug for her favorite gory TV series as well). In case you’re still not convinced, here are some additional nuggets:
- The market for big data technology worldwide is expected to grow from $3.2 billion in 2010 to $16.9 billion in 2015, a compound annual growth rate (CAGR) of 40% (hat tip to IDC).
- The amount of big data being generated continues to grow exponentially, now being expected to double in two years. This is largely driven by social networks, smartphones, and really cool IP-enabled devices like the Fitbit and this IPhone-based brain scanning device by our new Strata buddy Tan Le at Emotiv Lifesciences. Yes, she is much smarter than us but we like her anyway!
- The White House is even doing its share, investing $200 million a year in access and funding to help propel big data sets, techniques, and technologies while giving a shout out to our friends at Data Without Borders.
All you need is text, Text is all you need (sing to the tune of The Beatles’ All you need is love). If you are one of our regular readers you will remember that several months ago I wrote a manifesto on what the perfect analytics system would look like. One of the last points was:
It must be as accessible as Excel (still the number one analytics tool in the world).
I was wrong – Excel is the number one non-specialized analytics tool in the world but in terms of usage, it is dwarfed in comparison to a very well know specialized analytics toolkit. The creators of this tool are a little company that you may have heard of: it does no evil and analyzes the Internet to bring you back everything on the web based on a simple text query. But behind that simple text box, Google has one of the most sophisticated analytics infrastructures in the world:
- It can deduce your interests.
- Give you the most relevant results.
- And show you appropriate information based on them, as well as bring back highly personalized ads.
Google is not only the largest big data analytics company in the world, but it also has the easiest to use tools—proof that text is all you really need!
We have recently made a big architectural change concerning our storage back-end and I wanted to talk about it.
Storage is key to any Big Data problem. As we’ve mentioned in prior posts, most of our performance bottlenecks and optimizations have to do with storage performance and architecture, as opposed to computation. Our architecture for the last few years has consisted of a hybrid approach with “no-SQL” analytics storage using MongoDB and “non-transactional” data stored in a traditional RDBMS, primarily SQL Server. There were a couple of reasons for this architecture. First, we started off entirely in RDBMS-land, because our initial design was done before no-SQL systems were really at a production-level of maturity. Second, most of our customers and prospects had traditional schemas and data organization – making integration easier if we could just use the same object model. (more…)
As you all know, Tim and I spoke at MongoSF recently. Our session was focused on how to build a streaming analytics system with Mongo. For those of you who might have missed this post thread, here are the highlights (with the appropriate links):
- We wanted to make our beta version of PatternBuilders Social Media Analytics demo publicly available on the web.
- We looked at cloud-based deployments as a way to make this economically viable.
- As part of our move to the cloud, we made significant changes to PatternBuilders Platform architecture—which included MongoDB (a choice that the PatternBuilders development team is very happy with).
Our session was videotaped and I am happy to announce that it is now available on the 10gen site. You’ll notice that we got a lot of great questions. If, after viewing the video, you have some thoughts or questions please send them my way through comments or email—it may take me some time (we are, as Mary said in her last post, crazy busy right now), but I will follow up!