Big Data is Coming of Age in the Capital Markets—Wall Street and Technology’s Deep Dive into “Everything You Need to Know to Unlock Big Data’s Secrets” is a Must Read for All
In the “it’s a small world” category while we were in the midst of launching FinancePBI, the first financial services big data solution built for the cloud and designed to address the needs of the industry, Terence chatted with Melanie Rodier (@mrodier), a Senior Editor at Wall Street and Technology. The topic: big data and the capital markets. That 33-page report is now available and it’s a must read for anyone interested in big data and business.
Why a must read for all? Well, similar to the McKinsey report on big data in 2011, Wall Street and Technology’s big data deep dive covers a lot of ground that applies to any business or organization. In other words, specific industry requirements may be different but big data technology and process challenges are very similar. For example, Wall Street firms—like so many others—find themselves dealing with unstructured data from a variety of sources, including the web, social media, and mobile devices. While there’s value in that data, there are infrastructure issues and a looming talent shortage. Sound familiar?
When we talk to prospects, in or out of financial services, we consistently hear the following:
- “I don’t want technology, I want a solution.” No one wants to DIY their big data solution because it has a very high price tag, especially when you factor in the requisite army of programmers needed to build and then “service” the application—they are looking for companies like ours that deliver fully-fledged solutions.
- “I don’t want to sink all of my money into building out my own infrastructure.” As Andrew Rubin, CEO of Rubin Worldwide, points out in “Why Your IT Spending Is About To Hit The Wall” (page 8 of the report), we are moving towards a technology-driven economy where IT spending will move from 8% to 12% of revenue to the mid-teens or higher. Rubin argues that companies need to shift to disruptive technology options like the cloud in order to deal with increasing demand and rising costs. To prove his point, Rubin discusses Moore’s law and Jevons paradox (it’s a fascinating read and for all of you Moore’s purists out there, I would be very interested in your thoughts so read it and comment on this post!) By the way, this is one of the reasons we offer multiple deployment solution options (cloud, hybrid, or on-premise).
- “I don’t have enough skilled talent.” Yes, we are all painfully aware that there is a severe shortage of folks with deep experience in statistics and machine learning. But as Editorial Director Greg MacSweeney (@gmacsweeney) points out in “Skills to Pay the Bills” (page 3 of the report), “Big data solutions and methodologies are so new and evolving so quickly that financial firms need to develop these skills on the fly. Firms need to invest more in training and workforce development to bridge the knowledge gap if they want to take advantage of what big data can offer.” And they need big data tools that aren’t in the Stone Age but are end user-, analyst-, and programmer-friendly (like ours) to really make their workforce effective.
Certainly, these are challenges that cross industries and it’s clear that Wall Street, like everyone else, is trying to understand:
“…how to ingest, index, and integrate structured and unstructured, streaming, and static data from a variety of sources.” (Melanie Rodier, “Wall Street is Scratching Its Head Over Big Data,” page, 6)
Big data is certainly “top of mind” for enterprise, organizations, and individuals. Consider this:
- Google searches for “big data” are up 1,200% in the past year.
- 90% of all data on the planet was created over the last two years AND that “analysis of the data dwarfs the original data set” (according to IBM).
- All of the data contributed by our smartphones is one of the primary levers for this explosion.
In the report’s cover story, “Big Data: Buyer Beware” (page 11), Melanie Rodier takes a look at some of the companies and technologies in the space as Wall Street searches for the alpha, or as Rob Passarella, vice president at Dow Jones Financial Markets, puts it:
“Everyone who works as a quant wants massive amounts of data in different areas to see if you can play it out. The cheaper and more distributed [the data] becomes, the greater the returns will be.”
Banks and hedge funds are analyzing big data for risk management, price discovery, fraud management, and industry trends. Others are analyzing it for storing and processing derivatives, customer relationship management, and increased regulation and the need to provide granular reporting to the regulators. As Terence points out in the article, grey box traders:
“…who use a system that reveals some or all of the decision-making process, as opposed to an automated black-box system that conceals the trade decision process, have been particularly interested in PatternBuilders’ product.”
Of course, the ability to “layer” (or, as we like to say, mashup) social media or news data on top of equity or bond price data is central to deriving value from big data. As Passarella points out in the same article:
“We used to think of data sets as a single vector. Now we’re thinking of multiple types of data and trying to see if there are any correlations. The real challenge when you accumulate data is being able to line it up so that it’s meaningful.”
This is certainly what we are seeing in our work with the University of Sydney to research the influence of traditional media sources (like The New York Times) with social media (like Twitter) on a company’s stock price. Or, as Terence points out:
“If the Wall Street Journal publishes a negative article on IBM, for example, what is the impact on pricing and volume of that article when it’s been magnified by social media?”
Of course, we are also looking at whether the location of the Twitter user has an impact as well as other factors—but I’ll save this for an upcoming post that fully discusses the research project! For another example of the power of mashups, take a look at an interesting mashup of road sensor data and ticker info that was created by Insight Voices, one of our partners.
Like McKinsey, Wall Street and Technology’s report is a treasure trove for those of us seeking to understand and address the challenges of big data and the tremendous value it represents. Although some of the articles may be specific to the financial sector’s business issues, much of the report applies to any company that is embarking, or has embarked on, the big data journey.
And in a nice hat tip to previous research by McKinsey, be sure and check out the video (“The Big Data Advantage,” page 7) that features Kathy Burger, editorial director for Bank Systems and Technology, interviewing McKinsey’s director, Allen L Weinberg, on what it’s going to take for banks to compete on the strength of big data. In the video, Weinberg talks about the key role that big data can play in narrow margin environments as in: How valuable is this customer to me and how profitable is my next customer? He argues that those banks that can make better decisions about the customer will, over time, make better decisions about the bank itself.
As I said at the beginning of this post, when you have some time (it is 33 pages long) read the report. I would love to hear what you thought of it as well as any key takeaways that I may have missed. And (marketing hat now firmly on) to find out more about FinancePBI, check out our special page or contact us.