A Sneak Peek at Our New HTML 5 UI and Geek Love for Some of the Libraries Used in Building AnalyticsPBI4Azure

November 8, 2013 at 2:07 pm 1 comment

By Terence Craig

AnalyticsPBI Coming SoonDrumroll please! After nearly a year of development work, we are about to offer early access to the first real-time/streaming analytics solution software appliance for the cloud – AnalyticsPBI for Azure.  There will be more forthcoming on the product launch but the new UI is so cool I had to show it off a bit.

We will be following up with a formal launch and Early Access Program (EAP) signups in the next couple of weeks so watch this space and patternbuilders.com for details – the big data analytics market is about to change in a big way! Here’s a sneak peek on what we’ve been working on.

For the geek part of my blog I am going to give a shout out to three libraries that we are using – all have made a huge difference in the product’s performance, scalability, and usability. The first two libraries come from Microsoft – Reactive Extensions and TPL Dataflow.  The third library is the open source math and statistics library, Math.Net.

Rx has been around for a while but, except for financial services, most projects that used it focused on using it for the client side.  However, Rx’s functional deferred execution paradigm is perfect for complex manipulations of streaming data. One of Rx’s coolest features is its Generate statement. With Generate it is trivial to generate real-time streams for testing. (Anybody that has worked with streaming data on the server side knows exactly what a boon that is to both developers and QA.) That ease of generating test data has allowed us to push both the quality and performance envelopes.

Here is an example from Eugene Prystupa’s Weblog. His post was the first time we realized the expressive power that Rx could have in our domain.

// simulate market data
var rnd = new Random();
var feed = Observable.Defer(() =>
    Observable.Return(Math.Round(30.0 + rnd.NextDouble(), 2))
   .Delay(TimeSpan.FromSeconds(1 * rnd.NextDouble())))
   .Repeat();

In just a few lines of code you have a tick simulator.  Mr Prystupa has some other cool uses of Rx on his blog so check it out. We have done similar things simulating tweets (using a Markov text generator) among others.

The other Microsoft library that we have been blow away by is TPL Dataflow (a push-based, asynchronous Dataflow library). With TPL Dataflow it is trivial to put together complex, runtime configurable data meshes (think agent-based programming if you are an old guy like me).  One way we use it is to customize data stream loading based on user/organization preferences at runtime.  For example, here is how we dynamically manage whether encryption is done for a particular feed or a particular user by using a simple predicate supplied to the LinkTo function that links blocks.  (Warning: This is pseudo code written on a plane with no access to our repo so there WILL be some syntax boo-boos.)

//Anything the encryption block processes must be passed on so no
//predicate.
EncryptionBlock.LinkTo(PersistenceBlock);

//If the type of data event or the user requirements means this
//data needs to be encrypted, NeedsEncryption will return true.
//The configuration can be changed at run time.
DataReaderBlock.LinkTo(EncryptionBlock, (DataEvent d)=>
  {return runtimeConfiguration(d, context.user).NeedsEncryption());

//If NeedsEncryption returns true the PersistenceBlock needs to
//not store the data until its encryption is completed.
//In that case the predicate makes sure that the persistence block
//is skipped while the encryption block's predicate link to the
//DataReader block will return true making sure that it is executed.
//The lack of a predicate on the link between the EncryptionBlock
//and the PersistenceBlock guarantees that data is stored
//after encryption.
DataReaderBlock.LinkTo(PersistenceBlock, DataEvent d)=>
{return !runtimeConfiguration(d, context.user).NeedsEncryption());

//Anything the encryption block processes must be passed on so
//no predicate for link.
EncryptionBlock.LinkTo(PersistenceBlock);

//From there all that is needed is to start feeding data events
//to the DataReader block
DataReader.Post(d)

And we are off to the races.

The great thing about this is that with a couple of simple predicates we can now turn encryption on or off at very granular levels without bringing the system down. The applications for multi-tenancy are obvious. Truly geek coolness.

In an upcoming release, users will be able to add their own custom dataflow blocks to their workflow on both a user and instance level. With these two libraries, the productivity gains are amazing.  I will definitely be writing more about how we used them in future posts.

The third library, Math.NET, is an amazing piece of validation for open source libraries. We spent a great deal of time looking at various math and statistics libraries to use as our default (we support any .Net capable math library but Math.NET is fully integrated into our new model scripting library) and were blown away by its quality and capabilities.  If you are a .NET developer looking for a great math library, check it out!

Needless to say, there are exciting times ahead! Post-launch we will be blogging about many of our technology decisions and I will finish off my long delayed architecture series but for now, I hope that you enjoy this small “taste” of how we are raising the bar on big data analytics.

All trademarks belong to their owners.

Entry filed under: big data, PatternBuilders Technology, Technology. Tags: , , , , , , .

Events to Measures – Scalable Analytics Calculations using PatternBuilders in the Cloud Ways to Help the Philippines Recover

1 Comment Add your own

  • […] Crowdsourcing of this nature extends the idea that the analytics process and team isn’t just made of big data sources and data geeks: small data can be just as important to answering questions as big data is. Regardless, crowdsourcing, or other equally imaginative approaches, also require an analytics tool that is both powerful and easily accessible to all stakeholders. This is why we created AnalyticsPBI for Azure. […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Video: Big Data Made Easy

PatternBuilders Corporate

Follow us on Twitter

Special privacy section!

Enter your email address to subscribe.

Join 58 other followers

Recent Posts

Previous Posts


Follow

Get every new post delivered to your Inbox.

Join 58 other followers

%d bloggers like this: