How to reconcile Real-Time and Batch processing using In-Memory technology: A demo at the AFCEA Cyber Symposium Plugfest

As you might remember, we(*) participated in a "Plugfest"(**) earlier this year in San Diego. Here is the summary post of what we built for that occasion: http://fsanglier.blogspot.com/2013/04/my-2013-afcea-san-diego-plugfest.html.

This time around, we entered the plugfest competition as a technology provider at the AFCEA Cyber Symposium, which happened last week (June 25-27 2013) in Baltimore. We not only provided technologies components and data feeds to the challengers (San Diego State University, GMU, Army PEO C3T milSuite), but also built a very cool Fraud Detection and Money Laundering demo which was 1 of the plugfest use case for this cyber event.

Our demo was centered around a fundamental "Big Data" question: How can you detect fraud on 100,000s transactions per seconds in real-time (which is absolutely critical if you don't want to lose lots of $$$$ to fraud) while efficiently incorporating in that real-time process data from external systems (i.e. data warehouse or hadoop clusters).
Or in more general words: How to reconcile Real-time processing and Batch processing when dealing with large amounts of data.

To answer this question, we put together a demo centered around Terracotta's In-Genius intelligence platform (http://terracotta.org/products/in-genius) which provides a highly scalable low-latency in-memory layer capable of "reconciling" the real-time processing needs (ultra low latency with large amounts of new transactions) with the traditional batch processing needs (100s of TB/PB processed in an asynchronous background jobs), all bundled in a simple software package deployable on any commodity hardware.

Here is the solution we assembled:

Cyber Plugfest Software Architecture
How to reconcile real-time and batch processing

A quick view at how it all works:

1. A custom transaction simulator generates pseudo-random fictional credit card transactions and publish all of them onto a JMS topic (Terracotta Universal Messaging bus)
2. Each JMS message is delivered through pub/sub messaging to both real-time and batch track:
1. The Complex Event Processing (CEP) engine which will identify fraud in real-time through the use of continuous queries.
1. See "Real-Time fraud detection route"
2. Apache Flume, an open source platform which will efficiently and reliably route all the messages into HDFS for further batch processing.
1. See "Batch Processing Route"
3. Batch Processing Route:
1. Apache hadoop to collect and store all the transaction data in its powerful batch-optimized file system
2. Map-Reduce jobs to compute transaction trends (simplified rolling average in this demo case) on the full transaction data for each vendors, customer, or purchase types.
3. Output of map-reduce jobs stored in Terracotta BigMemory Max In-Memory platform.
4. Real-Time fraud detection route:
1. CEP fraud detection queries fetch from Terracotta BigMemory Max (microsecond latency under load) the hadoop-calculated averages (in 3.2), and correlates those with the current incoming transaction to detect anomalies (potential fraud) in real-time.
2. Mashzone, a mashup and data aggregation tool to provide visualization on detected fraud data.
3. For other plugfest challengers and technology providers to be able to use our data, all our data feeds were also available in REST, SOAP, and Web socket formats (which were used by ESRI, Visual Analytics, and others)

As I hope you can see in this post, having a scalable and powerful in-memory layer acting as the middle man between Hadoop and CEP is the key to providing true real-time analysis while still taking advantage of all the powerful computing capabilities that Hadoop has to offer.

In further posts, I'll explain in more detail all the components and code (code and configs are available on github at https://github.com/lanimall/cyberplugfest).

Notes:


(*) "we" = The SoftwareAG Government Solutions team, which I'm part of...
(**) "Plugfest" = "collaborative competitive challenge where industry vendors, academic, and government teams work towards solving a specific set of "challenges" strictly using the RI2P industrial best practices (agile, open standard, SOA, cloud, etc.) for enterprise information system development and deployment." (source: http://www.afcea.org/events/west/13/plugfest.asp)

My 2013 AFCEA San Diego Plugfest Participation - Showing off Terracotta In-Genius

A couple of weeks ago (end of January 2013), 2 colleagues and I participated (under our company banners, Terracotta and SoftwareAG) in a government "plugfest"...and we won first place! Check out this other articles that also talks about our win: http://ctovision.com/2013/02/plugfest-at-afcea-west-terracotta-wins-first-place - and if you're in a rush, jump directly to the short 3 minute video demo below.

What is that, you may ask? As explained on the AFCEA website (http://www.afcea.org/events/west/13/plugfest.asp), a plugfest is a "collaborative competitive" challenge where industry vendors, academic, and government teams work towards solving a specific set of "challenges" strictly using the RI2P industrial best practices (agile, open standard, SOA, cloud, etc.) for enterprise information system development and deployment.

The idea is to "plug" technologies together (technologies provided by the various players, not necessarily within your team) as opposed to rebuild everything from scratch. And indeed, "plugging" is almost mandatory since the scenario is only announced 24 hours before the event, giving the teams a mere 72 hours to create something based on the scenario provided. 

Overall, it's the government effort to encourage/push for more interoperability and reuse of IT components across projects and/or even agencies.

This particular January 2013 plugfest was about solving a Humanitarian Assistance and Disaster Relief (HADR) use case problem where technology:

• Helps track in real-time what's happening on the ground (data streams about hazardous materials, first responders, sensors, injured civilians, etc...) and report it in an actionable, geospatially-enabled, format
• Provides real-time decision support based on pre-defined emergency protocols
• Correlates various "BigData" streams (sensors, social feeds, etc...) to perform real-time analytics in order to predict movements and/or identify "flash mobs" / criminal hotspots taking advantage of the confusion.

The end result of what we put together was a real-time "map" dashboard that shows everything that's happening on the ground, and provide contextual highlights to help decision support.
Here is a short 3 minute video showing the nuts and bolts of that demo:

What you particularly see in the video demo:

• Moving actors on the disaster zone (first responders, plumes of toxicity, drones, etc...). Each of these actors are "broadcasting" their current geolocations (lat, long + metadata) at various time interval using the nirvana universal messaging (1000s of message per second)
• Terracotta's Complex event processing (CEP) engine performing continuous "geo" queries identifying in real-time the distance, speed and direction of the hazardous plumes in comparison to the various red-cross shelters on the map. The CEP engine automatically generates alerts if hazardous plumes are indeed forecasted to impact shelters...providing critical decision support to the commander in charge.
• All events and metadata are stored in-memory, using Terracotta BigMemory for faster, micro-second access and analytics.
• The ability to drill into the moving drones, planes and responders to see a ground view in real-time.
• A triaged based causality tracking and available blood supply.
• Availability of shelters, red cross centers, blood banks, and other supporting organizations(DOD types).

List of what we "plugged":

• Terracotta's In-Genius in-memory intelligence platform (http://terracotta.org/products/in-genius) to provide real-time analytics  (continuous queries ) and decision support based on the incoming streaming data
• SoftwareAG/Terracotta's low-latency low-bandwidth universal messaging platform (http://www.my-channels.com/products/nirvana.html) as the fast messaging communication engine between the visualization front-end and the intelligence platform backend
• SoftwareAG webMethods ESB (http://www.softwareag.com/webmethods/) to integrate various data feeds and store and make it available to us and rest of the players
• Google (https://developers.google.com/maps/) and ESRI (http://www.esri.com/software/arcgis/developers) mapping technologies to display the events in a geospatially-enabled format.
• Note: We also use Jackbe Presto (http://www.jackbe.com/) platform to display the same events onto a different platform and demonstrate interoperability and ease of message ingestions.
• Data feeds from SimTable (http://www.simtable.com/), IBM, Red Cross, San Diego state university, and Nokia.

Thanks for everyone who organized this event. It's been a blast to participate as part of Terracotta team, and I'm looking forward to participating in the next "plugfest" event!