The Shift from Batch Data to Streaming Data

On Medium, we published an article that chronicles the shift from batch- to streaming-data processing. A few snippets from the original are included below:

"...The fact of the matter is that in today’s world, too many fast-breaking events occur within any given hour: events that we need to register and have the ability to react to in real time. Just as batch data doesn’t provide actionable, on-the-ground intelligence that can actually prevent deforestation, neither does it give programmatic buyers any leeway to change their campaigns if new, split-second, impactful developments arise (which inevitably they do). With the latency that goes into processing a batch data report, advertisers are left flying blind. As a result, oversights begin to pile up at a fast clip: advertisers begin throwing their media budgets at websites where there’s no longer enough of an audience to justify the spending. And companies are at a disadvantage if they want to test new strategies at a moment’s notice and see if they can perform better.

In contrast to batch data processing, data streaming is a child of our own age, the age of big data. According to a report by (the modern-day) IBM, 90% of the world’s information was collected within the past two years alone. While a stat like that might seem bewildering to some, for businesses, organizations, and government bodies with the necessary processing power and with smart data scientists ready to analyze it, big data streams give us a level of granular insight that none of us would have thought possible only 20 years ago.

By using the right algorithms on a streaming data set, we can actually quantify event-based changes in a matter of microseconds. And, while event-based data might not always let us know why changes occur, it does let us know what changes are occurring at any given instant with minimal latency. Presented with enough event-based, real-time data, we can suddenly notice patterns of correlation and causation where we’d never thought to look beforehand. By “mining” this stream of big data and by studying frequent correlations between events, we’re given the ability to forecast the probability of event-based outcomes with a surprising level of accuracy..."

To learn more, head over to Medium to read the whole piece. Additionally, watch this space over the coming weeks to learn more about AppNexus' streaming data capabilities and applications.

  • Industry Perspectives
  • Technology