Stream Processing POC

Kappa/Lambda Architecture POC Sample

Here's a description of folder contents:

Databricks

Samples of Stream Processing using Databricks Spark Structured Streaming

• Spark Structured Streaming
• Databricks Structured Streaming

RealTimeAzFunc

Stateful Streaming sample using Azure Functions.

Two functions are deployed to Azure:

• RealTimeAzFunc_EH: Get called by EventGrid on newly created Blob in the monitored Azure Blob Storage. Reads Blob content and the send it ito the EventHub used to support Stream Processig
• DataProcessingAzFunc: Implement a stateful processing on the Stream (EventHubs) to calculate the PlannedTimeTurn KPI for a specific Tail Number.

SqlDatabase

Azure SQL Database used to

• Store Azure Function state
• Be the Serving Layer in Azure Function and Stream Analytics examples

Feature used that may be useful to JetBlue:

• Temporal Tables (to automatically store all changes made to data)
• JSON Support
• Column-Store In-Memory Tables (not used in the POC is stream volume was very low, useful for Near-Real Time Operational Analytics)

StreamAnalyticsQueries

This example show how complex JSON can be manipulated using Stream Analytics and how data can be aggregate using Temporal Windowing Functions. The output is send to PowerBI and Azure SQL at the same time.