Spark Streaming - Stream Processing in Lakehouse - PySpark

Explore Apache Spark development environments across cloud and on-premise platforms, focusing on Databricks and Cloudera, browser notebooks, and local IDEs like PyCharm and VSCode.

Set up the Databricks community edition account, verify email, and log in to the cloud workspace, then explore the UI for Spark development.

Learn to use the Databricks community edition workspace to create a spark project, write and run code in notebooks, and manage data with dbfs and csv files.

Explore when batch versus stream processing is needed and master key challenges—back pressure, incremental processing, checkpointing, fault tolerance, and late data—using Spark structured streaming.

Develop a streaming word count solution by turning batch processing into streaming, ingesting landing zone data, applying quality rules, and storing word count results.

Transform a batch word count into a streaming application using spark structured streaming, adopting read stream, write stream, incremental data processing, and a checkpoint location.

Explore how Spark structured streaming incrementally executes streaming plans through a streaming query, micro-batches, and checkpoints, reading new data with read stream and writing results with write stream.

Create a spark streaming application to ingest json invoices from a landing zone, explode and flatten line items, and write denormalized records to a delta table.

Explore spark structured streaming triggers: unspecified, fixed interval (processing time), and available now. Learn how microbatches start, collect data for fixed intervals, and enable incremental batch processing with available now.

Master a single Spark structured streaming application that runs as batch or streaming, using triggers such as available now or processing time and max files per trigger.

Explore spark structured streaming sources and sinks, including directory and delta table inputs, Kafka sources, delta tables as sinks, and for each with external connectors.

Learn to chain delta table streaming with two Spark jobs, bronze to silver in a medallion lakehouse, including archival, checkpoints, and flowing data through bronze and silver layers.

Apache Kafka is an open source distributed event streaming platform for real-time data collection, integration, and processing, with producers, brokers, and consumers.

Learn to create a free Confluent Cloud Kafka cluster in the cloud, including signing up, provisioning a basic cluster, and creating a topic named invoices with four partitions.

Understand Kafka core concepts, including brokers and clusters, producers and consumers, and how topics, partitions, keys, values, timestamps, and offsets organize data.

Learn to build a Python Kafka producer that reads a JSON invoices file and sends each line as a message to the invoices topic using the Confluent Kafka client.

Read data from the invoices kafka topic with the spark kafka connector. Load into a spark data frame and inspect the binary key, value, and timestamps.

Connect to a Kafka cluster using Spark Structured Streaming with the Kafka connector to ingest topic data into a bronze layer, applying max offsets per trigger.

Implement idempotence in a Spark structured streaming pipeline by upserting with a merge into the invoices table using a for each batch sink to prevent duplicates during restarts.

Learn to build a Spark Structured Streaming Kafka sink by reading json invoices from a landing zone, applying a where filter, and writing key-value messages to a Kafka topic.

Learn real-time aggregates with Spark Structured Streaming, state store, and bronze-to-gold pipelines to compute customer totals and rewards, plus append and complete output modes.

Implement incremental aggregation in Spark structured streaming by using update mode with per-batch upserts via a merge statement, and enable RocksDB state store provider for performance.

Learn how Spark Structured Streaming uses microbatches and three output modes—append, update, complete—while inspecting streaming query stats in the Spark UI and managing state store memory for incremental aggregates.

Explore stateless and stateful transformations in spark structured streaming and how the state store enables incremental aggregates across micro-batches. Learn about unbounded versus time-bound aggregations and state cleanup.

Implement stateless streaming aggregation by moving the aggregation after the write stream, using a bronze and gold approach to compute unbounded incremental aggregates without a state store.

Explore time bound windowing in Spark streaming to compute 15-minute tumbling aggregates on traded data, producing a trade summary with total buy and total sell from a Kafka source.

Learn how watermarking enables state cleanup in Spark structured streaming, handle late records in time-based windowing, and implement withWatermark to manage 15-minute windows and aggregate buy and sell totals.

Implement sliding window aggregates in PySpark to compute moving maximums over a 15-minute window sliding every five minutes, using event time, watermark, and state cleanup.

Databricks brings Apache Spark to the cloud with Delta Lake transactions, Unity Catalog, and on-demand clusters, offering notebooks, a Photon engine, and Azure integrations for analytics and machine learning.

Create your Azure free account to access 12 months of popular services and over 40 always free services, plus $200 credit for 30 days while you complete the course.

Explore the Azure portal overview, sign in with your Microsoft account, and navigate subscriptions, resource groups, and dashboards. Use search, cloud shell, and settings to manage Azure services efficiently.

Create an Azure Databricks service, set up subscription, resource group, region, and pricing tier, then launch and explore a Databricks workspace with Azure single sign-on for Spark development.

Explore the Databricks workspace, including the left side menu, and learn how data warehousing, data engineering, machine learning capabilities, job runs, and workflows are organized with data ingestion partner tools.

Explore the Azure Databricks architecture, detailing control plane and data plane with a single consolidated bill. See how DBFS abstracts storage and enables seamless access to Azure Data Lake Gen2.

Learn to create and configure an Azure Databricks cluster, choosing multi-node or single-node setups, and manage runtime, auto scaling, and essential tools like notebooks, libraries, and Spark UI.

This lecture introduces Databricks notebooks in Azure Databricks. It shows how to create and name notebooks, set Python as the default language, attach to a Spark cluster, and run cells.

Explore Databricks notebook magic commands and multi-language cells in PySpark notebooks. Learn to use percent sequel, percent scala, and percent python alongside markdown documentation and file system commands.

Explore Databricks utilities and dbutils for notebook workflows and file system operations. Use notebooks and widgets to build modular, parameterized workflows with data, credentials, and secrets in Databricks.

Unity Catalog is a Databricks premium service for metadata and user management that stores catalogs, schemas, tables, volumes, and storage locations in a metastore, using SQL grants for access control.

Explore how to design and manage Databricks workflow jobs to build reliable data pipelines with tasks, parallel execution, dependencies, cluster configurations, parameters, and schedules.

Discover how the Databricks Rest API automates Databricks project activities and enables deploying workflow jobs across environments via a Python example.

Explore the Databricks CLI, a command line tool that automates the Databricks platform from your terminal via the REST API; learn to create jobs with shell scripts and JSON definitions.

Design a Databricks lakehouse with bronze, silver, and gold layers to ingest device, profile, bpm, login, and workout data and deliver gold tables for workout bpm and gym summaries.

Design a lakehouse workflow to process five input datasets for analytical use, syncing profile updates from a Kafka topic to the cloud database while decoupling operational workloads.

Design a lakehouse storage layer on Azure ADLS Gen2 within the Databricks cloud, using separate containers for metastore, managed data, and unmanaged data, with bronze, silver, and gold layers.

Implement data security by inserting a Unity Catalog layer between storage and compute, enforcing fine-grained access, blocking direct directory access, and granting read/write on databases and tables via groups.

Implement resource policies with workspace management and cluster policies to control cluster type, size, per-user limits, costs, configurations, and logging in a lakehouse environment.

Describe decoupled data ingestion for a lakehouse, loading Azure SQL CDC and Kafka topics into ADLS data zones via Azure Data Factory and the Kafka ADLS Gen2 Sink.

Design bronze layer in a lakehouse medallion architecture. Ingest raw data from landing zones into a single bronze table with Spark, including user, login, gym, and Kafka data, preserving duplicates.

Designs the silver and gold layers of a lakehouse medallion architecture, adding five silver tables for five entities and two flat gold reporting tables, with quality checks and CDC handling.

Set up your project’s source control with Azure DevOps by creating an organization, a project, and a repository; initialize the main branch and add release and feature branches.

Set up a Databricks lakehouse on Azure by creating a premium workspace, configuring an Azure Gen2 storage layer, and establishing the Unity catalog, storage containers, and access connectors.

Create and launch an Azure Databricks workspace, configure subscription and resource group, select region and premium tier for Unity Catalog, then access Spark development from the workspace.

Create a storage layer for your lake house by creating a storage account, then containers, then a Databricks access connector, and grant the connector the storage blob data contributor role.

Create a Unity Catalog metastore in Databricks by configuring the storage path, Azure Databricks Access Connector ID, and workspace connections, then define a developer group with access.

Set up data directory locations and a Unity Catalog metadata store; create external locations for managed and unmanaged datasets mapped to storage containers.

Launch coding in a Databricks development workspace, connect to Azure DevOps, and create a feature branch to develop setup notebooks and DDL scripts for bronze, silver, and gold tables.

Learn to build a config-driven setup script and SetupHelper that parameterizes environments, creates a database for bronze, silver, and gold tables, and validates and cleans up with unit-tested DDLs.

Load historical and lookup data with a history loader that populates the date_lookup dimension in the silver layer, using a one-time data load from cloud storage.

Ingest raw data from three landing zones into the bronze layer of the lakehouse using Spark streaming with autoloader, loading into bronze tables without transformation.

Code the silver layer by streaming from the bronze layer, eliminating duplicates and transforming data, and upserting into eight silver destinations via readStream, writeStream, and merge statements using Upserter classes.

Handle multiple updates for the same record in spark streaming silver layer by ranking by updated timestamp and keeping latest. Use foreachBatch upsert to merge changes into target.

Implement the gold layer by upserting workout_bpm_summary from the silver layer, using readStream, joins with user_bins, aggregations, and foreachBatch to merge results.

Define and run bronze, silver, and gold jobs from a single run notebook where one consumption method triggers all three pipelines, with parameterized environment, run type, and processing time.

Outlines an end-to-end integration testing strategy for spark streaming in lakehouse, using two payloads, test data preparation, and three notebooks to validate gold layer reports.

Develop a test data producer by implementing a producer class with five data set methods that copy test data from the adls directory to the landing zone, and validate.

Automate batch mode integration testing for a Spark Streaming lakehouse pipeline in PySpark by setting up, loading history, producing data, and validating bronze, silver, and gold layers with cleanup.

Learn to implement streaming mode integration tests for Spark in a lakehouse using Databricks Rest API to create, trigger, and monitor bronze, silver, and gold layer jobs with payloads.

Implement a two-pipeline CI/CD workflow with build and release pipelines in Azure DevOps. Connect your Databricks workspace to a Git repository, run PyTest, and deploy artifacts.

Learn to build an Azure DevOps pipeline using a yaml file, configure all-branch triggers on Ubuntu 22.04, install Python 3.10, and publish a notebooks zip artifact.

Learn to create and configure a release pipeline in Azure DevOps, linking a build artifact, using a ubuntu-22.04 agent, and deploying via Databricks CLI with integration testing.

Create a databricks cli script to automate a deployment workflow: define a notebook task, create and run a job, poll for completion, delete the job, and publish results.

Recognize that an end is a new beginning and embrace ongoing learning beyond the curriculum. Connect on LinkedIn or YouTube to share feedback and keep learning.

Set up the Spark development environment by installing the JDK and Hadoop WinUtils on Windows, then Spark binaries and environment variables, and configure PyCharm to run a test Spark application.

Install Java JDK on Windows to satisfy Spark prerequisites, set JAVA_HOME and PATH, verify with java -version, and configure Hadoop WinUtils with HADOOP_HOME for Windows environments.

Install Spark on Windows by downloading Spark 3.0 with Hadoop 2.7, set up Spark home and PATH, configure Python path and pyspark, and verify with spark-shell or pyspark.

Configure and test your spark development IDE on Windows with PyCharm, a conda environment, and Python 3.7. Install PySpark, open the Hello Spark SQL project, and verify the setup.

Install and configure Apache Spark on mac by setting up JDK 8 or 11 and JAVA_HOME, installing Spark and SPARK_HOME, updating PATH, and enabling pyspark with python3.

Set up Spark development on Mac using PyCharm: configure Python interpreter, create a virtual environment, install PySpark, run the Hello Spark example from the IDE, and inspect logs and tests.

Install and configure a single-node Apache Kafka cluster on your local machine, including zookeeper and server properties, then test with a topic, producer, and consumer.

Explore how spark streaming processes data in micro-batches by reading streams into dataframes, transforming, and writing to sinks, with configurable trigger intervals.

Read JSON files with Spark readStream, flatten invoices by exploding line items, and write the results to an output directory with checkpointing and a one-minute trigger.

Explore how a Spark streaming app reads json files from a directory with readStream, uses checkpoint, and writes outputs with append, update, or complete modes.

Explore fault tolerance in Spark Structured Streaming, detailing exactly-once processing with checkpointing, read position and state information, write-ahead logs, and how to restart using the same checkpoint and re-readable sources.

Explore how stream processing extends batch processing with Spark, addressing late-arriving records and incremental calculations, while supporting scheduling and fault handling across streaming workloads.

Compare DStream and Structured Streaming, highlighting micro-batch processing, event-time semantics, and a unified dataframe-based model running on the Spark SQL engine.

Create your first Spark streaming application by reading text from a tcp/ip socket, counting words with split and explode, then writing results to the console.

Learn to read from Kafka as a streaming source with Spark Structured Streaming, flatten invoice items, apply from_json to value, explode items, and sink to a file system.

Read invoices from Kafka and transform them into customer notifications. Write to Kafka as key/value, using InvoiceNumber as key and json value of CustomerCardNo, TotalAmount, and EarnedLoyaltyPoints (20% of total).

Learn how to run multiple streaming queries in a single Spark application by reading invoices from Kafka once, then output a notification to Kafka and flatten data to file sink.

Explore reading and writing Kafka streams with Spark, deserializing key and value using from_json, from_csv, or from_avro, and producing JSON, CSV, or AVRO outputs with to_json, to_csv, and to_avro.

Create a kafka avro sink by reading json from kafka, applying a schema, flattening invoices, and writing key-value with avro payload using spark-avro.

Read avro records from Apache Kafka using from_avro and the spark-avro package. Aggregate PRIME customers by card number, summing sales and rewards, then write the notification back to Kafka.

Compare stateless and stateful transformations in Spark Structured Streaming and describe how a state store tracks results across micro-batches.

Explore event time and windowing in Spark Streaming, mastering tumbling and sliding windows, and learn how trigger time and event time shape 15-minute time-bound aggregates.

Implement tumbling window aggregates in spark streaming on kafka data to compute buy and sell sums in 15-minute windows, including late arrivals, noting that analytical windowing isn't supported.

Learn how to use watermarking to manage late data in Spark streaming windows, enabling window state cleanup and accurate aggregates via withWatermark and event-time windows.

Explore spark streaming output modes: complete, update, and append, and their effects on state cleanup, windowed aggregates, and watermark behavior with event-time windows.

Learn to implement sliding window aggregates in Spark streaming with PySpark, using event time, watermark, and state cleanup to compute moving maximums over a 15-minute window sliding every 5 minutes.

Learn to join a streaming dataframe with a static dataframe in Spark Structured Streaming, using a login event stream and a Cassandra user table, with foreachBatch to sink results.

Demonstrates that joining streaming dataframes is a stateful stream-to-stream join using a state store, highlighting design considerations like unique IDs to avoid duplicates.

Apply watermarks to both streams in Spark stream-to-stream joins to clean the state store and prevent out-of-memory issues, using impression-id to join impressions with clicks.

Learn how Spark Structured Streaming enables left and right outer joins, with strict watermark and time-range rules for stream-to-stream joins. See how unmatched records and state cleanup illustrate the process.

Celebrate your achievement and embrace ongoing learning as this final word signals a new beginning; share reviews and feedback to guide future updates and new courses.