PySpark Development

Course Description

Learn to develop distributed data-processing solutions with PySpark, the Python interface to Apache Spark. This four-day course covers the Python foundations and big-data concepts needed to work with Spark, before introducing its architecture and application execution model. Participants will explore resilient distributed datasets (RDDs), transformations, actions and caching, then progress to Spark SQL and DataFrames for querying, manipulating and writing data.
4 Days
Contact us for pricing
 

Prerequisites

Participants should have a basic understanding of a programming language and a development framework.

Who Should Attend

This course is suitable for software developers, programmers, data engineers, data scientists, data analysts and technical consultants who want to work with PySpark. It is also relevant to Hadoop developers and professionals building big-data applications.

Learning Objectives

Use core Python syntax, functions and data structures in PySpark development.
Explain the roles of Hadoop and Spark in big-data processing.
Describe Spark architecture and how applications execute.
Work with the PySpark shell and Python API.
Create RDDs and apply transformations, actions and caching.
Build DataFrames from external data sources and RDDs.
Query and manipulate data with Spark SQL and DataFrame operations.
Write processed data using output operations.

Python Foundations

Reviewing essential Python features
Understanding language syntax and fundamentals
Defining and using functions
Working with Python data structures
Writing comprehensions

Big Data and Hadoop

Understanding the big-data landscape
Introducing Hadoop and its role
Exploring the components of Hadoop

Starting with PySpark

Introducing Apache Spark
Understanding Spark architecture
Following the application execution process
Working in the Python Spark shell
Accessing Spark through the PySpark API

RDD Foundations

Understanding resilient distributed datasets (RDDs)
Creating RDDs
Working with distributed collections

RDD Transformations

Applying transformations to RDDs
Transforming key-value datasets

RDD Actions, Caching and Jobs

Executing actions to obtain results
Caching datasets for reuse
Understanding how Spark jobs fit together

Spark SQL and DataFrame Creation

Introducing Spark SQL
Working with the Spark SQL API
Loading data sources into DataFrames
Converting RDDs into DataFrames

Working with DataFrames

Performing core DataFrame operations
Expressing queries through the DataFrame API
Using RDD operations alongside DataFrames
Writing results with output operations
PySparkApache SparkPythonBig DataHadoopSpark SQLDataFramesRDD