What is PySpark?

Understanding PySpark

PySpark is the Python API for Apache Spark, a powerful open-source distributed computing framework. It allows you to process terabytes of data across hundreds of machines using familiar Python syntax. PySpark is the backbone of modern big data engineering and analytics.

Core Architecture

• Driver — Your main program that defines operations
• Cluster Manager — Allocates resources (YARN, Mesos, or Standalone)
• Executors — Worker processes that run tasks on each node
• Partitions — Data chunks distributed across executors

SparkSession — Your Entry Point

from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder \
    .appName("MyApp") \
    .master("local[*]") \
    .getOrCreate()

# Read data
df = spark.read.csv("sales_data.csv", header=True, inferSchema=True)
df.show(5)

DataFrames — The Core API

PySpark DataFrames are distributed, immutable datasets organized in named columns — similar to Pandas DataFrames but running across a cluster:

from pyspark.sql.functions import col, sum, avg, count

# Select and filter
df.select("product", "revenue") \
  .filter(col("revenue") > 1000) \
  .show()

# Group and aggregate
summary = df.groupBy("category") \
    .agg(
        sum("revenue").alias("total_revenue"),
        avg("revenue").alias("avg_revenue"),
        count("*").alias("num_orders")
    ) \
    .orderBy(col("total_revenue").desc())

summary.show()

Transformations vs Actions

• Transformations (lazy) — select(), filter(), groupBy(), join(), withColumn() — define a plan but don't execute
• Actions (eager) — show(), count(), collect(), write() — trigger actual computation

Joins

# Inner join
result = orders_df.join(
    customers_df,
    orders_df.customer_id == customers_df.id,
    "inner"
)

# Broadcast join (for small tables)
from pyspark.sql.functions import broadcast
result = big_df.join(broadcast(small_df), "key")

Writing Data

# Write as Parquet (columnar, compressed)
df.write.mode("overwrite").parquet("output/sales")

# Write partitioned by column
df.write.mode("overwrite") \
    .partitionBy("year", "month") \
    .parquet("output/sales_partitioned")

🧪 Try It Yourself — Cluster Partition Simulator

Adjust the number of data records and cluster workers below. The simulator shows how Spark distributes partitions across worker nodes:

Best Practices

• Avoid collect() on large datasets — it pulls all data to the driver
• Use broadcast() for joins with small lookup tables
• Partition data by frequently-filtered columns to avoid full scans
• Cache DataFrames you use multiple times: df.cache()
• Monitor your jobs in the Spark Web UI (localhost:4040)
• Write output in Parquet format for optimal compression and read speed