High Performance Spark

Apache Spark is amazing when everything clicks. But if you haven't seen the performance improvements you expected or still don't feel confident enough to use Spark in production, this practical book is for you. Authors Holden Karau, Adi Polak, and Rachel Warren walk you through the secrets of the Spark code base and demonstrate performance optimizations that will help your data pipelines run faster, scale to larger datasets, and avoid costly antipatterns.

Ideal for data engineers, software engineers, data scientists, and system administrators, the second edition of High Performance Spark presents new use cases, code examples, and best practices for Spark 4.x and beyond. This book gives you a fresh perspective on this continually evolving framework and shows you how to work around bumps on your Spark and PySpark journey.

With this book, you'll learn how to:

Accelerate your ML workflows with integrations including PyTorch
Handle key skew and take advantage of Spark's new dynamic partitioning
Make your code reliable with scalable testing and validation techniques
Make Spark high performance
Deploy Spark on Kubernetes and similar environments
Take advantage of GPU acceleration with RAPIDS and resource profiles
Get your Spark jobs to run faster
Use Spark to productionize exploratory data science projects
Handle even larger datasets with Spark
Gain faster insights by reducing pipeline running times