Debugging Spark Code Locally Like A Boss

Anup Vasudevan
Analytics Vidhya
Published in
6 min readJan 4, 2021

--

https://flic.kr/p/sbEHMt

Stepping through Spark internals can be helpful. If anything, it helps make sense of what your code is doing under the hood. In this post, I’m going to explain how I set up my debugger to hit breakpoints within the Spark codebase to be able to debug Spark Scala, Java and Python code.

The Prep

--

--

Anup Vasudevan
Analytics Vidhya

Hi there! I’m a software developer based out of Boston, MA. Worked at DSW, T-Mobile, HMH, Wellington Management, Thesys CAT and Autodesk to name a few.