Spark performance tuning scenario-based interview questions for data engineers and architects.
Complete course available here.
scholarnest.in/courses/apache…
Prashant Pandey
@prashant_pdy
Big Data Consultant @ Tata Consultancy Services Ltd. Creating Big Data Tutorials @ youtube.com/learningjourna…
- How to grow from data engineer to a solution architect? Do you know the answer or know the career path? I know one thing. To be a solution architect, you need to start being a solution architect. No one is offering you a role that you are not already…
- Big data, data lake, data ingestion and lakehouse architects! Here is a design scenario and a bunch of interview questions for data ingestion. You have data arriving in a Kafka cluster and you are asked to design a data ingestion pipeline to bring data…
- What are the 5-S of Spark performance tuning? These are the critical areas of performance bottlenecks in a Spark application and are subject to tuning and need special care and follow best practices. Here is the list. 1. Skew 2. Spill 3. Shuffle 4. Stora…
- Common misconception. Spark Dataframe supports structured and semi-structured data. But for unstructured data, you must use RDD APIs. Here is another way of saying the same statement. Spark Dataframe supports CSV and JSON. But for reading a text and imag…
