About the role
Responsibilities:
- Develop and maintain data pipelines using PySpark.
- Process and analyze large-scale datasets in distributed environments.
- Design and implement ETL/ELT workflows.
- Optimize Spark jobs for performance and scalability.
- Work with data stored in HDFS, Hive, or cloud storage (S3 ADLS).
- Collaborate with data engineers, analysts, and business teams.
- Ensure data quality, integrity, and governance.
- Debug and troubleshoot data processing issues.
- Automate workflows using scheduling tools (Airflow, Oozie, etc. ).
- Write clean, scalable, and efficient code.
Requirements:
- Strong proficiency in Python and PySpark.
- Good experience with Apache Spark (RDDs, DataFrames, Spark SQL).
- Knowledge of Hadoop ecosystem (HDFS, Hive).
- Experience in ETL pipeline development.
- Familiarity with SQL and database concepts.
- Experience with data formats (Parquet, ORC, JSON, CSV).
- Basic understanding of distributed computing concepts.
- Exposure to version control tools (Git).