About the role
• Designing and developing robust PySpark applications for large-scale data processing.
• Building and optimizing data ingestion, transformation, and storage processes.
• Implementing efficient algorithms and data structures for distributed computing.
• Collaborating with cross-functional teams to integrate data-driven solutions into business processes.
• Troubleshooting performance bottlenecks and ensuring high availability and reliability of data pipelines.
• Writing and optimizing SQL queries for data extraction and manipulation.
Required Skills and Qualifications:
• Bachelor’s/Master’s degree in Computer Science, Engineering, Proven experience (3-10 years) in Python development with a focus on PySpark.
• Strong understanding of distributed computing principles and experience with Apache Spark.
• Proficiency in SQL and experience with relational databases (MySQL, PostgreSQL, etc.).
• Experience with data serialization formats such as JSON, Parquet, Avro.
• Familiarity with cloud platforms (AWS, Azure, GCP) and containerization technologies (Docker, Kubernetes) is a plus.
• Excellent problem-solving skills and ability to work independently or as part of a team.
• Good communication skills with the ability to effectively collaborate with stakeholders.