Lead Software Engineer - Python + Spark

MSA Role: Lead  Software Engineer – Python and Spark

Experience: 8 - 12 Years

• Design, develop, and maintain scalable data processing applications using Python, PySpark, Spark SQL, and distributed computing frameworks.
• Build, optimize, and support large-scale batch and real-time data pipelines using Apache Spark and cloud-native technologies.
• Develop high-performance ETL/ELT solutions for processing structured, semi-structured, and unstructured data from multiple sources.
• Write clean, reusable, and efficient Python code following software engineering best practices, design patterns, and coding standards.
• Optimize Spark applications through partitioning, caching, memory management, join optimization, and performance tuning techniques.
• Develop and maintain data ingestion frameworks, APIs, and integration services to support enterprise data workflows.
• Design and implement scalable data models and data transformation logic to support analytics, reporting, and machine learning use cases.
• Work with SQL and relational databases to develop complex queries, stored procedures, and data validation processes.
• Collaborate with Data Engineers, Architects, Data Scientists, and Business Analysts to understand requirements and deliver robust solutions.
• Build automated testing frameworks and implement unit, integration, and end-to-end testing for Python and Spark applications.
• Develop and maintain CI/CD pipelines using Git, Jenkins, Azure DevOps, or similar tools to enable automated deployment and release management.
• Leverage cloud platforms such as AWS, Azure, or GCP for application deployment, scaling, monitoring, and operational support.
• Implement logging, monitoring, alerting, and observability solutions to ensure application reliability and operational excellence.
• Troubleshoot production issues, conduct root cause analysis, and drive performance and stability improvements.
• Participate in code reviews, technical design discussions, and architecture reviews to maintain high-quality engineering standards.
• Follow Agile/Scrum methodologies and actively contribute to sprint planning, estimation, development, testing, and deployment activities.
• Ensure compliance with security, governance, and data privacy standards while developing enterprise applications.
• Mentor junior engineers and contribute to technical knowledge sharing and best practices across the team.

Profile Required:
Senior‑level Software Engineer with deep Databricks, AWS, and Python data platform expertise, capable of designing and leading enterprise‑scale AI‑enabled data solutions.

Minimum Qualifications:

• Strong experience in Python development (5+ years preferred).
• Hands-on expertise in Apache Spark / PySpark and Spark SQL.
• Strong knowledge of data structures, algorithms, and software design principles.
• Experience with ETL/ELT development and large-scale data processing.
• Strong SQL skills and experience working with relational and NoSQL databases.
• Experience with cloud platforms (AWS/Azure/GCP).
• Familiarity with CI/CD pipelines, Git, and DevOps practices.
• Experience with REST APIs and data integration frameworks.
• Strong debugging, performance tuning, and troubleshooting skills.
• Excellent communication and stakeholder management skills.

Preferred Qualifications:

• Experience building and operating AI agents for workflow automation and decision support.
• Strong understanding of data observability, platform reliability, and FinOps practices.
• Experience working in regulated or financial services domains.
• Exposure to enterprise‑scale AI governance, compliance, and audit requirements.
• Ability to influence architecture decisions and align stakeholders.

• Excellent communication skills with both technical and non‑technical audiences.