About the role
About the Role
Data Engineer (Python / ETL )
We are seeking a Senior Data Engineer to help refactor and optimize our existing data pipeline ecosystem. The current pipelines extract data from MS SQL Server (CRM system) and load it into Elasticsearch and MongoDB. As part of this initiative, you will lead the transition to extract data from a new data store — PostgreSQL (CURA) — while maintaining existing load semantics, data quality, and performance integrity.
This role demands deep technical expertise in Python-based ETL frameworks, distributed data systems, and hands-on data manipulation using libraries such as Pandas and NumPy. You will work closely with cross-functional teams to ensure a seamless migration with zero disruption to production workloads.
Key Responsibilities
- Analyze existing ETL pipelines (Python-based) that read from MS SQL Server and load into Elasticsearch and MongoDB.
- Redesign and refactor ETL components to extract data from PostgreSQL (CURA) as the new source system.
- Ensure no regression in data quality, latency, or system performance post-migration.
- Develop, test, and deploy changes in a containerized environment (Docker, Kubernetes).
- Write efficient SQL queries and optimize database interactions for performance.
- Collaborate with platform and QA teams to validate data consistency and ensure production stability.
- Monitor data pipeline performance and proactively identify optimization opportunities.
- Document data flow, transformation logic, and system dependencies.
Required Skills & Experience
- Strong hands-on experience (3+ years) building and maintaining large-scale ETL pipelines and distributed data systems.
- Expert-level proficiency in Python, with solid understanding of libraries like Pandas, NumPy, and preferably Polars.
- Proven ability to work with SQL Server and PostgreSQL — including schema understanding, performance tuning, and query optimization.
- Familiarity with Docker and Kubernetes for deploying and managing data workloads.
- Experience with Elasticsearch and MongoDB as data sinks.
- Strong debugging, troubleshooting, and performance analysis skills.
- Solid understanding of data quality assurance practices and impact analysis in production systems.
Preferred Qualifications
- Exposure to data migration or data source refactoring projects in production environments.
- Familiarity with CI/CD practices for data pipelines.
- Understanding of data versioning, metadata management, and monitoring frameworks.