About the role
Key Responsibilities:
- Develop, optimize, and maintain data pipelines using Python.
- Work with large datasets and ensure efficient extraction, transformation, and loading (ETL).
- Integrate data from various internal and external sources.
- Build reusable Python modules and automation scripts.
- Work closely with data engineers, analysts, and stakeholders to understand requirements.
- Optimize performance of data processes and troubleshoot issues.
- Ensure data quality, validation, and integrity across systems.
- Deploy and monitor data workflows in production environments.
Technical Skills Required:
- Strong hands-on experience in Python (Pandas, NumPy, PySpark preferred).
- Experience with ETL development and building data pipelines.
- Good understanding of SQL, complex queries, and stored procedures.
- Experience with databases (PostgreSQL, MySQL, MongoDB, or similar).
- Exposure to big data frameworks (Spark/Hadoop) is a strong advantage.
- Experience with cloud platforms (AWS/Azure/GCP) for data services.
- Familiarity with API integrations, automation, and scheduling tools (Airflow, Cron).
- Knowledge of version control (Git) and CI/CD practices.
Preferred Qualifications:
- Advanced experience with Python libraries used in data workflows (Pandas, NumPy, PySpark, FastAPI/Flask for data services).
- Hands-on experience with data warehousing concepts and building scalable data models.
- Practical understanding of ML model pipelines and supporting data science teams with clean, production-ready datasets.
- Experience working with workflow orchestration tools (Airflow, Prefect, Luigi).
- Familiarity with containerization and deployment (Docker/Kubernetes) for data applications.
- Strong understanding of cloud-based data services (AWS Glue, Redshift, Azure Data Factory, BigQuery).
- Bachelor’s degree in Computer Science, Engineering, or a related technical field.