About the role
Experience range- 3-5 yrs
Job Summary:
We are seeking a skilled Airflow Developer to lead the migration of ETL workflows from Talend to Apache Airflow. The ideal candidate will have hands-on experience in designing, developing, and deploying data pipelines using Airflow, and a solid understanding of Talend ETL processes. This role involves analyzing existing Talend jobs, re-engineering them in Airflow, and ensuring seamless integration with data sources and destinations.
Key Responsibilities:
- Analyze existing Talend ETL jobs and understand business logic and data flow.
- Design and implement equivalent workflows in Apache Airflow using Python.
- Develop DAGs (Directed Acyclic Graphs) for scheduling and monitoring data pipelines.
- Ensure data quality, error handling, and logging in Airflow pipelines.
- Collaborate with data engineers, analysts, and business stakeholders to validate migrated jobs.
- Optimize performance and scalability of Airflow workflows.
- Document migration processes, pipeline architecture, and operational procedures.
- Provide support during UAT and production deployment.
Required Skills:
- Strong experience with Apache Airflow (2+ years).
- Hands-on experience with Talend ETL tools (Open Studio or Talend DI).
- Proficiency in Python for scripting and pipeline development.
- Experience with SQL and working with relational databases (e.g., PostgreSQL, MySQL, Oracle).
- Familiarity with cloud platforms (AWS, GCP, or Azure) and cloud-native data services.
- Understanding of CI/CD pipelines, version control (Git), and DevOps practices.
- Knowledge of data warehousing concepts, ETL best practices, and data governance.
Preferred Qualifications:
- Experience with containerization (Docker) and orchestration (Kubernetes).
- Exposure to Airflow plugins, sensors, and custom operators.
- Experience with monitoring tools (e.g., Prometheus, Grafana).
- Talend certification or Airflow certification is a plus.
Soft Skills:
- Strong analytical and problem-solving skills.
- Excellent communication and documentation abilities.
- Ability to work independently and in a team environment.
- Agile mindset and experience working in Scrum teams.