About the role
Data Ingestion Specialist
Role Summary
We are looking for a Medior Data Ingestion Specialist to support development, operation, and maintenance of data ingestion platform orchestrated by Apache Airflow. The role focuses on onboarding and maintaining data integrations from multiple source systems, troubleshooting operational issues, and supporting release and governance processes
Key Responsibilities
Data Integration & Pipeline Development
Configure and maintain data ingestion pipelines
Develop and support Python-based ingestion components
Integrate data:
Relational databases (Oracle, PostgreSQL, MS SQL, MySQL, DB2, etc)
Apache Kafka
AWS S3, flat files
Elasticsearch, Hive, Impala, hBase
Troubleshoot data flow, connectivity, and integration issues
Ingestion to Databricks, Databricks loader
Platform Operations
Support incident, problem, and change management activities
Maintain Python libraries and dependencies
Create ETL/ELT pipelines focused on source ingestion
Monitor pipeline execution and resolve operational issues
Assist with platform upgrades, patching, and maintenance activities
Release & Governance
Support Dev/Test/Prod deployment processes
Review configuration and template changes
Ensure compliance with governance, security, and operational standards
Maintain technical and operational documentation
Collaboration
Work closely with architects, engineering teams, and platform owners
Collaborate with security, audit, and infrastructure teams
Participate in release planning and communication activities
Nice-to-Have Skills
Knowledge of ETL Python frameworks (e.g. Meltano)
Experience with other ETL/ELT tools
Experience with Git, CI/CD pipelines, and DevOps practices
Knowledge of Linux environments
Experience with Jira and ITSM tooling
Exposure to cloud-based data platforms
Candidate Profile
Medior-level engineer capable of working independently on standard tasks
Able to create and resolve customers' requirements
Able to analyse integration issues and identify root causes
Comfortable working in an enterprise environment with governance and security requirements
Good communication skills and ability to collaborate with multiple stakeholders
Must-Have Skills
2+ years of experience with Python development
Practical (hands on) experience with Apache NiFi
Practical (hands on) experience with Apache Airflow
Experience integrating data from relational databases
ETL/ELT performance tuning, optimalizations
Basic experience with Apache Kafka
Basic experience with Elasticsearch
SQL knowledge and understanding of data integration concepts
Experience troubleshooting technical and connectivity issues
Understanding of incident and release management processes
Ability to create and maintain technical documentation
Strong analytical and problem-solving skills