Data Engineer

Responsibilities:

• Set up processes for data management, working on automated analytical modules.
• Continuously focus on improvement and automation by partnering with different teams.
• The person would lend data engineering support to the creation and execution of omnichannel campaigns for the bank.
• Prepare data layer and support for the analytics and data science team.
• Help in the integration of new sources of data and pipelines.
• Lead the data development of new analytical modules and automated customer journeys.
• Design, build, and optimise data pipelines using AWS S3 EMR, and Glue.
• Develop and maintain scalable ETL workflows for campaign, customer, deposit, and credit card domains.
• Automate tasks that are repetitive, and design ETL pipelines to reduce manual efforts.
• Bring in the innovation quotient that can assist in driving business digitally.
• Write and maintain complex data transformation logic using Python or PySpark scripts for ETL, data quality, and automation tasks.
• Monitor, troubleshoot, and resolve issues in scheduled data jobs (root cause analysis, escalations, bug fixes) using Airflow.
• Automate failure notifications, job monitoring, and maintenance history tracking.
• Coordinate with the source and Datalake teams for missing data or fixes.
• Design and implement automated data quality checks and reporting solutions.
• Investigate and resolve data quality issues.
• Collaborate with QA and production support teams for continuous process improvement.
• Work on the end-to-end data lifecycle from the data ingestion, data transformation and data consumption layers.
• Versed in API and its usability.
• Ability to work independently and handle your own development efforts.
• Excellent oral and written communication skills.
• Learn and use available analytic technologies.
• Identify key performance indicators and create an educational/deliverables path to achieve the same.
• Use background in data engineering and perform analysis.
• Work with BI analysts/engineers to create prototypes.
• Engage in the delivery and presentation of solutions.

Requirements:

• 2-5 years of work experience in data engineering with experience in cloud and big data.
• Strong hands-on knowledge of Python and PySpark.
• Has experience in processing large amounts.
• Proficiency in writing SQL queries is a must.
• Worked on the AWS/Azure/GCP cloud platform. AWS preferred.
• Experience in Airflow or similar orchestrations would be an added advantage.
• Experience with Jira or similar project management tools.
• Banking domain experience is a plus.
• Graduation: Bachelor of Science (B. Sc. ) / Bachelor of Technology (B. Tech. ) / Bachelor of Computer Applications (BCA).
• Post-Graduation: Master of Science (M. Sc. ) / Master of Technology (M. Tech. ) / Master of Computer Applications (MCA).

Technical Stack:

• Programming: PySpark, Python/Scala.
• AWS: S3 EC2 EMR, Glue, Athena.
• Orchestrator: Apache Airflow.
• Data Integration: Kafka, API, flat files, vendor feeds.
• DevOps: GoCD, Bitbucket, Jira.