About the role
Responsibilities:
- Design, build, and optimize scalable data pipelines for batch, streaming, and API-based data ingestion.
- Develop and maintain enterprise data platforms using Snowflake, ensuring high performance, governance, security, and efficient data sharing.
- Build and manage transformation workflows using dbt, including reusable models, testing, documentation, and data contracts.
- Develop scalable data processing solutions using Apache Spark, PySpark, and Databricks.
- Implement modern Lakehouse (Medallion) architecture with Bronze, Silver, and Gold data layers.
- Design robust ingestion frameworks supporting CDC, batch, event-driven, and API integrations.
- Optimize complex SQL queries and large-scale data transformations for performance and reliability.
- Build orchestration workflows using Apache Airflow, Azure Data Factory, or equivalent tools.
- Implement data quality validation using Great Expectations, dbt tests, or similar frameworks.
- Develop CI/CD pipelines for data engineering workflows using Git and automated deployment practices.
- Collaborate with data scientists, AI engineers, business analysts, and client stakeholders to deliver reliable and scalable data solutions.
- Support AI and GenAI initiatives by preparing high-quality datasets and enabling data pipelines for RAG-based applications.
Requirements:
- Experience: 4-12 years.
- Strong hands-on experience with Snowflake, including architecture, performance tuning, data sharing, security, and governance.
- Expertise in Databricks, including Delta Lake, Spark, Unity Catalog, and notebooks.
- Production experience with dbt for data transformations, testing, documentation, and data contracts.
- Strong programming skills in Python, including PySpark, pandas, and custom data engineering frameworks.
- Extensive experience with Apache Spark for large-scale distributed data processing.
- Experience with cloud platforms: AWS: S3 Glue, Redshift, Lambda; Azure: Azure Data Factory (ADF), ADLS Gen2 Azure Databricks, Synapse Analytics
- Advanced SQL skills with expertise in query optimization and complex transformations.
- Experience implementing CDC, batch, streaming, event-driven, and API-based ingestion pipelines.
- Strong understanding of Lakehouse/Medallion architecture.
- Experience implementing data quality frameworks such as Great Expectations, dbt tests, or equivalents.
- Hands-on experience with workflow orchestration tools such as Apache Airflow or Azure Data Factory.
- Experience with Git, version control, CI/CD pipelines, and automated deployment practices.
Required Technical Skills:
- Strong hands-on experience with Snowflake (Mandatory).
- Advanced SQL programming, Python, dbt (Data Build Tool), and Apache Airflow.
- Data Modeling (Dimensional and Relational), ETL/ELT Pipeline Development, Spark/PySpark.
- Cloud Platforms: AWS, Azure, or Google Cloud Platform (GCP), Git, and CI/CD practices.