About the role
We are seeking an experienced Senior Data Engineer to design, build, and optimize enterprise-scale data platforms on AWS using Databricks and Terraform. In this role, you will architect robust ETL/ELT pipelines, implement event-driven architectures, and enforce Infrastructure as Code (IaC) best practices. You will collaborate closely with data science, analytics, and platform engineering teams to maintain automated, reliable, and secure data ecosystems.
Responsibilities:
- Data Platform and Pipeline Engineering: Architect and deploy high-performance batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.
- Infrastructure as Code (IaC): Provision, manage, and scale cloud infrastructure, Databricks workspaces, clusters, storage (S3), and access controls using Terraform.
- Cloud and Event-Driven Integration: Integrate cloud-native AWS services (S3 Lambda, IAM, EKS/Batch, CloudWatch) and streaming platforms (Apache Kafka/Kinesis) into enterprise data pipelines.
- CI/CD and Operational Excellence: Implement automated CI/CD deployments using tools like GitHub Actions, Jenkins, and Databricks Asset Bundles (DAB).