Data Engineer - Databricks

Job Description – Data Engineer (AWS + Databricks)

Experience: 5+ Years

Location: Flexible / Hybrid

Key Responsibilities

• Build scalableETL/ELT pipelinesusingDatabricks (PySpark, Spark SQL, Delta Lake).
  

• Design and implementMedallion Architecture(Bronze, Silver, Gold) for Lakehouse.
  

• Developbatch and real-time pipelinesusingSpark Streaming,AWS Kinesis, orGlue Streaming.
  

• Manage data ingestion from diverse sources – RDBMS, APIs, S3, Kafka.
  

• Performdata profiling, validation, and quality checks.
  

• OptimizeSpark jobs, partitions, and cluster configurations for performance.
  

• Implement and manage data infrastructure usingAWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR.
  

• Set upCI/CDpipelines (GitHub Actions,CodePipeline, Jenkins).
  

• Collaborate with analysts and data scientists to deliver analytical datasets.
  

Technical Skills

• Databricks:PySpark, Spark SQL, Delta Lake, UnityCatalog, Job Workflows.
  

• AWS:S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch.
  

• Programming:Python (pandas), SQL (window functions, optimization).
  

• IaC& DevOps:Terraform / CloudFormation, Git, Jenkins, GitHub Actions.
  

• Performance Tuning:Spark optimizations, Z-ordering, partitioning, caching.