Data Engineer - Azure

We are seeking an experienced Data Engineer to build and refine robust big data solutions. You will be responsible for designing high-performance ETL pipelines using Apache Spark across major cloud environments (Azure, AWS, or GCP), with a heavy focus on data engineering excellence and system optimization.

Responsibilities:

• Lead the end-to-end design and optimization of Databricks ETL pipelines to ensure high-quality data delivery.
• Develop scalable data ingestion and transformation workflows by leveraging the full power of the Apache Spark framework.
• Manage complex data lifecycles involving both structured and unstructured datasets from diverse originating sources.
• Fine-tune Spark jobs to balance peak performance with cost-efficiency and long-term scalability.
• Establish robust CI/CD patterns and version control standards for all Databricks-based workflows.
• Collaborate with stakeholders across engineering, data science, and analytics to translate business needs into technical solutions.
• Enforce rigorous security protocols and access management policies within the Databricks environment.
• Diagnose and resolve complex bottlenecks related to big data processing and system scalability.

Requirements:

• Proven track record of developing high-performance solutions using Databricks and Apache Spark (proficient in PySpark, Scala, or Java).
• Expert-level command of SQL for complex data modeling, transformation, and analytical querying.
• Deep familiarity with cloud ecosystems (Azure, AWS, or GCP), with specific expertise in storage and warehousing solutions like ADLS, S3 or BigQuery.
• Significant experience building end-to-end ETL/ELT pipelines and managing scalable Data Warehousing environments.
• Strong grip on CI/CD pipelines, Git version control, and DevOps methodologies tailored for data engineering.
• In-depth knowledge of Delta Lake and Lakehouse architecture, with a focus on advanced performance tuning and optimization.
• Skilled in managing massive-scale datasets within distributed computing environments.
• 5-7 years of specialized experience in Databricks development and big data architecture.
• Mandatory background in the Retail sector, with a deep understanding of industry-specific data challenges.

Preferred Skills:

• Skilled in leveraging Databricks MLflow to track, deploy, and scale machine learning models.
• Familiarity with modern data stack components such as Apache Kafka for streaming, Airflow for workflow automation, or Snowflake for cloud warehousing.
• Deep understanding of data privacy standards, access control policies, and corporate compliance requirements.
• Bachelor's or Master's degree in Computer Science, Data Engineering, or a strictly related technical discipline.