About the role
- Strong experience with Databricks platform and PySpark for big data processing.
- Proficiency in Python programming, especially for data engineering tasks.
- Deep understanding of SQL, including ability to write and optimize complex queries and develop stored procedures.
- Hands-on experience with AWS cloud services and cloud-based data engineering solutions.
- Expertise in building and managing scalable data pipelines using Spark, Databricks, and associated technologies.
- Familiarity with data warehousing, ETL/ELT processes, and big data ecosystems such as HDFS, Spark, and Kafka.
- Experience with Azure Data Factory (ADF) is often an added advantage.
- Knowledge of DevOps practices such as CI/CD, version control (Git), and automation for data pipelines.
- Experience in migrating and managing cloud data platforms.
- Strong communication skills to collaborate with business users and technical teams.
- Designing, developing, and optimizing scalable data pipelines and ETL workflows on Databricks and cloud environments.
- Converting SQL stored procedures into PySpark-based transformations using DataFrames.
- Collaborating with cross-functional teams to understand data requirements and deliver robust data solutions.
- Maintaining documentation for data processes, ensuring accuracy and accessibility.
- Supporting data analytics and real-time data streaming solutions.
- Ensuring data quality, governance, and efficient data extraction strategies.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- 5 to 8 years of relevant experience with demonstrated competency in Databricks, Python, PySpark, SQL, and AWS.
- Experience in agile development environments.
- Good understanding of modern data architectures like data lakes and data warehouses.