About the role
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
- Build and optimize ETL/ELT workflows for ingesting, processing, transforming, and loading structured and semi-structured data.
- Develop and maintain data processing solutions using Apache Spark and Databricks.
- Integrate Databricks with AWS services to build secure, scalable, and cost-effective data platforms.
- Work with AWS services such as Amazon S3 and other relevant data services for data storage, ingestion, and processing.
- Implement data transformation, cleansing, validation, and quality checks to ensure data accuracy and consistency.
- Optimize Spark jobs, SQL queries, and data pipelines for performance, scalability, and cost efficiency.
- Work with Delta Lake and lakehouse architecture to support reliable data storage and processing.
- Implement appropriate data access controls and security practices across cloud-based data environments.
- Troubleshoot data pipeline failures, performance bottlenecks, and data quality issues.
- Collaborate with data architects, analysts, BI developers, and other engineering teams to understand requirements and deliver data solutions.
- Maintain technical documentation, data pipeline specifications, and development standards.
- Strong hands-on experience in data engineering and data pipeline development.
- Practical experience with Databricks and AWS cloud services.
- Strong programming skills in Python and PySpark.
- Proficiency in SQL, including complex queries, joins, aggregations, and data transformations.
- Good understanding of Apache Spark architecture and distributed data processing.
- Experience developing ETL/ELT pipelines and handling large volumes of data.
- Knowledge of data modelling, data warehousing concepts, and data integration patterns.
- Experience with cloud-based data storage and processing, particularly Amazon S3.
- Strong troubleshooting, analytical, and problem-solving skills.
- Ability to work collaboratively with technical and business stakeholders.
- Experience with Delta Lake, Lakehouse architecture, and Databricks Unity Catalog.
- Knowledge of AWS Glue, AWS IAM, AWS Lambda, and other relevant AWS data services.
- Experience with workflow orchestration and scheduling of data pipelines.
- Familiarity with CI/CD, Git, automated testing, and deployment practices.
- Understanding of data governance, metadata management, and data quality frameworks.
- Experience with streaming data processing or incremental data ingestion.