About the role
We are looking for a skilled Data Engineer with 3-4 years of experience in developing, maintaining, and optimizing data pipelines and data processing solutions. The ideal candidate should have strong hands-on experience in Python, SQL, PySpark, and big data technologies, along with exposure to cloud-based data platforms.
Responsibilities:
- Design, develop, and maintain scalable data pipelines for processing structured and unstructured data.
- Develop data transformation and processing workflows using Python, PySpark, and SQL.
- Work with large datasets using Apache Spark and related big data technologies.
- Build and maintain ETL/ELT pipelines to extract, transform, and load data from multiple sources.
- Work with data storage and processing technologies such as Hive, Hadoop, and cloud-based data platforms.
- Optimize SQL queries and data processing jobs for improved performance.
- Perform data validation, troubleshoot pipeline failures, and ensure data quality.
- Collaborate with data analysts, data scientists, and other engineering teams to support business requirements.
- Monitor scheduled jobs and ensure the reliability and efficiency of data pipelines.
Requirements:
- 3-4 years of relevant experience in data engineering.
- Strong problem-solving and analytical skills.
- Ability to work independently on data engineering tasks.
- Good communication and collaboration skills.
Required Skills:
- Python: Strong hands-on experience in Python programming and data processing.
- SQL: Good knowledge of complex SQL queries, joins, aggregations, and query optimization.
- PySpark / Apache Spark: Experience in distributed data processing and large-scale data transformations.
- ETL/ELT: Hands-on experience building and maintaining data pipelines.
- Big Data Technologies: Exposure to Hive, Hadoop, Spark SQL, or similar technologies.
- Data Processing: Understanding of data warehousing concepts, data modeling, and file formats such as Parquet and JSON.
- Workflow Orchestration: Exposure to Apache Airflow or similar scheduling tools.
- Cloud Platforms: Exposure to Azure, AWS, or GCP is preferred.
- Version Control: Working knowledge of Git.
Good to Have:
- Experience with Azure Databricks, Azure Data Factory, or similar cloud data engineering services.
- Familiarity with data lake and data warehouse architectures.
- Experience with performance tuning and optimizing Spark jobs.
- Understanding of data quality checks and pipeline monitoring.