About the role
- Design, build, and maintain ETL/ELT pipelines for large-scale data ingestion, transformation, and loading.
- Develop and optimize Spark and PySpark jobs for batch and real-time data processing.
- Work with AWS services (S3, Glue, Lambda, Redshift, Athena, EMR) to manage the data ecosystem.
- Support the design and implementation of Data Lake and Data Warehouse architectures.
- Implement data validation, partitioning, and schema management for efficient query performance.
- Collaborate with data analysts and BI teams to ensure data availability and consistency.
- Maintain data lineage, metadata, data quality, and governance.
- Implement monitoring and alerting for data pipelines.
- Use Git and CI/CD tools to manage code and automate deployment of data workflows.
- Education: Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.
- Experience: 3–5 years of hands-on experience in data engineering, pipeline development, or cloud-based data systems.
- Core Skills: Strong knowledge of SQL and Python/PySpark.
- Cloud & Data Stack: Practical experience with AWS (S3, Glue, Lambda, Redshift, Athena, EMR, Step Functions).
- Architecture: Solid understanding of Data Lake architecture, ETL/ELT frameworks, and data warehousing concepts.
- Frameworks: Familiarity with Delta Lake, Spark SQL, or other big data frameworks, along with data modeling and performance tuning.
- Exposure to GCP (BigQuery, Dataflow) or Azure (Data Factory, Synapse, Databricks).
- Experience with Databricks, PostgreSQL, MySQL, NoSQL, or Airflow.
- Understanding of DevOps practices, CI/CD pipelines, and infrastructure automation.
- Prior experience in an EdTech or public data ecosystem.