About the role
What will you be doing?:
- You will be discovering trends in the data sets and developing algorithms to transform raw data for further analytics.
- Create data pipelines to bring in data from various sources, with different formats, transforming it, and finally loading it to the target database.
- Implement ETL/ELT processes in the cloud using tools like Airflow, Glue, Stitch, Cloud Data Fusion, Dataflow.
- Design and implement data lakes, data warehouses, and data marts in AWS, GCP, or Azure using Redshift, BigQuery, PostgreSQL, etc.
- Create efficient SQL queries and understand query execution plans for tuning queries on engines like PostgreSQL.
- Perform performance tuning of OLAP/OLTP databases by creating indices, tables, and views.
- Write Python scripts for orchestration of data pipelines.
- Have thoughtful discussions with customers to understand their data engineering requirements. Break complex requirements into smaller tasks for execution.
What do we need from you?:
- Strong Python coding skills with basic knowledge of algorithms/data structures and their application.
- Strong understanding of data engineering concepts including ETL, ELT, data lake, data warehousing, and data pipelines.
- Experience designing and implementing data lakes, data warehouses, and data marts that support terabyte-scale data.
- A track record of implementing data pipelines on public cloud environments (AWS/GCP/Azure) is highly desirable.
- A clear understanding of database concepts like indexing, query performance optimization, views, and various types of schemas.
- Hands-on SQL programming experience with knowledge of windowing functions, subqueries, and various types of joins.
- Experience working with big data technologies like PySpark/Hadoop.
- A good team player with an ability to communicate with clarity.
- Show us your Git repo/blog!
Qualification:
- 3+ years of experience working on data engineering projects.
- 1-5 years of hands-on Python programming experience.
- Bachelor's/Master's degree in Computer Science is good to have.
- Courses or certifications in the area of data engineering will be given a higher preference.
- Candidates who have demonstrated a drive for learning and keeping up to date with technology by continuing to do various courses/self-learning will be given high preference.