About the role
As a Cloud Engineer II, you will be responsible for designing, building, and maintaining large-scale cloud-based data systems and infrastructure. You will develop data pipelines, implement data quality measures, and optimize systems to ensure efficient processing. This role involves collaborating with clients and stakeholders, translating business requirements into technical solutions, and applying best practices in cloud computing, data engineering, and security.
Responsibilities:
- Showcase strong analytical and problem-solving capabilities by consulting with clients and other stakeholders to understand business problems.
- Design, store, process, and maintain large-scale data systems and related infrastructure.
- Build data pipelines, implement data quality, develop data models, monitor, and optimize pipelines to minimize runtime while adhering to organizational and customer standards.
- Demonstrate strong conceptual understanding of data warehousing, ETL, data governance, security, cloud computing, and batch & real-time data processing.
- Apply strong execution knowledge of data modeling, SQL/NoSQL databases, SDLC practices, unit testing, and functional programming.
- Work with Medallion architecture patterns.
- Work on at least one major cloud platform.
- Execute multiple end-to-end data engineering projects.
- Manage conversations with client stakeholders, understand requirements, and translate them into technical outcomes.
Requirements:
- 2-4 years of relevant work experience.
- Strong proficiency in SQL.
- Experience working on any major cloud platform (AWS, Azure, or GCP).
- Working knowledge of ETL/orchestration tools such as IICS, Talend, Matillion, Airflow, Azure Data Factory, AWS Glue, GCP Composer, etc.
- Working knowledge of OLTP databases (Postgres, MySQL, SQL Server, etc. ).
- Working knowledge of data warehouses such as Snowflake, Redshift, Azure Synapse, Hive, BigQuery, etc.
- Proficient in at least one programming language used in data engineering, such as Python (or Scala/Rust/Java).
- Strong execution knowledge of data modeling (star schema, snowflake schema, fact/dimension tables).
- Proficient in Spark and related applications like Databricks, GCP DataProc, AWS Glue, EMR, etc.
- Experience with code and SQL query optimization.
- Strong knowledge of version control systems such as Git and designing CI/CD pipelines for continuous delivery.
- Understanding of code optimization and security best practices.
- Strong problem-solving skills and ability to analyze problems to develop effective solutions.
- Proficient in written and verbal communication.
- Exercises independent judgment to determine the best course of action within established practices and procedures.
- Good to have: cloud certification at an architect level, knowledge of unit/functional/integration testing, Kafka and real-time streaming, and CI/CD pipeline familiarity.