About the role
The Lead Developer on this team should be a data engineer, familiar with SQL, Spark, and either Databricks or Athena/Iceberg/Trino. They should understand how to write and optimize ETL/ELT jobs to ingest raw data from a variety of sources into bronze tables, refine that data into scrubbed/silver tables, and optimize/tune those datasets into production/gold tables. They should be comfortable with relational and non-relational databases like SQL and PostgreSQL, big data systems such as Spark, and cloud deployments of software and ETL jobs in AWS. Airflow experience is a bonus. Fluency in Python/Scala is ideal.
Responsibilities:
- Architect and develop well-designed, testable, efficient, and secure code.
- Analyze complex problems and devise innovative solutions, adapting existing approaches to resolve diverse challenges with limited information.
- Exercise evaluation, judgment, and interpretation to independently select appropriate courses of action, with work reviewed at key milestones.
- Engage in the entire software development lifecycle, including CI/CD processes. Mentor and support new and junior team members.
- Creatively assess and resolve a wide range of issues, suggesting alternative approaches as needed.
- Work closely with data consumption stakeholders and display strong analytical knowledge to define data models that are both efficient and cost-effective.
Requirements:
- Bachelor's or advanced degree in Computer Science, Software Engineering, or a related field.
- A minimum of 10 years of full-time work experience as a software developer.
- Proficiency in technologies like Apache Spark, Databricks, Kafka, SQL, and Terraform.
- Strong programming experience in Python, Golang, Java, Scala, or another advanced object-oriented language.
- Hands-on experience with creating robust ETL pipelines.
- In-depth knowledge of data structures and algorithms.
- Skills in writing unit tests, data quality checks, and automated data testing frameworks.
- Proven backend development experience, including work on scalable, high-availability services.
- Excellent communication and interpersonal skills, with a track record of effective collaboration across cross-functional teams.
Additional Skills:
- Experience with AWS technologies.
- Practical experience with Kubernetes for orchestration and containerization.
- Proficiency in designing, implementing, and optimizing data models for data warehouses, databases, and data lakes.
- Familiarity with data warehousing concepts and platforms like Databricks, Iceberg, Snowflake, Amazon Redshift, or Google BigQuery.
- Experience with Schema Registries and Data Catalogs, e. g. AWS Glue, Unity Catalog.
- Experience with real-time processing systems like Apache Kafka Streams and Kinesis.
- Experience with monitoring tools like Prometheus, Grafana, or cloud-native monitoring solutions, and skills in optimizing performance for large-scale data processing.