About the role
Responsibilities:
- You will be responsible for streamlining and tuning existing Big Data systems and pipelines, and building new ones.
- Ensuring the systems run efficiently and at minimal cost is a top priority.
- You will be making changes to the underlying systems, and if an opportunity arises, you can contribute your work back into the open source.
- You will also be responsible for supporting internal customers and on-call services for the systems we host.
- Ensuring we provide a stable environment and a great user experience is another top priority for the team.
Requirements:
- 10+ years of production experience building big data platforms based on Spark, Trino, or equivalent.
- Strong programming expertise in Java, Scala, Kotlin, or another JVM language.
- A robust grasp of distributed systems concepts, algorithms, and data structures.
- Strong familiarity with the Apache Hadoop ecosystem: Spark, Kafka, Hive/Iceberg/Delta Lake, Presto/Trino, Pinot, etc.
- Experience working with at least 3 of the technologies/tools mentioned here: Big Data / Hadoop, Kafka, Spark, Trino, Flink, Airflow, Druid, Hive, Iceberg, Delta Lake, Pinot, Storm, etc.
- Extensive hands-on experience with public cloud AWS or GCP.
- BS/MS degree in CS or equivalent.
- AI Literacy /AI growth mindset.