About the role
Responsibilities:
- Create and maintain existing Scala/Spark jobs for data transformation and aggregation, from simple to complex data transformations involving structured and unstructured data.
- Produce unit tests for Spark transformations and helper performance optimization methods.
- Develop data processing pipelines, data storage, and management architecture.
- Define scalable calculation logic for interactive and batch use cases.
- Interact with infrastructure and data teams to produce complex analysis across data.
- You will be working on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, etc., and to help better measure the quality of map data.
- You will work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.
Requirements:
- Minimum of 5 years of work experience in Big Data platforms and building large-scale distributed systems with high availability.
- Experience in developing Spark applications using Spark RDD API, Spark-SQL, Spark GraphX API, Spark Streaming API, Spark MLlib API, and DataFrames APIs.
- Should have broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark workflows, Spark query tuning, and performance optimization.
- Solid in data structures and algorithm basics and should have good hands-on experience with the Scala programming language and strong investigative and problem-solving skills.
- Data ingestion, optimization techniques, data transformation, and aggregation pipeline design/development knowledge are required.
- Experience working on cutting-edge big data storage systems and technologies like Hadoop, HDFS, AWS S3, AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, and/or MySQL, etc.