About the role
Responsibilities:
- Will collaborate closely with our data science and ML teams across different cybersecurity domains to define ML infrastructure requirements and build critical data services.
- Can leverage MLOps best practices to design and develop scalable model training, evaluation, experimentation, and deployment workflows.
- Has extensive experience in ML training (local and distributed), feature extraction, and dataset creation.
- Is comfortable deploying software with CI / CD tools, including Jenkins, Harness, Terraform, etc.
- Is an expert at developing and deploying assets in the cloud, preferably AWS and Kubernetes, using IAC (infrastructure as code).
- Can build a workflow orchestration platform to be used by other developers.
Requirements:
- Has hands-on experience of 2+ years implementing data pipeline infrastructure for data ingestion and transformation near real-time availability of data for applications and ML pipelines.
- Has experience designing optimized solutions for ingestion, curation of large datasets.
- Has working knowledge of Data Lake technologies, data storage formats (Parquet, ORC, Avro), and query engines (Athena, Presto, Dremio), and associated concepts for building optimized solutions at scale.
- Maintains a proficient level in one of the following programming languages or similar- Python, Java, Go.
- Has experience with data pipeline tools (Flink, Spark, or Ray) and orchestration tools such as Airflow, Dagster, or Step Functions.
- Is an expert in implementing data streaming and event-based data solutions (Kafka, Kinesis, SQS/SNS or the like).