About the role
This role focuses on building and maintaining scalable data pipelines and ETL processes to support our company-wide Data and Analytics solutions, with particular emphasis on our OpenSearch (AWS) and Elasticsearch (Azure) environments. You'll be responsible for designing and developing performant, maintainable data processing systems that handle large volumes of data efficiently across our multi-cloud search platforms.
Responsibilities:
- Design, develop, and maintain scalable ETL pipelines that process data from various sources into our OpenSearch (AWS) and Elasticsearch (Azure) platforms.
- Implement and optimize indexing strategies, mapping configurations, and query patterns for search performance.
- Build reliable data workflows with appropriate error handling, monitoring, and alerting specific to search platforms.
- Configure and maintain OpenSearch/Elasticsearch clusters, including sharding strategies, replication, and security settings.
- Create data transformation processes that prepare data for optimal search indexing and retrieval.
- Develop data validation and quality control procedures to ensure accuracy and consistency in search indices.
- Implement and manage data refresh processes to maintain index currency while minimizing performance impact.
- Design and implement proper data retention policies and archiving solutions for search data.
- Ensure search data is optimized for their analytical needs.
- Create and maintain documentation for search platform architecture, data models, and pipeline processes.
- Provide production support for search infrastructure and troubleshoot performance issues.
- Mentor team members on best practices for working with search technologies.
Requirements:
- 5-8 years of experience in data engineering with a significant focus on search technologies.
- Proven expertise with OpenSearch on AWS and/or Elasticsearch on Azure environments.
- Strong proficiency in Python for data manipulation and ETL process development.
- Advanced knowledge of search query DSL and optimization techniques.
- Experience designing and implementing large-scale data pipelines for search platforms.
- Hands-on experience with data ingestion from various sources (databases, streams, files) into search indices.
- Familiarity with AWS and Azure cloud services related to data processing and search.
- Understanding of distributed search architecture and scaling considerations.
- Experience with monitoring and performance tuning of search clusters.
- Knowledge of data modeling for search optimization (field mappings, analyzers, tokenizers).
- Ability to balance search relevance with performance requirements.
- Strong problem-solving skills and attention to detail.
- Excellent communication skills to work effectively with cross-functional teams.
- Self-motivated with the ability to work independently in a dynamic environment.
Our Minimum Requirements:
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
- Proven experience with OpenSearch on AWS and Elasticsearch on Azure.
- Proficiency in Python and search query DSL.
- Experience building ETL pipelines that feed search platforms at scale.
- Understanding of search indexing concepts and optimization techniques.
- Experience with version control.