About the role
The core responsibilities for the job include the following:
Ingestion Operations and Pipeline Management:
- Operate, monitor, and maintain the telemetry ingestion stack across multiple data center sites, including Node Exporter, JMX Exporter, OpenTelemetry agents and collectors, Vector agents and aggregators, SNMP, Blackbox, and syslog collection.
- Manage Vector pipelines, including file sources, multiline parsing, VRL transformations, disk buffering, and sink health, ensuring reliable log collection during service restarts and log rotation.
- Manage Apache Kafka streaming infrastructure, including topic configurations, partitions, retention, replication health, and consumer-group offsets.
- Troubleshoot consumer lag, stalled consumers, and ingestion failures while ensuring minimal telemetry loss.
- Maintain consistent telemetry labels and resource attributes to enable correlation across metrics, logs, and traces.
Onboarding, Upgrades, and Data Validation:
- Deploy, configure, and upgrade telemetry agents across application, Kafka, Cassandra, KeyDB, PostgreSQL, network, and security hosts.
- Validate network and security requirements, including routing, firewall rules, port connectivity, DNS resolution, and TLS certificates, before onboarding.
- Perform regular health checks, capacity reviews, certificate renewals, patching, and version upgrades.
- Validate configurations and rollback procedures in lower environments before production deployment.
- Verify telemetry completeness after every change and troubleshoot data gaps, and count mismatches and cardinality issues.
Monitoring, Incident Management, and Security:
- Monitor ingestion health through scrape success rates, ingestion volumes, consumer-group lag, buffer utilization, queue depth, and data freshness.
- Identify and resolve silent source loss, missing telemetry, and pipeline failures.
- Build and maintain Grafana dashboards and alert rules for infrastructure and ingestion monitoring.
- Respond to incidents within defined SLA timelines, perform root cause analysis, and escalate unresolved issues through established escalation procedures.
- Ensure secure collection endpoints by maintaining appropriate OTLP receiver bindings, read-only log mounts, and preventing unauthorized access to exposed ingestion endpoints.
Automation and Documentation:
- Automate operational activities such as agent deployments, configuration management, verification checks, log analysis, and shift reporting using Bash and Python.
- Develop and maintain operational runbooks, Standard Operating Procedures (SOPs), onboarding guides, and troubleshooting documentation.
- Maintain incident records, RCA reports, and shift handover documentation.
- Identify opportunities to improve operational efficiency and reduce manual intervention.
24×7 Support & SLA Management:
- Provide continuous operational support across morning, evening, and night shifts, including weekends and public holidays.
- Ensure structured shift handovers covering open incidents, ingestion health, suppressed alerts, and ongoing activities. All handovers must be documented.
- Follow defined incident response and escalation timelines:
- P1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, with further escalation up to the Incident Commander at 30 minutes.
- P2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.
- Participate in structured shift overlap and backfill arrangements to ensure uninterrupted operational coverage during planned and unplanned absences.
Requirements:
- Experience: 3+ Years.
- Linux Administration: Strong experience with RHEL/Ubuntu, command-line operations, systemd, networking, and performance troubleshooting.
- OpenTelemetry (OTel): Hands-on experience with Java agent instrumentation, Collector pipelines, OTLP, receivers, processors, and exporters.
- Vector.dev : Experience with file sources, multiline parsing, VRL transforms, disk buffering, and sinks.
- Prometheus Ecosystem: Knowledge of Node Exporter, JMX Exporter, SNMP Exporter, Blackbox Exporter, and PromQL.
- Prometheus / VictoriaMetrics: Experience with scrape configurations, relabeling, and remote write.
- Log & Trace Management: Experience with VictoriaLogs (LogsQL) or OpenSearch, along with exposure to trace storage systems.
- Apache Kafka: Understanding of topics, partitions, consumer groups, offsets, lag and retention.
- Containerisation: Hands-on experience with Docker and working knowledge of Kubernetes.
- Grafana: Ability to build dashboards, configure panel queries and set up alert rules.
- Scripting & Automation: Proficiency in Bash and Python for operational automation.
- Networking: Strong understanding of TCP/IP, routing, DNS, TLS, firewall rules and port analysis.
- Version Control: Working knowledge of Git and configuration-as-code practices.
- Incident Management: Experience in incident triage, escalation, root cause analysis (RCA) and documentation.
- Willingness to work in a 24×7 rotational shift environment, including night shifts, weekends and public holidays.
Additional Requirements:
- Strong troubleshooting and analytical skills.
- Ability to work independently in a production support environment.
- Good communication, documentation, and incident coordination skills.
- Comfortable working in a critical 24×7 production environment with strict SLA commitments.