NextHire Try for free

Wits Innovation Lab

Posted 17 Nov 2026

SRE Engineer

Apply with NextHire Auto Apply fills in the application for you.
Employment
Full time
Team
IT Operations
Posted
17 Nov 2026

About the role

The core responsibilities for the job include the following:


Ingestion Operations and Pipeline Management:

  • Operate, monitor, and maintain the telemetry ingestion stack across multiple data center sites, including Node Exporter, JMX Exporter, OpenTelemetry agents and collectors, Vector agents and aggregators, SNMP, Blackbox, and syslog collection.
  • Manage Vector pipelines, including file sources, multiline parsing, VRL transformations, disk buffering, and sink health, ensuring reliable log collection during service restarts and log rotation.
  • Manage Apache Kafka streaming infrastructure, including topic configurations, partitions, retention, replication health, and consumer-group offsets.
  • Troubleshoot consumer lag, stalled consumers, and ingestion failures while ensuring minimal telemetry loss.
  • Maintain consistent telemetry labels and resource attributes to enable correlation across metrics, logs, and traces.


Onboarding, Upgrades, and Data Validation:

  • Deploy, configure, and upgrade telemetry agents across application, Kafka, Cassandra, KeyDB, PostgreSQL, network, and security hosts.
  • Validate network and security requirements, including routing, firewall rules, port connectivity, DNS resolution, and TLS certificates, before onboarding.
  • Perform regular health checks, capacity reviews, certificate renewals, patching, and version upgrades.
  • Validate configurations and rollback procedures in lower environments before production deployment.
  • Verify telemetry completeness after every change and troubleshoot data gaps, and count mismatches and cardinality issues.


Monitoring, Incident Management, and Security:

  • Monitor ingestion health through scrape success rates, ingestion volumes, consumer-group lag, buffer utilization, queue depth, and data freshness.
  • Identify and resolve silent source loss, missing telemetry, and pipeline failures.
  • Build and maintain Grafana dashboards and alert rules for infrastructure and ingestion monitoring.
  • Respond to incidents within defined SLA timelines, perform root cause analysis, and escalate unresolved issues through established escalation procedures.
  • Ensure secure collection endpoints by maintaining appropriate OTLP receiver bindings, read-only log mounts, and preventing unauthorized access to exposed ingestion endpoints.


Automation and Documentation:

  • Automate operational activities such as agent deployments, configuration management, verification checks, log analysis, and shift reporting using Bash and Python.
  • Develop and maintain operational runbooks, Standard Operating Procedures (SOPs), onboarding guides, and troubleshooting documentation.
  • Maintain incident records, RCA reports, and shift handover documentation.
  • Identify opportunities to improve operational efficiency and reduce manual intervention.


24×7 Support & SLA Management:

  • Provide continuous operational support across morning, evening, and night shifts, including weekends and public holidays.
  • Ensure structured shift handovers covering open incidents, ingestion health, suppressed alerts, and ongoing activities. All handovers must be documented.
  • Follow defined incident response and escalation timelines:
  • P1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, with further escalation up to the Incident Commander at 30 minutes.
  • P2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.
  • Participate in structured shift overlap and backfill arrangements to ensure uninterrupted operational coverage during planned and unplanned absences.


Requirements:

  • Experience: 3+ Years.
  • Linux Administration: Strong experience with RHEL/Ubuntu, command-line operations, systemd, networking, and performance troubleshooting.
  • OpenTelemetry (OTel): Hands-on experience with Java agent instrumentation, Collector pipelines, OTLP, receivers, processors, and exporters.
  • Vector.dev : Experience with file sources, multiline parsing, VRL transforms, disk buffering, and sinks.
  • Prometheus Ecosystem: Knowledge of Node Exporter, JMX Exporter, SNMP Exporter, Blackbox Exporter, and PromQL.
  • Prometheus / VictoriaMetrics: Experience with scrape configurations, relabeling, and remote write.
  • Log & Trace Management: Experience with VictoriaLogs (LogsQL) or OpenSearch, along with exposure to trace storage systems.
  • Apache Kafka: Understanding of topics, partitions, consumer groups, offsets, lag and retention.
  • Containerisation: Hands-on experience with Docker and working knowledge of Kubernetes.
  • Grafana: Ability to build dashboards, configure panel queries and set up alert rules.
  • Scripting & Automation: Proficiency in Bash and Python for operational automation.
  • Networking: Strong understanding of TCP/IP, routing, DNS, TLS, firewall rules and port analysis.
  • Version Control: Working knowledge of Git and configuration-as-code practices.
  • Incident Management: Experience in incident triage, escalation, root cause analysis (RCA) and documentation.
  • Willingness to work in a 24×7 rotational shift environment, including night shifts, weekends and public holidays.


Additional Requirements:

  • Strong troubleshooting and analytical skills.
  • Ability to work independently in a production support environment.
  • Good communication, documentation, and incident coordination skills.
  • Comfortable working in a critical 24×7 production environment with strict SLA commitments.

Similar jobs

See every job

85% of jobs are filled through networking

The AI Outreach Agent writes to recruiters and hiring managers at the companies you pick, from your own Gmail.

Try AI Outreach