About the role
Roles & Responsibilities
- Work with the engineering teams to design, build, monitor and maintain the system for high availability and scalability.
- Define SLIs and SLOs with the engineering teams, track them, and use error budgets to guide reliability work and release decisions.
- Take part in design reviews and make architecture decisions that improve reliability, scalability, security and operability before systems reach production.
- Build security into the infrastructure and delivery pipelines, including access control, secrets management, network policies and image and dependency scanning.
- Work with the security team to track vulnerabilities and audit findings, and close them within agreed SLAs.
- Write production-grade software, deployment frameworks, tracers, monitors and auto remediation tools, and automate the processes.
- Follow an AI-first approach to operations. Use AI tools and agents for incident triage, log and alert analysis, runbook execution, change review and documentation, and build our own where off-the-shelf tools fall short.
- Manage cloud infrastructure as code (Terraform / Pulumi) and keep environments consistent, reviewable and reproducible.
- Lead and participate in performance, load and stress tests; identify bottlenecks, opportunities for optimization and capacity planning.
- Participate in the on-call rotation and lead incident response as needed. Write postmortems and follow through on the fixes.
- Identify repetitive operational work and engineer it away.
- Review existing processes and recommend changes or institute new processes as necessary, including the areas of monitoring, alerting, upgrades, and tuning, etc.
- Create quality project documentation of architecture designs, implementation plans, design documents, runbooks, test plans, etc.
Mandatory Skills
- Excellent programming skills in at least one language (Go, Python, Java, Node or similar), with production code you can walk us through. Experience with Go is a plus.
- Experience applying SRE practices: defining SLIs and SLOs, error budgets, and using them to drive design and prioritization decisions.
- Experience designing and building reliable, scalable systems on Google Cloud Platform (GKE, Compute Engine, Cloud Storage, Pub/Sub, VPC, IAM). Working knowledge of AWS or other clouds.
- Experience with infrastructure as code using Terraform or Pulumi.
- Experience with Container and Cloud technologies like Docker/Kubernetes.
- Experience with CI/CD using Argo CD, Jenkins, helm charts etc.
- Experience in performance diagnostics, capacity planning, performance architecture design, performance tuning, performance monitoring.
- Experience with the use, maintenance and configuration of monitoring, metrics and logging infrastructure (Datadog, Grafana, Prometheus, New Relic, etc.)
- Experience leading incident response and writing blameless postmortems, with a focus on removing the cause rather than handling the symptom.
- Comfortable working with modern nosql databases and big data platforms (MongoDB, Redis, etc.)
- Total experience of 10+ years, with a software development background and 3+ years working with GCP, Kubernetes and large-scale distributed systems.