About the role
Responsibilities:
- Design, build, and operate a production-grade AI Gateway / shared LLM infrastructure layer serving multiple downstream consumers.
- Implement multi-provider LLM routing across Anthropic, Gemini, OpenAI, and other providers, with intelligent fallback logic.
- Build usage and cost monitoring, quota management, rate limiting, and centralized logging for LLM traffic.
- Establish evaluation pipelines, prompt/version management, guardrails, and PII protection mechanisms.
- Build and maintain MCP-based tool-use and agent infrastructure to support downstream agentic applications.
- Own production operations for the platform — reliability, scaling, incident response, and performance tuning.
- Partner with applied ML and product teams to expose LLM capabilities as reliable, governed infrastructure.
Requirements:
- Strong, hands-on Python engineering skills, with production-grade code quality.
- Deep experience with GCP, GKE, and Kubernetes for deploying and scaling services.
- Proven experience building multi-provider LLM routing systems (Anthropic, Gemini, OpenAI, etc. ).
- Experience implementing usage/cost monitoring, quotas, rate limits, logging, and fallback mechanisms in production.
- Experience building evaluation frameworks, prompt/version management, guardrails, and PII protection.
- Hands-on experience with MCP, tool-use patterns, and agent infrastructure.
- Strong production operations background — this must be demonstrable, not POC-only.
- Avoid: prompt engineers, POC-only consultants, architecture-only profiles with no hands-on ops experience.