About the role
Responsibilities:
- Model serving performance and efficiency.
- Own throughput and cost per unit of work for self-hosted model serving stacks (vLLM, SGLang, TensorRT-LLM, TGI) at the level of scheduler configuration, batching policy, and memory layout: continuous/in-flight batching, chunked prefill, max_num_seqs / max_num_batched_tokens, admission control, preemption.
- Manage KV cache economics: PagedAttention block sizing, memory-utilization headroom, KV quantization, cache offload, and prefix/radix caching. Our extraction prompts carry long fixed schemas across every page, so cache hit rate is a first-class operating metric here.
- Quantize models to FP8 / INT8 / INT4 (AWQ, GPTQ, W8A8) and prove quality held using an accuracy harness on real production documents. Building and maintaining that harness is part of the role.
- Serve vision-language models at scale: image token budgets, tiling and dynamic resolution, DPI/resize tradeoffs, the vision encoder as an independent bottleneck, encoder output caching, page-level batching. Our workload is batch document throughput, not interactive chat.
- Make and defend parallelism and topology decisions: tensor vs pipeline parallelism vs replica scale-out, including when tensor parallelism hurts (e. g., PCIe-only nodes without NVLink, where N replicas at TP=1 often beat one TP=N).
- Improve accelerator packing and multi-tenancy: MIG, MPS, time-slicing, multi-LoRA serving, model load and cold-start times, weight streaming so our small-model zoo runs co-resident efficiently rather than one model per node.
- Serve non-LLM models (layout detection, embeddings, rerankers) on a real inference server such as Triton, with dynamic batching and ensembles, rather than a Python web process.
- Own capacity and autoscaling policy: scale on queue depth, KV-cache utilization or TTFT rather than accelerator utilization, and be able to explain why. Spot and capacity blocks, Karpenter node pools, warm pools, and bin-packing across L4 / A10G / L40S / A100 / H100 and selecting the cheapest instance that fits after quantization.
- Establish benchmarking rigour: load generation using real input/output length distributions from production traffic, saturation curves and latency-throughput Pareto plots were reported per-accelerator-normalized, plus performance and quality regression gates in CI.
- Platform, reliability, and governance.
- Build document AI platforms using generative AI with asynchronous architecture, queues, and event-driven workers for elastic scaling and non-blocking inference.
- Architect self-hosting infrastructure for LLMs on Kubernetes/EC2 with accelerator orchestration and scheduling.
- Oversee correct LLM usage, including gateways (LiteLLM, Portkey), prompt engineering support, and guardrails to prevent misuse or hallucinations.
- Implement observability for AI/ML pipelines: distributed tracing with OpenTelemetry and X-Ray, drift and hallucination monitoring via Prometheus/Grafana, and LLM-specific metrics via Langfuse.
- Manage AI/ML workflows with MLflow (including serverless SageMaker MLflow setup) and Kubeflow for experiment tracking, versioning, and deployment.
- Add security layers: Bedrock Guardrails (PII/toxicity), KMS encryption, IAM least-privilege, VPC endpoints, and CloudTrail auditing.
- Optimize AWS resources (Bedrock, SageMaker, EKS) for cost, security, and performance in production AI workloads.