About the role
Dataset Quality AutomationAutomate quality assurance pipelines for SFT transcripts and RLHF preference pairs.
Implement schema validation, semantic overlap checks, and embedding-based deduplication.
Integrate filters for safety, toxicity, and reward-signal balance in datasets.
Training & BenchmarkingExecute proxy fine-tuning (LoRA/QLoRA) on open-source LLMs using QA-approved datasets.
Train lightweight reward models and track performance via public/internal benchmarks and calibration metrics.
LLM EvaluationOrchestrate human and LLM-as-judge evaluations, including generation of critiques and scoring.
Design evaluation rubrics focused on consistency, helpfulness, and alignment with reward models.
Calculate and interpret statistical measures like binomial confidence intervals for evaluation scores.
Annotation & Rater ManagementBuild a continuous feedback loop with annotation teams, resolve disputes, and maintain high annotation quality.
Manage human evaluation workflows to maximize consistency and throughput.
Research & ToolingPrototype new signal-to-noise metrics (e.g., reward model entropy, preference flip rate).
Package tooling into reproducible notebooks and integrate into CI pipelines (Airflow/Dagster).
End Value to the Company
You will serve as the client-end MLE advocate, ensuring that all training and evaluation datasets are aligned with downstream needs. Your work will directly influence model performance, client satisfaction, and data-driven improvements to our ML systems.Required Skills & Qualifications
Strong understanding of LLM training and evaluation pipelines (SFT, RLHF, reward modeling).
Experience with model performance diagnostics, identifying root causes in model behavior (e.g., data flaws, prompt issues).
Skilled in prompt engineering, dataset schema design, and annotation guideline development.
Proficient in Python, with experience using PyTorch, Hugging Face Transformers, FastAPI.
Comfortable building evaluation frameworks, including leaderboards and domain-specific test sets.
Familiarity with model evaluation metrics, clustering techniques, embedding models, and data drift detection.
Strong communication skills, especially in translating technical findings into actionable client insights.
Self-starter with a consultative mindset who can operate across technical and business domains.
Nice-to-Have
Experience with embedding similarity, data deduplication, or dataset filtering for toxicity/safety.
Prior work in LLM-as-a-Judge systems or human alignment evaluations.
Familiarity with CI/CD for data workflows and orchestration tools like Airflow or Dagster.