You will get a calibrated LLM eval harness with bias audit and CI regression gate
Project details
Most LLM eval setups use a raw GPT-4 judge with no reliability check. That's a black box scoring another black box. Then teams trust the numbers and ship anyway.
This project replaces that with a judge calibrated against a held-out human-labeled set. Cohen's kappa target over 0.7. If your judge can't agree with humans, the numbers it produces don't belong on a dashboard. We measure agreement first, then trust the metric.
You get a 50-100 question regression suite tailored to your app, a calibrated judge, bias auditing for position and length effects, and a GitHub Action that fails the PR check when scores drop below your gate. After delivery, two weeks of judge re-calibration if you change models or prompts.
I built the open-source eval-kit Python library this same workflow runs on. 161 tests, 87% branch coverage, mypy strict. Six years writing production AI for Société Générale, Allianz, and Decathlon, currently shipping a Claude support agent at Decathlon under the same eval discipline.
You give me access on day 1. Eight days to delivery. If the calibrated judge cannot reach kappa 0.7 on your data, I tell you so directly and refund the difference. No excuses.
This project replaces that with a judge calibrated against a held-out human-labeled set. Cohen's kappa target over 0.7. If your judge can't agree with humans, the numbers it produces don't belong on a dashboard. We measure agreement first, then trust the metric.
You get a 50-100 question regression suite tailored to your app, a calibrated judge, bias auditing for position and length effects, and a GitHub Action that fails the PR check when scores drop below your gate. After delivery, two weeks of judge re-calibration if you change models or prompts.
I built the open-source eval-kit Python library this same workflow runs on. 161 tests, 87% branch coverage, mypy strict. Six years writing production AI for Société Générale, Allianz, and Decathlon, currently shipping a Claude support agent at Decathlon under the same eval discipline.
You give me access on day 1. Eight days to delivery. If the calibrated judge cannot reach kappa 0.7 on your data, I tell you so directly and refund the difference. No excuses.
AI Algorithms
Large Language Model, Transformer ModelAI Applications
AI Chatbot, AI Mobile App Development, AI Text-to-Image, AI Text-to-Speech, AI-Enhanced Medical Imaging, AI-Generated Art, AI-Generated Code, AI-Generated Music, AIOps, Conversational AI, Natural Language Generation, Natural Language UnderstandingAI Development Language
PythonAI Tools
Azure OpenAI, Bing AI, Copy.ai, GitHub Copilot, Hugging Face, NVIDIA AI Platform, PyTorch, Replit, Streamlit, TensorFlowAI Models
AlphaCode, BERT, ChatGPT, DALL-E, GPT-3, GPT-4, GPT-Neo, LaMDA, LLaMA, OpenAI Codex, Stable Diffusion, WhisperWhat's included
| Service Tiers |
Starter
$1,200
|
Standard
$3,200
|
Advanced
$6,500
|
|---|---|---|---|
| Delivery Time | 4 days | 8 days | 16 days |
Number of Revisions | 1 | 2 | 2 |
AI Model Integration | - | - | |
Batch Normalization | - | - | - |
Database Integration | - | - | |
Detailed Code Comments | |||
Image Upscaling | - | - | - |
MLOps | - | ||
Model Deployment | - | - | - |
Model Documentation | |||
Model Monitoring | - | - | |
Model Testing & Optimization | |||
Model Tuning | - | - | |
Natural Language Processing | |||
NLP Tokenization | - | - | - |
Pre-Training | - | - | - |
Prompt Engineering | |||
Setup File | - | ||
Source Code |
Optional add-ons
You can add these on the next page.
50 additional eval cases hand-labeled
(+ 2 Days)
+$400
30-day judge re-calibration window
(+ 1 Day)
+$500Frequently asked questions
About Amir
Senior AI/LLM Engineer | RAG, AI Agents, LangChain, LLM Evals
Paris, France - 4:38 am local time
Currently at Decathlon building a Claude support agent with RAG over the product catalogue: three Sonnet calls (classify, draft, escalate) through Anthropic's tool-use API, per-call cost accounting, a 50-row eval set running in CI. Before that, three years at Allianz taking the insurer's first LLM applications from prototype to production: a document Q&A system for claim adjusters, a multi-step research agent on LangGraph, and a vendor-neutral wrapper around Anthropic and OpenAI. Three years before that at Société Générale on classical ML and NLP: a transformer-based document classifier for compliance, OCR for scanned regulatory filings, the team's first embeddings-based search.
What I publish on GitHub:
- A production RAG reference that refuses to answer when retrieval does not support the claim. 50-question eval gate blocks merges on more than 3 points of regression.
- A customer-support agent: three Sonnet calls, calibrated confidence, real cost accounting, eval gate in CI.
- A 7-node LangGraph DAG with a parallel fan-out and a critic retry loop. Streams per-node telemetry over SSE.
- An open-source Python library for LLM evals: calibrated LLM-as-judge, synthetic adversarial data, regression diff that fails the PR.
Stack: Python, FastAPI, Anthropic Claude, OpenAI, LangGraph, Voyage embeddings, pgvector, Postgres, Modal, Langfuse, Next.js, Tailwind. Bar: mypy strict, ruff, pytest with a coverage gate, golden-set evals in CI, structlog, per-call cost tracking. Every repo runs make ci clean from a fresh clone.
Best fit for:
- Building a RAG system that has to cite, not invent.
- Taking a multi-step agent prototype to production.
- Setting up the eval and quality engineering layer your team has been meaning to build.
EU timezone, 30+ hours per week, async-friendly.
Steps for completing your project
After purchasing the project, send requirements so Amir can start the project.
Delivery time starts when Amir receives requirements from you.
Amir works on your project following the steps below.
Revisions may occur after the delivery date.
Day 1: setup and access
I receive your repo, sample I/O pairs, and the quality criteria you ranked. Local dev environment up. Baseline reproduction confirmed before any eval work begins.
Days 1-2: tailored regression suite
I write a 50-100 question regression suite covering your specific app: positive cases, hard negatives, edge cases, and adversarial examples sized to your team's tolerance for false alarms.