You will get AI Agent Evaluation for Software Engineers — Mercor/Anthropic Method
Project details
You will get a production-grade AI agent evaluation or training system built using the same methodology applied in Mercor and Anthropic-style coding agent assessments. This is not theory or generic AI content — it is a practitioner-built system grounded in real 20-turn evaluations, structured rubrics, and measurable scoring.
What sets this apart is execution. Every evaluation is built across 6 dimensions: task success, interaction quality, code quality, thoroughness, trajectory, and comparative preference. For training, engineers don’t just learn — they practice on real tasks, calibrate scoring, and leave able to run submission-ready evaluations independently.
I bring hands-on experience from AI annotation, evaluator workflows, and building structured training systems that bridge technical depth with business outcomes. Whether you need a one-off evaluation, to upskill an engineer, or to build a full evaluation team, the output is always practical, structured, and ready for real-world use.
What sets this apart is execution. Every evaluation is built across 6 dimensions: task success, interaction quality, code quality, thoroughness, trajectory, and comparative preference. For training, engineers don’t just learn — they practice on real tasks, calibrate scoring, and leave able to run submission-ready evaluations independently.
I bring hands-on experience from AI annotation, evaluator workflows, and building structured training systems that bridge technical depth with business outcomes. Whether you need a one-off evaluation, to upskill an engineer, or to build a full evaluation team, the output is always practical, structured, and ready for real-world use.
AI Development Type
Deep Learning, Model TuningAI Tools
SonnetAI Development Language
PythonWhat's included
| Service Tiers |
Starter
$250
|
Standard
$850
|
Advanced
$1,600
|
|---|---|---|---|
| Delivery Time | 1 day | 10 days | 14 days |
Number of Revisions | 1 | 1 | 1 |
AI Model Integration | |||
Detailed Code Comments | |||
Knowledge Graph | - | - | - |
Model Documentation | |||
Ontology | - | - | - |
Source Code | |||
Taxonomy | - | - | - |
Frequently asked questions
About Fred
AI Systems Engineer | LLM Evaluation (RLHF) | Full-Stack Engineer
Nairobi, Kenya - 2:42 am local time
Most engineers stay on infrastructure.
Most annotators stay on task execution.
I’ve worked across both.
That means I don’t just design pipelines or deploy models in isolation—I understand how data is labeled, where quality breaks, and what actually improves model performance.
Result: systems built for real-world performance—not theory.
🛠 Software Engineering
I design and deploy full-stack systems built for real operations—not demos.
WeDemo Africa — AI + Human Talent Ecosystem
A production ecosystem I built to move users through:
Mindset → Skills → Monetization
This is a multi-layered operational system, not a content platform.
Core system design:
User progression: onboarding → learning → execution → earning
Hybrid AI + human workflows (mentorship, assessments, guidance)
Integrated platforms: learning, work, referrals, leadership
Gated progression: certification, assessments, controlled access
Operational layer:
Structured training pipelines
Dashboard-driven user tracking
Real-world outcome focus (not just completion)
✔ Live system handling real users and workflows
Mini Gigs Hub — Team Operations Dashboard
A production system for managing distributed teams:
Role-based access control
AM/PM shift reporting
Automated email alerts (cron jobs)
Admin analytics & performance tracking
Infrastructure & Systems
RustDesk relay & ID servers (VPS deployment)
Secure remote desktop environments
Browser isolation + proxy/session workflows
Core Stack
Next.js · React · TypeScript
PostgreSQL · Prisma · REST APIs
Vercel · Docker · VPS
Clerk · Role-based access
Cron jobs · Automation
JSON · CSV · JSONL pipelines
🤖 AI Systems Engineering
I design the operational layer between AI systems and the people training them.
That includes:
Annotation pipelines (RLHF, ranking, classification, etc.)
Task design + annotation guidelines
QA systems for consistent output at scale
I also build AI-powered APIs:
LLM integrations
Annotation workflow endpoints
Prediction-serving APIs
✔ Production-ready, documented, and integration-ready from day one
Training Systems
I’ve built a complete AI annotation training program:
5 structured modules
Covers text, image, audio, video + RLHF
Includes ethics, bias, and PII handling
Comes with:
Lesson scripts
Slides
Exercises
Assessments
✔ Ready for teams of 3–50
🧠 AI Evaluation
I’ve worked directly on:
Remotasks
Handshake
Mercor
So I understand:
How tasks are designed
Where they break in practice
What actually produces high-quality data
What I evaluate:
RLHF preference ranking
Prompt/response quality
Instruction-following accuracy
Hallucinations & factuality
Safety, bias, compliance
I also train evaluators in judgment—not just mechanics.
🎯 Who I Work With
AI companies building or refining LLM systems
Teams setting up annotation / RLHF pipelines
Organizations deploying training + workforce systems
If you need someone who understands:
systems + data + people + execution
—that’s where I come in.
🚀 Let’s Work
Send me a message with what you're building.
I’ll respond with a clear, technical breakdown of how I’d approach it—usually within a few hours.
Steps for completing your project
After purchasing the project, send requirements so Fred can start the project.
Delivery time starts when Fred receives requirements from you.
Fred works on your project following the steps below.
Revisions may occur after the delivery date.
Project Intake & Alignment
You provide a short description of your goal and choose a package. I send a quick intake form to gather repo, team details, or evaluation objectives so I fully understand your requirements before starting.
Scope Definition & Planning
I review your inputs and define the evaluation scope, including task design, rubric structure, and delivery timeline. Everything is confirmed with you before execution to ensure clarity and alignment.