You will get a local eval harness to test, score, and track your LLM prompts
Rising Talent
Rising Talent
Project details
Shipping prompts without testing them is like pushing code without tests — you only find out something broke when a client complains. I build a local evaluation harness in Python that lets you run your prompts and agents against a structured test suite, score every output, and catch regressions before they reach production.
Each harness includes: a CLI runner you can invoke locally or in CI, custom scoring functions (exact match, regex, semantic similarity), an HTML report showing pass/fail per test case with diffs, and a regression baseline so you know the moment quality drops.
From a simple 5-case starter harness to a full FastAPI evaluation server with multi-model comparison and a live dashboard, I scope each tier to where you are in your LLM development journey. You will own the code, the test cases, and a repeatable quality process from day one.
Each harness includes: a CLI runner you can invoke locally or in CI, custom scoring functions (exact match, regex, semantic similarity), an HTML report showing pass/fail per test case with diffs, and a regression baseline so you know the moment quality drops.
From a simple 5-case starter harness to a full FastAPI evaluation server with multi-model comparison and a live dashboard, I scope each tier to where you are in your LLM development journey. You will own the code, the test cases, and a repeatable quality process from day one.
Machine Learning Tools
ChatGPT, PythonWhat's included
| Service Tiers |
Starter
$350
|
Standard
$650
|
Advanced
$1,100
|
|---|---|---|---|
| Delivery Time | 4 days | 5 days | 7 days |
Number of Revisions | 0 | 0 | 0 |
Model Validation/Testing | - | - | - |
Model Documentation | - | - | - |
Data Source Connectivity | - | - | - |
Source Code |
About Ignazio
Software Engineer working on AI systems. Retrieval, agents, evaluation
Padova, Italy - 7:58 pm local time
I build reliable AI systems for teams that have outgrown the prototype phase: RAG with measurable retrieval quality, agent workflows with persistent state and approval gates, FastAPI backends purpose-built for LLM products, and the evaluation layer most teams skip between prototype and production.
WHAT I DELIVER
Retrieval Contracts for RAG
Hybrid retrieval (pgvector + BM25) with reranking, instrumented end-to-end. Built for the gap between a demo that retrieves and a system that survives the next 10,000 queries.
LLM Agent Infrastructure
Multi-step agent workflows on LangGraph with persistent state, tool orchestration, approval gates, retry logic, and deterministic evaluation.
FastAPI Backends for AI Products
Async FastAPI backends purpose-built for LLM apps. pgvector, Alembic migrations, health checks, and operational readiness on day one.
AI Reliability Engineering
Deterministic evaluation frameworks, structured output validation, cost and latency observability, and regression detection.
Everything I ship runs on four laws: deterministic evaluation (no "it worked when I tried it"), cost control by architecture, latency as a feature, and operational reliability under load and failure.
Tech stack: Python, FastAPI, LangGraph, pgvector, Anthropic Claude API, OpenAI API, React, TypeScript, Next.js, Tauri, Rust, PostgreSQL, Docker.
My work ships as source in your repo, and all my systems are public on GitHub with live deploys. If you need AI systems that actually work in production, let's talk.
Steps for completing your project
After purchasing the project, send requirements so Ignazio can start the project.
Delivery time starts when Ignazio receives requirements from you.
Ignazio works on your project following the steps below.
Revisions may occur after the delivery date.
Harness Setup & Test Case Design
Configure the test runner, define input/output schemas, write scoring functions (exact match, semantic similarity, regex), and create initial test cases from your production prompts.
Evaluation Run, Report & Regression Baseline
Run the full suite against your model, generate an HTML/JSON report with per-case scores, establish a regression baseline, and hand over source code with CI/CD integration instructions.