AI Platform Audit Engineer — Agent Loop Optimization and Runtime Observability

I have an existing AI platform with agent workflows, RAG/document retrieval, API integrations, task execution, and runtime services. I’m looking for an experienced AI systems engineer to audit the current AI loop, identify performance bottlenecks, and build a lightweight audit and observability system. This is not a from-scratch chatbot project. You will work with an existing codebase and help us understand why certain AI workflows are slow, expensive, unreliable, or difficult to debug. Scope of work: - Trace the full request lifecycle from user input to retrieval, prompting, model calls, tool execution, retries, and final response; - Identify slow steps, unnecessary model calls, repeated retrieval, oversized context, inefficient prompts, and retry loops; - Measure latency, token usage, model cost, failure rates, retry counts, and tool execution time; - Build an audit system with correlation IDs, timestamps, step status, duration, model usage, errors, and trace relationships; - Protect sensitive data and avoid storing API keys or unnecessary raw user content; - Produce a prioritized bottleneck report with evidence from real traces; - Implement and verify at least one high-impact optimization; - Document the architecture, metrics, findings, and recommended next steps. Expected deliverables: - AI loop architecture and trace map; - Working audit/observability layer; - Trace data for representative workflows; - Bottleneck and root-cause report; - Before-and-after measurement for one optimization; - Setup instructions and technical documentation. Acceptance criteria: - One request can be traced across all major AI workflow steps; - Each step exposes status, duration, errors, and relevant metadata; - The top three bottlenecks are supported by trace data; - At least one optimization demonstrates measurable improvement; - The audit system is maintainable and documented. Required experience: Python, FastAPI, LLM APIs, AI agents, RAG, PostgreSQL, API integrations, debugging, and production observability. Experience with OpenTelemetry, Prometheus, Grafana, LangChain, LangGraph, or custom AI tracing systems is a plus.