LLM / RAG Architecture Consultant — Review & Advise on-Prem Local AI Build

We're building an internal AI system that runs entirely on our own hardware (no cloud inference) against our own company data. We have a working proof-of-concept and want to get the architecture right. We need an experienced consultant to review what we've built, pressure-test our decisions, and tell us where we're wrong. This is an advisory/validation role first — we have someone doing the hands-on work; what we want is a senior second opinion to make sure we're building this the right way. What we're running today: Inference: RTX 5090 (32GB, Blackwell), Ubuntu 24.04, running llama-server (llama.cpp + CUDA) serving Gemma 4 31B-it (Q4_K_M GGUF) at a 262,144 context window. Also hosts our MCP retrieval server, PostgreSQL, and Qdrant. Embeddings: separate machine with an RTX 3060 running vLLM serving Qwen3-Embedding-4B. RAG: hybrid retrieval — Postgres full-text search + Qdrant semantic search with RRF fusion, exposed through a custom MCP server with tool-calling. Data: ingesting our own internal operational data into Postgres + Qdrant. Planned stack: LiteLLM for model routing, n8n for automation, Open WebUI for the interface, Langfuse for observability, Vault or Infisical for secrets, Keycloak/Azure AD for SSO. What we need help with: Validating our two-machine split (inference vs. embeddings) and whether our VRAM/context budget holds up under real load — specifically whether a 256K context window is real and performant on a single 32GB card or just nominal. Model selection and routing strategy: which open-weight models for which tasks, and how to structure LiteLLM routes. RAG quality: chunking, embedding dimensionality, hybrid search tuning, reranking — making retrieval actually accurate on messy real-world data. Sanity-checking our overall architecture and telling us our blind spots. You should have done: Stood up local LLM inference in production — llama.cpp/llama-server and vLLM, not just Ollama on a laptop. You understand GGUF quantization (Q4_K_M, IQ-series), KV cache, KV-cache quantization, and how context length maps to actual VRAM consumption. Real fluency in GPU sizing math — given a model, a quant, and a context window, you can tell us whether it fits on a given card and what throughput to expect. Bonus if you've worked with Blackwell / sm_120a. Built production RAG — vector DBs (Qdrant, pgvector), hybrid search, RRF fusion, embedding model selection, reranking, evaluation. Worked with agentic/tool-calling systems and ideally MCP servers. Know the open-weight model landscape (Gemma, Qwen, Llama, Mistral, Phi, Nemotron, Hermes) and their licenses well enough to advise. Production ops: systemd, Docker, model gateways (LiteLLM or similar), observability (Langfuse), secrets management, SSO.