AI Voice Engineer for Real-Time Chat

I run a production voice AI platform used by organisations in Australia, handling live call volume across two operational modes: daytime knowledge base retrieval with call transfer, and after-hours urgent query triage with automated outbound dispatch. The system uses a deterministic logic architecture (not a pure LLM-driven conversational loop), and that architecture must be preserved throughout this engagement. I need an experienced engineer to enhance the existing (or replace whatever is required) voice pipeline to achieve: Mandatory Sub-second response latency (target: under 1000ms end-to-end from end of caller speech to start of system response) More natural conversational handling, without replacing the deterministic decision logic with non-deterministic or purely model-driven behaviour, including interruptions, barge in etc Improved speech recognition accuracy via keyword bias/boosting tuning, specific to domain terminology relevant to council service queries - i have 100's of calls available in supabase for farming info Verified compatibility across both operational modes (daytime KB retrieval + transfer, and after-hours triage + outbound dispatch trigger — the outbound call mechanism itself is pre-built and out of scope) A shadow mode deployment, running in parallel with the live system, logging comparative performance data without affecting live callers, to prove out improvements before cutover Tech stack: live kit, Twilio, Deepgram (including Flux), OpenAI/Azure OpenAI, AWS Sonnet, Haiku ElevenLabs, Fly.io, Supabase. - What I Need From You Before Hiring To move quickly and avoid another mismatch, please include in your proposal: Specific prior experience with speech recognition latency optimisation and keyword bias/boosting configuration. A brief technical explanation, in your own words (not ai garbage), of how you would approach preserving deterministic logic while improving conversational flow. Generic answers will be deprioritised. Confirmation you can deliver a working technical proposal within the first few days of engagement, referencing the actual current codebase (I will provide access on award), not a generic framework. Your realistic estimate for reaching a live shadow mode deployment, and what could cause that estimate to shift. I am looking to have a protype shadow framework in place within the next 2 weeks maximum with a cutover scheduled for no later than 4 weeks. Please also know i am very well versed with claude, fable, codex etc so please don't insult me by trying to baffle me with rubbish, i am looking for someone with genuine and real experience building these systems not someone who just wants to put my repo in claude and try to fix it. Experience is mandatory. I am fed up of freelancers who give 'detailed audits' of a claude run and think that it is an acceptable quality to move on, it is not! please let me know, happy to jump on a call when you are free - there is ongoing work available for a good fit