Real-Time AI Orchestration Engineer

Consumer AI Mobile Product · iOS & Android · Full-Time or Contract Early-stage, stealth. Product details shared under NDA. The Role We're building a consumer AI product with a real-time, photorealistic AI persona: live voice conversation, persistent memory across sessions, camera-based perception, and a purpose-built harness around a frontier LLM — all orchestrated so the user experiences one continuous, seamless presence. This is not a chatbot wrapper and not a standard mobile build with AI features bolted on. It is an AI systems build with a mobile front end. The hardest work is in the real-time orchestration layer: managing multiple streaming AI vendors, hard latency budgets, cost-aware routing, and invisible transitions between them. You will be the lead engineer. What You'll Own • Real-Time Orchestration Layer: A backend router and state machine that determines, in real time, which technology is active, and switches between them invisibly. This is the architectural core. • Session Lifecycle Management: Starting, stopping, and resuming live streaming sessions on demand, under strict latency and cost constraints. • Real-time Voice Pipeline: Wake-phrase, hands-free, sub-1.5-second end-to-end response in noisy ambient conditions, with real turn-taking and interruption handling. • LLM Harness: Middleware enforcing persona consistency, structured memory injection, safety guardrails, and server-side deterministic rules. Not prompt engineering; a production system. • Persistent Memory Architecture: Structured session records and living summaries that compound across sessions and days, injected into every subsequent interaction. • Camera Perception Layer: Including confidence gating and graceful fallback. Reliability over false precision. • Full Mobile Product: React Native (iOS + Android), Firebase, serverless backend, subscriptions, push, analytics, error monitoring, GDPR, and store submission. • Vendor Co-Development: Working directly with our vendors' engineering teams on capabilities that don't yet exist off the shelf. Stack Real-time avatar/perception platform · real-time speech models · Anthropic Claude · ElevenLabs · computer vision · image/video generation · React Native · Firebase (Firestore + Auth) · Vercel serverless · RevenueCat · Sentry · PostHog Non-Negotiable Requirements Real-time AI Systems: • Production experience with real-time voice AI / conversational agents: turn-taking, barge-in, interruption handling, sub-second latency • Streaming media / WebRTC, or frameworks built on it (LiveKit, Daily, Pipecat, or similar) • Orchestrating multiple AI vendors into one coherent, session-level experience, with fallbacks and graceful degradation • Demonstrated latency optimization — you have profiled a real-time pipeline and driven a number down • Cost engineering — you think in cost-per-user-per-minute, not just "does it work" • Production LLM integration, including the deterministic scaffolding around the model Engineering Foundation: • Strong backend / serverless architecture • Firebase Firestore — you have designed production data models, not just queried them • Production mobile delivery; React Native strongly preferred (shipped to both App Store and Google Play) How You Work: • Live, direct communication. Daily overlap with US Eastern hours. • You state hard technical truths plainly. We would far rather hear "that's not possible yet — here's what is" than be told what we want to hear. • You are resourceful about building what doesn't exist yet. Strong Advantages • Real-time avatar / talking-head platform experience • Wake-word detection / always-on listening in noisy environments • Vision/perception work where knowing the model's limits mattered • LLM harness or middleware architecture • RevenueCat Engagement Target Launch: October 2026 Type: Full-time or contract. Fixed-price milestones or hourly for the right candidate. Potential to grow into a long-term technical leadership role. Location: Remote, with meaningful US Eastern overlap. Process: Application → NDA → full product brief → technical conversation → offer. To Apply, Send: 1. One production system you built where a human talks to an AI in real time. What it does, your specific role, link or demo if possible. 2.Your answer to this: You're running live, GPU-backed streaming sessions billed per-minute for as long as a session is open. The vendor has no pause or resume. Most of the time the user doesn't need the live session, but when they do, it must feel instant. How do you architect that — and how do you hide the spin-up? 3. One production React Native app you shipped (what it does, your role, link). 4. Your rate expectation and timezone.