Full stack AI Agent Developer — AI Data Pipeline, Vector DB & Real-Time Push API. Web crawlers.

Title: Backend Developer — AI Data Pipeline, Vector DB & Real-Time Push API Post: We are building an automated backend system that continuously crawls public web sources, processes and indexes content using AI, and delivers updates via webhooks. Looking for someone who has built this type of system before and can move fast. NDA required before project details are shared. What you’ll build: • Web crawler network —. • AI processing pipeline — cleans, deduplicates, chunks, and embeds ingested content into a vector database using an LLM embedding model. Quality scoring and incremental updates required. • Push API — monitors for significant content changes and delivers updates via webhook endpoints automatically. Includes configurable push schedules per subscriber, REST query endpoint, API key authentication, and token usage tracking per key. Tech stack (flexible — use what you know best): • Python (FastAPI) or Node.js • Any vector DB — Pinecone, ChromaDB, Supabase • Any LLM API — Anthropic or OpenAI • Any scheduler — n8n, APScheduler, cron • Redis for queue management • Railway, Render, or AWS for deployment Requirements: • NDA signed before kickoff — non-negotiable • Must have built RAG pipelines or vector DB systems in production — not tutorials • Must have experience with web crawlers and scheduled job pipelines • Must have experience with webhook delivery systems • GitHub or portfolio showing relevant deployed work required • 95%+ Job Success Score preferred • Individual contractors only — no agencies To apply include: • Example of a similar system you’ve built — web crawler, RAG pipeline, or push notification API • Your preferred stack for this type of build • Brief technical approach in 3–5 sentences • Hourly rate and availability to start Budget: $50–$80/hr Timeline: 3 weeks — focused sprint with daily check-ins