Brand Collaboration Data of 1M+ Influencers

We need our influencers data enriched with brand-collaboration history — which brands a creator has worked/collaborated with, or been sponsored by. This is high volume: 1M+ records in ~2 weeks, a third week for fixes and migration, then ongoing work for updates. Tell us how you'd architect it, and if part isn't realistic, where and why. We value an honest plan over an optimistic one. What you'll do - Build a standalone pipeline for collaboration detection and enrichment, separate from our existing ones - Ingest collaboration signals via APIs and licensed data providers, and use AI/context detection to identify the brands involved - Source and clean brand data from multiple places into one reliable reference set - Resolve messy brand data — duplicates, inconsistent names, parent/child brands, ambiguous names — AI-assisted, with manual validation where it matters - Collect and match brand logos - Catch hard cases, e.g. a brand named like a common word or referenced indirectly - Write to our DB efficiently under load (Supabase / PostgreSQL; pipelines run concurrently and we've hit CPU bottlenecks) Required: Python; PostgreSQL; REST/JSON APIs Nice to have: - 1M+ row datasets - AI/LLMs for data work; DBs and servers (Supabase, RDS, EC2, DigitalOcean, Cloud sql) Engagement - Weeks 1–2 enrich the full dataset, Week 3 buffer, ongoing after for the right person. Start within a few days, significant weekly hours. To apply: Answer the questions below and tell us plainly whether 2-week / 1M+ enrichment is realistic. Copy-paste proposals skipped. 1. Have you processed a 1M+ row dataset? Throughput, and how you held performance under heavy read/write? 2. How would you architect for 1M+ profiles in 2 weeks — parallelization, batching, API throughput, DB efficiency? 3. Which platform APIs or data providers have you integrated for creator data? 4. Can you start ASAP, and how many hours/week over the next 2–3 weeks?