We have a Symfony2 based system for managing Lists (lists of subscribers) and distributing them among our partners.
The system is for lead distribution and managing bulk "full record" data which includes basic contact details about each user such as "first name, last name, address1, address2, city, state, zip, phone number, email address, opt-in timestamp, source URL" and a few other fields.
Admins and List Owners will need the option to enable 3rd Party "data validation" services on the lists - to have the system run new incoming records against data validation APIs (for email data hygiene and to verify phone numbers are real, addresses are real, etc.) to confirm the quality/realness of the records (and scrub junk/fake data submissions out before redistributing).
There should also be a way to manage "suppression lists" and a dedicated "blacklist API" for retrieving suppressed records across the system and per list.
- A high-level explanation of the system scope is as follows -
A data "partner" will POST in realtime leads to our secure (and scalable) API, at a rate of upwards of 1mil per hour, per partner. The system needs to be able to scale/handle thousands of concurrent requests and not fail/drop incoming records. Logging should be intelligently thought out, and we will want real-time reporting of the list activity and user activity (with a caching layer and intelligent architecture for responsiveness and accuracy).
For the API - Something based on Go, Elixir, or Java may be good for this, with an in-memory storage system like Redis or NoSQL for the delayed writes and queuing (open to clever recommendations, as long as they make sense and are dead-simple to maintain).
On the UI side, admins should be able to create users with permissions based on lists (list owner, can view list, can view/retrieve records from lists(daily quotas on how many records they can pull, etc.) and other such "List Management" functionality.
List Owner and Admin users should be able to fully manage their lists and setup both Incoming and Outgoing posting specs. This means they can set an incoming API Auth Key or User/Pass combo for Data Partners to "Post in" new records, as well setup "Outgoing Feeds" that will be queued and sent out to other platform APIs automatically.
So, for a high-level flow: A data partner posts in records from their lander page. The system (for example) has an "Outgoing Feed" for that particular list, and as quickly as the records are coming in to the API, they are also being posted out to another 2-3 partner API's for distribution of the realtime records.
Users will need to be able to "export" or "pull" UNIQUE records from the lists (no duplicates, and should only be pulled once per day per partner). So for example, if 100,000 new records come in today for "List ID 1234" and there are 10 data partners who have "Export" permissions with 100k daily quota cap, they can login to the UI and pull those 100,000 new records ONCE each for the day. We also need the option to "Batch deliver" these data segments via email (via tokenized/secure download link) or automated FTP upload.
The system is currently in an unfinished state, and we would like to have a skilled high-scale developer familiarize themselves with the currently existing platform, form a clear vision of what it needs to become, and rewrite it from scratch - to comprehensively cover our use cases.
For the right candidate, this project will lead to much more work. We are an innovative and fast-moving company and have lots of fun engineering challenges and interesting platforms to cut your teeth on.