QA Reviewer: Rate AI Scoring Accuracy Against Conversation Transcripts

We built an AI system that reads transcripts of customer service conversations and scores how well the employee handled each one. We need a sharp, detail-oriented reviewer to tell us where the AI is right and where it is wrong. This is judgment work, not data entry. You will read what was actually said, look at the score and flags the AI assigned, and give us an honest assessment. WHAT YOU WILL DO For each conversation, answer five questions and log them in a spreadsheet we provide: 1. Is this a real customer interaction, or is it background noise or unrelated chatter? 2. Did the AI correctly identify what type of interaction it was? 3. Is the score fair for what was actually said? Too harsh, too generous, or about right? 4. Do the flags the AI raised actually match what is in the transcript, or are they false alarms? 5. Overall, did the AI get this one right? Plus a one-line note explaining anything notable. WHAT WE ARE LOOKING FOR - Excellent written English comprehension. You must be able to read a natural conversation and judge tone and intent accurately. - Genuine attention to detail. We need you to notice when a score does not match what actually happened. - Willingness to be blunt. If our system is wrong, we need you to say so plainly, with the specific example. - Consistency and reasonable speed. A focused reviewer completes about 20 items per hour. WHAT WE ARE NOT LOOKING FOR Please do not apply if you plan to skim and mark everything as correct. We check the work against our own review, and a batch that rubber-stamps everything is immediately obvious and will not be paid beyond the trial. HOW THIS STARTS We will start with a paid trial of 20 items (about 1 hour). If your judgment is good, we will move to ongoing batches of 120 items. CONFIDENTIALITY This involves real business conversation transcripts. Personal and payment details are automatically removed before you see anything, but you will be required to keep all content confidential and you will agree to Upwork's standard confidentiality terms.