STEM Researchers Needed for Evaluation of AI Responses to Questions about Scientific Papers

You will be shown a series of queries about a scientific paper on a custom website. For each query, we provide a model response to the query and its referenced associated figures and tables, and we ask you to assess several qualities of the response and the figures and tables per a defined rubric that we will provide to you. You will also have access to the paper, which you should use to validate the quality of the provided model response. You will work in batches of up to 10 queries, and should plan to spend around 1 hour per batch of 10. For the initial round of annotations, we are looking for pilot annotators to give us detailed feedback on what is and is not working well with the process. For this, you will experience only 1 batch of 10 queries which we will provide you, and in addition to completing the annotations, you may be asked to create a short report (at most 1 page) on what needs to be altered about the annotation process to make it smoother for future annotators. We expect this feedback within 1 week of you accepting the job. You will work on this independently but are welcome to reach out to us for clarifying questions. Ideal candidates for this task are STEM researchers (PhD students, industry researchers, academics) who have 3+ years of experience reading and crafting academic papers. This project is subject to the terms and conditions of the attached Upwork Project Agreement (the “Agreement”). By agreeing to participate in the project, you expressly accept and agree to the terms of the Agreement.