Why Upwork Relies On In-House AI Research To Power Uma, Upwork’s Mindful AI

While many builders in the AI space are looking towards off-the-shelf solutions to inform their AI strategic roadmap, we at Upwork have decided to embark on the complete opposite path: We know that the core pieces of our AI transformation must be driven by new technology we develop in-house. This strategy allows us to push the state-of-the-art within AI in a way that specifically benefits our products and services—and ultimately our customers—and is supported by an expert team of AI engineers and researchers who were added to our deep bench of tech talent over the last two years and have frontline experience within the hyperscaler AI labs. In this blog post, we’ll discuss how we think about and develop AI and why our in-house technology and team play a pivotal role in the future of humans and machines.

Last year, we shared how Upwork is scaling AI models to deliver better work outcomes across our marketplace. We outlined our growing emphasis on building domain-specific models that power our flagship AI product, Uma, Upwork’s mindful AI—models fine-tuned on proprietary data and designed to help businesses and freelancers work faster, smarter, and with greater confidence.

As we continue evolving Uma, we are embracing the fact that Uma needs to be more personalized and customized for our platform to optimally engage with our customers. As such, we are building Uma with proprietary datasets and training techniques to make it more context-aware and to allow it to handle increasingly agentic workflows. Because of our efforts, Uma can now take meaningful action on behalf of clients and talent to move hiring and collaboration forward. As highlighted in our Summer 2025 Upwork Updates release, Uma now acts across the full hiring flow: crafting effective job posts using platform insights, surfacing top matches, autonomously coordinating live interviews, evaluating candidates, and delivering tailored recommendations.

Let’s explore how Upwork’s own technology is driving Uma’s abilities for intelligent action—from our focus on custom-trained models, to our research into techniques for deep customization, and ultimately to our efforts supporting increasingly capable agentic AI. 

The benefits of custom-trained AI in business-critical workflows

Think of off-the-shelf LLM models like a mass-produced frozen meal from your grocery store — it doesn’t account for your unique ingredients, tools, or tastes. At Upwork, we have observed that fine-tuned models will consistently outperform off-the-shelf alternatives and enable faster iteration, greater reliability, and better product experiences. Our firsthand experience is that off-the-shelf models, while convenient, often fall short when applied to business-critical tasks and in complex domain-specific environments like hiring. Understanding the need for custom models starts with recognizing these domain knowledge gaps and building specifically to close them.

Here are some places where custom-trained models shine:

1. Dynamic and extended business dialogues: Conversations with customers to solve their problems without stiff prompting guardrails.
2. Complex multi-step workflows: End-to-end training means all our models know where they fit into the bigger machine, which leads to significantly better user experiences.
3. Incorporating expert domain knowledge: We have a wealth of platform data on what success looks like for clients or freelancers on Upwork. This information is directly trained into our custom Uma models.
4. Fast and controlled model iteration: See a problem in the model? Update the dataset and/or retrain the model. Model retraining is much more interpretable and controllable than updating a prompt, and doesn’t add any bloat for final model inference.
5. You own your model weights: No matter what happens, you have a copy of your model weights. No need to worry about model versioning hiccups from a third-party provider.
6. Cost and efficiency: Custom-trained models often get away with much smaller numbers of weight parameters and thus can reduce costs (often by 10x or more) and also latency.

Importantly, generic off-the-shelf models also always pose a performance tradeoff: They perform reasonably well overall but at the cost of excelling at specific domains. They are a true manifestation of the oft uncited second half of the idiom: jack of all trades, but master of none. For the same reason, these models often error out on edge cases. Anybody who’s played with readily available LLMs know that even small changes in input phrasing can lead to dramatically different LLM outputs, which makes them hard to use in production. A custom model exceeds these crucial limitations by allowing a model builder to assertively expose the model to important specific behavioral patterns that they consider to be non-negotiable must-haves, which leads to higher stability and performance where it counts.

That’s why we’re fully invested in fine-tuning models specifically for the Upwork environment. Whether it’s understanding what makes a job post effective or identifying key qualifications in a freelancer profile, this approach enables Uma to deliver more relevant, personalized support through training our models directly on the data that matters most. Rather than the frozen meal (which can certainly do in a pinch but often leaves us desiring more), Uma is prepared like a chef who not only knows how to cook like a pro, but also knows your pantry inside and out and can tailor every dish to your exact tastes.

Key ingredients behind Uma’s intelligence

To train our chef, we taught Uma with real-world examples, expert guidance, and platform context to master the Upwork “kitchen.” Uma’s ability to provide personalized responses to each individual business and freelancer is derived from two core ingredients: the right data and the right training approach.

Our internal data innovations at Upwork come in two principal flavors:

1. High-Quality Custom Data from Freelancers Hired on Upwork: We hired Top Rated freelancers directly through Upwork to script ideal customer conversations for a variety of work scenarios, such as hiring a website developer or managing a remote project. These freelancers have specific expert domain knowledge including writing on specific topics or categories of work that we plan to use to train our model (e.g. math, coding, etc.). These handcrafted interactions reflect actual user behavior and platform expectations, helping Uma learn what success looks like in context. We focus our data collection efforts specifically on the business contexts we most care about and where we know conventional LLMs struggle, such as longer conversations that require targeted planning. We have already open-sourced some of our human-crafted data and will soon publish an accompanying manuscript exploring this rich dataset. [dataset link]
2. High-Quality Synthetic Data Anchored in Platform Reality: Our proprietary platform data and human-created examples serve as the foundation for synthetic data generation. Using custom synthetic dataset algorithms developed by our AI R&D team, we increase our dataset sizes by orders of magnitude over just our human data. Our synthetic dataset algorithms are tunable and steerable, which allows us to cover a diverse set of edge cases and real-world situations.

Building on that high-quality data foundation, we also employ the following training techniques:

• Custom Fine-Tuning on Open Source LLMs: At the core of our approach to building Uma is open-source fine-tuning. Our fine-tuning utilizes both our human datasets encompassing business-critical modalities (like longer conversations) as well as synthetic data built around anonymized platform signals such as task completions, interaction patterns, and successful collaboration outcomes. Results from our tests of our personality models (that we highlight in a below section) show that training a model this way leads to dramatically better results in both content and style accuracy, with our Uma “personality” baseline model substantially outperforming other off-the-shelf models. Our internal infrastructure is also flexible and resilient, allowing us to switch between all major open-source model classes (e.g. Llama, Mistral, Qwen) at the drop of a hat. 

• Iterative Training and Continual Learning: Our internal training process for Uma is built with the assumption that we will have to train many models quickly. Analytics workflows provide fast feedback on how models are performing in public with the help of autonomous AI agents that provide customizable evaluation. Since the inception of our Uma training program, our infrastructure and processes have evolved to the point where a new and improved model version for any given Uma product surface can be brought online within a few days. We are also in exploratory stages for continual learning, through which this iteration process is completely automated and any need of human intervention is abstracted away.

• Inference-Time Techniques Like Custom Chain-of-Thought: We also use internally developed variants of inference-time techniques like chain-of-thought (CoT). These techniques allow Uma to consolidate information before providing final recommendations, especially in complex workflows such as candidate evaluation or proposal critique. For example, instead of instantly ranking a set of freelancer profiles, Uma works through a structured reasoning process—considering factors like past job success, relevant skills, responsiveness, and client preferences—before generating a shortlist. This approach improves both reliability and transparency in high-stakes decisions.
• Custom AI Safety and Evaluation Algorithms: Uma is evaluated continuously for safety and effectiveness. We’ve developed proprietary algorithms for red-teaming, bias detection, hallucination mitigation, and success scoring tailored to hiring workflows. These tools ensure Uma performs responsibly and is aligned with both platform values and user expectations. You can learn more about the safety and transparency behind Uma here. 

Developing agentic behavior through deep customization

Where do agents fit into this picture? Now Chef Uma isn’t just cooking a meal, but also planning the menu, coordinating the kitchen, and managing the food budget.

Together, the training techniques above — custom fine-tuning, iterative training, inference-time reasoning, and safety-aligned evaluation — lay the foundation for Uma to operate not just as a helpful companion, but as an increasingly agentic AI system. Through our Uma journey, we have been wading into the agentic pool one layer at a time. Uma is already beginning to understand user intent, adapt to sophisticated workflows, and we have already started building the intricate connections between various models to solve some truly complex problems. 

From a product perspective, Uma has already enabled agentic workflows through instant interviews and our Uma QnA system. In instant interviews, Uma actively interprets the client’s project needs, and based on that analysis, will call tools to identify talent that is a strong fit, reach out to qualified freelancers, and run real-time interviews. Uma’s Q&A model works hand-in-hand with the custom RAG tools and our in-house intent recognition systems to ensure the user is getting the most relevant information they need. This shift shows Uma starting to exceed the bounds of a single model and evolving into a system of various AI models — a whole kitchen of staff — working in tandem to drive a work outcome. 

We believe AI agents will play a major role in the future of work, and we continue to evolve Uma to become a meta agent, an agent that can support coordination or management of other agents. Reaching full agency requires new capabilities, behaviors, and training techniques that we’re actively developing and embedding into Uma today and in the near future. Here’s a snapshot of this work:

• Intent Handling: Uma understands high-level goals and properly sends the right requests to the right endpoints. For example, Uma breaks down a user’s intent into sequenced subtasks and triggers the next logical steps.
• Context Awareness: Uma tracks state and memory over time to manage multi-step workflows. We’re developing a memory architecture that allows Uma to recall relevant past interactions and adjust its behavior on the fly based on personalized context.
• Action & Execution: Uma doesn’t just offer suggestions, but takes action by calling tools and platform capabilities. This is enabled by a function-calling layer and policy engine that determine when and how Uma should take action, such as scheduling an interview or sharing curated candidate insights with a client.
• Adaptability: Uma adjusts its behavior in real time based on changing user input or evolving task conditions. We make this possible through self-evaluation loops, critique-and-revise strategies, and reinforcement learning techniques. 
• Collaboration: Uma supports ongoing interaction and works across multiple sessions and stages. We support this capability with orchestration frameworks that allow Uma to delegate subtasks and monitor progress as a central coordinator working on behalf of both clients and freelancers.
• Autonomy: Uma does not wait for explicit instructions. It initiates next steps when it detects opportunities to move users closer to their goals, and is explicitly trained to always proactively help the user based on defined success criteria. For example, if a client needs to hire quickly, Uma might proactively suggest and initiate an interview with a freelancer who fits the project criteria.

Uma is already beginning to demonstrate these agentic behaviors in production, and we continue to train, test, and evolve its capabilities so our customers can achieve their best outcomes alongside a true AI collaborator.

Uma’s present and future: AI that elevates human brilliance

Just like a truly great chef enables their kitchen staff to cook at their best, Uma is designed to amplify the creativity and strategic brilliance of the people using it. Uma’s evolution from a helpful companion to a more agentic collaborator is a technical milestone our AI & machine learning team is proud of and excited to continue evolving. It reflects a product philosophy centered on partnership, and a belief that the future of work belongs to dynamic teams that combine human creativity with AI systems designed to amplify their impact.

AI systems are extremely useful but are still poor replacements for the creativity and intelligence that define the human experience. With Uma, we strive to empower people by amplifying their best qualities, and to build autonomous agents that work alongside exceptional talent by supporting faster workflows and higher-quality outcomes—achieved together.