Agent Evaluation Engineer — CAD/CAE/PLM

Project Overview We're building AI agents that automate CAD, CAE, and PLM workflows. Not copilots. Not chatbots bolted onto a toolbar. We are building autonomous agents that drive Creo, CATIA, NX, Ansys, Abaqus, and Windchill the exact way a senior engineer would—modeling, meshing, simulating, releasing, and managing the full product lifecycle. We need a Senior Adversarial Domain Expert whose job is to prove, every single day, that the agent isn't ready yet. The Role You are not QA. You're not writing test scripts for button clicks. You are an adversarial domain expert—a senior engineer who knows exactly what "right" looks like across the product development lifecycle, and who will systematically expose every single place the AI agent gets it wrong. Your success is measured by how often you break the agent and how precisely you can explain why it broke. Core Responsibilities Build the Evaluation Signal: Every time you interact with the agent, you generate structured training data. When the agent makes a modeling decision, changes an assembly constraint, kicks off a simulation, or touches a BOM, you will capture what happened, why it's wrong (or right), and what a competent engineer would have done instead. You'll articulate the exact reasoning: the design intent, the manufacturing constraint, or the simulation boundary condition the agent missed. You're writing the ground truth. Audit Agent Behavior at the Trace Level: You will spend significant time reviewing agent execution traces (Logfire) and model outputs, line by line. Every discrepancy between what the agent did and what it should have done gets documented (e.g., wrong feature order, bad mesh refinement, incorrect tolerance stack, botched ECN workflow, misapplied load case). You are the final authority on whether the agent's engineering judgment is sound. Who You Are Highly Experienced: You have 10–20+ years working hands-on in CAD/CAE/PLM across real product programs (automotive, aerospace, heavy equipment, consumer hardware, or similar). CAD Master: You've built complex models and assemblies in at least two of the following: Creo, NX, CATIA, SolidWorks. CAE/Simulation Expert: You've set up and run production simulations in at least two of the following: Ansys Mechanical/Fluent, Abaqus, COMSOL, HyperMesh/OptiStruct, ANSA. PLM Administrator/Power User: You've worked deeply within Windchill, Teamcenter, or ENOVIA. You haven't just checked parts in and out; you have configured workflows, lifecycles, and change processes. Cross-Functional Visionary: You know what it means to hand off a model from design to simulation to manufacturing, and you can easily spot where intent gets lost at each boundary. Highly Opinionated: You have strong opinions about feature order, datum strategy, and why a specific sketch constraint is going to cause a regeneration failure three revisions down the line. Adversarial Mindset: You find deep satisfaction in finding the failure mode no one else thought to test. Nice to Have Experience with ANSA or HyperMesh for pre-processing / meshing workflows. Familiarity with MBD (Model-Based Definition) and annotation-driven manufacturing handoffs. Exposure to API-driven automation (J-Link, NX Open, CATIA VBA/CAA, Ansys APDL/Workbench scripting). You don't need to be a developer, but understanding what's programmable in these tools is extremely valuable. Experience writing engineering standards, design review checklists, or process documentation.