The main thing I think about with agentic AI these days is trust. Agent capability is growing faster than our capacity to digest and audit what agents produce, so we need ways to delegate work that keep oversight efficient. Trusting an agent means knowing where its weak spots are and where we can be confident in its work. Public software benchmarks have given us confidence that agent-written code will probably compile and run on the first try, but none of them measure whether an agent chose the right covariate structure or picked a model consistent with a drug's mechanism of action. To build trust, we need a benchmark that aligns agent scores with pharmacometric judgement. I've also come to think a benchmark might be the last thing we'll need to build. The rules we write for agents get outgrown with each model release, but the definition of a good analysis doesn't, and no model can write it for us.
TL;DR: PMxbench is a pharmacometrics benchmark you can run today. It's where I think our field's expertise should go. It needs better scenarios and better ways to score them, and I'd like your help with both.
Three elements of trust
I've divided trust in agents into three elements. They are human skills, agent skills and benchmarks, and they map directly to the objectives of our working group.
Human skills build with education and consistent exposure to AI agents. Every modeler should be prompting, iterating, and breaking these systems, because that hands-on understanding is the foundation the other elements of trust are built on.
Agent skills are the context we engineer for the agent. Context engineering means deciding what the agent sees at each step of the work, so that its instructions carry the expertise we'd expect from a modeler. I've covered this in detail in our CPT: PSP paper, notes on building Modus and the working group walkthrough video.
This post is about the third element, benchmarks. Benchmarks evaluate an agent's performance against a scenario where we know what a good answer looks like. The better an agent performs on our benchmark, the more we can trust it with similar tasks. Benchmarks are also where the other two elements meet, since people write the scenarios and the scores rank our agent skills. I'll start by introducing the context engineering workflow from our paper as the test framework and then show how a numerical score can begin to capture a pharmacometrician's judgement. I recently presented this in a talk at ACoP 2026 (slides, PDF), and this is the longer written version.
Agent skills in practice
You can think of the workflow like a computer. The agent is the CPU, processing instructions and doing the work. The deterministic shell script that drives the loop is the operating system, queuing the next agent process and managing inputs and outputs. The task library is the program, and in this case it encodes pharmacometric expertise into the agent. The shared state saved to the workspace is the hard drive, where each step leaves its outputs for the next one to read.
The workflow from our paper, mapped to the computer analogy. A person works through an orchestrator agent that starts the script, and each task agent commits its work before the loop moves on.
The task library is a single JSON file that describes each step of an analysis, from raw data to report. For each step, the script hands the fresh agent that step's rules, a defined set of outputs, and a short list of things to verify before it finishes. If you've used agent skills, each task entry is close to a SKILL.md. As models get better some rules become redundant, and I expect most of them to go eventually. For now there always seems to be some nuance or edge case where a modeler would do something different from the base agent, and this is where that nuance lives.
The covariate task, unrolled as one unit operation. It gets the same base prompt as every task, its own task entry, and the outputs of earlier tasks, and it leaves a script, results and a summary.
Every task leaves a folder with a script, results, and a summary of what the agent decided and why, all committed with git. A key payoff of this structure is that you can easily rerun the whole analysis without the agent. The summaries let us check the agent's reasoning and push back when it doesn't match ours, but reading every summary doesn't scale. Instead, a benchmark gives us a numerical score that shows at a glance how well the agent's judgement lines up with our own.
What a benchmark measures
Software has a very active field of agent benchmarks we can borrow from. SWE-bench scores an agent's fix to a real historic GitHub issue with that project's own test suite, so every task is realistic and has a known answer. We don't have a public database of solved pharmacometric analyses on that scale, so for now we can build scenarios where we know the ground truth because we simulated it. The classic version is simulation and estimation, where you simulate clinical trial data from a known model, fit it, and compare the parameters.
Simulation and estimation is useful, but it's only the bottom rung of a ladder, and tells us less about agent performance than we may think. Once the agent has set up the model, the parameter estimates come from the estimation tool, so a good score there is partly a statement about NONMEM or nlmixr2 and identifiability in the data. Modeling decisions sit a rung above and reflect more of the agent's own judgement. Did the agent catch the bad PK records, test the covariates that are clinically meaningful, and pick a model structure that reflects the biology? No estimation tool makes those choices for it, but they're harder to score because two modelers can disagree based on instinct or taste. At the top are key design decisions, where one choice changes every analysis after it, like whether new trial data justify refitting the model, or which exposure metric should drive the dose recommendation. These say the most about the agent and are the hardest to put a number on. So the higher we climb, the more the score measures the agent and the harder it gets to score at all. A benchmark we can trust needs scores on every rung.
The three rungs. Going up, the score measures more of the agent and gets harder to score. Scenario 00 scores only the bottom two.
A worked example
PMxbench scenario 00 is a simple PopPK scenario I built with three traps an agent might fall into. It simulates a first-in-human population PK study of a monoclonal antibody in 120 subjects, who each get a single IV infusion at one of three dose levels and have weight, albumin and creatinine clearance recorded. The agent gets an analysis plan, the data, and a template for its answers.
- One subject per dose group has a concentration with a moved decimal point.
- Albumin tracks body weight but has no effect of its own, shown only by testing it with weight already in the model.
- Creatinine clearance has no effect either but is included as a column in the dataset.
Scenario 00 data, with the decimal slips in red. Albumin tracks weight (r = 0.40) and creatinine clearance doesn't.
The scoring is a first pass at putting numbers on all of this. It checks five components, the structural model compartments, parameter values, error model, covariates and outliers. Each component scores between 0 and 1, and the overall score is their average. Numbers lose credit smoothly as they drift from the truth, and sets lose credit for anything missed or anything extra. Parameter values sit on the bottom rung, and the structural model, error model, covariates and outliers sit on the second, because they do require the agent to make decisions about the model. Nothing in scenario 00 reaches the top rung yet.
Each item is scored against the answer key. Numbers lose credit smoothly with relative error, categories are all or nothing, and sets lose credit for both misses and extras.
Proof of concept
I ran scenario 00 forty times, ten runs for each of two models with and without the task library.
Even on a scenario this simple, the benchmark separated the models. Claude Opus 4.8 averaged 0.87 against 0.73 for Claude Sonnet 5. Every run got the structural model right and caught the decimal slip outliers.
Overall score for all 40 runs, with one circle per run and a tick at each arm's mean. Changing the model moves the score, and adding the task library doesn't.
The task library made no difference to the score for either model. One reason may be that the scenario (straightforward PopPK fit) fits in one context window easily, so modular tasks are unnecessary. The other reason may be that the task library I used was written by an agent, so its rules were mostly the agent's own defaults. A library like that can't teach the agent anything it doesn't already do.
The score also can't see the workspace, where runs without the task library left oddly named scripts and no record of why anything was decided. Reproducibility is the obvious next item to score, once we agree on what that rubric looks like.
The subscores also show where a rule would help. The true model has proportional residual error only, yet 33 of the 40 runs kept an additive term. Plenty of modelers keep a combined error model by default too, which is exactly why a standard rule helps. The one you'd write by hand might be something like keep an additive error term only if the low-concentration data support it.
The last thing we need to build
That last finding is where benchmarks and agent skills connect. Rules in the task library (or SKILL.md files, if you prefer) are where benchmark feedback gets applied. The next step is to add the error-model rule, rerun the scenario and see whether the score improves.
Writing that rule is tuning the prompt by hand to improve one subscore. Now imagine running another batch of 40 agents, recording their actions and scoring them. A separate agent could then search those records (agent traces) for the patterns behind high and low scores and propose changes to the rules. The AI field already tunes prompts this way (see GEPA, available in DSPy), and it's where we're headed once we have meaningful scores.
A second agent reads the traces and proposes a rule edit, which a person approves before the next run.
This is where the bitter lesson comes in. In 2019 Rich Sutton looked back on seventy years of AI research and found the same pattern again and again. Researchers built their own knowledge into a system, it helped for a while, and then general methods that scaled with computation overtook it. Chess programs built on grandmaster strategy lost to brute search, and speech recognition built on linguistics lost to statistics. The lesson is bitter because the hand-built knowledge was real expertise, and it lost anyway. Andy Stein has a great post on what it means for pharmacometricians.
Hand-written rules in a task library are that kind of knowledge, and the null result above is a small version of the lesson. Every model release absorbs some of the rules, and an optimization loop with enough runs could find most of the rest. What doesn't get absorbed is the score.
A model can't tell us what counts as a good pharmacometric analysis, because that's a judgement our field makes, and often there's more than one good answer. Agents diverge on decisions like which technique to use or which model is right, the same way experienced modelers do. An agentic workflow doesn't have to pick one branch. It can run every defensible path to the end and show whether the choice changes a parameter estimate, a dose, or nothing at all. I don't have a way to score the top rung yet, but the answer key will probably be a set of defensible paths rather than one right answer, and only people who have done the work can say which paths belong in it.
However capable agents get, how far we can trust them is capped by how well the score separates a good analysis from a weak one. The scaffolding around the agent is transient, but the benchmark grows in value with every scenario we add. Each one becomes a regression test that every new model, tool or task library has to pass, and if the scenarios cover the pathologies we care about, an agent that scores 100% is exactly what we want. The benchmark is what keeps agents aligned with expert judgement, and I think it may be the last thing we need to build.
How to join
A benchmark is only as good as the human skill that goes into its scenarios and scores, and no one person should decide what counts as the right answer, me included. There are two ways you can get involved. You can run scenario 00 by hand or with your own agent and submit your answers. You can also join the Agentic Pharmacometrics working group and bring a pathology from your own work. The data behind it may be proprietary, but the pathology isn't, and if you can describe what went wrong we can simulate a study that reproduces it. Scoring ideas, tools, skills and stories of where agents went wrong are just as welcome.
What do you see agents constantly failing to get right?
Thank you to everyone who came to the ACoP session and the ISoP AI/ML SIG lunch afterwards.
This post was developed with assistance from AI. Views expressed are my own and do not represent my employer.