Audit of Concordare's Digital Protocol AI Conversion Engine Achieves 99.6% Accuracy
Everything Concordare generates, from source documents to EDC builds to CTMS configurations, comes from one place: the CTPM (Clinical Trial Protocol Model) extracted from your protocol. That makes extraction quality the single number that matters most in our platform. So we measured it formally.
In June, we completed a third-party structured quality assessment of Apta, our AI-powered protocol conversion engine. This post covers how we ran it, what we found, and what we're doing about the gaps.
How we measured
We took a real clinical trial protocol and built a reviewed ground truth: our clinical team went through the document and identified every discrete data element a complete extraction should capture, including visit windows, procedures, eligibility criteria, dosing rules, and assessment mappings.
Then we scored Apta's extraction against that ground truth, element by element, where each populated value counts as one element. Every element landed in one of three buckets: correct, present but wrong, or missing.
What we found
Apta's baseline run produced 3,171 scored elements. 3,159 were correct and 12 were present but wrong: 99.62% precision. Another 82 elements that should have been captured were missing, putting recall at 97.47% and the balanced F1 score at 98.53%.
We scored the same results three ways, because most of the errors repeat. The strictest method counts every error on its own; the other two count each recurring known issue once, or set known issues aside since they are already on the fix list.
- Raw, every error counted individually: 99.62% precision, 97.47% recall, 98.53% F1. 12 present but wrong, 82 missing.
- Deduplicated, each recurring issue counted once: 99.84% precision, 98.78% recall, 99.31% F1. 5 present but wrong, 39 missing.
- Novel, known recurring issues excluded: 99.91% precision, 98.97% recall, 99.43% F1. 3 present but wrong, 33 missing.
Terms, precisely: present but wrong means content was produced that does not match the source, a false positive. Missing means content that should have been captured was absent, a false negative. Precision is the share of what the pipeline extracted that was right; recall is the share of what should have been extracted that was caught.
The distinction between those two numbers matters. In clinical research, the failure mode that carries the most risk is confidently wrong data: an incorrect visit window or a wrong dosing threshold that looks plausible and flows downstream. That is the error class Apta almost never produces. The remaining gap is completeness, occasionally missing an element rather than distorting one. A missed element is visible during review; a plausible-but-wrong one is what quality systems exist to catch.
The engine almost never produces incorrect data. The remaining errors are clustered, repeatable, and fixable — engineering problems with clear solutions, not fundamental limitations.
What we're doing about the gaps
The errors weren't random. A small number of repeating patterns account for most of the error weight: visit scheduling assignments, placement of conditional footnotes, administration attributes on assessments, and completeness of certain reference lists. Each is a known behavior with a planned fix, and because the errors cluster, each fix closes a category of misses rather than a single instance. Counting each recurring issue once lifts the scores to 99.84% precision and 98.78% recall; setting known recurring issues aside entirely, the novel errors that remain amount to 3 wrong and 33 missing.
This assessment is also not a one-off. Evaluation against human-validated reference digitizations is part of how we operate: feedback rates are tracked over time and across protocol types as a core stability metric. And every CTPM is still reviewed and validated by the study team before a single output is generated. The audit measures the engine, but human sign-off remains the gate.
Why we're publishing this
AI vendors in clinical research make a lot of accuracy claims. Few say what they measured, against what reference, or where the errors were. If our platform is going to sit at the foundation of how trials get built, the people relying on it deserve the denominator, not just the percentage. We'll keep publishing these numbers as the engine and the evaluation set grow.
Want to see how your protocols extract? Get in touch, we'd love to run one with you.
