Beyond FHIR records

Abstract

Slothwise develops AI for interpreting information distributed across a person’s health history. In a same-model development comparison on 100 questions, Slothwise with Gemini 3 Flash Preview achieved a 79% pass rate versus 51% in direct mode, a 28-percentage-point improvement. Answers graded largely incorrect or incorrect fell from 23% to 8%, a 65% relative reduction. The comparison evaluates complete answering configurations with different access to record evidence.

Across three repeated evaluations, Slothwise with GPT-5.5 achieved a 95.0% mean pass rate and passed 89/100 questions in every run. In a separate adapted MedAgentBench evaluation, Slothwise passed 150/150 read-only tasks (100%) using its adapted interface and interaction budget. GPT-5.5 alone scored 119/150 in Stanford’s own harness.

Engineering measurements showed a p95 chart-read latency of 67.6 ms across 222,007 requests, 0.65 GB of queryable storage for 20.5 million rows, and engine ingestion of approximately 21,600 FHIR lines per second on one 16-vCPU worker.

The research program includes an initial source pool of 11,540 synthetic histories with approximately 4.97 million extracted facts, a 1,000-question pool, and a manually reviewed 100-question evaluation core. Five models within Slothwise produced 1,500 responses across three repetitions. Synthetic-benchmark passes combine fully and substantially correct answers; MedAgentBench measures required-output task completion. Clinical effectiveness remains a prospective research question.

Introduction

Slothwise has built a patient-facing system for interpreting connected health records and is extending this work into clinician workflows. The intended clinical application is to help clinicians evaluating complex, unexplained symptoms reconstruct changes across fragmented histories and locate the supporting evidence. The technical objective is to preserve the meaning, timing, and source of each fact.

Structured records still require interpretation. A medication order, a report of medication use, and a record of administration represent different evidence. An observation may contain multiple components rather than a single scalar result. These distinctions are part of the FHIR standard and affect normalization and retrieval. [1, 2]

Our evaluation program characterizes the source histories and task complexity, compares complete answering configurations, and measures response quality and repeatability. It combines source-record review with structured grading and statistical analysis.

This report presents Slothwise’s aggregate experimental findings, methods, and statistical analysis.

Methods

Benchmark data and reference review

Synthetic histories were generated using Synthea [3] and represented in FHIR. The initial July source-pool characterization recorded 11,540 histories and 4,965,918 extracted facts in 11 categories. The generated question pool contained 1,000 questions associated with 1,000 histories. A 100-question core was randomly selected, and its reference answers were checked and corrected against source records. The core was subsequently used for development and repeated evaluation.

Question construction combined focal facts with surrounding patient context. Corpus and construction statistics describe the initial July snapshot, preceding later extraction refinements and reference review. Internal benchmark results refer to the separately reviewed 100-question core.

Experimental comparisons

In the Gemini comparison, direct mode received selected record context, while Slothwise could consult the connected history with tools. This is a comparison of complete configurations with different evidence access, not an isolated test of one architectural component. The five-model repeated study uses Slothwise in every arm. Model names throughout are the labels recorded for those historical experiments.

Endpoints and statistical analysis

Responses were classified as fully correct, substantially correct, partially correct, largely incorrect, incorrect, or no answer. Fully correct denotes an accurate and complete answer under the rubric; substantially correct permits limited deficiencies. A pass combines these two categories. The repeated study used GPT-5.6-sol as the evaluator. We report the pass rate and the fraction graded largely incorrect or incorrect; the latter measures answer quality rather than adjudicated clinical harm.

Response-level percentages pool the three repetitions, with 300 responses per model. Question-level consistency classifies each of the 100 questions as passing all three times, passing some but not all times, or never passing.

For the October reanalysis, 95% percentile bootstrap intervals used 20,000 resamples of the 100 questions (random seed 20261027). Each resample retained all three responses per question in the repeated study and paired responses in the Gemini comparison. Model differences used the same sampled question indices. Intervals are descriptive, conditional on the study and grading, and are not adjusted for multiple comparisons.

Results

System improvement with the same answer model

Using the same Gemini 3 Flash Preview model and the same 100 questions, Slothwise increased the pass rate from 51% to 79%. Answers graded largely incorrect or incorrect fell from 23% to 8%, a 65% relative reduction.

The 28-percentage-point pass-rate improvement had a paired 95% bootstrap interval of 16 to 40 points. Slothwise could consult the connected history with tools, while direct mode received selected record context. The experiment measures the complete configuration, including its evidence access, rather than isolating normalization, retrieval, or prompting.

Task completion on MedAgentBench

MedAgentBench is a physician-authored evaluation environment built around de-identified electronic health records [4]. We evaluated its 150 read-only tasks across five categories. Slothwise passed all 150 tasks (100%), with 30/30 in every category, using its adapted interface and interaction budget. The order-placing and other write tasks were excluded because Slothwise’s patient reasoning system is read-only and never writes to records.

Slothwise completed 31 more tasks than GPT-5.5 alone, a 20.7-percentage-point difference under the respective evaluation configurations. Required-output task completion includes retrieval, protocol execution, and answer format. All three standalone models failed the 30-task category requiring a specific output shape; the remaining failures involved patient lookup, retrieval, or protocol execution.

Slothwise’s evaluation adapters supported record search, patient binding, the benchmark’s fixed time, and answer formatting. Patient lookup used an evaluation-only broader search scope. All configurations used the same reference-checker implementation, obtained from four byte-identical upstream forks.

Record size and task complexity

The initial source histories were strongly uneven in size: the mean exceeded the median, and the largest history contained 8,905 extracted facts. The constructed tasks combined multiple facts within a larger context. These statistics motivate evaluation of completeness and cross-record synthesis alongside individual-fact correctness.

Model selection and answer quality

Slothwise with GPT-5.5 achieved a 95.0% mean pass rate (285/300 responses), compared with 93.0% for the Opus configuration and 76.3% for Sonnet. Each model was evaluated on the same reviewed questions within Slothwise.

Claude Sonnet 5 had 60/300 responses (20.0%) graded largely incorrect or incorrect, compared with 9/300 (3.0%) for GPT-5.5 and 20/300 (6.7%) for Opus. This endpoint distinguishes the frequency of these errors alongside the overall pass rate.

Together, the distributions support model selection based on pass rate and error frequency, with repeatability examined in the next section. Each result describes a candidate model operating within Slothwise on the same development questions.

Consistency across repeated evaluations

Slothwise with GPT-5.5 passed 89 questions in every run, compared with 83 for the Opus configuration and 59 for the Sonnet configuration. Sonnet had 31 questions with mixed outcomes and 10 that never passed, so 41/100 had at least one non-pass. These measurements provide a stronger basis for selecting a configuration than its best single run.

The paired mean pass-rate difference between GPT-5.5 and Opus was 2.0 percentage points (95% bootstrap interval, -2.0 to 6.0); this study does not resolve a pass-rate difference between them. The GPT-5.5–Sonnet difference was 18.7 points (12.7 to 25.0). The paired comparisons quantify differences between the evaluated Slothwise configurations on the shared development set.

Discussion

The studies provide complementary evidence: improved answer quality with the same model, repeated evaluation of candidate models within Slothwise, task completion on MedAgentBench, and engineering performance on large record workloads. Together, they support an approach that evaluates the complete answering system, selects models using multiple quality endpoints, and measures the infrastructure needed to serve longitudinal histories.

Engineering feasibility

Limitations and clinical translation

The internal synthetic-benchmark studies used a repeatedly evaluated 100-question development core. Their reference answers were manually reviewed; response grading was automated and was not independently adjudicated for every answer. The bootstrap intervals quantify question-level sampling variation within this design. MedAgentBench is a separate evaluation using an external task set and de-identified clinical records.

The comparisons apply to the reported model versions, resource settings, and benchmark adaptations. The same-model experiment evaluates the complete configuration. Clinical effectiveness and cost savings remain outcomes for prospective evaluation.

The next research step is independent reference review on held-out records, followed by prospective clinician evaluation of review time, clinically relevant facts identified or missed, and correction burden. That would test whether technical improvements translate into useful care workflows.

Data provenance and availability

Slothwise conducted the experiments reported here. The internal benchmark uses histories generated with Synthea; MedAgentBench supplies a separate task set and de-identified clinical-record environment [3, 4]. July source-corpus statistics describe the initial extraction snapshot. August grade distributions and repeatability counts were recalculated from archived response-level labels. October engineering figures are separate company measurements.

The tables and figures present aggregate results, with sample sizes, endpoint definitions, and comparison conditions stated alongside them. Slothwise’s internal benchmark questions, answer keys, and patient-level study data remain private, as do production prompts and proprietary implementation code. MedAgentBench materials are provided by the benchmark’s authors [4]. Full reproduction of Slothwise’s experiments would require access to its private evaluation materials.

The report synthesizes the dated experiments above, with statistical reanalysis completed on 26 October 2026. Public references describe the underlying standards and benchmark methods. Slothwise’s experimental methods and aggregate results are reported directly in this document.

References

1. HL7 International. MedicationRequest. FHIR Release 4, version 4.0.1. Resource semantics and relationships among medication orders, dispensing, administration, and reported use. Accessed 26 October 2026. https://hl7.org/fhir/R4/medicationrequest.html
2. HL7 International. Observation. FHIR Release 4, version 4.0.1. Observation values, components, and relationships to diagnostic reports. Accessed 26 October 2026. https://hl7.org/fhir/R4/observation.html
3. Walonoski J, Kramer M, Nichols J, et al. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association. 2018;25(3):230–238. doi:10.1093/jamia/ocx079. https://doi.org/10.1093/jamia/ocx079
4. Jiang Y, Black KC, Geng G, Park D, Zou J, Ng AY, Chen JH. MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents. NEJM AI. 2025;2(9). doi:10.1056/AIdbp2500144. https://doi.org/10.1056/AIdbp2500144 Open preprint: https://arxiv.org/abs/2501.14654 Benchmark implementation: https://github.com/stanfordmlgroup/MedAgentBench

Acknowledgments

This work was conducted by Jacob Berkowitz, Sofia Sigal-Passeck, Justin Zheng, and Behzat Acele. We also thank Ivan Zhang and Natalie Ma for their help and inputs.