What validation usually means

The textbook definition asks a lot. The AIAA’s 1998 guide defines validation as “the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.”¹ Two phrases carry the weight: the real world, and intended uses.

In practice, the evidence that a model is fit for a purpose usually arrives as a validation report written by the people who built it, on cases they selected, sometimes against the same data its parameters were fitted to. None of that implies bad faith. It is how the work gets done when there is no shared record.

Regulators have started to draw the line explicitly. In its 2023 guidance on computational models in medical device submissions, the US FDA interprets validation “as comparison against data that is independent of the data used to create the model”, and states that calibration “is not considered validation”. Its advice is direct: “Be cautious not to present or confuse calibration evidence as/with validation evidence.”²

What happens when the answer is hidden

The cleanest test of a model is a prediction made before the result exists. Several engineering communities have run exactly that experiment on themselves.

The Sandia Fracture Challenge

In mid-summer 2012, Sandia National Laboratories set out to test a gap its organisers named directly: methods in computational mechanics “are rarely validated against problems with an unknown outcome.” Teams were asked to predict how a specimen of a common commercial engineering alloy would tear: the force and crack opening at first cracking, the path the crack would take, and the force and opening when it started to grow again. Thirteen teams from 22 institutions submitted predictions before the experimental results were released.³

Nearly every team predicted the first-crack force within experimental scatter, and every team picked one of the two crack paths seen in the tests. Further into the failure, agreement fell apart. The organisers reported “a striking lack of consistency across research groups” and called for “additional blind prediction-based assessments.”³

They got them. The second challenge, issued in 2014, drew 14 teams. Its authors found the predictions “more consistent and in line with the experimental results than in previous challenges”, though four teams still predicted the wrong crack path in at least one loading case.⁴ By the third, 21 teams took part and “all blind predictions identified the nominal crack path and initiation location correctly”, an improvement the authors credit to the “adoption of best practices from previous Challenges.”⁵

The Drag Prediction Workshops

Aerodynamics saw the same pattern. At the first AIAA CFD Drag Prediction Workshop in 2001, 18 participants using 14 codes computed the drag of the same wing-body configuration. For a single flight condition, the results spanned “a range of over 270 counts in the drag data, which is quite unacceptable.”⁶ A later review of the series put the industry’s goal at ±1 drag count, and notes that the fourth workshop was run as a blind test.⁷

What blind scoring built

Protein structure prediction shows where this can lead. Since 1994, CASP, the Critical Assessment of protein Structure Prediction, has run every two years. Participants receive the sequences of proteins whose structures have just been determined but are kept secret, submit predictions, and have them scored. Its organisers describe the core of the exercise as “rigorous blind testing of methods and evaluation of the results by independent assessors.”⁸

At the fourteenth round, in 2020, AlphaFold2 reached a median score of 92.4 GDT across all targets, where a score of around 90 is informally considered competitive with experimental methods.⁹ The Nobel committee later wrote that when CASP’s organisers evaluated those results, “they understood that biochemistry’s 50-year-old challenge was over.” In 2024, Demis Hassabis and John Jumper shared half of the Nobel Prize in Chemistry for protein structure prediction.¹⁰

The model was remarkable. The reason the field believed it was the format: predictions made before the answer was public, scored by independent assessors, the same way every round.

Why batteries need it now

Battery and materials teams face the same question with three complications.

Each check is slow

“Long battery lifetime entails delayed feedback of performance, often many months to years,” as Severson and colleagues put it in 2019.¹¹ Attia and colleagues made the same point a year later: “conducting even a single experiment to evaluate lifetime can take months to years.” Their closed-loop method found high-lifetime fast-charging protocols among 224 candidates in 16 days, compared with over 500 days for an exhaustive search without early prediction.¹² Every test that confirms or refutes a model is expensive, so the record of those tests is worth keeping.

Behaviour shifts with conditions

In the same 2019 study, 124 commercial LFP/graphite cells of a single model, cycled in a 30 °C chamber under 72 fast-charging policies, lasted anywhere from 150 to 2,300 cycles.¹¹ The conditions that matter most to a product decision, such as fast charging, cold operation and late life, are where behaviour moves fastest.

Models are anchored narrowly

Physics-based models are often parameterised in a narrow window. A commonly used parameter set for the LG M50 21700 cell, with an NMC811 positive electrode and a graphite-SiOx negative electrode, was validated against discharge and relaxation voltage profiles at room temperature.¹³ It ships with PyBaMM as “Chen2020”.¹⁴ Reviews are direct about what happens beyond a model’s window: equivalent circuit models cannot “operate outside the regime in which they are parameterised”, and the single particle model’s “accuracy decreases as the C-rate increases”.¹⁵ One comparison against experimental data found that “the predictability of the DFN model deteriorates when trying to predict the voltage response at low state of charge”.¹⁶

Inputs are uncertain too. A 2022 review of parameterisation found that “experimentally obtained exchange current densities vary by several orders of magnitude”.¹⁷ Degradation predictions can change shape, not just size: for the same LG M50 cell, raising a single cracking-rate parameter moves the predicted capacity fade from quasi-linear to a knee and then to sudden failure, though the authors note this matters only if such values occur in practice.¹⁸

Put together, the conditions where decisions are made are often where models are least anchored, and checking them takes months. That combination is where a sealed, running record earns its keep.

What the standards already ask for

None of this is new to standards bodies. ASME V&V 40-2018 sets out a framework in which the credibility a model needs should be “commensurate with the degree to which the computational model is relied on as evidence” and with “the consequences of that decision being incorrect”.¹⁹ It ties credibility to a context of use, “a statement that defines the specific role and scope of the computational model used to address the question of interest.”²

NASA-STD-7009, published as a standard in 2008 and revised in 2016 and 2024, was developed in response to NASA’s 2004 report on applying the findings of the Columbia accident investigation across the agency. It defines credibility as “the quality to elicit belief or trust in M&S results”, and assesses it through factors that include validation, uncertainty characterisation and results robustness.²⁰

The standards describe what credible evidence looks like. What they leave to each team is the record itself: where predictions are kept, how they are dated, and who can check them later.

A lesson from pre-registration

Science met a similar problem in how results were reported, and addressed part of it by changing the order of events. Registered Reports, launched at the journal Cortex in 2013, review a study’s question and methods before the results exist.²¹ In the Center for Open Science’s words, “peer review occurs prior to observing the outcomes of the research.” More than 300 journals have used the format, either as a regular option or in a special issue.²²

The principle carries over directly. Commit to the prediction and its uncertainty first. Score it after. Keep both.

What a track record needs

From the challenges, the standards and the reporting reforms, five requirements follow.

1. Sealed before the outcome. The prediction, its interval and the test conditions are fixed and timestamped before the result exists, and cannot change afterwards.
2. Scored on the model’s own terms. Each prediction is judged against the interval its model stated, so calibration is visible: a model that claims 90% should contain about 90% of outcomes.
3. Mapped to where it was tested. Results are tracked across the operating envelope, so validated regions are kept apart from extrapolation. This is the context of use, made concrete.
4. Kept over time. Versions, parameter sets and drift stay on the record, and a claim of fitness lapses when the evidence does.
5. Independent of any one vendor. Like CASP’s assessors, whoever keeps the record should have no stake in which model wins.

How Lilienthal puts this into practice

Lilienthal is our implementation of these requirements for teams that run several models against the same lab program. The screens below are from the product, shown with sample data from a fictional cell program.

References

Quotations are reproduced from the sources below. Figures are as reported by their authors.

1. AIAA, Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, AIAA G-077-1998. Link
2. US Food and Drug Administration, Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions, final guidance, 17 November 2023. Link
3. Boyce et al., “The Sandia Fracture Challenge: blind round robin predictions of ductile tearing”, International Journal of Fracture 186, 5–68 (2014). Link
4. Boyce et al., “The second Sandia Fracture Challenge: predictions of ductile failure under quasi-static and moderate-rate dynamic loading”, International Journal of Fracture 198, 5–100 (2016). Link
5. Kramer et al., “The third Sandia fracture challenge: predictions of ductile fracture in additively manufactured metal”, International Journal of Fracture 218, 5–61 (2019). Link
6. Levy et al., “Data Summary from the First AIAA Computational Fluid Dynamics Drag Prediction Workshop”, Journal of Aircraft 40(5), 875–882 (2003). Link
7. Morrison, Kleb and Vassberg, “Observations on CFD Verification and Validation from the AIAA Drag Prediction Workshops”, AIAA 2014-0202 (2014). Link
8. Kryshtafovych et al., “Critical assessment of methods of protein structure prediction (CASP), Round XIV”, Proteins 89(12), 1607–1617 (2021). Link
9. Google DeepMind, “AlphaFold: a solution to a 50-year-old grand challenge in biology” (2020). Link
10. The Nobel Prize in Chemistry 2024, popular information, nobelprize.org. Link
11. Severson et al., “Data-driven prediction of battery cycle life before capacity degradation”, Nature Energy 4, 383–391 (2019). Link
12. Attia et al., “Closed-loop optimization of fast-charging protocols for batteries with machine learning”, Nature 578, 397–402 (2020). Link
13. Chen et al., “Development of experimental techniques for parameterization of multi-scale lithium-ion battery models”, Journal of The Electrochemical Society 167, 080534 (2020). Link
14. Sulzer et al., “Python Battery Mathematical Modelling (PyBaMM)”, Journal of Open Research Software 9(1), 14 (2021); parameter sets in the PyBaMM documentation. Link
15. Brosa Planella et al., “A continuum of physics-based lithium-ion battery models reviewed”, Progress in Energy 4, 042003 (2022). Link
16. Arunachalam and Onori, “Full Homogenized Macroscale Model and Pseudo-2-Dimensional Model for Lithium-Ion Battery Dynamics: Comparative Analysis, Experimental Verification and Sensitivity Analysis”, Journal of The Electrochemical Society 166(8), A1380–A1392 (2019). Link
17. Wang et al., “Review of parameterisation and a novel database (LiionDB) for continuum Li-ion battery models”, Progress in Energy 4, 032004 (2022). Link
18. O’Kane et al., “Lithium-ion battery degradation: how to model it”, Physical Chemistry Chemical Physics (2022). Link
19. ASME, V&V 40-2018: Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices. Link
20. NASA, NASA-STD-7009B: Standard for Models and Simulations (2024). Link
21. Chambers, “Registered Reports: A new publishing initiative at Cortex”, Cortex 49(3), 609–610 (2013). Link
22. Center for Open Science, “Registered Reports”, accessed 18 October 2026. Link