The Validation Gap: Why Engineering Models Never Perfectly Match Reality
During ICOE-OEE 2026 in The Hague, Blackfish Operations Director Tim Warren delivered a presentation exploring what he described as the "validation gap". While the examples focused on ocean energy technologies, the underlying challenge is one that extends across almost every area of engineering.
How do we know whether our models, simulations and analyses accurately reflect what will happen in the real world?
At first glance, that might seem like a straightforward question. Modern engineering tools are incredibly powerful. Engineers can model structural behaviour, simulate environmental loading, predict fatigue life and analyse system performance long before a prototype is ever built.
Yet anyone who has worked on a real engineering project knows that predictions and reality rarely align perfectly.
That is not because the models are wrong. It is because reality is complicated.
Every Model Is a Simplification of Reality
Engineering projects begin with assumptions.
Before hardware is manufactured, deployed or operated, engineers rely on analysis, simulations, testing and previous experience to understand how a system is likely to behave. These techniques are essential because it would be impossible, both financially and practically, to build and test every possible design variation under every conceivable operating condition.
However, every model is ultimately a simplification of reality.
Loads are estimated. Boundary conditions are assumed. Environmental conditions are represented by datasets and statistical models. Material properties are based on available information. Certain physical effects may be simplified, ignored or combined to produce a practical engineering solution.
None of this is a flaw in the engineering process. It is simply the reality of engineering.
The challenge is that when a device eventually encounters the real world, it also encounters countless variables that are difficult, and sometimes impossible, to represent perfectly.
This is where the validation gap emerges.
The validation gap is the difference between what engineering models predict and what is ultimately observed when a system is tested or operated in practice. Understanding that gap is often where the most valuable engineering learning takes place.
The Goal Is Not Perfect Prediction
It can be tempting to think that the solution is simply to create increasingly sophisticated models. Larger simulations, higher-fidelity analysis, more detailed assumptions and greater computational power can all provide valuable insight.
However, there is an important distinction between increasing model complexity and improving engineering decision-making.
The real question is not:
"Can we create a perfect model?"
The real question is:
"Do we understand the uncertainty well enough to make a good decision?"
Engineering is rarely about achieving perfect certainty. More often, it is about achieving an appropriate level of confidence.
In practice, the challenge is often determining how much confidence is enough. The answer depends entirely on what decision is being made and the consequences of getting it wrong.
The level of confidence required naturally depends on the consequences of being wrong. A small discrepancy in predicted energy production may be tolerable. A structural failure offshore is not.
Understanding where uncertainty exists, how significant it is and how it might influence the design is often more important than endlessly pursuing ever-greater analytical complexity.
Why Validation Matters
Validation is the process of comparing predictions with reality.
It is how engineers determine whether the assumptions made during design remain valid when confronted with physical testing, operational experience and measured data.
While the examples discussed during Tim's presentation focused on ocean energy technologies, the same principles apply across many forms of technology development. For example, Blackfish recently helped Solead develop a structured Verification and Validation strategy for its novel energy storage technology, creating a roadmap that helped the team communicate development plans, manage technical risk and build confidence with investors, regulators and other stakeholders. Validation is not simply about testing hardware. It is about understanding what evidence is needed to support the next decision.
One of the examples discussed during Tim's presentation concerned the analysis of hinge-angle behaviour within a wave energy system. Understanding the loads and movements experienced by the hinge required a combination of numerical modelling, physical tank testing, site data and operational measurements.
The most interesting lesson was not whether the analysis was correct, but what could be learned by comparing predictions with measured performance.
Meaningful validation requires planning.
Too often, opportunities to compare prediction and reality can disappear as projects evolve. Teams move on. Priorities change. New challenges emerge. By the time real-world data becomes available, the original assumptions may no longer be actively reviewed.
Validation therefore should not be treated as a single stage within a project. It is a process that needs to be considered throughout the development journey.
The most valuable insights often come not from proving that a model was right, but from understanding why reality behaved differently.
One of the more subtle challenges is that meaningful validation often takes place over long timescales. In one example discussed during the presentation, there was a gap of several years between early testing and later operational data. By that point, priorities had changed and the opportunity for a direct comparison had become more difficult than expected. It serves as an important reminder that validation needs to be planned from the outset. If the right questions are not identified early, the necessary evidence may never be collected.
The Offshore Environment Adds Complexity
Marine energy technologies provide particularly good examples of the validation challenge because the offshore environment is inherently difficult to predict.
Engineers may undertake finite element analysis, laboratory testing, tank testing and full-scale trials. Yet once equipment is deployed, it encounters factors that are extremely difficult to replicate perfectly, including corrosion, biofouling, manufacturing tolerances, installation conditions, operational realities, environmental variability and human interaction.
Each introduces additional complexity.
A design may perform exactly as expected under laboratory conditions while still producing unexpected results offshore. Equally, a system may demonstrate performance that exceeds initial expectations.
Both outcomes provide valuable learning.
The important point is that engineering decisions are strongest when they are informed by both prediction and evidence.
Understanding Differences Is Often More Valuable Than Eliminating Them
When testing produces results that differ from simulations, the immediate reaction is often to identify which result is "correct".
In practice, a more useful question is usually:
"Why are they different?"
That investigation can reveal important insights into the design, the assumptions underpinning the analysis and the behaviour of the system itself.
Differences may arise from environmental conditions, measurement uncertainty, manufacturing variation, simplifications within the model or operational factors that were not originally considered.
Understanding these differences helps engineers refine future designs, improve confidence in results and make better-informed decisions.
The objective is not necessarily to force predictions and reality into perfect alignment.
The objective is to understand the relationship between them.
In reality, validation is rarely about proving a model is perfect. More often it is about building sufficient confidence to support a decision. The level of confidence required depends on what is at stake. Predicting energy production within a few percentage points presents a very different challenge from demonstrating that a critical component will survive offshore for twenty years. Understanding what decision is being made, and the consequences of being wrong, is often more important than endlessly increasing model complexity.
Validation and Verification Are Not the Same Thing
During the presentation, Tim also explored the relationship between validation and verification, terms that are often used together but serve different purposes.
Put simply:
Validation asks whether we designed the right thing.
Verification asks whether we built the thing right.
Both are important.
A technology can be built perfectly to specification while still failing to solve the problem it was designed to address. Equally, an excellent concept can fail if it is implemented incorrectly.
Together, validation and verification provide a structured framework for managing uncertainty, planning testing activities and building confidence in engineering decisions.
They help organisations understand what evidence they need, where risks remain and which questions still need to be answered.
Engineering Is About Understanding Uncertainty
One of the most useful takeaways from the discussion was that engineering is not about eliminating uncertainty entirely.
That is rarely possible.
Every model contains assumptions. Every test has limitations. Every deployment introduces new variables.
Good engineering is not the pursuit of perfect certainty.
It is the process of understanding uncertainty sufficiently well to make informed decisions.
Whether developing a wave energy converter, a tidal turbine, a test rig or any other complex system, the challenge remains remarkably similar. Engineers must balance evidence, analysis, experience and testing to build confidence in the decisions they make.
The validation gap will always exist in some form.
The goal is not to remove it completely.
The goal is to understand it, learn from it and use it to make better engineering decisions.
Because ultimately, successful engineering is not about predicting the future perfectly. It is about understanding reality well enough to move forward with confidence.
For technology developers, the real challenge is often not whether uncertainty exists, but whether enough has been done to understand and manage it before making the next important decision. Whether that decision relates to design progression, investment, testing, certification or deployment, confidence is built through evidence, not assumptions.
That is why validation planning, testing strategies and structured Verification and Validation (V&V) approaches remain such valuable tools. They help organisations focus effort where it matters most, reduce unnecessary risk and build confidence in both the technology and the decisions surrounding it.
At Blackfish, these are exactly the conversations we enjoy having. If you're developing a new technology, planning a test programme or trying to understand how much confidence you really have in your design assumptions, we'd be happy to discuss the challenge.
Comments