Before You Test the Technology, Decide What You Need to Learn
Why good testing starts with the question, not the test
If reality always gets a vote, testing is one of the ways we give it a voice while there is still time to respond. Technology development is full of moments when testing feels like the obvious next step. A concept has progressed through modelling and analysis, a prototype exists, funding has been secured or another development milestone is approaching. Perhaps an investor, customer or regulator is beginning to ask for evidence that the technology performs as intended. At that point, attention quite naturally turns towards the test itself: what should be built, where should it be tested, what facility is required and how closely can the eventual operating conditions be reproduced?
Those are all reasonable questions, but there is another one that should probably come first: what exactly are we trying to learn? It sounds incredibly simple, but the answer can fundamentally change the way a test programme is approached. Testing is not valuable simply because a test has been completed, data has been collected or a prototype has survived. Its value lies in whether it reduces an important uncertainty, challenges an assumption or generates evidence that allows somebody to make a better decision about what happens next. Sometimes the difference between conducting a test and generating useful engineering evidence can be surprisingly large.
Testing Is Not Simply a Development Milestone
Technology-development programmes are often described through a reassuring sequence of concept, analysis, prototype, testing, demonstration and ultimately commercialisation. The sequence is useful because it provides structure, creates milestones and allows teams, investors and other stakeholders to understand where a programme sits. The danger comes when the structure begins to imply that development itself progresses just as neatly, with testing treated as another box that can be ticked before the programme moves confidently to the next stage.
As we explored in Reality Always Gets a Vote, real technology development rarely behaves quite so obediently. Testing does not sit independently between design and demonstration. Results travel backwards as readily as they move forwards. They can challenge models, change requirements, expose unexpected behaviour and occasionally force teams to revisit decisions that appeared settled several stages earlier. That is not necessarily evidence that development has gone wrong. In many cases, it is evidence that development is doing precisely what it is supposed to do: replacing assumptions with progressively better understanding.
There can nevertheless be a subtle pressure to treat a test as something that must be passed rather than something from which the team needs to learn. A successful test provides visible progress. Hardware operates, data is generated, photographs can be taken and a milestone can be reported to investors, funders or customers. An unexpected result is considerably less comfortable, but if the purpose of testing is to improve understanding, a test that exposes a weakness may ultimately be more valuable than one that simply confirms what everybody already believed. The objective should therefore not be to make a test successful at all costs. The objective should be to make the learning useful.
Start With the Decision, Not the Test
Before specifying a physical test, it is worth understanding what decision will eventually be made using the evidence it generates. A development team might be trying to establish whether an architecture is viable, whether a particular component will withstand a defined load, whether a numerical model adequately represents physical behaviour or whether several individually proven systems can operate successfully together. Elsewhere, the uncertainty might relate to manufacturing consistency, degradation after repeated operation, fatigue life, maintainability or whether the technology is sufficiently mature to justify further investment.
These questions are related to technology development, but they are not interchangeable and they do not necessarily require the same form of evidence. Some uncertainties can be addressed effectively through analysis. Others may require inspection, comparison with existing evidence, simulation, peer review or physical testing. In many programmes the strongest evidence will come from several of these activities working together rather than from one definitive test at the end.
Blackfish's recent work with Solead on a Verification and Validation strategy for its energy-storage technology illustrates this broader way of thinking. Requirements and risks were considered against different methods of building evidence, including analysis, inspection, physical testing, design review and similarity assessment. The intention was not to produce the largest possible test programme, but to create an efficient development roadmap capable of reducing technical risk and supporting the decisions required as the technology moved towards commercialisation.
That principle travels remarkably well between sectors. Whether the technology is an energy-storage system, an aerospace mechanism, an autonomous machine or an offshore device, the first engineering challenge is not necessarily deciding how it should be tested. It is understanding what needs to be known, why it matters and what evidence will provide enough confidence to support the next decision.
More Testing Does Not Automatically Create More Confidence
There is an understandable temptation to associate confidence with quantity. More tests, more sensors, longer campaigns, larger prototypes and increasingly realistic environments can all feel like stronger evidence, and sometimes they genuinely are. However, engineering evidence derives much of its value from relevance and quality rather than its volume. If the uncertainty concerns the behaviour of a particular component, immediately testing an entire integrated system may introduce additional variables that make the result harder rather than easier to interpret. If a question can credibly be resolved at component or subsystem level, progressing straight to an expensive full-system demonstration may simply add cost and complexity.
A similar question applies to the amount of data generated. Modern test systems can measure enormous numbers of variables, but collecting data is not the same as creating understanding. The useful question is whether the measurements provide information that changes, supports or challenges an engineering decision. If the result of an expensive test could move significantly in either direction without affecting what the development team does next, there is a reasonable argument that the purpose of that test has not yet been defined clearly enough.
This does not mean testing should be minimal or that engineering teams should avoid comprehensive programmes. Safety-critical, fatigue, certification and reliability questions can demand extensive evidence, sometimes accumulated over thousands or millions of load cycles. The point is simply that testing should be purposeful. A well-designed programme understands why evidence is being collected and how that evidence will be used before the hardware is switched on.
How Much Reality Does the Test Actually Need?
One of the more interesting challenges in test planning is determining how faithfully the eventual operating environment needs to be reproduced. Instinctively, it can seem that the more realistic the test becomes, the more valuable the result must be. In practice, absolute realism is rarely achievable and is not always necessary. The engineering skill lies in understanding which aspects of reality are important to the question being investigated and reproducing those conditions sufficiently well to generate meaningful evidence.
Blackfish encountered exactly this type of challenge when developing a test system for evaluating a 500 kg hydrogen fuel cell for aerospace use. The relevant requirement was not to reproduce an entire aircraft or every aspect of flight. The system needed to enable the fuel cell to be safely pitched and rolled through angles of up to 50 degrees, recreating the dynamic conditions necessary to investigate its suitability for the application. Blackfish therefore completed the concept and detailed design, analysis, manufacture, assembly and testing of a system capable of recreating the conditions relevant to that particular engineering problem.
The distinction between reality and representative reality matters because every physical test is, in its own way, another model. Decisions are made about the loads to apply, the boundary conditions to create, the measurements to record, the environmental conditions to reproduce and the behaviours that can reasonably be excluded. Time can even become part of that representation, with accelerated fatigue or lifecycle testing compressing years of expected operation into a shorter test campaign.
The question is therefore rarely whether a test recreates reality perfectly. It is whether it recreates enough of the relevant reality to answer the question being asked, while remaining controlled enough for the result to be understood.
A Good Test Does Not Have to Answer Everything
A complex development programme may carry dozens or hundreds of technical uncertainties. Trying to resolve all of them through a single physical test can appear efficient at first, but it can easily produce the opposite result. As additional components, systems, environments and behaviours are introduced, the number of potential explanations for an unexpected result increases as well. The most representative test is not necessarily the easiest test to learn from.
A recent Blackfish project concerning the development of a standardised offshore-wind gangway interface provided a useful illustration during an internal engineering discussion. Operational simulations had been carried out to demonstrate how the proposed interface would behave during gangway engagement, normal operation and retraction. When the work was presented, a question was raised about whether those simulations had also been used to determine the maximum sea state in which the interface could operate. They had not, because establishing the ultimate operational limit was not what that particular activity had been designed to do.
That does not make the simulation incomplete. It makes its purpose defined. The important discipline is understanding what conclusion an exercise supports and resisting the temptation to stretch that conclusion beyond the evidence available.
Not every test needs to answer every question; it needs to answer its own question well.
In fact, part of good test planning is being equally clear about what a particular activity cannot tell you, because that boundary determines which questions remain for later stages of development.
The Biggest Test Is Not Necessarily the Best Next Test
Technology developers are ambitious by nature. That ambition is one of the things that allows an idea to travel from a sketch, calculation or research project towards physical hardware in the first place. Understandably, attention therefore tends to gravitate towards the major milestones: the first substantial prototype, a prestigious test facility, a full-scale demonstration or the first opportunity to put the technology into its eventual operating environment.
There is considerable value in reaching those milestones, but reaching the biggest possible test is not the same as answering the next important engineering question. Recent conversations with organisations operating technology-testing facilities have returned repeatedly to the value of intermediate stages, particularly where a development team can introduce real-world complexity progressively rather than simultaneously. Controlled testing can isolate a particular behaviour; component testing can address a risk before integration; smaller-scale trials can expose problems while significant design freedom remains; and increasingly representative environments can then introduce operational factors once some of the underlying uncertainties have already been reduced.
The principle is especially visible in offshore technology because the consequences of moving from a controlled environment into the sea can be dramatic, but there is nothing inherently marine about the logic. The founder of a clean-tech hardware company, an aerospace engineer, a university research team and a developer of an autonomous system ultimately face versions of the same question: how much additional cost, complexity and risk needs to be introduced in order to learn the next important thing?
Testing several unresolved assumptions simultaneously may produce a realistic experiment, but it can also make the result much harder to diagnose if something does not behave as expected. Building confidence progressively gives a team a much better chance of knowing which assumption has failed, which model needs revisiting and where engineering effort should be directed next.
Sometimes the Most Valuable Test Is the One That Finds a Problem
Nobody particularly enjoys discovering that something does not work. An unexpected deformation, failed component, disappointing performance result or disagreement between measurement and simulation can feel like a setback, especially within programmes operating against funding milestones, investor expectations and tight development schedules. Yet the value of that discovery depends enormously on when it happens.
As we explored in Before You De-Risk the Hardware, De-Risk the Decisions, the cost of learning changes substantially as technology matures. A problem identified during analysis might require another calculation or a change to a model. The same issue uncovered during component testing could require redesign and another manufactured part. Once it is embedded within a full prototype, the consequences increase. Discovering it during a major demonstration can affect an entire programme, while finding the same issue after commercial deployment can be more serious again.
Viewed in that context, the definition of a successful test begins to change.
Confirmation is valuable, but so is contradiction. A test that disproves an important assumption while the development team still has the technical freedom and financial headroom to respond may be doing an extremely valuable job. The objective is therefore not to engineer uncomfortable results out of the programme. It is to create opportunities for them to emerge while the consequences remain manageable.
This also affects the culture around testing. If every unexpected result is interpreted as failure, teams can become unconsciously incentivised to design demonstrations around showing what they already know will work. If testing is regarded instead as part of the learning process, an unexpected result becomes information. That does not make failure desirable, but it makes useful failure very different from wasted effort.
Evidence Should Build, Not Simply Accumulate
Technology development rarely depends on one decisive test that suddenly transforms uncertainty into certainty. Confidence normally develops incrementally. Analysis informs early testing, test results refine models, component evidence supports subsystem development, subsystem behaviour informs integration, and increasingly representative environments introduce variables that could not reasonably have been recreated earlier.
The strongest programmes therefore build evidence rather than simply accumulate it. Each activity should answer something that matters and create a better foundation for the next stage. The evidence does not all need to say the same thing, either. Differences between prediction and measurement can themselves be highly informative if the team understands why those differences occurred and allows them to influence subsequent engineering.
Blackfish's work supporting the SenseWind 2 MW demonstrator provides one example of this approach. Rather than relying on a single demonstration, the programme involved more than 40 full-load tests under up to 150 tonnes of axial loading, investigating load sharing, proof loading, failure cases and contamination. The tests were also witnessed by a certification organisation, meaning that the evidence served a wider purpose than demonstrating to the engineering team that the mechanism could operate.
This is where testing begins to intersect with confidence beyond the immediate development team. Investors, customers, certification bodies, regulators and commercial partners may eventually rely on engineering evidence for very different reasons. A good development programme therefore needs to understand not only what the engineers need to know, but what evidence other decision-makers may reasonably require before they can support the next stage.
Know What the Test Cannot Tell You
Successful test results naturally increase confidence, but that confidence has to remain within the boundaries of the evidence. A component surviving a proof load does not automatically establish its fatigue life. A prototype operating successfully for several hours does not demonstrate decades of reliability. A subsystem working independently does not guarantee straightforward integration, while good laboratory performance does not necessarily reproduce every aspect of the eventual operating environment.
This is one of the reasons that documenting limitations is as important as recording successes. After a test programme has been completed, an engineering team should be able to explain what was tested, under which conditions, why those conditions were chosen, what the result supports and which questions remain unresolved. A good result should therefore narrow uncertainty rather than create the impression that uncertainty has disappeared entirely.
The same discipline applies before testing begins. A team that can explain what it does not expect a test to prove has usually thought more carefully about what the test is actually for. It also makes the transition into the next activity clearer because unresolved questions remain visible instead of being buried beneath the confidence created by an apparently successful demonstration.
What Should Be Clear Before Testing Begins
Before committing to a significant test programme, a development team should be able to describe the uncertainty it is trying to reduce, why that uncertainty matters and which future decision depends upon it. It should have considered the evidence required to create sufficient confidence, rather than pursuing more evidence simply because more appears safer. It should understand why the proposed method is an appropriate way to generate that evidence, what assumptions are built into the test itself and which conclusions the results will not support.
Equally importantly, the team should have thought about what happens if reality does not cooperate. If an unexpected result would simply be dismissed, repeated until the preferred result appears or explained away without influencing the design, the programme is unlikely to gain the full value of testing. A useful test programme allows the evidence to change something. It creates a route through which new information can influence models, assumptions, requirements, hardware or the development plan itself.
These questions are not particularly complicated, but answering them honestly can fundamentally change the shape of a development programme. They shift the emphasis from completing a test to creating understanding, which becomes increasingly important as hardware gets larger, programmes become more expensive and the consequences of discovering an incorrect assumption begin to grow.
Testing Is Part of the Thinking
Testing is sometimes described as the point where engineering theory finally encounters reality, and there is some truth in that. Models contain assumptions, simulations contain simplifications and physical hardware has an inconvenient habit of revealing things that were not obvious when the design existed primarily on a screen.
However, treating testing simply as the point at which a finished design is checked misses much of its value. Good testing influences engineering long before the test begins because it forces teams to articulate what they believe, identify what they do not know, understand which uncertainties genuinely matter and decide what evidence would be sufficient to move forward. It also forces uncomfortable but useful discussion about what happens if reality behaves differently from the model.
Seen in that way, testing is not merely a technical activity at the end of a design process. It is part of engineering decision-making throughout the development journey. The best programme is not necessarily the one with the largest rig, the most sophisticated facility, the longest campaign or the greatest volume of data. It is the one that creates the evidence required to answer the important questions at a point when those answers can still influence the technology.
That is why, before choosing the facility, designing the test hardware or planning the next major demonstration, it is worth returning to the apparently simple question at the beginning of this article and taking it rather more seriously than its simplicity might suggest: what exactly do we need to learn, and what will we do differently once we know the answer?
Reality will always get a vote. Good engineering simply tries to hear what it has to say before the most expensive decisions have already been made.
If you're developing a new technology and working out what needs to be proven before the next stage, Blackfish can help structure the engineering questions, identify the evidence required and determine the most appropriate route to building confidence.
Comments