The Expensive Test Should Earn Its Place

Why the sequence of learning matters in technology development
There is a natural tendency in technology development to look towards the next big milestone. The larger prototype, specialist test facility, offshore trial or operational demonstration can represent years of work and provide a tangible sign that a technology is moving forward. Eventually, some questions genuinely can only be answered by putting increasingly complete hardware into increasingly realistic conditions.
The difficulty is that these are also usually the points at which learning becomes more expensive. Hardware is larger, more organisations are involved, facilities need to be booked and equipment may need to be transported. Procedures, permissions and safety arrangements become more substantial. In offshore applications, vessels, ports, weather windows and installation teams can enter the picture. Elsewhere, specialist laboratories, certification bodies or manufacturing partners may be required.
The question is therefore not simply whether larger or more representative testing provides valuable information. Of course it can. The more interesting question is:
Which uncertainties genuinely deserve to travel that far through the development programme before they are resolved?
A comment from Jay Sheppard of Marine Energy Wales on our previous article, Before You Test the Technology, Decide What You Need to Learn, put this particularly neatly: “Answer the cheap questions cheaply.”
That idea captures something easily lost when attention is focused on the next visible milestone. If a simple experiment can answer a question credibly, there is little advantage in waiting for a far more expensive test to discover the same thing. If analysis can eliminate an unsuitable design before anything is manufactured, there may be no reason to carry that uncertainty into the prototype. If a small-scale test can expose a fundamental assumption, discovering the same issue during a full-system demonstration may simply make the lesson more expensive.
The objective is not to avoid the expensive test. It is to make sure that, when you reach it, you are using it to learn something that genuinely needed an expensive test to discover.

Learning Has a Cost
Technology development is often represented as a progression through increasingly mature stages, but another way to think about it is as a progression through increasingly expensive forms of uncertainty. At the beginning, changing an idea may require little more than altering a requirement, calculation or CAD model. Once a component has been manufactured, the same change has a physical cost. Once that component forms part of an integrated system, the consequences spread further. Once complete hardware is being transported, installed, demonstrated or certified, even a relatively small change can affect suppliers, programme, equipment and schedule simultaneously.
That does not mean every important technical question can or should be answered early. Real operating environments introduce interactions that no calculation or laboratory experiment can reproduce perfectly. Integrated systems sometimes behave differently from their individual components, and operational realities have an inconvenient habit of exposing assumptions that appeared entirely reasonable during development.
The discipline lies in distinguishing between the uncertainty that genuinely requires the next level of realism and the uncertainty we have simply postponed.
A good development programme therefore does not necessarily progress from cheap tests to expensive tests in a perfectly linear sequence. Instead, each activity should introduce enough complexity to answer the next important question. Analysis may remove one uncertainty, inspection another. A simple physical mock-up may expose something they cannot reveal conveniently. Component tests can investigate controlled behaviour, subsystem tests introduce interfaces and integrated testing brings those systems together. Representative environments then introduce aspects of reality that could not sensibly be recreated elsewhere.
Eventually, operational demonstration answers the questions that genuinely belong in the real world.

Sometimes Plywood Is Enough
Sophisticated technology does not always require sophisticated equipment to answer its first question.
During the development of a novel air-valve system, Blackfish encountered an awkward sealing problem. A hinged flap needed to seal effectively around its perimeter, but the geometry meant that seal compression close to the hinge differed considerably from compression farther away. The seal also contributed to damping as the flap repeatedly opened and closed, while the available published information did not provide enough confidence to select a solution from datasheets alone.
There were many things that would eventually need to be understood, including pressure performance, behaviour under airflow and long-term durability. But before committing to that work, the engineering team needed something much more fundamental: to see how different seal profiles actually behaved within the geometry being considered.
The first tool for exploring that question was not an elaborate test facility. It was a simple plywood representation of the seal groove and flap.
Different commercially available profiles and materials could be handled, compressed and compared quickly. That allowed unsuitable options to be removed before more formal testing was justified. As the questions became more demanding, so did the evidence. The programme progressed into dedicated small-scale rigs, static pressure testing, wind-tunnel testing and repeated cycling to assess durability.
The plywood model was not an inadequate attempt to reproduce the final valve. It was an appropriately simple way of answering an early engineering question. Had the team tried to make that first experiment representative of the complete system, considerably more time and money could have been spent investigating seal concepts that were capable of being rejected much more easily.
That distinction matters well beyond this particular example. The credibility of a test should not be judged by how impressive the equipment looks. It should be judged by whether the evidence is credible for the decision being made.

Sometimes the Cheapest Useful Evidence Is Not a Test
Sequencing learning properly also means recognising that not every uncertainty deserves physical hardware.
When Blackfish worked with Solead on a Verification and Validation strategy for its energy-storage technology, risks and requirements were considered against several different ways of generating evidence: similarity, design, analysis, inspection and physical testing. The purpose was not to minimise testing. It was to understand which requirements genuinely needed physical evidence and which could be addressed credibly through another route.
That matters because every prototype and test consumes engineering time and development budget that cannot then be spent elsewhere. If analysis can remove a poor design option before hardware exists, or if several development questions can be answered through one well-planned validation activity, the efficiency of the programme changes considerably.
But the opposite mistake matters too. Analysis should not become a convenient substitute for physical evidence where the underlying uncertainty cannot credibly be resolved through modelling.
The interesting question is therefore not whether analysis or testing is better. It is which form of evidence is proportionate to the decision at that particular stage.
This is also where modelling becomes an important part of the testing conversation. A well-chosen model can allow engineers to explore far more possibilities than could reasonably be manufactured and tested, helping physical testing concentrate on the areas where assumptions, uncertainty or consequence justify it.
Good modelling and good testing are not competing approaches. Each should make the other more valuable.
Cheap Does Not Mean Careless
There is an understandable danger in talking about answering questions cheaply. A lower-cost experiment can sound like a request for approximate engineering, when the opposite should be true.
A simple test still needs a defined purpose. The conditions need to be understood, measurements need to be meaningful and the limitations of the result need to be recognised. A crude-looking mock-up used deliberately to investigate one behaviour can produce extremely useful evidence. A sophisticated test undertaken without understanding its boundary conditions can produce impressive data without creating much understanding.
Blackfish's work on a sand-based testing system illustrates the distinction. The existing arrangement had difficulty recreating the sand bed consistently between tests. Research and early trial work demonstrated that the way the sand was reset and deposited influenced the results, leading to the introduction of a simple pluviator into the setup. That change improved the quality and repeatability of the resulting test data.
Once that principle had been demonstrated, the next question was different. A much larger 1:10 system test required the same underlying control of sand distribution at industrial scale. The eventual pluviator therefore needed capacity for up to two tonnes of sand, controllable mass flow, remote opening and closing, and the ability to be lifted and repositioned.
What is interesting is the journey between the two. Development moved through simple timber and cardboard prototypes, sketches, rapid-prototyped components, detailed design, fabrication, commissioning and eventual operation.
The early hardware did not need to resemble the eventual industrial solution. It needed to generate sufficient understanding to justify developing one.
Early prototypes helped establish the principle before that learning was translated into robust industrial-scale equipment.
More Reality Should Have a Reason
Eventually, controlled testing reaches its limits. A component that behaves predictably on its own still has to work as part of a larger system, where other components, software, manufacturing variation and the operating environment can all affect what happens. At that point, testing in more realistic conditions becomes valuable because the question has changed.
The mistake is assuming that greater realism is automatically better. Every additional system or environmental condition introduces another variable. If mechanical hardware, controls, electrical systems, software and unfamiliar operating procedures are all brought together for the first time, an unexpected result can be difficult to untangle. You may know that something went wrong without immediately knowing why.
This is why there can be real value in testing technologies through progressively more realistic stages. Offshore renewable energy makes the principle particularly easy to see because the jump from a workshop or laboratory into the sea can introduce enormous additional cost and complexity. The Marine Energy Wales 'State of the Sector' report, for example, recorded more than 55,000 hours of testing across 15 programmes at the Marine Energy Test Area (META), where developers can trial technologies in real marine conditions before taking on the demands of a larger offshore deployment.
Larger facilities can then answer questions that controlled testing cannot. At the European Marine Energy Centre (EMEC) in Orkney, for example, testing is not simply about whether equipment functions. It can help generate performance data, environmental evidence and the wider confidence needed as technologies move towards commercial deployment and investment.
The point is not that every technology needs to follow exactly the same sequence or use the same kind of facility. A battery system, aerospace mechanism, university prototype or offshore device will need different evidence at different stages.
What matters is that every increase in realism, integration and cost has a reason behind it.
Arrive With the Right Unknowns
No responsible engineering team expects to arrive at a major demonstration with no uncertainty remaining. If everything were already known, there would be little reason to conduct the demonstration.
The more realistic aim is to arrive with the right unknowns.
Before committing to a large-scale test, the basic behaviour of important components should ideally be understood. Principal analytical assumptions should have been challenged. Instrumentation should be designed around the evidence that needs to be generated. Failure modes that can reasonably be explored in safer or cheaper conditions should already have been investigated.
That leaves the expensive environment to do the job only it can do.
The distinction becomes particularly important when several developing technologies are being integrated. If too many unresolved questions are carried forward simultaneously, a full-system test can become difficult to interpret. By resolving what can be separated beforehand, the eventual result becomes more diagnostic. Rather than simply asking whether the demonstration worked, the team can understand which assumptions held, where system interactions differed from prediction and what decision should follow.
That is a far richer outcome than a simple pass or fail.
Failure Is Priced Differently at Different Stages
The sequence of learning also changes how failure should be viewed.
A plywood mock-up demonstrating that a seal profile does not behave as hoped is not an expensive development failure. It is a cheap answer.
A small prototype exposing inconsistent test conditions may prevent months of unreliable experimentation.
An analytical model identifying unacceptable stresses before drawings are released to manufacture is doing exactly what the analysis was intended to do.
The same discovery made after manufacture, transport, integration or deployment remains useful engineering information, but considerably more has already been committed before receiving it.
This is why sequencing is about more than reducing the testing budget. It preserves freedom to respond.
Early in development there are usually multiple possible solutions and relatively little sunk cost. As the programme progresses, supplier commitments, manufactured components, schedules, certification strategies and stakeholder expectations narrow the range of practical responses.
Resolving an important uncertainty while several options remain open can therefore be considerably more valuable than resolving exactly the same issue when only one expensive option remains available.
The Expensive Test Still Matters
None of this is an argument for staying safely inside analysis, simulation and laboratory testing.
Eventually, technology has to meet reality.
Models need validation. Components need to function together. Complete systems need representative loading. Installation methods need to be attempted. Maintenance assumptions eventually encounter actual access constraints. Technologies intended to operate for years need more than a successful afternoon in controlled conditions before customers, investors, regulators or operators can reasonably place confidence in them.
There is a danger at both extremes. Progressing to full-scale demonstration too quickly can make learning unnecessarily expensive, but remaining in controlled environments for too long can create confidence in conditions that no longer represent the real problem.
The engineering judgement lies in recognising when the next level of realism has become necessary rather than simply attractive.
The answer will differ enormously between an energy-storage system, aerospace mechanism, university research prototype, industrial machine and offshore device.
But the underlying question is similar:
What can we learn at this stage that we could not learn credibly enough at the previous one?
If there is a clear answer, the next test may well have earned its place.

Development Budgets Buy Information
Perhaps this is the more useful way to think about the whole process.
Technology-development budgets do not simply buy engineering hours, analysis, manufactured parts, prototypes and test campaigns. They buy information.
Each activity consumes some of the available budget in exchange for a better understanding of the technology and greater confidence in the decisions that follow. Sometimes spending more to answer an important question properly is exceptionally good value. An expensive test can prevent an even more expensive failure, unlock certification or investment, or establish evidence that could not be produced any other way.
The goal isn't therefore to make every stage cheap.
It is to avoid paying premium prices for information that was available much more economically earlier.
That requires thought about the sequence in which uncertainty is reduced. Some questions belong in calculations and simulations; others on the bench or in controlled component testing. Some require integrated hardware, specialist facilities or representative environments. Eventually, some can only be answered by putting the technology into the world and seeing what actually happens.
Good development planning means arriving at each of those stages carrying the questions that belong there, rather than every question the programme happened not to answer previously.
Last week's article started with a deceptively simple question: what exactly do we need to learn?
The natural question that follows is just as important:
What is the simplest credible way of learning it, and when does the next, more expensive step genuinely earn its place?
The expensive test should teach you something.
Just make sure it is teaching you something you could not reasonably have learned for less.
If you're deciding what to analyse, prototype or test next, Blackfish can help identify the questions that matter and choose a proportionate route to answering them, before more time and cost are committed.









Comments