Discussion » Advocacy - a call to stop using the Water Boiling Test
Some of you might have heard, but a few other stove practitioners and I have decided to start an advocacy initiative, and a call to the main stakeholders of the stove sector, to stop using the Water Boiling Test when testing stoves.
There is a growing number of stakeholders in the cookstove sector who are increasingly concerned about issues related to stove testing.
We have now enough proof, from many studies, that the most widely used testing protocol, the Water Boiling Test 4.2.3 (WBT) is utterly unreliable in predicting the performance of a stove.
There are reasons to believe that using the WBT have had and can have large negative impacts on stove programmes.
This is the list of signees of the call:
Please find the last version of the text of the call below, and attached as well. Also attached is the list of studies and examples of alternative protocols.
If you have comments, questions, remarks, please share them, and let's discuss them on this thread.
If you believe this is important, please share these documents around, and join us! And contribute to a much needed change in stove testing and research.
"Dear Sir, dear Madam,
Locally obtained solid fuels play a major role in meeting the cooking and heating needs of about 3 billion people worldwide. For at least 40 years, stove designers have been innovating combustion devices to improve fuel efficiencies, improve general performance and reduce pollutant emission. Many improved stoves programmes have been funded and implemented over the years throughout Africa, Asia and Latin America, but they have not scaled rapidly. Many have failed and proven unsustainable.
A principal culprit is the testing methods used to rate such stoves.
The Water Boiling Test (WBT) protocol is the most widely used. Multiple versions have been used by the cooking stove community of implementers, manufacturers, and testers. Testing protocols are important tools: their role is to report the performance of a stove in terms of fuel savings and reduction of pollutants. Multiple versions of the WBT have been disseminated since its creation in 1982, the most commonly used being the WBT, the WBT 2.0, WBT 3.0, WBT 3.1, WBT 4.1.2. and WBT 4.2.3.
A number of papers have been published in recent years about serious issues with the WBT. And there is now sufficient proof showing that the WBT, including its latest version 4.2.3, is unable to correctly assess the performance of an improved cookstove in a reliable manner.
Further, many stakeholders have complained about the large variations in stove performance between what was reported in the lab, and in the field. Testers have described the test results as “irreproducible”. It is now well established in the literature that the WBT should never be used to assess how a stove performs in any context.
In February 2017, a study by Lombardi and al. summarized the issues:
“Some of WBT critical issues remain unsolved. In particular, the main weakness of the WBT concerns its real-life relevance. […] Criticism about WBT concerns also the repeatability of the protocol, with a number of researchers claiming that it would need to be reviewed in terms of accuracy. […] As a matter of fact, uncertainties related to temperature reading and vaporisation in the boiling region lead to high variability between test replicates.”
A lot of debate has been made around formulation of metrics, primarily on thermal efficiency, which is often interpreted as the most immediate and distinctive stove performance parameter. Studies from Bailis et al. highlighted how relying on WBT thermal efficiency outputs, regardless of the relative importance of high and low power cooking tasks among the target population, can lead to misleading interpretations. Furthermore, Zhang et al. and Jetter et al. questioned the scientific meaningfulness of thermal efficiency at simmering.
Finally, some unsolved issues concerning statistical significance of data are worth mentioning. WBT 4.2.3 includes “Statistic Lessons for Performance Testing”. The appendix specifies that the minimum number of test replicates for each model of stove should be three, […] Wang et al. investigated this topic using a simplified version of the WBT 3.0 and demonstrated that more than 5 replicates are likely to be required to avoid impractically large 95% confidence intervals and that even more replicates may be required to demonstrate a statistically significant difference in performance between two or more stoves.”
Despite the several updates of the WBT protocol over the years, none of the core issues could be resolved.
The recent discussions on the cooking stove discussion list Bioenergylists.org, specialized in state-of-the-art stove science, and at the ISO Technical Committee 285 showed that there are no solutions to “fix” the WBT protocol and in any case there exist other valid, reviewed test methods. Good science and reliable testing is at the core of the development of every stove product and every stove dissemination project. It is difficult to measure the damage through the use of this faulty testing protocol. What do we really know about the lab or field performance of stoves rated using it? How many test results are useless? How many stove projects or companies failed because of a performance claim that was unsupportable in the first place? How much effort, how much money was lost, how many disappointed customers?
Before being able to do meaningful and impactful stove projects, we need to get the testing right.
Alternative testing methods to the WBT exist already. They allow accurate measurement of the stove performance, both in the laboratory and in the field, they allow for repeatability. They have been reviewed, while the WBT never was. These alternative testing methods are listed below in this document and links are provided.
We are a group of concerned implementers and researchers. We have launched a call to stop using the Water Boiling Test 4.2.3. To stop using it to certify, select, as well as develop stoves.
Your organization is one of the stakeholders of the improved cookstove sector.
Today, we need your help to make sure cookstoves are developed and selected according to reliable testing methods.
We are asking your organization to support this effort and stop using the WBT in its projects and programmes, as well as, starting from now, not to fund new projects, programmes, and companies which plan to use the WBT to develop, select, or disseminate stoves.
We are asking your organization not to make decisions based on the WBT.
We are asking your organization to support the use and development of scientifically valid alternative protocols.
Your role is critical in promoting good science, good stoves and ultimately making a positive impact in the field."
And attached here is the list of studies and examples of alternative protocols.