The challenges of experimentation
Richard Feynman, the physicist, wrote that the “principle of science, the definition, almost, is the following: The test of all knowledge is an experiment. Experiment is the sole judge of scientific “truth”.”
Feynman was able to explain the most complex of issues. His energy and wisdom have benefited so many of us, and he pointed out how easy it is to get mislaid and fool oneself. Understanding the challenges of experimentation and overcoming biases when interpreting results is particularly important in our efforts not to fool ourselves and others.
Invert, always invert
The approach in this essay is that of Charles Munger, who urged us to “invert.” Munger explained this when he gave the Harvard School Commencement Speech in 1986. Munger referred to Jonny Carson and said:
“What Carson said was that he couldn’t tell the graduating class how to be happy, but he could tell them from personal experience how to guarantee misery.
What Carson did was to approach the study of how to create X by turning the question backward, that is by studying how to create non-X. The great algebraist, Jacobi, had exactly the same approach as Carson and known for his constant repetition of one phrase: ‘Invert, always invert.’
This essay uses the Munger-Jacobi method by seeing how not to conduct experiments and how not to interpret experimentation. There were many examples to chose from, ranging from astrology, ideologies, and theologies. I finally selected the fiasco of Pons and Fleischmann’s cold fusion experiments since it illustrates best ‘how to conduct scientific experiments by turning the question backward and by studying how not to experiment.’
The Scientific Fiasco of the Century
In 1989, two chemists, Stanley Pons and Martin Fleischmann announced that they had successfully created a sustained nuclear fusion reaction at room temperature in their laboratory.
Scientists around the world immediately tried to replicate the experiment. However after many attempts and much confusion it ultimately turned out that no one could get the same results. This however occurred only after millions of dollars were spent and a number of scientific careers were ruined.
Ignore previous work – leading to calculation mistakes
Pons and Fleischmann (hereafter “P&F”) lacked any deep knowledge of what had previously been discovered. This prevented them from realizing that they were perhaps on the wrong path and prevented them from catching serious mistakes in calculations. These calculation mistakes would have easily been seen if they had learned about previous work or had their reasoning checked by those with the relevant experience.
No full disclosures and no public scrutiny
P&F’s announcement was made at a press conference at the University of Utah. They provided only sketchy information on their actual experimentation. The information was akin to a political announcement and lacked any details that would have let others check the results.
When P&F did submit an article to the science magazine Nature, there were still too many questions answered. Rather than reply to the magazine’s queries, P&F withdrew its submission to Nature.
Philip Morrison, a great and wise physicist, commenting about scientific investigations, wrote:
“What science is deeply concerned about is something closer to self-deception. It is genuinely hard to make out how the natural world works; any investigator, even a long string of them, can go wrong. That is why it is the evidence, the experience itself, and the argument that gives it order that we need to share with one another, and not just the unsupported final claim.”
Only justify, with no attempt to falsify
Feynman pointed out that to a priory, “decide upon the answer is not scientific. It’s not right to pick only what you like, but to take [into account] all of the evidence.” P&F made every effort to justify cold fusion, and no efforts were made to understand the contradictory aspects of their experimental results.
Rather than try hard to find out if a mistake has been made or some error in measurement has occurred, the focus of P&F was on justification, using any available means in support of their claims. Huizenga wrote: “During the short lifetime of cold fusion, its advocates have required belief in miracle after miracle for one to swallow their discovery, suspending each time some existing knowledge in nuclear physics.”
John R. Huizenga was a member of the Panel set up by the U.S. Department of Energy to investigate cold fusion claims. The Panel tried to see cold fusion first hand. Huizenga wrote:
“From the point of view of seeing a working fusion cell, our Panel’s visits to the different sites were very disappointing. In none of our meetings with the various groups claiming positive results, did we see an operating cell that was producing excess heat at the time of our visit! Even when experiments were run under identical conditions, the positive results were not reproducible.”
The pressure of crowds and other errors
P&F made many other errors, amongst which they:
- Selectively used data – this meant that data against their claims were not used and others could not replicate their experiment
- Ignored details of the experimental apparatus – giving results to greater accuracy than their instruments
- Made no statistical error analysis of the data – thus failing to show that their observations were not statistically significant and did not exceed the background level.
- Invented a story – that fit in with their selective use of data. Even Julian Schwinger, a famous physicist, invented a phantasmagorical “castles in the air” story without any regard for thoughtful fact-finding.
There are many more details on how we should not conduct experiments and interpret data so that we can maximize our chances of errors. Perhaps the most dangerous behavior occurred when the adherents of cold fusion started becoming aggressive against those asking questions. At cold fusion conferences, the skeptics were either not given speaking time or were shouted down. Either one accepted the authority of the cold fusion priests or one was not welcome to the gathering.
The cold fusion fiasco clearly showed these human tendencies to believe in an unquestioning manner, to resort to an argument based on some kind of “authority,” to shout down dissent, and to argue without reasoning. Spending time understanding these pernicious propensities, as much in ourselves as in others, is perhaps the key lesson from Pons and Fleischmann’s failed experiments in cold fusion.
“The world,” said Richard Feynman, “looks so different after learning science.” Learning how to avoid errors in experimentation is a key aspect of this journey. Without such learnings we humans will continue going down our present road, believing in myriad cold fusion type beliefs and deluding ourselves about a fantastic future - until disasters strike. Will we wake up in time?
© Kaikhushru Taraporevala