Science's reproducibility problem: what can we learn?

A bit of op-ed.

As every schoolchild learns, one of the keys to the scientific method, and one of the reasons it allows us to know things rather than merely believe them is that the description of a method allows another scientist to go out and reproduce the results, either by using the same method or by using a slightly different one: the latter method provides consilience, whereby different approaches confirm results. For example, when Michael Mann published his hockey stick warming research (1) other researchers were able to confirm the result using slightly different methods but reaching the same conclusions, thus improving on Mann's work (2). In this case, the system worked and worked well.

It's becoming increasingly widely known that not all research can be properly reproduced. In pharmacology only one “positive” result is required to gain approval from the US Food and Drug Administration, and the regulatory bodies of many other countries simply follow the FDA's lead. There is good evidence that there are many trials (which we know of because of the trials' participants, if nothing else) locked in file drawers. The big biotech companies specifically ban the use of their seed for research purposes, making attempts to replicate impossible: anyone publishing would find herself in court.

This is a problem that affects some fields more than others. This recently led Nature, one of the world's top scientific journals, to ask over 1500 scientists about the problem of reproducibility. You can find an article on the survey here: http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

“More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments.”

Does this mean that all, even most, science is unreliable?

Not necessarily. For one thing, the participants in this study were self-selecting. The researchers who took part are more likely to be those who, for one reason or another, are more concerned about the issue. There may well be many thousands more who don't consider it a big issue, or who don't consider it a big issue in their field. Of dozens of studies conducted in a professional scientist's career, at least one of those conducted by the minority who took part in this study could not be reproduced. That doesn't stop the majority of research being sound.

It's also worth bearing in mind that Nature is a commercial concern: it has to make a profit, and headlines such as this draw in readers, and in this they have just been successful!

Nevertheless, the majority of researchers who took part in this study think there is some sort of reproducibility crisis. On the other hand, they are probably biased. As studies go, this is not a good one. I suspect there is a problem and lessons to be learned. I don't think we need to end up having some sort of hysterical reaction to the problem.

I think the first lesson it's worth being aware of is that the fact that something worked for one person doesn't mean it will work for someone else. Not only that, but we need to be aware that's okay. When we are dealing with a squishy science like agronomy there are so many uncontrollable variables that replication isn't always going to work. Biological systems are incredibly complicated. Even if half a dozen people try to replicate results and half of those fail to replicate that does not mean something interesting didn't happen the other four times: they key question then becomes why. It doesn't mean accusing someone of lying (fraud is often a possibility, and does happen, but it's not usually the most likely possibility).

For example, let's say five people undertake an otherwise identical protocol involving a compost tea on five different plots of land.
1) Predominantly sand: This may already have a simple microbiota, but the minerals required for plant growth simply aren't there to begin with. In this case the compost tea condition might show a slight but not statistically significant improvement over the control plot. (What might be needed is a load of organic matter.)
2) Loam with a history of organic cultivation: in this case the microbes are already there. The application of more does nothing by comparison with the control plot.
3) A patch of lawn treated until recently with the usual agrochemicals: no improvement over the control is seen because the remaining chemicals wipe out the bacteria.
4) A similar patch of lawn treated until a couple of years ago with the usual agrochemicals. Still no improvement is seen against the control condition, because bacteria and other microorganisms have recolonised the soil naturally.
5) Yet another similar patch of lawn just after the agrochemicals have broken down into inert substances. “Miraculous” results are seen: miraculous results that would have happened anyway by allowing the birds to forage on it.
That doesn't mean the fifth trial was an aberration, or even that the person reporting the first set of results was “lying”, but that she had unique conditions that facilitated a “positive” result – but it's no more than a useful case study, not to be overgeneralised. I'm not saying this is what would happen: merely that it seems plausible at 2am when I wrote this.

Soil testing and checks with a microscope would explain the lot.

Equally, if there are persistent failures to replicate, it seems to remain important to figure out why.

The thing is, we are going to be out at the cutting edge of polyculture agronomy, at least in temperate zones. Not all results are going to be robust. Not all explanations are necessarily likely to be good ones, although understanding relevant aspects of plant biology and of ecology is going to be important. If we don't understand the basics of what is known about these systems we are more likely to make mistakes.

One lesson I think it's important to draw is that people need to be willing to come forward and say that they tried something and then something happened – and be willing to have others check their work and query their methods and conclusions, but not to make overblown claims until they've been checked. That's how we avoid pseudoscience. Checking under multiple conditions is what allows us to develop good practice.

Part of the problem is that humans are masters of self deception. Unfortunately, instincts that worked well in Olduvai Gorge may be less useful when trying to figure out how to feed ten billion humans while addressing the threat of climate disruption and avoiding the worst extinction event since the Chixculub Impact. “It worked last time” may well be of survival benefit when working out that the proto-impala are likely to be near the watering hole, but may be less useful when trying to establish a forest garden. That's not a criticism of anyone in particular: we all make these mistakes, except me me included. See also: http://www.nature.com/news/how-scientists-fool-themselves-and-how-they-can-stop-1.18517

We all have a tendency to jump to conclusions. I do it too. I know there is a rule that you are not allowed to imply someone is less than perfect: I'm going to stick my neck out and say that we are all less than perfect. I'm less than perfect.

Even when one of us tries to conduct unbiased research, we're likely to have a stake in the outcome, and this applies as much to us as it does to a postgrad in a lab, if for different reasons. We all want to prove that permaculture works, by some standard of “works”. I want to prove that a multi-strata agroforest is the best (by several measures) way to produce a balanced animal-free diet. A rancher will want to prove that it's possible to produce meat while sequestering carbon. One solution to this is to declare an experimental design and invite those who might disagree to comment on that design. I might plan to measure net primary productivity, biodiversity change, total yield and protein yield (see me look for something that will prove what I want it to prove!).

If I don't have a decent control plot and report all the results and not just the ones that suit me I'm doing nobody any favours.

This article http://www.nature.com/news/how-scientists-fool-themselves-and-how-they-can-stop-1.18517 discusses problems ranging from the failure to consider other explanations, p-hacking (where you look for numbers to confirm your pet hypothesis), to disconfirmation bias (checking unlikely results but giving expected ones a free pass), among others. All these hit the accelerator, leading to potentially interesting, but also potentially wrong, conclusions that lead us to waste time and energy on practices that are not as helpful as we want to think they are.

One thing that I think would benefit most of us (anyone without a lot of experience in agroecological research) is decent mentoring. The Permaculture Association has a list of researchers who might be interested in some degree of collaboration: https://www.permaculture.org.uk/research/list-permaculture-friendly-academics With luck this also means help with design. This probably also wildly improves your chances of publication.

In conclusion, I don't think science as a means of understanding the universe is necessarily in crisis. I think there is a problem that I think the scientific community will be able to overcome. I think there are simple but important lessons that we can learn from the process that will help permaculturalists improve our own practices. We're still going to make mistakes, go down unhelpful paths and so on. We're never going to achieve some pristine “truth”. We can learn to avoid some pitfalls.


1) doi:10.1029/1999GL900070
2) see for example  doi:10.1177/0959683608098952

I must admit to some bias myself. After reading many of Neil's posts, I tend to assume that he knows what he's talking about and that his detractors do not. This is based on many examples where I have agreed with Neil.

My favourite was the discovery of N rays https://en.wikipedia.org/wiki/N_ray a novel form of radiation . So novel it could only be seen in France
For me the answer is in the question -a good experiment only tests one variable at a time . So the choice of control is of utmost importance . This is why biological systems are most difficult to do science with .
The soil , weather and seeds for instance are not uniform

David

Just a quick thing on scientific reproducability. I saw an interesting quote from a scientist a while back, something to the extent of "there is no money or glory in being the 2nd person to discover something" thus summing up a big part of the problem for many of the sciences. Who is going to fund testing if something someone else discovered is reproducible if their scientist and their company wont profit or gain glory from it.

Now in the permaculture world, I would hazard that farmers don't tend to make ideal scientists. Ask the old farmer why he does something a certain way "well cuase it works, I did it like that 20 some years ago and it started growing like crazy, so I keep doing it that way."

Not saying adding more true scientific principal and testing reproducability isn't a good idea for permaculture to advocate and try to incorporate. Just guessing that for many, farming and gardening is already a lot of work. Adding in trying to be scientific about it and cut out variables and set up controls, might just turn some off to the idea of incorporating scientific process.

I agree entirely that there are issues with comparability, which is why I brought it up.

That said, "I did x and things grew like crazy" is also not necessarily reproducible, or even necessarily a good idea: if you drench your fields in artificial fertilisers and other agrochemicals things will grow like crazy - for a while, and that led us to damaged soils.

A more controlled approach might lead us to a more nuanced situation, where we might be able to say that if you have soil type A then it seems better to do X, whereas if you have soil type B it seems generally better to do Y.

Neil I agree, as mentioned I think it is good for permaculture to advocate and add some scientific method.

However remember your audience too. Not everyone will be able to spend the extra time, effort, and money to set up multiple patches to separate and test different things to find out what is really causing things to work.

That while advocating scientific methods, we also shouldn't discourage personal experiences being reported. It is through sharing personal experiences of what does and doesn't work that folks who can devote themselves to more scientific methods can find ideas to test and report results.

*edit to add. I guess I am just advocating a middle ground where there is both personal experience and scientific method as I feel both can be beneficial.

Devin Lavign wrote:Neil I agree, as mentioned I think it is good for permaculture to advocate and add some scientific method.

However remember your audience too. Not everyone will be able to spend the extra time, effort, and money to set up multiple patches to separate and test different things to find out what is really causing things to work.

That while advocating scientific methods, we also shouldn't discourage personal experiences being reported. It is through sharing personal experiences of what does and doesn't work that folks who can devote themselves to more scientific methods can find ideas to test and report results.

I said this in another thread. I agree. We need more people coming forward to say that they tried X and then Y happened, because it's useful to know that Y will always follow X and if so why.

I also think it's important not to overgeneralise before we confirm it.

Before the "invention" of science mankind only had remembered experiences to use for understanding the workings of the universe.
In those days the "Universe" was probably more like Immediate surroundings, then some brave soul headed out on the first walkabout, came home and told of what he had found over the mountain, across the desert, on the other side of the river or what not.
Science is a method of understanding as well as a method of discovery. It is not what almost everyone thinks it is (Mostly it is the people who call themselves scientist that have convoluted the true meaning so they can feel/ seem more important or more educated).

If you are a scientist (the real kind, not the kind portrayed in "The Big Bang Theory"), you are inquisitive to the Nth degree.
This makes you listen more than you speak, because, if you are not listening to others experiences, how are you going to get the next big question?

Many people seem to think that 'science" wants to act the know it all, and I must say that much of what is touted today as science actually is not.
To me, science is the way I come to understand and find answers to my questions, in a manner that others could perform the same investigation and come to the same conclusion (repeatability, which is a prime requirement of real science).

As an example;
If someone mentions that they added "bio char" to their soil last year and now all their plants are growing great guns, I want to find out what the mechanisms are that make this work so well.
My next step is to find out how they created this "bio char" and how much of it they used, in what size space they put it.
I need this base data so I can work out an experiment to find out why this worked, what all it actually does to the soil chemistry and biology and come to the why it works answer.
I will devise and execute as many experiments within these perimeters as necessary to come to a conclusion.
This conclusion might be yes this works or it could be I could not get the same results ever or perhaps it would be found that this one method of use works and all others fail, or it could be something in-between.
To me this is science. We search for answers to our questions, that is what founded the scientific method in the first place.

Today's science has two types of investigation.

The true scientific method, where a question comes up and experiments are devised to find out the true answer.
This answer will disregard any pre-conceptions of what the answer is or might be.

Then there are those who have to justify their grant money by coming up with the answer the funder wants to hear.
The latter are not using science, they are manipulating science to their own ends through false means.
In many cases these people will actually have their answer prior to devising their experiments.
This leads to experiments that are actually reverse engineered.

A true Scientist doesn't actually care what the answer is, they just want to know the answer.

Bryant RedHawk wrote:

A true Scientist doesn't actually care what the answer is, they just want to know the answer.

Exactly right. Good science doesn't have an agenda, it is just looking for answers. You can tell someone has a great mind when they are just as happy to be proven wrong as right, because they learned something they didn't know.

I think a lot of the reproducibility issues are coming from inadequate initial pre experiment data collection.

If you don't make sure everyone knows the absolute conditions you started with (soil makeup, chemistry, biology, month started, duration, and on and on) how is anyone going to be able to do the same experiment, in the same soil conditions?

One of the things that gets missed (even in schools) is to document the exact conditions you are conducting your experiment in.
If you really want to perform and be able to reproduce any experiment all variables must be recorded prior to starting the experiment, during the experiment and after the conclusion of the experiment.

When you publish your paper, the top scientific tomes are going to want to know everything is in order in your house, if it isn't or if something is left out, you will not be published.

Some of the magazines that look like scientific reporting and publishing are not pure scientist directed, there are some that are more for the layperson to read and become interested.
After all, pure scientific papers tend to be focused on and meant to be read by others in that particular field or one closely related.
If you are a first year chemistry student you will have a hard time comprehending a paper written by a PHD on the relations atomic sub-particles when exposed to multi-angled radio waves ( for an example)
First off the math that would be reported in such a study would be two levels above your current math abilities and understanding, making you have to learn a lot just to be able to work out the reported equations.

It would be a matter of getting your head to the knowledge base level needed before hand so you could comprehend what was being said in such a paper.
(I had this very thing happen to me once, it was two weeks before I knew the math that was being used in the experiments). When this happened to me the first few seconds were "Duh" moments then I realized that I didn't have the knowledge needed to be able to comprehend his post doctoral work.
I corrected my short comings and ended that quarter as his assistant ( a really big deal at that school )

@Bryant R.: "I think a lot of the reproducibility issues are coming from inadequate initial pre experiment data collection. If you don't make sure everyone knows the absolute conditions you started with (soil makeup, chemistry, biology, month started, duration, and on and on) how is anyone going to be able to do the same experiment, in the same soil conditions? .....If you really want to perform and be able to reproduce any experiment all variables must be recorded prior to starting the experiment, during the experiment and after the conclusion of the experiment".

I don't disagree with this, but neither do I entirely agree with it. Keeping in mind that our faculties, and complementing intrumentation that substitute for faculties that we do not possess, are used to make measurements of phenomena that we deem *are* (not representations of, but *are*) facets of the universe, one must first question to what extent we know the "absolute conditions" of any given experiment. That said, it clearly would be advantageous to observe and document as much as possible....but it can become unwieldy and there would be some point of diminishing returns on one's efforts I suspect. It perhaps goes without saying that there were pre/non-literate peoples who practiced a form of science based on observation of what worked and rejecting what didn't, even if they did not have a term for it.....nor a written system of documentation. The maize story is a good example:

"The most impressive aspect of the maize story is what it tells us about the capabilities of agriculturalists 9,000 years ago. These people were living in small groups and shifting their settlements seasonally. Yet they were able to transform a grass with many inconvenient, unwanted features into a high-yielding, easily harvested food crop. The domestication process must have occurred in many stages over a considerable length of time as many different, independent characteristics of the plant were modified......The most crucial step was freeing the teosinte kernels from their stony cases. Another step was developing plants where the kernels remained intact on the cobs, unlike the teosinte ears, which shatter into individual kernels. Early cultivators had to notice among their stands of plants variants in which the nutritious kernels were at least partially exposed, or whose ears held together better, or that had more rows of kernels, **and they had to selectively breed them**. It is estimated that the initial domestication process that produced the basic maize form required at least several hundred to perhaps a few thousand years." -- http://www.nytimes.com/2010/05/25/science/25creature.html (my asterisks added)

Ultimately, the results of most experimentation seem to offer a place for other investigations to begin. Just considering the Materials and Methods alone of most original scientific publications reveals that these are *not* optimized conditions, but rather the conditions that lead to the results in that particular study. Reproducibility is an understandable goal, especially given a cultural heritage that craves control. But science "estimates" the universe. Given the right nail, it's a pretty good hammer. But we shouldn't feel so frustrated when hitting our thumbs because we're pounding it on a staple or screw.

Science, in it's basic form, follows thus:
- Hypothesis : an educated guess as to why a phenomenon is occurring.
- Observation and statistical data collection related to said phenomenon.
- Data analysis
- Conclusion
If the conclusion supports the Hypothesis then you have a Theory. If it doesn't then you must adjust the hypothesis and find more data.
In the sense, we have always had science. We just didn't call it science.

A hypothesis or Theory is only scientific when it can be proven false. Note: proven FALSE, not proven TRUE
With Physics, Chemistry, and maybe geology, we can produce experiments to test one variable at a time. We can sometimes get a 99.99% confidence result over repeatable experiments.
Biology, however, has so many unknown variables that getting above a 70% correlation is good enough to say the hypothesis is supported.
Repeated biology experiments will not give you the exact same result every time as will a well set up physics or chemistry experiment will. But a good biology experiment/observation will give you a correlation good enough to say "X has Y effect on Z"

Devin Lavign wrote:Just a quick thing on scientific reproducability. I saw an interesting quote from a scientist a while back, something to the extent of "there is no money or glory in being the 2nd person to discover something" thus summing up a big part of the problem for many of the sciences. Who is going to fund testing if something someone else discovered is reproducible if their scientist and their company wont profit or gain glory from it.

This appears to be one of the big problems in modern science (to me).
A scientist (of my understanding) should(!) not (have to) care about money or glory.

Rather then being forced to out compete all other scientist in the area, they could be working together, sharing unpublished experience, equipment and ideas.

The application of the economic model to the discipline of science is corroding it until nothing is left. Science requires collaboration not competition of market shares.
The devotion to spend a lot of time and possibly money to understand the hidden connections and relations, and then sharing them with others is not compatible with economic principles.

To try to answer the title of this post, "Science's reproductibility problem: what can we learn?", I would tend to say: we can learn that the scientific method, used to break down complicated matters into small pieces (or elements) to then by manipulation try to understand that one element to reach conclusions on how it works, is a tool of very limited value that carries great potential dangers. The scientific method can only be useful if we realize its severe limits and dangers. I would even dare to say that many of the problems in the world we face today are either partially or completely caused by mankind's believe in the scientific method to be the best tool for understanding our world and for civilization to progress.

I will try to explain. After graduating from university (social sciences) I worked in research for 4 years. During that time we developed a model to predict migration flows within the country. It was based on several sets of census data, so we had a very good database to build it from. We used the model as a tool to help policy makers. These people of course had questions, to which I could only answer "according to the model we would expect X or Y to happen". Policy makers (or anyone else for that matter) do not like that type of answer that shows limitations on what research actually tells us. The "according to the model" part illustrates what the scientific method is. And as such it explains why it is so damned difficult to replicate results obtained by other researchers.

I left science after 4 years. Although the scientific method is a usable tool, I found it not all that great to work with. When many years later I came across Permaculture I found it to be a much more logical and understandable approach to life and our planet than the scientific method. My own thinking leads me to believe that science as the 'best method' to understand our world (or even our universe) borders on religious belief.

I think the main flaw of science is that it fails to recognize how everything affects everything. Even if we step in to carefully observe something, just the mere fact that we observe it changes it. Any living thing is an organism that feeds on perception of outside influences. Something happens and it reacts. Us humans are the same, we respond to outside input, and that response is often different even if the input is the same, the reason being that the whole of any situation is never the same. There are too many factors at play to fully understand everything.


Bill Mollison fully realized this when he wrote the designers manual. In chapter 2, he wrote about this under the heading "Science and the thousand names of God" on pages 11 and 12. Permaculture explicitly states that there is way more which we don't understand in nature than there is what we do understand. To pretend otherwise is in my view misleading. Rather than stating that Permaculture has a problem being insufficiently 'proven to work' I would say the scientific method has a problem being the most praised method we humans have in understanding our world.

I don't think the scientific method is a bad thing by itself. It can for sure help us to structure experiments. From experiments we can learn things. It might even help us understand the effects of some of our actions. But it is not going to make us so knowledgeable that we can impose ourselves and our actions on everything else claiming that we know what we're doing. The farmer that does X because he knows that it works, because 'look for yourself' has in my view a much better understanding of what life is than the scientist who found a cure for Y but who has no clue how that cure might affect another thousand things that he never even thought of. The reason for that is that the farmer has the wisdom that he does not know why exactly it works, where the scientist has the mistaken belief based on his very limited methods that he does know.

I would conclude that science and the scientific method are tools. Observation and experimenting are other ones. Understanding that we don't really know much about life and that we probably never will is all the more reason to be very careful with our actions and with our claims on 'truth' and 'proven' and 'legitimate'. I hope this helps to further our thinking about our beautiful planet. Knowing is not our main goal in life, because we won't ever know everything. Enjoying life in all its forms seems more important to me.

Rene Nijstad wrote:The farmer that does X because he knows that it works, because 'look for yourself' has in my view a much better understanding of what life is than the scientist who found a cure for Y but who has no clue how that cure might affect another thousand things that he never even thought of. The reason for that is that the farmer has the wisdom that he does not know why exactly it works, where the scientist has the mistaken belief based on his very limited methods that he does know.

It seems to me that the farmer's observations may be just as faulty as the scientist's, but because there is no peer review process of the farmer's observations and methods - he may continue with the same damaging practice decade after decade.  I see this in my own region, in which the intuitive knowing and practice of the farmer and rancher has diminished the carrying capacity to one fifth of its historic level, and extirpated numerous species of animals and plants.

Tyler Ludens wrote:

Rene Nijstad wrote:The farmer that does X because he knows that it works, because 'look for yourself' has in my view a much better understanding of what life is than the scientist who found a cure for Y but who has no clue how that cure might affect another thousand things that he never even thought of. The reason for that is that the farmer has the wisdom that he does not know why exactly it works, where the scientist has the mistaken belief based on his very limited methods that he does know.

It seems to me that the farmer's observations may be just as faulty as the scientist's, but because there is no peer review process of the farmer's observations and methods - he may continue with the same damaging practice decade after decade.  I see this in my own region, in which the intuitive knowing and practice of the farmer and rancher has diminished the carrying capacity to one fifth of its historic level, and extirpated numerous species of animals and plants.

But that's the thing isn't it? We just don't really know. BTW I should have said permaculturist instead of farmer because what I tried to imply was doing X while following how we understand natural processes. We're not promoting bad practices on permies. Since the topic is why science has trouble reproducing results, the farmer example was merely to compare something so different to a scientific outcome that I could make a point. But yes, you're right.

Hi, I'm new.

This might be a good thread to work on an easy how-to guide for collecting data.
I think we should try to aim for a balance between easy to follow and accuracy.

Similar to a lab logbook, my proposal to get the discussion started:

Needed per individual:
A way to easily take notes (I always have a pen and notebook on me)
A project

Later on we will need for the whole group:
A central place to collect and sort the data

Method:
Write down the goal of your project.
Write down location & date.
Write down at least 3 traits that you think may be relevant, (more is better).
Write down the results.

Example:
Proposal
Try to cross strawberry (Fragraria _ unknown species) with mock strawberry (Potentilla indica) by planting them in close proximity.

Goal
Tasty mock strawberries.

Info
Almere, Flevoland, The Netherlands. april - july 2016
Soil with high sand content and a bit of clay.
Drainage of the soil is too strong so it dries out quickly.
Plenty of sun.
Protected from the wind by a 2,5m wooden fence all around the garden.
Mock strawberry plant was transplanted with plenty of very good soil from a different part of the garden.
Real strawberry plant was a gift, it came with a small amount medium quality soil.

Results
Mock strawberry is doing very well and spreading very far with plenty of fruit.
It is doing a bit better than the mock strawberries on the other side of the garden.
It still tastes like water though.
Real strawberry is alive and healthy, but not spreading out and has no fruit.


End of Example.

As you can see, it doesn't have to be a well thought out or well executed experiment as long as it is well documented.
I kinda cheated here because i wrote this from memory, maybe that should be noted as well?
It would be even better with a day to day log.

Once plenty of these experiments have been registered, a simple targeted meta-analysis could lead to interesting results.

I just read this article on permaculturenews.org. http://permaculturenews.org/2016/12/16/science-technology-permaculture-much-really-need-know/

It explains the differences between 'pure science', 'applied science' and technologies. The writer then continues to argue is a very clear and concise way how the 'permaculture lacks science' position some people hold is not only pure bogus, but also quite irrelevant. He does not turn down science, but he does manage to point out the difference between misuse of science and our ability to properly reason without science.

How much do we really know, and how much knowledge or details do we really need?