Research Integrity

Greetings Permie Friends, I wonder how do you verify the integrity of research. In many instances when someone tries to make a case he/she normally include a research that back ups his/her claim. Who makes this "research" or more precisely who fund it? And what is the motive for the "research". Can you honestly say that the results they publish are true for the most part? What percentage will you give to the reals and non reals?

Any valid answer to this question is, by its nature, going to be very long and detailed. And it has already been done much more effectively that I could manage.

I recommend the book "Bad Science" as a starting point. It will help you recognise good and bad evidence, and know what questions you should be asking when someone puts a research result in front of you.

Nice book but I have no way of having. So I'll find another route to the same end.

https://www.amazon.co.uk/gp/offer-listing/000728487X/ref=tmm_pap_used_olp_sr?ie=UTF8&condition=used&qid=1552736553&sr=8-1

You could try a second hand copy, or use a library. But you won't get a better answer than in those pages.

Here is his website as well.

https://www.badscience.net/

From this article:  https://hbr.org/tip/2018/10/should-you-trust-that-research-study-you-read-about

pay attention to a study’s sample size. Researchers are less likely to successfully replicate others’ results if the study looked at a small number of people or companies, for example. Similarly, consider the peculiarities of the sample or the study’s methods. How did the researchers come to study the specific sample they did? What kinds of factors could have skewed the results? Finally, always remember: If a study’s results sound too good to be true, they probably are.

Anne Miller wrote:From this article:  https://hbr.org/tip/2018/10/should-you-trust-that-research-study-you-read-about

pay attention to a study’s sample size. Researchers are less likely to successfully replicate others’ results if the study looked at a small number of people or companies, for example. Similarly, consider the peculiarities of the sample or the study’s methods. How did the researchers come to study the specific sample they did? What kinds of factors could have skewed the results? Finally, always remember: If a study’s results sound too good to be true, they probably are.

The internet has make publishing way to easy for anyone. How do I know that research actually existed in the first place. Right now I could make a bogus research and publish it. I can also publish many that corroborate. And then when it comes to scientific research that requires laboratory and expensive gadgetry, how do I verify the alleged results and claims? Its very risky to believe by faith. There should be a better way. I'm reading bad science perhaps that will give an idea.

Thanks for trying to help I appreciate your effort.  

Ultimately you live inside your own head. Your own senses can trick you into thinking all sorts of things. How do you know anything at all? Are you absolutely certain you aren’t merely a character in the Red King’s dream? (“Through the Looking Glass” by Lewis Carrol) My brother, who has since died, believed he saw and heard all sorts of things no one else saw or heard. He *did* see them, too. His brain made him see them. It’s quite a thing, isn’t it? He genuinely saw them even though they weren’t there. That is the way his brain interpreted the data from his receptors (eyes & ears.) That was the picture, those were the sounds his brain produced from that data. Or maybe he was right and we were deceived. Maybe his world was the real one and WE were “seeing things.” How could anyone prove, really prove, that we were right and he was wrong?

At some point you have to simply trust, just so you can get on with life, but with regard to “studies:” Study the studies, look at the data they present, the sample sizes, the previous work of the researchers involved, other studies of similar or the same topics. Look at the past work of the researchers, their experience, qualifications, motivations... who/what group(s) funded their work and possible motivations of the sponsors, their political affiliations and those of the researchers, their investments, what they stand to gain or lose depending on the results of the study, and on and on. Most of us are skeptical of studies we disagree with. The truth is though, that we should all, always be skeptical of all studies.

Today nearly all scientific research is done by people who are not entirely free. Their research grants come from the government and from business interests that are looking to prove this or that thing that would help them in their own goals. If the research doesn’t yield the desired results, guess who will NOT be getting that funding next time around? I’m not saying “scientific” research is worthless. I’m saying it is not free to come to true conclusions UNLESS those true conclusions happen to please the scientists and those who pay the scientists. Deciding whom to believe and whom to doubt is a LOT of work, and even if one does the work, one can never be certain.

Cindy Skillman wrote:Ultimately you live inside your own head. Your own senses can trick you into thinking all sorts of things. How do you know anything at all? Are you absolutely certain you aren’t merely a character in the Red King’s dream? (“Through the Looking Glass” by Lewis Carrol) My brother, who has since died, believed he saw and heard all sorts of things no one else saw or heard. He *did* see them, too. His brain made him see them. It’s quite a thing, isn’t it? He genuinely saw them even though they weren’t there. That is the way his brain interpreted the data from his receptors (eyes & ears.) That was the picture, those were the sounds his brain produced from that data. Or maybe he was right and we were deceived. Maybe his world was the real one and WE were “seeing things.” How could anyone prove, really prove, that we were right and he was wrong?

At some point you have to simply trust, just so you can get on with life, but with regard to “studies:” Study the studies, look at the data they present, the sample sizes, the previous work of the researchers involved, other studies of similar or the same topics. Look at the past work of the researchers, their experience, qualifications, motivations... who/what group(s) funded their work and possible motivations of the sponsors, their political affiliations and those of the researchers, their investments, what they stand to gain or lose depending on the results of the study, and on and on. Most of us are skeptical of studies we disagree with. The truth is though, that we should all, always be skeptical of all studies.

Today nearly all scientific research is done by people who are not entirely free. Their research grants come from the government and from business interests that are looking to prove this or that thing that would help them in their own goals. If the research doesn’t yield the desired results, guess who will NOT be getting that funding next time around? I’m not saying “scientific” research is worthless. I’m saying it is not free to come to true conclusions UNLESS those true conclusions happen to please the scientists and those who pay the scientists. Deciding whom to believe and whom to doubt is a LOT of work, and even if one does the work, one can never be certain.

This one I totally agree except for the last sentence.

Can it be duplicated?  Is there a way to video it?  Is there a way to prove it?

When you read someone's documentation, you should doubt it.  The question is:  how much should you doubt it.  It is a matter of probability and trust.  And then there are the reviews - how much do you trust them?  

Well said Paul, that is most of what I was planning to say while reading through the thread.

The one other thing I would like to mention, is how much it sucks so much science is effectively hidden behind pay walls. While you can access it, you can only do it at a cost. Of course there are multiple pay walls, so one payment does not give you access to all.

There are ways around this, using someone else's access is the most common. Most often by visiting a college and using their computers. Or some public libraries have access in bigger cities.

but it still is effectively a way to separate the average person from the information. Even extremely old stuff, like some of Darwin's papers is held behind pay walls. It is really absurd, I feel.

I am a research scientist to pay the bills, while also trying to be a permie at home. Sadly, a recent review of published research found that only about 14% of PUBLISHED results were found to be repeatable. I heard this on NPR a while back. That makes one question the very foundation of science and the scientific method. One of the things they hammer into your head in grad school is that your results must be repeatable. Depending on the assay, I typically do 3 replicates of my experiment and average the results before submitting it for publication. Some journals specify these parameters, as what you submit will be peer-reviewed before acceptance of your manuscript. Not everything is caught in peer review.  Recently Duke University had to pay back the NIH over $100 million because a researcher falsified data to obtain grants. That's a big black eye for every honest scientist.

paul wheaton wrote:Can it be duplicated?  Is there a way to video it?  Is there a way to prove it?

When you read someone's documentation, you should doubt it.  The question is:  how much should you doubt it.  It is a matter of probability and trust.  And then there are the reviews - how much do you trust them?  

Trust is always going to be an issue. But then, as with Cindy's point, how can you even trust your own senses?

I think the falsifiability thing is huge, as is the sample size issue. But the biggest, I think, might be the number of independent research projects doing the same experiment, at different times and on different experimental and control groups of sufficient size. Having only two independent studies can't be as helpful as having a half-dozen.

-CK

I think this is some first class research:



She documented her techniques and results so well, that you can understand the entire experiment in very little time.   The sample set it quite small, but that is appropriate for the earliest steps.  

Others can attempt to duplicate - with conclusions that support or refute.  Others can attempt a larger sample set.   Clearly there are opportunities for dozens of branches of tests from this humble beginning.  

Frankly, I think this is some of the best scientific research available.  I suspect the scientist in this case doesn't even have a phd.  

We live in an era of video.  We can build our knowledge set faster than ever before.   Of course, the flip side is that we can also build propaganda faster than ever before.  So for each piece of research we find that our skepticism engine is more powerful than our logic engine:  What were the motivations for this research?   Are the results of this research paid for?  Is there nefarious motivation behind this presentation thus making it propoganda.

Because of this, we then turn to trusted sources who will judge the research, presentations and bodies of knowledge.  

I think that if dozens, or a dozen, kids do this exact same experiment, in the same way, and record it, that's some pretty good science. It's like they are each their own test group, each with a control being kept in the same environment as the experiment.

This is sort of like an open-sourced, modularly designed experiment that can be made more accurate with more participants. I like it. More participants become the factor that makes it more accurate, or at least gives us a better idea of the experiments' dynamics in subtly different environmental conditions. People-scale science sounds very permaculturally aligned to me.

I also think that this approach won't work for some experiments, and that's okay. The integrity of the research still depends on the individual doing it. We have no guarantee something nefarious didn't happen off-camera to ensure that the organic sample did better than the control. I am not saying that happened, just that without an agreement to act with integrity, it doesn't matter how many letters of whatever kind the experimenter has behind their name, the research will be tainted by their bias.

-CK

"Trust is always going to be an issue. But then, as with Cindy's point, how can you even trust your own senses?"



Well if you dont trust your senses then you got a real problem. Better not try to prove anything. Because ultimately it is our set of senses that can prove anything for us. The problem I am seeing is that most people of the scientific age will write off their set of senses in favor of alleged scientific study. A grave mistake. me thinks. For example how many people believe that all sugars is bad including the naturals? A very big majority. How many taste buds say that sugar is sooo good. When you call the dearest person sweetheart; honey and sugar thats  enough proof I guess . Then when you add a little logic you get. natural sugar is the best food process sugar is deadly. I am probably the only person who believe in this despite all the taste buds believes the same. IMO our set of senses should be the corner stone of all the proofs hunting. Say gather the facts that can be proven by our own senses and make that as point of reference. Anything that debates from it should be written off. Anything that supports it should be the next level of reference. Until you are able to connect the dots. The problem with scientific  results is that it can not be duplicated by ordinary folks who havent seen a microscope in their entire life. Unless its a potato experiment. And as I have said on another thread. Logical reasoning can be verified by logic, which unlike a microscope, we all have. So I am taking advantage of whatever I have. This way I dont have to duplicate or video tape but still get proof. Reminds me of the school where people go every Sunday or on regular basis to be trained to believe by "faith" in the higher authority. All the control freaks need is to set up authorities in each field then you get a pyramid control system where the ones in the lower echelons believe by "faith" in the higher ones. "never mind our sense". Grave mistake. people should start honoring their senses. As Gandhi said "truth is self explanatory".