Essay:What is an experiment?

There seems to be a bit of confusion over what an "experiment" actually is. I'm mostly talking, of course, about the people who like to say variants of "you can't prove something by experiment" or a phrase that basically boils down to that when you remove fancy words. But I think skeptics and scientists too can sometimes over-complicate matters and forget the basics. Simply put, an experiment is like this:

jdsgkhoaipodgopasdgnvskjpsad

Wait, what?

Well, it's difficult to get across in just text but I have simply banged on a keyboard (and put some formatting around the result to differentiate it) and nothing much more than that. It doesn't seem like an experiment at first glance, there are no test tubes, there's no discussion section, it doesn't start with "an experiment was carried out..." But this would be to confuse what is what is just good practice in science and what science actually is. An experiment simply tests an hypothesis. Fancy words aside, it takes our beliefs, what we predict and expect to happen due to them, and see if these predictions come true. "I predict that, if my keyboard is correctly set up and connected, and that I have a text-editor open, letters will appear on the screen." I thought this, and I tested it.

It's not a very good experiment as it is, it doesn't give us any information apart from that the keyboard is working, but then again that is what the experiment set out to show and it worked. There's nothing magical about it, or particularly special, you're simply looking for the effect that your beliefs anticipate and expect. If it didn't work, the keyboard would be most likely buggered or not plugged in - though the fact that I'm currently typing on a laptop excludes the latter possibility. A better experiment would be like this:

h

This is testing the hypothesis "if I press the 'H' key, then an h will appear on screen." What if I test the "if I hold down shift and press the 'H' key I'll get a capital H instead" hypothesis:

H

Again, nothing magical or special. It's simply looking for an effect by carrying out an experiment. Often, skeptics are too bogged down in details such as peer review, placebo control, randomisation and so on and forget what those are actually there for. There is no difference between a real effect and the test for that effect; we're just simply looking for the effect. These extra details are there to ensure that the effect we're looking for is the one we're looking for, and not something else entirely.

We don't need to build the Large Hadron Collider to see that there's a door standing in the way of us getting from one room to the other. We anticipate that, if there is a door, we will be able to see it, we will be able to feel it, and it will block our path if we try to walk through it. We believe there is a door, we anticipate the properties of a door, we look for those properties. Not seeing those properties leads us to conclude that there is no door blocking our path and we can freely walk from one room to the other. Again, there is no difference between the real effect and the test for that effect.

What proper controls do in experiments is allow us to make sure what we're testing is actually what we set out to test. This is simply a case of refining experimental design to give us more specifics; akin to banging on a keyboard to see if it's plugged in versus tapping each letter individually to see if each produces the right signal. How these work is to simply narrow down effects to give us more information, they don't change the basic nature of anticipating a particular effect and looking for that effect.

This is why you can't say that prayer, or homeopathy, or chiropractic works and then say you can't test for if. If they have an effect, it can be tested for. Sometimes our controls might have to be impressively subtle and well thought-out, but that's just detail, the basic point doesn't change. If we were to suggest that "prayer works" then we would anticipate that it should work. We should anticipate that, say, if you prayed for a group of 200 people going in for complicated surgery and didn't pray for 200 people going in for the same surgery (and controlled for all other factors to make sure it was only prayer that was being looked at), we should expect the prayer group to do better. However, those who say you can't "test" for prayer disagree with this. The problem here is that we can conclude that these people don't just act as if they believe it can't be tested, they are actually expecting that the result will be negative. They make excuses in advance (or even after the fact, but I don't see much difference) about why it won't work, or why it can't be tested.

This makes no sense. You cannot assert that something has a real effect but can't be tested because the real effect and the test for it are one and the same.