What is a fair test?



In ‘What’s the point of experiments?‘, we talked about why scientists do experiments. Here, we look at how to make sure a test is fair.

If a test is not fair, it means the experiment could be fatally flawed and the results are actually meaningless (rather than perhaps just seeming meaningless).

The basics

Imagine you have a die; if it is fair and rolled randomly, the chance of it landing on a six is one-in-six. If you had to design an experiment to prove this is true, there are a number of things you need to do to make sure the experiment gives you the right result:

First is the material (the die). If you use a die that is loaded, the experiment will probably not give you the right result. So you must use a ‘fair’ die.

Second is the technique. If you roll the die in exactly the same way each time, with it in exactly the same starting position, certain numbers will crop up more often than others. This can be corrected by making the starting position random by shaking the die first.

Third is the sample size (or number of ‘repeats‘) of the experiment. Say you roll the die three times and land on the number six once. These results would say that the chance of landing on a six is one-in-three instead of one-in-six. Ideally, you would repeat the experiment an infinite number of times, but since this is impossible, you should just repeat it as many times as you reasonably can.

There is also a distinction between repeats using the same material (throw one die lots of times) and using different materials (throw lots of dice a few times). Using more materials is generally better, because if you experiment on only 1 die that turns out to be defective, your results will be completely screwed up. If you use 10 dice, of which 1 is defective and 9 are normal, your results will be much better – the poor results of the defective dice are ‘diluted’ by the good results of the normal dice.

In real scientific studies, the sample size is often limited by the time, cost or available materials, but in this case it’s limited by how long you can be bothered to keep rolling the die (which is probably not long, unless you really have no life).

Finally, you need to interpret your results correctly. In this case, you check the chance of the die landing on a six by taking the number of times it landed on a six and diving it by the total number of rolls. If instead (for some strange reason) you divided it by the total of the numbers from one to five, you would get the wrong answer.

So in order to make the test fair, you have to have the right materials, use the right techniques, repeat the experiments as many times as you can and interpret the results correctly.

Making comparisons

SI Exif

Now we’re going to step it up a notch. Imagine you have developed a drug that cures the common cold. You take a group of 100 people, and give them all a sample of your drug. Two weeks later, only 20 of them still have a cold. Success! You’ve cured the common cold! You’re a millionaire!

Well, actually it’s not that simple. First, if you work for a pharmaceutical giant, you probably won’t make much money out of it personally, but more importantly, your cure may not have actually worked. People recover from colds naturally, so it could be that the 80 people who shook off their colds just got better by themselves – your drug was not directly responsible for their recovery.

If you wanted to prove that your drug was responsible for their recovery you have to do a control test. A control is a ‘standard’ situation to compare your new system against. In this case you would have two groups of 100. Group A you give your drug to, Group B you don’t (the control). If 20 people in Group A and 20 people in Group B still have colds after two weeks, then your drug probably doesn’t work. But if only 5 people in Group A have a cold, then your drug probably does work.

common cold control

So for a lot of experiments, a control test needs to be done in order to prove that your new system (it could be a chemical, device process etc.) has a significant effect compared to what would happen under ‘normal’ circumstances. In order for the tests to be fair, both the test and the control need to be as identical as possible (same number of people, similar ages, genders, same recovery time etc.).

The placebo effect: this is where it gets a bit complicated…

You may have heard of the placebo effect, and although scientists have known about it for ages, we still don’t really know exactly how it works. But we do know what it does: it’s a psychological effect that can cause someone to either feel better, or even to actually get better. It’s a weird quirk of the mind – if you believe you’ve been given a cure, even one that doesn’t work, you have a slightly better chance of recovering than if you have nothing at all.

The placebo effect means that the best control for your experiment is to give Group B a placebo rather than nothing at all and see how that compares to your drug. In order to keep it a fair test, your drug and placebo must appear identical, and the patients cannot know whether they are getting a placebo or the real drug as this would negate the effect.

When the patients don’t know whether they are receiving a real drug or a placebo, it is called a ‘single-blind’ test.

Sorry – it turns out the placebo effect is even more complicated than that…


To make the whole process even more of a pain in the backside, the placebo effect is not only manifest in the material (i.e. the pill or injection), but also in the technique. The person giving the drug to the person could (unintentionally) give away subconcious hints about whether the patient is receiving the drug or the placebo. For instance, they might appear a bit more cheerful when giving the drug because they expect it to work.

In order to prevent this from happening, researchers can use a ‘double-blind’ test where the person administering the drug/placebo does not know which one they are giving out.

Some experiments even use a ‘triple-blind’ system, where the researchers have nothing to do with the administration of the drug/placebo, but simply look at the data – this helps their interpretation of the results to be as objective as possible.

So, to sum up…

Making sure that an experiment is fair is critical to ensuring your results are not flawed. To do this, you have to make sure that your materials, techniques, sample size and interpretation are appropriate to what you’re trying to prove.

For some experiments, you want to compare a ‘new’ system against a control. To do this fairly, you need to make sure the conditions and the treatment of the results are the same for both.

In experiments such as drug testing, the placebo effect needs to be accounted for. You can use the placebo as a control, but again the test conditions for the drug and placebo need to be as similar as possible for the test to be fair. To eliminate the placebo effect completely, the test may need to be double- even triple-blind.

Getting experiments to work can be tedious and frustrating, but making sure they are fair is relatively simple.

All images are open-source/Creative Commons licence.
Credits: diacritica (first); unknown (second); thisscienceiscrazy (third); Dave Morice (fourth).

Text © thisscienceiscrazy. If you want to use any of the writing or images featured in this article, please credit and link back to the original source as described HERE.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s