POP SCIENCE LESSON 2: MYTHBUSTERS

PART A. CONTROL GROUPS

Good scientists are all control freaks, not huge-mustache-grooming, jaunty-beret-wearing freaks. Before I can explain to you how the Mythbusters often lack good controls in their experiments, I first have to briefly explain what controls are and why you need them.

Running an experiment requires that you exert control over the environment in which you conduct your experiment. That’s one kind of control– keeping everything in the universe constant so that you can manipulate only a single variable and see what happens. Of course, that’s the ideal, and everyone falls well short of it. The Mythbusters do reasonably well at this type of control most of the time, but they tend to be pretty terrible at the other type, control groups.

Control groups could also be called comparison groups. You want to know if your experimental variable changes anything, so you need to have something to compare it to, obviously, duh. For example, if you want to see if putting my hand in warm water makes me pee my bed, so you put my hand in warm water and then I pee my bed, you’ve proved nothing. Why? Because I pee my bed every night, OK? It’s a medical condition. The warm water had nothing to do with it. If you had a proper control (having me sleep through the night without putting my hand in warm water) you would know that! And furthermore, you don’t really know if the water had to be warm, or if it even had to be water. Maybe if you put my hand in cold salsa it would make me pee my bed. I’m telling you, I wet the bed on a hair trigger. Really you should probably do the whole experiment on a different person. The point is, you need a good control group (no warm water), and often multiple control groups (cold salsa, cold water, warm salsa), in order to know exactly what is causing the change you see (hot piss in my bed). And before you get on my case about it, I know they did an episode that included this myth, but they couldn’t fall asleep so they never really tested it so it’s not really worth talking about, now is it?

It took me about 5 seconds on youtube to find an example of the Mythbusters doing an experiment with a crappy control group. The first one I came across (and I know there are many more), was when they were testing whether mixing alcohols makes a hangover worse. In this one, Asian-dude and Jock-dude each drank beer until they got stupid drunk, and were tested for how bad their hangovers were in the morning. (Asian-dude only needed 5 beers. Lightweight.) A few weeks later, they drank the same amount of alcohol, but split between beers and hard-A, and were again tested. They did a decent job of keeping the universe constant in the two different conditions. The dudes ate and slept the same amount on both testing days, so any differences in hangover couldn’t be blamed on diet or the time of day, etc.  So they did pretty well on the first type of experimental control.

I couldn’t find a clip that showed the results (other than Asian-dude puking into a bucket), but that doesn’t matter, because the data cannot tell us anything about whether mixing alcohol makes hangovers worse. Even if they found that the beer + hard-A combo group had a worse hangover, would they have really demonstrated that the combination of beer and hard-A made hangovers worse? No, they wouldn’t have.  Could the data also support the alternate claim that just drinking hard-A makes hangovers worse? Yes, they can. Let me show you. Because we do not know how bad a hard-A alone hangover is by their measure, we cannot differentiate between many possible alternatives, three of which I have pictured below.

This is the assumption their experiment makes: that a beer alone or a hard-A alone hangover would be identical or at least similar. If this is true, then showing that the combo hangover is worse indicates that combining alcohol creates a worse hangover.

But it is also very possible that the hard-A hangover is actually much worse than the beer hangover, and worse even then the combo group. In this case, there doesn’t need to be an interaction between alcohols to explain the increased hangover in the combo group- it is just in between a beer hangover and a hard-A hangover.

A third possibility is that the hard-A hangover is identical or similar to the combo hangover. Perhaps having any hard-A at all brings you to a new level of hangover, and at that point it doesn’t matter whether or not you have beer.

We cannot distinguish between these alternatives because we do not know where that red bar should be, because their experiment has a faulty design. All they have shown is that either a mixed-alcohol-hangover, or a hard-A hangover, or possibly both, is worse than just a beer hangover. You can hardly make any claim at all based on their crappy experiment. And this is why my husband won’t watch Mythbusters with me. Because I’m always shouting things like “YOU CAN HARDLY MAKE ANY CLAIM AT ALL BASED ON YOUR CRAPPY EXPERIMENT, BUTTHEADS! USE A CONTROL GROUP! YOU ARE POLLUTING THE BRAINS OF THE PUBLIC WITH FALSE SCIENCE!” All because they lacked the appropriate control groups. So demand good controls out of your entertainment, people. Please… for the sake of my marriage.

P.S., they have failed to even demonstrate that the men even had hangovers. No, they really didn’t. I know you think they did, but they didn’t. Sure, common sense says that they performed worse on the hangover tests than they would have had they not been drinking the night before, but it has not been demonstrated by the experiment, because they never took the tests without drinking the night before. This is a huge science no-no. How bad is a hangover to start with? Where is baseline? Their experiment, because it is a giant piece of crap, can’t answer these questions.

I’m not done listing the many flaws in Mythbusters’ “experiments”!

Up next week: Sample Size!