This Thursday, Michael Balter at Science Magazine (and many others) reported that geneticists had found “that epigenetic effects, chemical modifications of the human genome that alter gene activity without changing the DNA sequence, may have a major influence on sexual orientation.” As Balter observes, such a finding could be welcome news for LGBTQ rights advocates, as it would combat those who argue that homosexuality is a “lifestyle choice.” On the other hand, such a finding could suggest the possibility of biomedical interventions, a prospect that I find very disturbing.
Fortunately, Ed Yong of The Atlantic dropped some common sense and statistics on the whole affair:
The problems begin with the size of the study, which is tiny. The field of epigenetics is littered with the corpses of statistically underpowered studies like these, which simply lack the numbers to produce reliable, reproducible results.
Unfortunately, the problems don’t end there. The team split their group into two: a “training set” whose data they used to build their algorithm, and a “testing set”, whose data they used to verify it. That’s standard and good practice—exactly what they should have done. But splitting the sample means that the study goes from underpowered to really underpowered.
If you use this strategy, chances are you will find a positive result through random chance alone.
There’s also another, larger issue. As far as could be judged from the unpublished results presented in the talk, the team used their training set to build several models for classifying their twins, and eventually chose the one with the greatest accuracy when applied to the testing set. That’s a problem because in research like this, there has to be a strict firewall between the training and testing sets; the team broke that firewall by essentially using the testing set to optimise their algorithms.
If you use this strategy, chances are you will find a positive result through random chance alone. Chances are some combination of methylation marks out of the original 6,000 will be significantly linked to sexual orientation, whether they genuinely affect sexual orientation or not. This is a well-known statistical problem that can be at least partly countered by running what’s called a correction for multiple testing. The team didn’t do that. (In an email to The Atlantic, Ngun denies that such a correction was necessary.)
So it seems that this is probably all just a case of bad scientific methodology; yet another instance of scientists abusing significance measures to manufacture spurious findings. Which should come as no surprise, given that it is absurd to think there could be such a simple genetic cause of such a complex trait as sexual orientation.