Quality Comment

Media reporting on this study has ranged from (mostly) non-critical repetition of the conclusions to (frequently) silly scatological humor to (thankfully) even a little skeptical investigation. The basic conclusion of the authors is that dogs line themselves up with the magnetic field of the Earth when defecating (and, sometimes, when urinating). As usual, there is a devil in the details. While the findings of this study are hardly earthshattering news, whether true or not, the questionable practices utilized to generate the conclusions most people are reading in the news are illustrative of how weak hypotheses can be made to seem stronger than they are.

The Study

The hypothesis in this case was that because there is evidence some animals can sense and orient to the Earth’s magnetic field (convincing evidence in some cases, less convincing in others), it might be the case that dogs can do so This would be cool in itself but also convenient for researchers interested in “magnetoreception” since dogs are a plentiful and convenient research subject.

The researchers recruited 37 private dog owners with a total of 70 dogs in Germany and the Czech Republic and had the owners determine the alignment of their dogs’ spine from tail to head during defecation and urination using a hand-held compass. The dogs were off leash and in locations where they were not obviously constrained in their movements by human structures or activities. Routes the dogs were walked on were supposed to be changed haphazardly by owners.

The Results

The authors hypothesized the dogs would orient preferentially in a North-South direction while urinating and defecating, suggesting they were detecting and responding to the orientation of the Earth’s magnetic field. This didn’t turn out to be true, and the dogs’ orientation was essentially random. This is where the authors decide to go fishing. As they put it,

"After sampling and the first analysis (which yielded negative or at least ambiguous results) had been completed, we decided to sort the data according to the geomagnetic conditions predominating during the respective sampling times….Second order analysis was performed on the data which yielded the higher significance in the first order analysis."

In other words, the authors started thinking of all sorts of different ways to group and organize the data for analysis, and then they threw out those ways that didn’t yield the statistically significant results they were looking for and kept the ones that did. After lots of re-arranging and multiple statistical tests on various arrangements, they finally found a significant result:

"The relative declination change proved to be the best predictor of alignment, i.e.,. sorting the data according to this parameter provided the most significant results. Analysis of pooled recordings as well as of mean vectors of recordings in dogs sampled during calm magnetic field conditions (relative change in declination = 0%; minimum of 5 observations per dog) revealed a highly significant axial preference for North–South alignment during defecation."

The technical term for this is data-dredging, and it’s kind of cheating. Testing numerous hypotheses only thought up after the initial hypothesis has turned out to be wrong and the data have already been explored virtually guarantees one will find something that can be spun into a validation of the original idea and justification for the current study and others in the future. This is the stock-un-trade of alternative medicine research, though it is by no means limited to that field, and it is one of the most important reasons bad ideas are able to survive in science.

The authors, to be fair, are quite up front about how they approached their analyses:

"…No one, not even the coordinators of the study, hypothesized that expression of alignment could have been affected by the geomagnetic situation, and particularly by such subtle changes of the magnetic declination. The idea leading to the discovery of the correlation emerged after sampling was closed and the first statistical analyses (with rather negative results, cf. Figure 1) had been performed.

The relative declination change proved to be the best predictor of alignment, i.e., sorting the data according to this parameter provided the most significant results."

Unfortunately, they don’t seem to realize that this makes most of their statistical analyses invalid, and rather seriously undermines their conclusions. The dividing up of the observations into tiny categories reduces the power of the statistical tests used, and failing to compensate for performing multiple unplanned comparisons renders the results unreliable.

The specific arrangement of data that was found to yield a non-random pattern of alignment during defecation was when only measurements taking during periods of stability in the magnetic field were considered. The Earth’s magnetic field fluctuates slightly much of the time, and the authors were only able to find a statistically significant pattern in the dogs’ position during periods when it was not fluctuating. As challenging as it is to explain why dogs’ would have evolved to orient themselves according to the magnetic field of the Earth while defecating, it is even harder to explain why they would only be able to do this during relatively brief and unpredictable periods when the field was not fluctuating.

There are obviously a large number of factors that one can guess might influence how dogs position themselves during elimination besides the magnetic field of the Earth. The authors acknowledge a few of these, but they don’t do a very convincing job account for many in their design, analysis, or discussion.

Despite the haphazard and unrecorded attempts of dog owners to vary their routes, there is no way to determine if the habitual walking patterns of either dogs or owners influences the direction the dogs faced during elimination. The authors address the possibility of the position of the sun influencing the dogs, but only with speculation.

"…generally, there are on average 1,450 sunshine hours per year at maximum in the Czech Republic and in Germany, on localities where measurements were done. Even if we would assume that these sunshine hours were evenly distributed over the daylight period and the year (as our observations were), there would only be a probability of 33% that the observation was made when the sun was visible. Hence, with high probability (67%) most walks during the daylight period were made when it was cloudy."

The sun position wasn’t actually recorded, just assumed to be irrelevant since it is usually cloudy there most of the time. The dogs’ orientation didn’t appear to vary significantly with time of day, but again given all the subdividing of data for analysis, this doesn’t reliably exclude the sun as a possible influence.

The authors also don’t do much to account for the possible influence of other sights, sounds, or most importantly smells on the position of the dogs. All of these variables seem at least as plausible as magnetic field orientation as influences on how dogs position themselves when defecating.

And finally, the question of what possible reason there might be for dogs to align themselves with the magnetic field of the Earth while eliminating isn’t convincingly addressed. The general suggestion is made that the ability to detect the Earth’s magnetic field might have some evolutionary benefits in terms of navigating through one’s territory, and that aligning with this filed when pooping might be a way of “calibrating” the system against visual or other landmarks. Of course, if the results of this study are accurate, they could only do this, or use the field for positioning at all, during the times when it is not fluctuating.

Conclusions

Despite the significant and rather obvious problems with the design and analysis of this study, it does touch on an interesting subject. Some animals do use magnetic field detection for orientation and navigation, and it would be interesting if dogs proved to have such an ability. At best, this study might be considered useful in generating some hypotheses for further testing, though the largely negative results don’t justify much optimism about the outcome of additional studies. Predictably, however, the authors spin their results on the most positive possible terms to claim a groundbreaking achievement.

"In this study, we provide the first clear and simply measurable evidence for influence of geomagnetic field variations on mammal behavior. Furthermore, it is the first demonstration of the effect of the shift of declination, which has to our knowledge never been investigated before."

The authors engage in some spectacular mental contortions when using the positive results wrung out of their data to suggest that they have not only discovered a revolutionary new phenomenon but explained the failure of past research to support the magnetic field detection abilities of animals.

"the findings that already small fluctuations in Earth’s magnetic field elicit a behavioral response and the fact that “normal” magnetic conditions under which dogs express their orientation behavior occur only in about 30% of all cases call for caution. When extrapolated upon other animals and other experiments and observations on animal magnetoreception, this might explain the non-replicability of many findings and high scatter in others."

Rather than acknowledge that the most likely explanation for the failure of their a priori hypothesis and most of the analyses they conducted, as well as the negative results of other studies, was that there is no underlying relationship to find, the authors choose to conclude everyone has simply been looking for the wrong thing, and that their creative data mining has finally stumbled across the right variable. Time (or replication, really) will tell, of course, but it strikes me as a bit of stretch.

Since it is theoretically impossible to definitively prove a negative, the positive findings of poorly controlled research and data dredging unfortunately make it possible to argue for more research on almost any topic, regardless of how implausible the underlying theory or how consistently negative the results (yes, Homeopathy, I’m looking at you).

This study, while not dealing with as serious a subject as a medical treatment, exemplifies some of the ways in which research can be structured and analyzed, to eliminate any chance of actually falsifying a hypothesis and to justify continuing research even in the face of repeated and consistent negative findings. Though I would be pleasantly surprised if the findings turn out to be correct, it wouldn’t alter the fact that the approach represents some common and significant problems in the process of finding the truth about nature through science.