Greetings everyone,
The post today is all about a new study on monarch overwintering numbers that was just published last week. More importantly though, it was published in a new online journal called "PeerJ", that apparently allows anyone to see what went into getting this paper published. This journal publishes not only the actual research study on its website, but also the word-for-word scientific reviews of the manuscript before it was published. I've never seen this before, and it is quite interesting. So this is a great opportunity to talk about what actually goes into getting a paper published in the scientific world. If you've always wondered how scientific publishing works, now is your chance to find out!
The study in question was conducted by a large group of researchers, some who are well-known monarch scientists (Oberhauser, Pleasants, Taylor), and all led by a guy named Wayne Thogmartin, of the USDA. I'll put a link to the paper here, and recall that it is freely-available to download. Apparently, this project came about from a working group meeting these people all attended, at the “Powell Center” which is run by the USGS. This is a place where scientists gather for meetings on specific topics and where they talk about ways to analyze data, etc. - very sciency stuff. The study appeared to be an analysis of monarch overwintering density in Mexico. As readers here know, when the monarchs go to Mexico for the winter, they form clusters in the oyamel trees on top of the mountains, and these clusters can be pretty thick. We actually don’t even know how thick they are, and that was the point of the paper (I think). The study was an attempt to come up with a specific estimate of the density of monarchs within the overwintering sites, so that way, when the overwintering colony size is measured each year (in hectares), we could then estimate the true number of adult monarchs by doing the math.
Just to make this point, can you tell how many monarchs are in this photo? Then try to guess how many monarchs would be on an entire tree! It's hard! By the way, I think I took this picture while visiting the overwintering sites in 2007 with Lincoln Brower. Fun times...
The researchers here attempted to come up with this estimate by assembling all of the prior estimates of monarch density over the years, and then conducting their own analyses of these numbers. There have been a number of early attempts to estimate monarch density, mostly by Dr. Bill Calvert and Lincoln Brower. These all varied in their approach - one was a count of the number of dead monarchs on the ground after a storm, one was by collecting whole branches of monarchs, one was an eyeball approach, etc. In all there were 6 prior estimates, and they ranged from 7 to 61 monarchs/hectare. The table below lists these, which I lifted directly from the paper. Apparently, there is some very wide variation in these estimates, which further proves just how difficult it can be to estimate monarch numbers.
Next, I'm going to skip over the nuts and bolts of the paper and jump right to the result, since today I want to talk more about the reviews, rather than the paper itself. From the prior estimates of monarch numbers in Table 1 above, the researchers did some very fancy math to come up with an overall estimate of 21million monarchs/hectare. If you're curious, you can read the paper to see more about how they came up with this number. Most of it is very technical and math-y though. Geek-alert.
I can see why the researchers wanted to come up with this estimate. This number is great for helping to interpret results from other projects, and for conservation purposes. For example, if the colony area from this past winter was 2.91 ha, then by this density estimate, there were approximately 61 million monarchs at the overwintering sites this past winter. Recall from a previous post I had done on the impact of road mortality to the monarch population, and where I estimated how many monarchs were killed each year on the journal south to Mexico. Based on the best available science, I had estimated that between 30 and 50 million monarchs die on roadways during the entire fall migration. Compare this then to the overwintering number of 61 million - yikes.
Now, let's talk about the peer-review of this paper. I mentioned that this particular journal is unique in that it allows readers to see the scientific review process that went into each of its papers. But first, let me explain how this whole business of publishing science works. It's not the same as publishing a book. When a researcher does an experiment, or conducts a study, or whatever, they need to get this work published in a scientific journal, before it can be considered 'real' science. They always teach you in graduate school that this is one of the most important elements of the scientific process - if it's not published in a scientific journal, then it is not science. Simply collecting the data is only one small part of the scientific process.
So the whole publishing process begins when the researcher submits a manuscript to a scientific journal. There are hundreds of different journals in existence (like I am the editor of one called, Animal Migration), and the researcher needs to pick one that best fits the study in question. There are behavioral journals, physiological ones, entomological ones, etc. Then the editor of that journal takes the manuscript and passes it to a "handling editor", who oversees the review process for that paper. The handling editor then selects 2 or 3 scientists to serve as 'reviewers' of the paper. These are people who have no connection with the project, but who are supposed to be well-versed in the science around that topic. These people are chosen because of their expertise, and their identities are (normally) kept hidden. The reviewers then read the paper and decide if there are any problems with the science, or the interpretation, or the figures, or whatever. There are ALWAYS problems and criticisms with any paper, but if there are too many really, really glaring problems, that's bad. The handling editor then makes a decision to accept or reject the paper, based on the comments from the reviewers. If the paper is not rejected at that stage, the authors then get a chance to revise it, and to fix any issues found by the reviewers. Then it may or may not get another round of review, or the editor will decide to accept. This whole process is what we refer to as "peer-review", and it is an essential component to the scientific process. The peer-review keeps bad science from getting published, or at least it is supposed to. Peer-review is designed to make science 'self-correcting.'
Now, normally, when a paper is published, no one but the reviewers, editors and authors see the peer review comments, but in this case, they are published along with the paper itself. There is a link to click next to the online paper, called "Peer review history", and I'll link it here. I strongly encourage you to read these, or at least skim them, before going further in this blog entry so that what I say next about them makes sense. The comments on the paper are listed here in reverse-chronological order, so you have to scroll all the way to the bottom to see the original submission of the manuscript, followed by the reviewer comments. As you go up the page you get closer to the final product.
I'm next going to walk through the review history of this paper, because it's pretty interesting.
It looks like there were 3 reviewers - Art Shapiro, a well-known butterfly expert from California, then Lincoln Brower, who needs no introduction here, and one other anonymous reviewer. From reading the comments from each person, it looks like Art and Lincoln had found some minor issues with the manuscript that needed to be addressed. For example, Lincoln indicated that he questioned if these density estimates that were collected decades ago are meaningful today, since he has personally observed a change in the cluster densities in recent years. It looks like Art wanted further clarification on a few things relating to how monarchs use milkweeds, etc. Neither of these comments are big, and could easily be addressed in a re-write of the paper. Art's comments were actually fairly positive toward this paper.
The most significant, and critical, comments apparently came from Reviewer 3, who appeared to have major reservations about this project. He or she also pointed out that the densities may have changed over time since these early estimates were made, similar to what Lincoln had said. Then he or she pointed out that there were parts of the paper that were just wrong, and should be removed. In looking back through the paper, it seems they are referring to a section near the end where the authors discuss the link between overwintering monarch numbers and milkweed. This is indeed a controversial issue, and the debate is ongoing among monarch scientists over it (see the MonarchScience post last month, even). There are some in the monarch community who firmly believe that milkweed is the key to saving this critter from imminent extinction. Other prominent scientists believe that the biggest problem facing monarchs today is their migration is becoming harder, which kills them before they reach Mexico. This reviewer seemed to think that because of this debate, this entire section should be removed from the paper, because it detracted from the real subject of the project, which was the overwintering numbers. From reading the final paper that was accepted, it looks like the authors did not do this, and kept this section in the paper.
This reviewer also made some comments at the end that were directed more toward these authors, than at the paper itself. Basically, he or she said that because the monarch is in no danger of extinction the authors should stop writing in an alarmist way and to stop with the headline-seeking news stories, or something to that effect. This made me laugh because the headlines last week regarding this paper were "Billions more milkweed needed to restore monarchs" And, from a quick search on Facebook, I can see that this news story (and this title) was shared a gizillion times. This was just what that reviewer was talking about - alarmist.
Next, if you continue scrolling up on the review page, you can see how the authors then submitted a revised version of the paper, and at the same time, they had to submit a letter that listed each of their responses to the reviewers. This is a standard practice in the peer-review process, and it allows the editor to see how they fixed certain parts, and/or argued for why they disagreed with certain comments. I read through this file (you can download it from the website) and I was a little surprised at how much they disagreed with the reviewers on a lot of points, rather than fix the issues that were pointed out. This is a little arrogant, and gives the impression that the authors were unwilling to admit their paper had problems, and/or unwilling to make changes. I was also surprised that the journal editor let them get away with that. In fact, from the lack of critical points made by either the handling editor or the editor, it looked to me like this paper got a free pass, especially given the critical comments by that one reviewer (that they did not fix).
Then, the paper was apparently reviewed again, then finally accepted after that round. It looks like Lincoln was one reviewer for the second round, and in his last batch of comments, he made a strong pitch to have this paper accepted. From my reading of this review history, I think Lincoln's comments here, and in the first round too, probably made the difference between acceptance and rejecting, given that the other two reviewers were split on the value of this paper.
After the paper gets accepted, it usually then goes through a formatting stage, which basically means it is made to look like a scientific paper in that journal, and then the authors usually get to have one final pass though it to look for any typos, that sort of thing. And then, viola - a scientific publication is born. From start to finish, this whole process can take many months, and often a year before a scientific paper is "in print". Most people are surprised at that.
So there you go - you just got an inside peek at what goes on behind the scenes when a scientific paper is published. It's a lot of back-and-forth between authors, editors and reviewers, and it can sometimes get intense. Let's face it, scientists often disagree on things, and with the science around the monarch butterfly, this is especially true.
That's all for now.
**************************************************************************************************************************************************************
Direct link to this blog entry:
**************************************************************************************************************************************************************