Saturday, January 30, 2016

Transparency, public arguments: a wrong recipe for arXiv rejections

OneDrive: off-topic: tomorrow, Microsoft will reduce the free 15 GB space by 10 GB and abolish the free 15 GB camera roll space. Old users may click here and after two more clicks, they will avoid this reduction if they act on Sunday!
A European blog critiquing science and its flavor appendix called Nature believe that it was wrong for the website of scientific preprints (100k papers a year, 1.1 million in total) to reject two submissions by students of quantum information who attempted to rebrand themselves as general relativists and argue that you can't ever fall into a black hole.

Thankfully, the European blogger and others agree that the papers were wrong. But they still protest against the fact that the papers were rejected. Or to say the least, there should have been some "transparency" in the rejection – in other words, some details about the decision which should be followed by some arguments in the public.

I totally disagree with those comments.

The arXiv (the website) was established in the early 1990s as a tool for researchers to share their findings more quickly, before they got published in the paper journals that mattered at that time. Paul Ginsparg created the software and primarily fathered the hep-ph and hep-th (high energy physics phenomenology and theory) archives – he also invented the funny, friendly yet mocking philosophy-like nickname "phenomenologists" for the people who were not formal (mainly string) theorists but who actually cared what happens with the particles in the muddy world of germs and worms.

The hep-th and hep-ph archives were meant to serve rather particular communities of experts. They pretty much knew who belonged to those sets and who didn't. The set got expanded when a member trained a new student. Much like the whole web (a server at CERN serving Tim Berners-Lee and few pals), the got more "global" and potentially accessible to the whole mankind.

This evolution has posed some new challenges. The website had to get shielded from the thousands of potential worthless submissions by the laymen. There existed various ways to deal with the challenge but an endorsement system was introduced for hep-th, hep-ph, and other experts' archives. It is much easier to send papers to "less professional" archives within but it's still harder than to send them to the crackpot-dominated of Philip Gibbs.

The submissions are still filtered by moderators who are volunteers. One of them, Daniel Gottesman of the Perimeter Institute, has made an important response to those who try to criticize the arXiv moderators when they manage to submit their paper previously rejected by the arXiv to a classic journal:
“If a paper is rejected by arXiv and accepted by a journal, that does not mean that arXiv is the one that made a mistake.”
Exactly. The arXiv's filtering process isn't terribly cruel – authors above a certain quality level can be pretty sure that their paper gets accepted to the arXiv if they obey some sensible conditions so it's not like the "bloody struggle for some place under the Sun" in some printed journals considered prestigious.

But the arXiv's filters are still nontrivial and independent and it may happen that the arXiv-rejected paper gets to a printed journal – which often means that the printed journal has poor quality standards. There is no "right" and there cannot be any "right" to have any paper accepted to the arXiv. There is no "guarantee" that the arXiv is always more inclusive than all journals in the world. The arXiv's filtering policies are different which doesn't mean "softer and sloppier than everyone else's".

In this case, the rejected papers were written by students of Dr Nicolas Gisin – a senior quantum information expert in Geneva. But these students didn't write about something they're expected to be good at because they're Gisin's students.

Instead, they wrote about black holes and it was bullšit. You can't ever fall into a black hole, a layman often thinks before he starts to understand general relativity at a bit deeper level. They made the typical beginners' mistakes. Then they realized they were mistakes and did some smaller but still embarrassing mistakes that allowed them to say that "you can't ever fall into a Hawking-eaporating black hole" which is still demonstrably nonsense.

My understanding is that these preprints about black holes should not be allowed in the quantum information archive where they would be off-topic; and these students should not have the "automatic" right to post to high-energy physics or general relativity archives because they're not experts and they're not in a close enough contact with an expert. So I think it's a matter of common sense that papers from such authors about this topic are likely to be rejected – and if everyone seems to agree that the papers are wrong, what are we really talking about?

The reason why some people talk about this self-evidently justified rejection is that there are some people who would love to flood the arXiv with garbage and dramatically reduce its quality requirements. These people want it simply because they can't achieve the minimum quality threshold that is required in hep-th, hep-ph, and elsewhere – but they want to be considered as experts of the same quality, anyway. So they fight against any moderation. If there is any moderation at all, they scream, at least, they should get some complete justification why their submission was rejected. It's clear what would be done with such an explanation. The rejected authors would show it to friends, posted on blogs, and look for some political support that would push the moderators in a direction and these moderators could ultimately give up and accept the garbage, anyway.

The louder and more well-connected you would be, the more likely it would be for the garbage you wrote to be accepted to the arXiv at the end.

In fact, this Dr Nicolas Gisin already shows us where this "transparency" would lead. The actually relevant comment that should be said in this context is that Dr Gisin has partially failed as an adviser. He failed to stop his students from embarrassing themselves by (nearly) publishing a preprint about a part of physics that they clearly don't understand at the expert level. It's really this Dr Gisin, and not the arXiv moderators, who should have been the greatest obstacle that his students should have faced while submitting wrong papers on general relativity.

Instead, he became a defender of the "students' right to submit these wrong papers". Why this right should exist? Once you try to demand such non-existent "rights" and scream things that make it clear that you don't give a damn whether the papers have elementary errors or not, you are a problem for the arXiv. You are a potentially unstoppable source of junk that may get multiplied and that the experts would have to go through every day. It doesn't matter that you have published lots of good preprints to another place of the arXiv. You just don't have credentials to flood every sub-archive at

We see that Dr Gisin tried to inflate his ego by co-organizing an article in Nature that tries to pretend that it's a scandal that two young people who have the honor to be students of Dr Gisin himself were treated in this disrespectful way by the archives dedicated to general relativity or particle physics. With this screaming in the public, lots of people could join Dr Gisin and send the message to the arXiv moderators: How dare you? Those were students of our great Dr Gisin. You must treat them as prophets.

Sorry but they're not prophets. They were just students who tried to send wrong papers to professionals' archives about disciplines at which they are clearly not too good and unsurprisingly, they have failed. Even if Dr Gisin had sent the papers about the "impossibility to fall to a black hole", these papers should have been rejected.

The rejection may depend on some personal opinions or biases of a particular moderator – but there's nothing wrong about it. At the end, science has to be evaluated by some individual people. Gisin's students' papers could have been rejected for numerous simple reasons. If you demanded the moderators to publish some detailed explanations, it wouldn't help anybody. Any suggestion that some "arguments" between the rejected authors and moderators should follow means that
  • someone believes that there is a "right" for everyone to submit preprints anywhere to, but there's none
  • the moderators must be ready to sacrifice any amount of time and energy, but they don't have to
  • the interactions between the moderators and the would-be authors are discussions between two equal peers.
But the last point is simply not true, either. The rejected authors are primarily meant to be – and it's almost always the case – people who just don't know the basics or don't require their own papers to pass even the most modest tests of quality. One may say that they're crackpots or marginal crackpots. You just don't want the moderators to spend much time by communication with these people – because to save the time of actual researchers is the main reason of the rejection in the first place. So if you forced the moderators to spend an hour with every rejected crackpot paper, you could very well "open the gates" and force every researcher to waste a few seconds by looking over the abstract of the bullšit paper instead. If the gates were opened in this way, the number of junk papers would obviously start to grow.

The main problem of this "transparency" is that the meritocratic decision – one that ultimately must be done by someone who knows something about the subject, or a group of such people – would be replaced by a fight in the public arena.

Let me give you a simple example. It's just an example; there could be many other examples that are much less connected with the content of this weblog in the past but whose issues are very analogous, anyway. I believe – or hope – that loop quantum gravity papers aren't allowed at hep-th (just at gr-qc) because these people are acknowledged to be crackpots at the quality threshold expected in high energy physics. Every expert knows that even if there were something OK about loop quantum gravity (and there's nothing), there's no way how it could tell us something meaningful about particle physics.

Now, whenever a moderator would reject a loop quantum gravity paper at hep-th, the "transparency" regime would force him to explain the steps. In one way or another, more explicitly or less explicitly, he would have to reveal that he considers all the loop quantum gravity people to be cranks. Pretty much everyone in high-energy physics does. But almost no one says those things on a regular basis because people are "nice" and they want to avoid mud. OK, so the loop quantum gravity author would get this reply. What would he do with it? Would he learn a lesson? No, loop quantum gravity folks can never learn any lesson – that's a big part of the reason why they're crackpots.

Instead, this rejected author would send the explanation by the arXiv moderator to his friends, for example clueless inkspillers in Nature (e.g. Zeeya Merali who wrote this Nature rant about the "high-profile physicist" whose students were "outrageously" rejected), who would try to turn the explanation by the arXiv moderator into a scandal. Could anything good come out of it? Not at all. At most, the loop quantum gravity crackpot could assure himself that just like him or the European blogger, way over 99.9% of the public doesn't have the slightest idea about issues related to quantum gravity.

But the arXiv must still keep on working – it has become an important venue for the professionals in high energy physics. It's serving the relatively small community whose knowledge – and therefore also opinions – dramatically differ from the knowledge and opinions of the average member of the public. Clearly, if the hep-th arXiv were conquered by the community of the loop quantum gravity crackpots or the broader public that has been persuaded that loop quantum gravity is an OK science, the actual experts in quantum gravity would have to start a new website because hep-th would become unusable very soon, after it would be flooded by many more junk submissions. But hep-th is supposed to be their archive. That's how and why it was founded. The definition of "they" isn't quite sharp and clear but it's not completely ill-defined, either.

If a paper is rejected, it means that there is a significant disagreement between a moderator and the author of the preprint. The author must think that the preprint is good enough or great (that's why the paper was submitted) while the moderator doesn't share this opinion. If the author is smart or has really found something unusual, he may be right and the moderator may be wrong. The probability of that is clearly nonzero. It's just arguably small. But you simply can't improve the quality of the rejection process by turning the process into a potentially neverending public argument. It's essential that the expertise needed to evaluate submissions to the professional archives is not "omnipresent" which is why the broader "publication" of the details of the rejection transfers the influence and interest on a wrongly, too inclusively defined subgroup of the mankind.

Those are the reasons why I think that the calls for transparency, however fashionable these calls have become, are misplaced and potentially threatening for the remainder of meritocracy in science.

No comments:

Post a Comment