Wednesday, May 14, 2014

BICEP2 vs Planck: nothing wrong with screen scraping

The BICEP2-Planck relationship is competitive and that's how things should be; BICEP2 claims remain robust

Two days ago, Adam Falkowski of Paris created a ministorm when he asked
Is BICEP wrong?
The idea of his was that there was a rumor that the BICEP2 folks have so seriously misinterpreted, misunderstood, understimated, and misunderestimated a sky map by the Planck Collaboration that when the bug is corrected, the significance of the BICEP2 discovery fully evaporates.

He tried to suggest that almost all the well-informed experts in this experimental enterprise converge to the opinion that most of the B-modes seen by the BICEP2 gadget is due to the polarizing dust in the Milky Way, not due to the gravitational waves from the era of cosmic inflation. Erik Verlinde tweeted that a Princeton workshop implies the same conclusion. In particular, these two physicists seem certain that Planck will refute the BICEP2 claims in their new paper to be published not later than October 2014.

I remained open-minded for half a day but once I saw the claims by Clement Pryke and John Kováč that they are not planning any revision of their papers at all, I decided that Adam's story is almost certainly just bullšit, a conspiracy theory mixing half-truths, dust, šit, and fantasy into a stinky whole – and Adam and Erik are just full of this, politely speaking, composite šit.

The people who are spreading bad mood and who are saying that something is wrong with the result or with the whole discipline of experimental science do so because of various reasons, assumptions, expectations, and speculations, and I will try to argue that every single one of them is just wrong.

The proposed bad mood is supposed to be justified by some or all of the following memes:
  1. Planck and BICEP2 are obliged to love one another and to cuddle one another.
  2. Planck is such a superior team that no one smaller, like BICEP2, is allowed to make a discovery without Planck's warm praise and Planck's generous endorsement.
  3. BICEP2 should have waited for Planck's detailed opinions.
  4. The BICEP2 analysis existentially depends on the precise form of the Planck map and its very accurate interpretation.
  5. Planck is obliged to publish all the currently evaluated raw data quickly because BICEP2 has made a big claim that is related to their work.
  6. Planck is obliged to publish all their raw data in a maximally transparent form, especially because it is obliged to help the BICEP2 folks.
  7. BICEP2 isn't allowed to reverse-engineer Planck's PDF files (to "scrape the screens") because it is a dirty method.
  8. Only high school students may want to slow down a competition or to circumvent the competition's attempts to slow them down.
OK, let's talk about these wrong assumptions in some detail.

Competition is healthy.

First, many kibitzers on the Internet suggest that the situation in experimental cosmology must be very bad because if it were good, the Planck collaboration would do their best to minimize the amount of work that others, like BICEP2, have to do to convert the data into numbers they need, in formats they prefer.

It's just nonsense. Planck and BICEP2 are two different teams – in fact, two competing teams. Their task is to measure the cosmic microwave background and to make physics discoveries – sufficiently justifiable findings that are true and that may be later independently verified by others to be true. Their task is not to spend their days with formatting or making all the things user-friendly, especially not user-friendly to their competition.

The total price of the Planck satellite was €700 million, almost a billion of dollars. On the other hand, BICEP2's expenses are comparable to $10 million, about one hundred times smaller. That's why Planck is the Goliath and BICEP2 was the David – or perhaps "a David". The budgets are totally independent and of course that the Planck budget should be used to allow the discoveries by Planck; and the BICEP2 budget is here to make discoveries by BICEP2 possible.

It is totally fair to compare the relationship between Planck and BICEP2 to the relationship between Google and [a small competing search engine or software company]. They sometimes fight and make each other angry. That's how things should be. The competitive struggle finally helps the progress; the consumers benefit, at least statistically. It is totally silly to think that experimental cosmology should be different from the software industry – or from any other part of the human activity where competition has reasons to exist.

The huge disproportion in the budgets is why many people have had expected that Planck should be the first one to discover things and the source of the most accurate information about all these data. But this is just an expectation, not a fact, and similar expectations about top dogs and under dogs often turn out to be wrong. And indeed:

David, BICEP2, morphed into a top dog.

If your budget is 100 times smaller than the budget of someone else, it may be and feel more likely that you won't win but it simply does not imply that you can't discover something important before the Goliath does. Even though "chance" could be enough to explain all these unexpected events, the victories of the under dogs, there are also detailed reasons why BICEP2 has apparently done the discovery before Planck.

BICEP2 had a vision. They were focusing on the B-modes, assuming from the beginning that there could be something new over there. Their devices were sufficiently optimized to do the job and they have apparently succeeded in the job. In comparison, Planck has gotten into a pessimistic mode in which people assume that "they can't discover something really new, anyway" which is why they don't even try so hard.

At any rate, the odds were dramatically changed two months ago. BICEP2 has probably made the Nobel-prize discovery. If it is right, it will be confirmed by someone else within a year or two (not necessarily by Planck: POLARBEAR and others have a chance as well; Keck Array might be viewed as a "not quite independent" experiment because the people are the same).

As far as I can say, it's obvious that only BICEP2, and not BICEP2 plus those who confirm the result, should get the experimental Nobel prize if the discovery is right. It would seem extremely unfair to claim that BICEP2 has discovered the B-modes "simultaneously" with someone else. I think that it's fair to say that BICEP2 has already been facing some jealousy and attempts of the competition to diminish their results which is another reason to consider them the "exclusive flag-bearers" here.

And yes, I think that the BICEP folks are – both individually and as a team – the world's top experts in measuring the polarization of the CMB. Even at a sociological level, I wouldn't consider a vague criticism by some people in Planck (I mean more vague than the BICEP2 papers) to be a sufficient reasons to abandon my belief that the BICEP2 discovery is almost certainly true.

And I don't think that BICEP2 has ever had any reasons to wait for Planck's confirmations.

When you're seriously good in some business, you don't wait for the "top dog" to beat you or declare a tie. Tomáš Berdych didn't want to play a balanced game or to achieve a tie with Rafael Nadal. Berdych doesn't want to "lose just a little bit" with Nadal. He wants to beat Nadal. It didn't work the last time but things like that sometimes work.

Similarly, BICEP2 didn't have a dream to publish an important discovery simultaneously with Planck. They wanted to win the race and it seems that they have won the race. It's totally sick, unscientific, and I would say deeply immoral to suggest that an experimenter or an experimental team (and the same thing may hold for theorists, of course) has the obligation to wait for another one who is considered stronger. It's just bullšit. Science doesn't contain any such rules. Science doesn't have these special rights of "top dogs" who are assumed to be stronger. Even in sports, they don't do such disgusting things so why should the scientists do them? In sports and sciences, it's the results, and not the expectations, that matter most.

Every experimenter should be measuring his things as independently as possible. He or she or they should publish the results when they feel sufficiently confident about them, according to their understanding of the standards in their scientific discipline. And that's exactly what BICEP2 has done. No one else matters to the zeroth order. Experimental discoveries in physics are primarily a race against the confidence levels – an analogy of the runners' race against the clock. By this sentence, I mean that looking at others is secondary. As an experimental physicist or a team, you must primarily be independent and follow the rules of the scientific method.

So I am really disgusted by those who try to suggest that BICEP2 should have waited for Planck's co-discovery. Such an important discovery is even cooler if it were done by someone who may have been considered an under dog – exactly because the discovery by BICEP2 could have seen less likely. But it was never impossible, of course. Everyone who has sufficiently good tools, intelligence, vision, and excitement can make discoveries. And it has turned out that $10 million is enough for such a discovery.

Various people who try to politicize science are bringing biases concerning top dogs and under dogs, biases in both directions. Sometimes the biases in both directions are introduced by the very same people. On one hand, some people say that only expensive experiments or large collaborations or famous, established scientists may be "allowed" to do important discoveries – or may be "trusted" when they claim to have made discoveries. On the other hand, some people – and indeed, sometimes it is the same people – demand quotas and affirmative action. They want everyone to have the same chance, to be assigned the same number of discoveries, and so on. I am talking about various Smolinian hippie jerks who have demanded for years that junk physicists and crackpots like themselves should be as influential as great physicists.

Is that really so hard to see that both attempts to rape the scientific process are blatant and immoral violations of the rules of the scientific method and attacks on the common-sense fairness in general? In the real world of science, people and teams have unequal chances to make important contributions, but above a certain level, the chances are never zero so it's always possible that someone unexpected will be the first one to make the actual discovery.

BICEP2's dependence on Planck maps isn't critical.

This is a slightly technical point in this mostly sociological essay. But let me comment on it, anyway.

The Planck collaboration hasn't described the meaning of a particular map terribly accurately and it's being suggested by the "creators of the bad mood" that this fact demolishes BICEP2 because BICEP2 must use a "magic key" generously provided by Planck, and if it doesn't use this key in the perfectly prescribed manner, everything that BICEP2 does is guaranteed to be worthless.

But that's simply not how the analysis and reasoning by BICEP2 worked and works.

The Planck maps haven't been used for the BICEP2 results in an existential way at all. BICEP2 has focused on a piece of the sky where the amount of dust is known to be much lower than elsewhere and their extracted values of the new parameters were based on the assumption that the dust doesn't contribute at all. Of course that it isn't true that the dust doesn't contribute at all. It has always been clear to everyone in the field – and especially to the people in BICEP2 – that the dust contributes something.

However, they have various reasons to see that the dust almost certainly (with the huge confidence levels they announce) cannot be responsible for the whole B-mode signal they are seeing. It is simply not true that a "quantitative announcement from a superior team, like Planck" is necessary for acquiring this kind of near-certainty. They have found and performed many tests that may decide whether it's plausible that the whole signal is due to the dust. And the answers are No, it can't be due to the dust.

Even if the dust doubled, they probably know that at a discovery-scale confidence level, it can't explain the whole signal. The vortices they are seeing are just way too similar to the pattern expected from the primordial gravitational waves. They would probably be much less isotropic and much less uniform if the dust were the cause. They have divided the data into pieces in many ways and if the dust or something else were the cause, they would probably see an inconsistency with the predictions of the primordial gravitational waves.

And let me tell you something. I think that before the BICEP2 discovery claim, they have probably had accumulated (preliminary, but completely real for them) data from the Keck Array (BICEP2.5) which will probably allow the confidence level to be even much higher than it was according to BICEP2. So I find it likely that their reasons to be certain are actually much stronger than what we have been shown. Note that at the level of apparatuses and events, the Keck Array is independent of BICEP2 so a simple confirmation by the Keck Array (even if its separate confidence level is "just" the same or even a bit lower) increases the probability that the BICEP2 discovery is right.

Again, the main wrong point by the "producers of bad mood" is the idea that the experimenters should rely and wait for other experimenters, especially the "theoretically superior" ones. It ain't so. In science, experiments should be as independent from others as possible, and that's how the BICEP2 folks are doing science, too. And it's actually plausible that even when Planck publishes its final polarization results, their accuracy and "amount of information" will be inferior to BICEP2's.

In fact, the Planck instrumental noise in the BICEP2 rectangle is probably stronger than the Milky Way dust in the same rectangle – which allows us to say that whatever Planck ultimately says about this piece of the sky is probably going to be irrelevant, anyway. It seems utterly unrealistic to promote some details about the Planck's opinions about the dust in this rectangle to the point that decides about the life and death of BICEP2. (Moreover, some partly censored map that Jester has mentioned may very well be more anti-BICEP2 as well as more pro-BICEP2; Jester seems to unjustifiably assume that the censorship may only go in one direction.)

Planck should wait with every result for the right time according to their own internal evaluation of the quality, too.

The previous section is meant to debunk the claims that Planck is obliged to "speed up" the process because their competitors in BICEP2 have said something important. That's just a piece of nonsensical bureaucratic šit, too. Planck is relasing many papers as well and they're released when the Planck people feel sufficiently good about their reliability, according to their internal standards. What happens with other experiments shouldn't materially modify the timing.

There's really no reason to expect that two very different experiments should reach the "level of the discovery" at the same moment. Perhaps, Planck may discover the B-modes as well. Perhaps, it may take a year longer than it took in BICEP2. Perhaps, Planck hasn't accumulated sufficient data to match BICEP2 at all. All these things are plausible. The idea that Planck must confirm the discovery almost immediately is just wrong. The idea that after the BICEP2 announcement, Planck has transformed itself into a group of assistants and janitors working for BICEP2 is wrong, too. They're still an independent team whose race is primarily against their own confidence levels.

So ignore the sourballs, Planck, and take your time.

Experimenters are employed to do physics, not paperwork.

The open/transparent science movement sometimes wants scientists to publish their data and results in the kind of perfectly standardized, user-friendly, optimized formats that mean that if someone else wants to use these results or raw data, she doesn't have to do any extra work.

I disagree with that. It's just not the job of scientists to focus on these superficial things. The experimenters' goal is to make physical measurements, to impose new upper bounds, and – if they are hard-working and lucky – to make discoveries. The pictures, graphs, and raw data that are published are not the primary goals. They're just tools serving the real goal and the real goal are physical measurements and discoveries, not perfectly readable formats.

It is totally normal that a researcher who wants to use the results or the raw data from someone else has to spend some time with data extraction and reformatting, too. The author of the original paper should only be expected to publish the raw data, the intermediate results, and the final results in a form that seems sufficient for sufficiently many others to see that the conclusions of the paper have been supported by some valid work. But a scientist needs to redo the things independently if he wants to independently confirm a result, anyway. So the intermediate results of a particular scientific paper aren't really a key that may permanently settle a dispute about a scientific claim.

So some formatting work is both up to the producers of the new science as well as its consumers – producers of the future science.

In particular, it's OK for BICEP2 to scrape the screens.

I have been reverse-engineering various graphs and plots many times, too. With Wolfram Mathematica, I became a bit better in this activity although I am confident that Phil Gibbs is, for example, much better than I am (well, I am referring to his abilities to reverse-engineer various Higgs-related graphs from the LHC teams), and I believe that many computer-savvy experimenters – including some folks in BICEP2 – must be even better than Phil Gibbs.

Why are they good at it?

Well, it's because the reverse-engineering of the PDF files isn't that different from the work that experimenters have to do to study Nature Herself! Extracting some theory-laden parameters from the experimental data is actually very similar to the reverse engineering of a PDF file. If you think that the screen scraping is a dirty method, let me inform you that the experimental science is dirty, too. In fact, it's probably much dirtier than the screen scraping! Nature rarely provides us with the fundamental information about Herself as an optimized and readable Excel file. What a bič!

And the dirtier environment an experimenter may inhabit while extracting clean enough data, the better experimenter he or she is! So I think that this whole idea that experimenters should always work with clean data shows people's complete misunderstanding what experimental science is and what it means for an experimenter to be good.

Competition exists between high school kids, adults, as well as teams

The members of Planck, especially most of them who are not members of BICEP2 as well, may feel a bit jealous after the BICEP2 discovery. The great expectations connected with Planck only make their psychological situation worse. They're still trying hard and their papers seem very good. It's just the extra X-factor – the discovery – that has been missing so far.

Nevertheless, the attempts of the Planck folks not to help the competition to get too far ahead of them is completely understandable. A tennis player doesn't start to play soft against his opponent once the opponent acquires some new fans. There's no reason for such a change. So of course that the Planck folks want to spend more time with their own work than with their technical help to BICEP2.

Some people say things like this:
The refusal of the Planck team to share their raw data is disturbing. This is potentially one of the most important scientific discoveries of the last century. I assume, as scientists, we outgrow this behavior in high school or college.
But this is a complete myth. This myth is being used by some adults to intimidate teenagers. Teenagers are those who are doing stupid things such as competitive tactics. The reality is that the adults are doing these things as well – and these constants of the human behavior arguably become much more important for adults than they are for teenagers! Edward Pig Measure surprisingly understands these matters:
Just a point of psychology. Anybody who thinks that the same instincts, impulses and motivations that impel us in middle school aren’t still at work in middle age hasn’t been paying attention. Read some history, or better, Watson’s version of the Double Helix. If Watson and Crick hadn’t surreptitiously swiped Franklin’s X-Ray diffraction, they wouldn’t have won the Nobel. If Pauling had seen it, he would have collected his third Nobel Prize.
He didn't have to pick Watson and Crick as villains but it's sensible that he has picked some example. Of course that such deeds are not exclusively owned by teenagers. On the contrary. A lot is at stake. It's arguably one of the most important physics discoveries in many decades. So of course that people want to be the "exclusive discoverers" if that's possible, and it it's impossible, they want to "share the discovery with the first ones". These desires are often – if not always – incompatible with the desires of the competition.

All of them still (hopefully) share the desire to make science advance and to allow the whole mankind to learn new things. But that's simply not the only consideration they have when they're deciding about down-to-earth questions e.g. whether or not they will make a preliminary dataset available before the publication of a paper.

Everything is just fine in experimental cosmology and the combative spirits are nothing else than a testimony that something really interesting is taking place in that science. They are symptoms of a change in the atmosphere that I personally find highly refreshing.

So please shut up, anti-science and anti-competition sourballs.

Totally off-topic: Vladimir Putin hired me to switch the Russian Federation to the Czech alphabet and the Lumocyrill script since July 1st. You have one of the last chances to send your feedback about the current transliteration StringReplace rules. From July 1st on, unfortunately, those who commit a grammar mistake will have to be put down.


  1. Lubos, excellent meta-analysis of the race for scientific knowledge. Competition is the backbone of science - let the best ideas win and let the losers carp about politics. They seem petty in the process.

  2. Lubos, I thought I'd add one clarification regarding "instrument noise". The reason why Planck can measure dust and "dust fraction" much better along the galactic lines of sight isn't because instrument noise itself is any smaller along those lines of sight, but simply because both the temperature and polarisation signals (of dust) are so much larger there. I would presume that their instrument noise is roughly constant. Even if the polarisation *fraction* is small within those patches, the absolute amplitude of the polarisation will still be large, because so is the temperature (of dust).

    Equivalently, even if the true dust polarisation fraction along BICEP2's patch is large (the potentially worrying scenario for BICEP2), this doesn't *necessarily* mean Planck could resolve it because the absolute temperature (of the dust) there is also smaller.

    As I've written elsewhere, I'm not an expert on this (really only Planck are experts when it comes to their own instrument!), but seeing as Planck aren't willing to clarify speculation (which is fine), I thought I'd clarify what I can.

  3. I totally expected "What a bič!" there ;)

  4. Jane, you're totally right! It was a typo ;-) which has been fixed, thanks.

  5. I liked what (you) Gene wrote but not that he (you) neglected to commend Lubosh for his delightful if slightly less distinKt new English-enhancing way of spelling "severely nutritionally depleted by digestion former food-stuffs". :-)

  6. A nice euphemism, Peter! I am using this alternative spelling so that search engines, robots, and other Silicon Americans don't abandon their certainty that this blog is the ultimate role model of aristocratic, fucking diplomatic speech.

  7. What's wrong with standardized formats? Obviously the first priority of any researcher is to get results, but in the long run releasing raw data in machine-readable formats will clearly benefit everybody. If it is really so difficult to do, than funding agencies should kick in a little more money to hire some grad student to do it - certainly the cost of releasing this data would be a rounding error in the Planck budget.

    Not to say that the data should be released before the Planck team wants to release it - but it should eventually happen.

  8. "bullšit" Czech on that! Some days are made so much brighter by TRF.

  9. I think it is fair to say that Planck should be able to extract some primordial signal even from the somewhat more dusty regions because it can measure the signal over many frequencies and therefore it can distinguish dust from CMB.

    What I've understood is that, because Planck was (initially, when first planned) primarily a temperature experiment, it isn't as good at measuring polarisation as the likes of BICEP2, Keck, etc are (which were built more recently and don't have to go into space, etc). Therefore, even though Planck will measure a number of frequency channels, it won't measure any of them as well as BICEP2 measured its 150 GHz channel. I might be wrong, but I haven't seen anyone explicitly claim otherwise.

    So, Planck's virtue is that it sees the whole sky and can do so firstly because it looks at the whole sky and secondly because it can see behind the dust because it measures multiple frequencies. It's vice is that it won't measure any single line of sight quite as well. Therefore any B-mode detection it makes will be based on an average over the full sky. And therefore, it will also be able to measure the largest scales, which has another "bump" in the expected B-mode signal.

    And yes, my expectation was that Planck's final \pm was supposed to be comparable to BICEP2's current one.

  10. The very first sentence of your article contains a factual error, Lubos!

  11. My thinking is that the tension between Planck and BICEP2 is due to some peculiarity in reheating after inflation.
    Read something on arxiv that invoked axionic dark energy as the culprit. But the paper also had weird claims about varying neutrino masses and unconventional gravity.

  12. If you mean "CERN", I moved him to Paris just like he did move to Paris. ;-)

  13. There's nothing wrong with it, I also like to be able to download data from papers in formats that are readable etc.

    Except that it's a fantasy that some of them are really, de facto "standardized". People just use different ways to encode data because they think differently, they have been trained differently, and so on. And in many cases, lots of intermediate results are inevitably written in very idiosyncratic ways whose meaning is only understood by the person who does it. Even if someone is using some standardized software etc., the details how the data are represented in it may have some idiosyncrasies.

    To force everyone to learn some external rules for everyone and to enforce the rules for all input, intermediate, and final data in all papers means to triple the amount of work that goes to research - and in many cases, subtracting the time and energy from science.

    Planck can almost certainly release all such relevant data in understood formats. In Planck's case, it's not a financial or work-hard-related matter, it is a matter of their deliberate decision not to release certain things yet.

  14. Hi Lubos,

    Long time reader, big fan of your blog! I was wondering if you still do some physics research with your time? If not, do you find you have to always be setting yourself things to do for the intellectual challenge?

  15. I just remembered this relevant discussion between McAllister and one of BICEP2 collaborators in the comment section of this TRF post.

    I think it is helpful and pretty much summarizes the situation

  16. I would vote for simple CMB magnetism as the remnant of the splitting dark matter fractal big bang .

  17. Dear Lubos, I understand that you want to keep architects like me way from your site, but what was wrongwith my former mail?

  18. Thanks Lubos, very generous:

  19. Unrelatedly, have you seen Weinberg's latest preprint (quant-ph) ? Thoughts ?

  20. You say the BICEP2 claims remain robust?

    Today they seem to go from robust to BUST.

    The PDF has just come out for today's Princeton talk on BICEP2 foregrounds. "Towards an Understanding of Foregrounds in the BICEP2 Region"

    Read it and frown if you’re a BICEP2 advocate.

    Go to page 39 of the presentation. One corrected model shows 7 out of 9 BICEP2 observation points within the area predicted by a combination of dust foregrounds and gravitational lensing. The other corrected model
    also shows 7 out of 9 BICEP2 observation points within the area predicted by a combination of dust foregrounds and gravitational lensing.

    If you use the BICEP2/Keck combination for the observation points, then it’s basically 9 out of 9, with nothing that needs to be explained by cosmic inflation.

  21. Yup, I plan to write a post about it. Thanks for reminding me.

  22. Why don't you try to read e.g. the conclusions on page 71 instead of trying to draw far-reaching conclusions out of a technical graph in the middle of the talk that you have absolutely no chance to comprehend with your crackpot brain?

  23. Can BICEP2 people modify their equipment to take data at different frequencies or is that too difficult and very expensive?

  24. Dear Kashyap, BICEP2 *means* a collection sensors that operate at 150 GHz, a hardwired frequency.. BICEP(1) and Keck array (will) have both 100, 150 GHz, and BICEP3 will have 95 GHz.