Thursday, June 28, 2012

Why we should combine Higgs searches across experiments

Aidan Randle-Conde of the U.S. LHC blogs argues
Why we shouldn’t combine Higgs searches across experiments.
On July 4th, the CMS and ATLAS will present their current state of the Higgs searches. There is no reliable information on whether or not one of these experiments (or both) will claim the discovery of the Higgs boson – at least a 5-sigma signal.

The data collected by each experiment may be calculated to be approximately enough for a 5-sigma discovery right now so we will have to be surprised whether they have been "lucky" or not.

Some rumors coming from extremely unreliable places say that they have been lucky enough. Chances are not far from 50% that the rumors are right. However, if you believe that the 4.5-sigma combined excesses from 2011 show that the Higgs boson is there, it's guaranteed that there will be enough evidence to discover the Higgs boson in the combined charts that include the collisions recorded both inside the ATLAS detector as well as the CMS detector. I am confident that the combined significance level will be above 6 if not 7 sigma.

Aidan finds it inappropriate to combine the data. He argues as follows:
The next obvious step would be to combine the results from the two experiments and count the sigma. Despite being an obvious next step, this is the worst thing we could do at the moment. The Higgs field was postulated nearly 50 years ago, the LHC was proposed about 30 years ago, the experiments have been in design and development for about 20 years, and we’ve been taking data for about 18 months. Rushing to get a result a few weeks early is an act of impatience and frustration, and we should resist the temptation to get an answer now. Providing good quality physics results is more important than getting an answer we want.
From these paragraphs, assuming that Aidan is a rank-and-file member who must obey his bosses, we are learning that ATLAS and CMS will not combine their charts on July 4th (although the two Tevatron experiments did exactly that when they officially discovered the top quark). However, I think that his explanations why the two experiments' data shouldn't be combined are pure emotions. The quality isn't lowered in any way if one combines the results.

It's actually the duty – the only right attitude – for any scientist outside CMS and ATLAS to combine the results. Why? Simply because a fair and impartial scientist must take all the existing evidence for a theory or against a theory into account. In total, the LHC has accumulated about 20 inverse femtobarns of proton-proton collisions at 7 or 8 TeV – that translates to 1.4 quadrillion collisions – in 2011 and 2012. These collisions may be evaluated by two different teams, for various reasons I will discuss momentarily. But all of them are valuable data and no impartial scientist outside these collaborations may justify any kind of cherry-picking.

Also, it's a red herring to say that we "want" the Higgs. I don't "want" the Higgs. I may be much happier if the LHC discovered the Motl fermion instead of the Higgs boson. I just see clear data showing that the Higgs boson is there and this certainty will probably become even more extreme on July 4th.

It's simply a fact that if ATLAS ended up with a 4.5 sigma signal and CMS would have the same, we could still combine them in quadrature to get a 6-sigma or 7-sigma excess. One may reconstruct the number of events in each channel – counting both the collisions inside ATLAS and inside CMS – and see that the evidence against the null hypothesis (a Standard Model without any nearby Higgs boson) has reached a 6-sigma confidence level (or higher).

It would be nothing else than a pure denial of scientific facts if someone would reject the claim that the higgsless hypothesis has been excluded with the certainty required by particle physics. One may offer lots of moralizing words but all of them are irrational gibberish. Including all the known evidence is the only honest and accurate enough way to give the best possible answer at this moment to the question whether the experiments show that there is a Higgs boson near 125 GeV.
If we combine measurements from two different experiments we end up losing the vital crosscheck. The best way to proceed is two wait a few more weeks until both experiments can produce a 5 sigma discovery and see if these results agree.
This "crosscheck" comment is also irrational because if both experiments will show a near-5-sigma excess in similar channels (just imagine that none of them will get "there" separately), then the crosscheck will have been done and its result will have been "passed". There is no other "crosscheck" one needs to do when he determines that two experiments are saying pretty much the same thing about the existence of the Higgs boson. Waiting for another "crosscheck" and not knowing what this "crosscheck" is exactly supposed to be – except for seeing that the experiments agree within the error margins which will have been achieved – is nothing else than obstructionism, a nonsensical justification of the denial of one-half of the data.

Finally, I want to discuss the "experimental duality cult":
The reason we have two experiments at the LHC looking for the Higgs boson is because if one experiment makes a discovery then the other experiment can confirm or refute the discovery. This is why we have both D0 and CDF, both Belle and BaBar, both ATLAS and CMS, both UA2 and UA1 (where in the interest of fairness the orders the names are chosen at random.) Usually these pairs of experiments are neck and neck on any discovery or measurement, so when one experiment sees an effect but its counterpart doesn’t then it’s likely due to a problem with the analysis. Finding such a problem does not indicate poor scientific practice or incompetence, in fact it’s part of the scientific method to catch these little hiccups.
This is just plain bullshit. There is nothing essential for the scientific method and its quality about the maintaining of such redundandies. The actual reason why there are two major detectors at the LHC, ATLAS and CMS, is that the price tag of the LHC ring was nearly $10 billion while one big detector only cost $1 billion or so. So it would be waste of space and an imbalanced investment if the $10 billion ring only harbored one detector.

Instead, people decided to build two detectors, to double the number of collisions. The remaining choice was whether the detectors would follow the same design (which would be a bit cheaper) or different designs. It just became fashionable sometime in the 1980s to have detectors of different designs on the same accelerator but there's nothing "vital" about this diversity, either. If the LHC hosted two ATLAS detectors or two CMS detectors instead of one ATLAS and one CMS, we would know pretty much the same thing today, with pretty much the same reliability and data quality.

A Time-Lapse Video Shows Creation of Giant Higgs Boson Mural: on the side of the ATLAS control room (2010)

It's a pure convention to build detectors according to two designs; and it's a pure convention to have two teams of physicists working on them. One could also have three or 47 detectors and 470 teams like that. But one could also live with one. In fact, most of the key discoveries in physics (and science) occurred without any such redundancy. There was no Rutherford II who would be discovering the nucleus; there was no Eddington II who would travel to a different forest to observe the 1919 solar eclipse; I could continue like that.

Before the 1980s, even particle physics worked without such redundancies. When SLAC was discovering quarks and QCD in the late 1960s (deep inelastic scattering), they weren't employing two competing groups of experimenters. And it just worked, too. One could argue that the redundancy of the experiments is just another manifestation of the reduced efficiency of the contemporary institutionalized science, an example showing how the money is wasted in the public sector, a demonstration that the Big Science also tends to behave in the same way as the Big Government and it tries to employ as many people as possible even though they're not necessary. The Big Government and bureaucracy – and the Big Science is unfortunately similar in many cases – is able to reduce the efficiency of the process to an arbitrarily low level.

All these people are employed – and someone responsible for that may think that he or she is "nice" because of that – but they inevitably end up somewhat frustrated because even when they discover the Higgs boson, and a similarly vital particle is only discovered a few times in a century, each of the ATLAS and CMS member will only "possess" 1/6,000 of such a discovery. She or he is an irrelevant wheel in an unnecessarily large machine. Everyone knows that. The members know that, too. It's a price for someone's decision to give "jobs" to a maximum number of physicists. It's not the only price, of course.

Someone wants to double the number of the people who are getting salaries from the public sector? No problem, why don't we just double the number of otherwise equivalent experiments?

Concerning the inefficiencies that automatically come together with the Big Government, one could also argue that there exists a deliberate effort to delay and slow down the discoveries – it's implicit even in Aidan's text – so that the experimenters don't look "useless" in the following year or years (assuming that nothing else will be discovered which is a huge assumption). But that's an unacceptable policy, too. If discoveries may be made quickly, it's just wrong to artificially slow them down. It's a bad testimony about the efficiency of the system if some people manifestly think that they will be rewarded if their work is extra slow.

Experiments may check each other but is this possibility to check the other folks' work really worth the money? I would say it's not. CDF claimed crazy things and bumps and D0 later rejected them. But if D0 hadn't done it, sensible people would have thought that CDF made an error, anyway (I have surely recognized all the bogus CDF papers as soon as they published them). And these claims would be pretty much experimentally rejected by the LHC by today, anyway.

Incidentally, I also strongly disagree with Aidan's claim that "Finding such a problem does not indicate poor scientific practice or incompetence, in fact it’s part of the scientific method to catch these little hiccups." The numerous erroneous claims that the CDF has made in recent years – "little hiccups" is an over-the-edge euphemism – surely do show that their research work was poor in recent years. If the redundancy and competition is good for anything, it's good exactly for knowing that D0 has done a better job in the 145 GeV bump and some bizarre top-quark-related claims than the CDF did. In other words, CDF did a poor job and if the errors boiled down to human errors, then the disproved observations by the CDF show someone's incompetence, too. The scientific method indeed expects people to search for errors in their and other people's work; but once such errors are found, it does mean something about the competence and all functional scientific institutions will inevitably reflect this competence or incompetence in hiring and funding decisions! This is a part of the separation of the wheat from the chaff that is so essential in science! If D0 is doing things in a better way than the CDF, it's desirable to strengthen "things and people working in the D0 way" and suppress "things and people working in the CDF way". Saying something else is a denial of the good management, denial of natural selection, denial of almost all mechanisms that improve the society in the long run.

One may also say that the redundancy is neither sufficient for a reliable crosscheck nor useful when the crosscheck fails. Even if there are two experiments, they may make an error which is especially likely if all the experimenters are affected by the same lore and habits (or the detectors are plagued by similar design flaws). And on the contrary, if one experiment debunks another, we really don't know who is right. So strictly speaking, when D0 rejected the CDF claims about the 145 GeV Wjj bump, we should have said that it was a conflict and we don't know who is right so having two detectors didn't really help us to learn the truth. To decide who is probably right, we need some "theoretical expectations", anyway. D0 is right because it's much more likely for someone (CDF) to make an error that deviates from the Standard Model in a random way – there are many such potential errors – than for someone else (D0) to make the right unique error that is needed to exactly cancel Nature's deviation from the Standard Model. But with similar "theoretical expectations", I didn't really need D0 to know that CDF was wrong, anyway.

The ongoing discussion about the "advantages of the competition" is much more general. Some people want to social-engineer competition in the markets. Whenever a company is successful and large or dominant in some business, they want to hurt it, cut it, destroy it, subsidize its competitors, and so on. The idea is that such an intervention improves the products and consumers will benefit etc. However, there is no valid general rule of this sort. In some cases, one company is simply doing a better job than others and that's why it's dominant. This dominant company knows the best recipe to transform an invested dollar to a profit in the future. So any different allocation than the allocation that makes it dominant is misallocation; by hurting the dominant company, it also reduces the profitability and efficiency of the system as a whole.

What makes capitalism superior isn't "forced competition" that is as diverse as possible; what makes capitalism superior is the "freedom for the competition to appear". But if the conditions are such that there is no competition and things can be produced more efficiently by one company which makes everyone satisfied, then this is how it should be (at least temporarily, but the market decides how much long this situation may last).

Imagine that you fund N otherwise (nearly) equivalent experiments and the question is for what positive value of N, you get the optimum ratio of "value of scientific information you get from the experiments" over "the price of the experiments". A simple question. Aidan clearly wants to suggest – with all the comments about "good quality" and "crosschecks" – that the best value is N=2. But that's rubbish, of course. The law of diminishing marginal returns implies that the optimum value of N is N=1. What the additional experiment gives you is inevitably less valuable than what the first one could have given you. In most cases, the experiments simply overlap so they must share the "scientific profit".

To summarize, the redundancy and diversity of the experiments is grossly overrated. By convention, people chose to evaluate the LHC data in two separate teams. It's a perfectly legitimate although not necessarily economically optimal design – like the design of a car with 6 or 8 cylinders. But this randomly chosen design shouldn't be worshiped as a key ingredience of the scientific method. It's certainly not one.

While ATLAS and CMS folks are supposed to work in isolation in order to check each other at the very end of the research, it's still true that the LHC has accumulated a certain amount of experimental data – 1.4 quadrillion collisions – and throwing away some of the data would mean to deny the experimental evidence. ATLAS and CMS employees may have internal rules not to look at other experimenters' data; that's just fine as an internal etiquette. But impartial scientists who are not constrained by similar internal rules just can't afford to ignore one-half of the collisions; or to pretend that the two sets of the collisions don't affect the answers to the same questions (i.e. pretend that the data can't be combined).

They definitely do. Whenever a physicist wants to know what the LHC is telling us today, he must combine all the collisions.

And that's the memo.


  1. I look forward to the new combination plots Phil will probably produce, even though Aidan oviously does not like them ... :-D

    I agree with you Lumo about the redundancy etc, but I thought the 4 LHC detectors (even ATLAS and CMS) have a bit different scientific goals and are built to detect or measure different things a bit differently, such that non of them is 100% redundant ...?

  2. Dear Dilaton, ATLAS and CMS have different design but they have exactly the same goals – to detect everything at the frontier of physics.

    There are various claimed differences in their advantages – CMS could be a bit better in detecting muons and other things may be better in ATLAS – but as far as I know, these comparisons are hypotheticals at this point. The papers from ATLAS and CMS could have been written by people from two copies of a CMS as well. Or two ATLASes.

    LHCb and ALICE are of course different and also much cheaper. They can't detect "everything". ALICE is only for collisions of heavy ions (Pb). LHCb only measures bottom-quark related physics, especially phenomena relevant for the CP-violation. It couldn't discover the Higgs and similar things. It doesn't even encircle the collision point from all sides!

    But the CMS and ATLAS have almost exactly the same goals, despite some design differences.

  3. Dilaton, and others, let me offer you a fun analogy for the CMS/ATLAS duality. In Czechia, we have some carmakers aside from the Volkswagen-owned Škoda.

    One of them is in Kolín – the Czech Köln ;-) – and it is joint company TPCA which stands for Toyota Citroen Peugeot Automobile.

    It produces three cars with very different logos, from totally different continents. Their names are Toyota Aygo, Peugeot 107, and Citroen C1. Cute brands, very different, you know. Toyota is Japanese, Peugeot has a lion, and so on.

    But just for the sake of completeness, look what the three models look like:

    Aside from the logos and names, they're 3 pretty much identical cars! ;-) Now, this is not wasteful because it doesn't cost much to produce 3 different logos and the work is proportional to the number of individual cars, anyway. But in ATLAS and CMS, one is literally paying a twice as high number of physicists than what would be enough to process all the data.

    Instead of 6,000, one could have just 3,000 physicists who would collect and evaluate data from two ATLAS detectors (or two CMS detectors, whatever). We would have exactly the same signals today as with ATLAS+CMS combined but the combination would already be done and we would save 3,000 annual salaries every year, among overhead and other things. It's about a personal taste which setup would look better. I would probably prefer the 3,000 physicists in the doubled ATLAS, not only because of the economy but also because of the efficiency and speed with which all the data are processed. But even if someone makes a different choice, it's preposterous to suggest that it's "necessary" to have diverse detectors at the LHC.

  4. Of course I agree with you that the data should be combined for the final significance.

    It is though good to see the individual analysis, as for example in fig 1 for the mass of the W. Note the four different LEP experiments and their differences.

    Now your argument : there should be one experiment ATLAS running twice by the same 3000 people does not really hold on economy. A lot of the time the 3000 people are baby sitting the instruments and the economy in man power would be small. ( building two same sky scrapers economizes on the architects, but the manpower to build them is by necessity twice than for one) .The crucial analysis is carried out by tens of people, not hundreds, and those are the ones one could economize on.

    The reason why SLAC did not have many experiments was because similar data were being run with neutrino beams and muon beams etc, so the physics was explored from different angles at that time.

    What if you did not have D0? Theorists would be going to town with the
    various results of CDF. One cannot be voting on the trust of the
    results. Experimental data has to be confirmed or refuted with
    experimental data, not gedanken experiments.

    That is the traditional experimental method for physics, to repeat an experiment and confirm or refute a result. Why, we even run labs teaching students that. Tradition was the reason for the four different experiments at LEP , a parallel processing instead of waiting serially for the results.

    That the experiments are large and cumbersome and expensive and needing too many physicists is true. It may also be true that there is an overproduction of hep physicists as a result and the sociological implications have to be worked through. I am curious to see what will happen with the ILC .

  5. i don't know much about the two projects but are ATLAS and CMS competing because those that funded them wanted to have competing teams? are they competing because they use two machines that work in a different way trying to get similar results? or are they competing because the funding is from different councils/countries etc?

  6. I would say the first one is the only "more true than untrue" description: the pair of competing teams was social-engineered.

    The machines are working in slightly different ways but the difference isn't too important; the differences in their work evaluating the data doesn't go too much beyond a slightly different jargon for some hypothetical new particles and for some filters etc.; it would be very simple for people from one team to switch to the other - many have done it. Both parts of the LHC experimenters are funded from the same sources.

  7. And, how credible are the Tevatron results for Higgs decaying to b quarks? Because LHC had still too much background noise in these channels and they are critical to decide if the Higgs is a full SM Higgs or simply an electroweak model Higgs.

  8. I wonder what these guys would say if one experiment had an excess and the other had a deficit. I bet combining the plots would be completely fine in that case. In fact, in any situation where a combination didn't make the difference between five sigma and not five sigma, they would probably say a combination is fine.

    Having two experiments as independent as possible is of course good for sanity checking results and guarding against mistakes, but unless they show different enough results that you suspect something's wrong, using all the data you have is just the obvious thing to do.

  9. The "crosscheck" boils down to the fact that 2 x 5 sigma evidence combines to ~7 sigma, so he's just arbitrarily raising the bar.
    Especially "the worst thing we could do" is over the top. They are just waiting because they can and it was the plan to collect enough data for individual discovery. If the LHC could for some unlikely reason not produce any more data this year, they would surely combine the results, instead of waiting until 2014.

  10. Exactly: they are delaying the discovery because they can - because they have no *genuine* competition outside the LHC. And yes, insisting on a crosscheck of 2 x 5 sigma means to arbitrarily raise the bar to 7 sigma.

  11. Sorry, Anna, I don't believe for a second that most of the time, the 6,000 people are babysitting the instruments. If the working hours are 1/4 of the average day, you are really saying that more than 1/2 of 6,000/4 i.e. at least 750 CMS+ATLAS members are babysitting the instruments at each moment, right? This is preposterous and you know that.

    Most of the work they do or are supposed to do is contributing to the analyses in the papers - which are redundant.

  12. This is why to cross check, instead of just combining data: ;-)

  13. Aidan's attitude is typical from people who love to withhold information and who are never enthusiastic about anything. They are the type of people I want to roar at and scratch :-[]

  14. if there is an overproduction of hep physicists what should have been done with the money? use the money for people and machines to create more experiments or fund other areas of physics? if the money could go to other areas, plasma and quantum information among other areas would be useful to have more people.

  15. I personally think there is just overproduction of routine experimental physicists. Lots of areas in theoretical HEP etc. seem underpopulated to me. Of course, there are many other fields of human activity that may be overpopulated or underpopulated. I think it is nonsensical to think about particular pairs of sides to transfer from one to another. If people move, they move to various places.

  16. ow ! it is a nice information related website. I already bookmark this website & visit regularly.
    I have also a can also find something new information from my site