Thursday, July 12, 2012

Should experimenters bet on their own experiments?

Tommaso Dorigo of CMS indirectly tells us that his ex-wife he recently divorced has a pretty good lawyer:
The SM Rules: Four Bets Won, $1200 Claimed
So one of the things he urgently needs is $1,200. He would like to declare victory in two or four of his 2006 bets. And no, they're not Higgs bets.

He describes the bets as follows:
So, to me this is pay day. If you recall, a full six years ago (September 2006 to be exact) I bet $1,000 with Gordon Watts (an experimentalist in ATLAS, who took the bet for $250) and Jacques Distler (a theorist from Texas university, who covered the remaining $750) that no new physics beyond the SM would be claimed by the ATLAS and CMS experiments upon analyzing 10/fb worth of data.
Distler and Watts have obviously taken bold positions.

Dorigo already wants to claim $750+$250 and twice $200 from Tony Smith – yes, that makes $1,200 in total (and he would lose about $1,950 if new physics were found) – for bets that are obviously too early to close. Whether new physics will be found in the first 10/fb worth of data remains to be seen. This is approximately the data that have been recorded by each detector so far but papers with conclusions about this dataset are music of a distant future.

If Italy doesn't make it, Galileo Galilei could finally return to banknotes again.

In fact, not only "many" searches using 10/fb are still waiting to see verdict; most of them are still waiting to be evaluated. Just look at the abstracts of the "full text" of ATLAS conference notes and CMS physics analysis summaries that most of the newest papers only use the 2011 data, i.e. 5/fb at 7 TeV. In fact, many recently published papers only use 2/fb or 1/fb and papers based on just 35/pb from 2010 are still being revealed these days!

The recent CMS paper with tantalizing hints of top partners which may have very well surpassed 3 sigma is another example of a paper that only uses 5/fb from 2011. An analogous paper using both 2011 and 2012 data – that may wait for 2013 to see the light of the world – may easily contain a 5-sigma discovery. It seems clear to me that Dorigo can't possibly claim victory even if he is right before 2013.

At any rate, Tommaso Dorigo is a member of the CMS statistics committee and the information about the fate of more than $3,000 (subtracting the positive and negative profit) has strengthened my previously already rather strong suspicion that the particular paper's conclusion that "we see no excess" is in fact fraudulent and I know the person behind it. Of course, I am eagerly waiting for papers on the same topic that use the 2012 data and/or that are prepared by ATLAS.

Cliff Harvey has expressed doubts whether experimenters should bet on the results of their very own experiments at all. It looks strange but in principle, I would probably not think that it is necessarily unethical. I still think that a good experimenter should be able to do his research independently of any financial pressures, including ones that are of completely personal nature, so he or she should be free to make the same bets as anyone else.

However, I admit that I may still be living outside reality and no such experimenters actually exist. In fact, it's plausible that even amounts that are smaller by an order of magnitude could encourage an average experimenter to change something in his or her paper. Of course, whether this "will" is enough depends on the status of an experimenter in his or her collaboration.

What do you think, is it OK for experimenters to make bets worth thousands of dollars whose resolution depends on the conclusions written by the same experimenters' papers? Or is it a conflict of interests that people should avoid?


  1. All professions inherently involve conflicts of interest. Doctors prescribing medical procedures, dentists recommending dental work that they will perform, lawyers counseling on lawsuits, etc. all have a financial stake in the matter at hand but they are expected to render judgements independent of their own financial interests.

    The very definition of a professional is one who does this and scientists are no exception. Of course experimentalists should be able to bet on outcomes but if their financial stake influences their work they are frauds. I don't know if Dorigo is a fraud but I am suspicious.

  2. Exactly, I totally agree. There are moral standards but there's also presumption of innocence that should hold for official decisions. Many activities overlap with each other so banning any combination of activities that *could* be done in a conflicting way just isn't the right solution. I have the same opinion about politicians etc.

    Politicians can't be forced to abandon friends in the commercial sector just because they're politicians. In fact, I would find it counterproductive because such contacts are among the things that make management more efficient, and it's true even for management by public officials.

    On the other hand, it's obvious that if someone would hypothetically distort research because of financial reasons, it's badly wrong. Just to be sure, I am not officially accusing Dorigo of anything and I don't claim to have any extra solid evidence that wasn't presented and everyone who thinks he's innocent should continue to think so. I am just revealing what uncertain probabilistic conclusions I am drawing from publicly available data.

  3. It seems analogous to a professional sportsman betting he is going to lose. I don't think it's ok at all. What if the people paying for CERN found out?

  4. On the one hand, it doesn't seem that ethically bad in cases like this where Tommaso is a small cog in a very big machine. The idea of these committees should be to suppress the possible effects of any particular person's bias. I would be much more worried if Tommaso was one of 2 or 3 or 4. However it still seems like it might be a good idea if the members of these collaborations would refuse these bets on principle. Also the ability of a person to maintain integrity would seem to be inversely proportional to the size of the bet relative to their own personal finances, as you're alluding to...

    Yesterday I was very foolish. I was trying to challenge on the same point about the lack of SUSY analyses from the 8 TeV dataset on another blog whose author rhymes with "hoit". I was informed that Gordon Kane's predictions in particular and string theory in general are already falsified, in addition to being unfalsifiable. And so on.... I've learned my lesson. Its just incredibly ironic that I cant get this guy to agree that the data from the 8 TeV run actually matters!

  5. Ah, now I probably see why CMS grayes out certain areas in specific result plots and applying TDs excuse ... :-P, LOL :-D

  6. I agree with you Cliff about TDs bet.

    And I always admire how bravely amazingly elegant you can fight zillions of trolls at the same time on other blog :-D

    Not sure it is always worth it ..? I at least have stoped clicking the two worst sourballs and I am much happier now ;-)


  7. The analogy seems very good - not finding anything new (beyond the Higgs) *is* (or would be) a kind of loss for CERN. This bet would surely look strange for an athlete in the eyes of his fans. But even with such bets, the athlete could still be winning the sport events and losing bets and the athlete may want to know whether such a mixture will be beneficial for him or not. ;-)

    But different occupations may still have different standards and logic and a scientist is not "working for his fans", I think. It's a much more independent an occupation.

  8. Dear Cliff, being falsified by the LHC data before they're actually evaluated and being eternally unfalsifiable is a mixture that only theories more powerful than logic may achieve. ;-)

    It's pretty incredible if there are people who can't see through this demagogy. After saying for decades that ST was constantly unfalsifiable, using not-fully-evaluated data with no truly new results as an argument that ST is already falsified - that's quite a jump!
    I understand your comment about the committees, it makes some sense, except that I don't believe that committees really work in this way. Individual biases are replaced by group biases, influence of committee opinionmakers, and group think. I don't believe you really get any improvement in the accuracy by transferring such responsibilities to committees.

  9. Thanks Dilaton. Sometimes I really cannot help myself, but you're right it does take a toll after a while. ;] I was just getting really tired of hearing people suggest Gordon Kane's work was built entirely around the 125 GeV Higgs rumor when I saw his presentation myself (thanks TRF) back in August.

    I resolved to stop visiting there a while ago. When the Higgs rumors leaked, I figured I cared about that sort of thing, but now that that business is over I might as well just remove it from my RSS reader again. I feel really silly now for ever believing there was any chance to hear a contrary opinion argued in a remotely serious way.

  10. Haha that sure is "more powerful than logic" alright. ;] Thats pretty poetic.

    I dont have enough experience with what committees on big experiments are like. Your assessment sounds pretty believable. Id like to think for such a prestigious project they can get a lot of very strong-willed and sharp people, but Im sure there must be a degree of group think that rules there as well.

    I dont know, but on the optimistic side, it seems like the worst such a person could do is delay the date when they acknowledge seeing something. Maybe Tommaso can just tip the scales enough that the excess isn't official until all of the 2012 run is analyzed!

    When I can finally begin to see the beautiful face of my dear Susy, I wont much care about these shenanigans. Everybody wins except Tommaso's counterparties, who should have been wise enough to challenge him. ;]

  11. Dear Lubos, Depends partly on the size of the bet obviously. Someone who bet their entire net worth should not be trusted. There is also the issue of "face" as they call it in China. People don't like to be wrong, some more than others. This would not be a factor for an athlete secretly betting against his own team naturally.

    Incidentally, what is the scientific status of these and similar observations about human psychology and sociology? A German sociologist of the 19th century whose name I forget once made a distinction between wissenschaft and verstehen (I think those were the words). Wissenschaft referred to scientific understanding; it was objective. Verstehen referred to "psychological" understanding based on putting ourselves in the place of the actors and imagining how they would feel and therefore act. Verstehen is the was we "understand" history and even economics. It's why they are called social sciences though I would argue the word "sciences" should be removed.