Thursday, April 12, 2012

Rating agency for research

Pat Michaels discusses an embarrassing paper published in Nature which claims that man-made aerosol emissions are behind the Atlantic Multidecadal Oscillation. He thinks that Nature is on the edge of losing its last droplets of credibility.

Dictating what is allowed to be researched

Another kind of silly article was printed in Nature:
Rating research risk
It's a shortened version of an August 2011 preprint; see also an August 2010 preprint on "portfolios".

Abraham Loeb, the chair of Harvard's astronomy department, proposes that a group of grad students will run a server that will rate the health of various disciplines of physics – and probably science in general – and their ratings would be used by students deciding about their fields, by granting agencies, by governments, and so on. I don't really know whether it's an April fool's joke but let me assume it is not.

It's difficult to sensibly evaluate the current state of research and it is even more difficult to predict the future. When people – for example, students who are just choosing their specialization – are trying to obtain the information about these matters, they are using a diverse variety of sources and methods to make up their mind. Some of them use more sophisticated methods, others use less sophisticated methods, many of them decide by chance and it's important that chance can have a say, too.

However, if one wants to pay attention to some authorities, it's rather likely and sensible that he or she will pick senior scientists whose job is secured (which reduces the potential for the conflict of interests) and who have made some achievements, too (which increases the chance that their recommendation is actually helpful). Loeb thinks that this is a bad algorithm because senior scientists are just like the rating agencies that gave an "AAA" rating to some unsafe papers before the financial crisis. Are they really analogous?

Needless to say, he doesn't have a glimpse of evidence that there is any analogy between senior physicists and top rating agencies. It's just a purely demagogic sleight of hand designed to say something bad about the senior physicists. Of course, if he wants to suggest that second-class scientists are better advisers than the first-rate scientists, trash talk is probably the only argument he can offer.

As you may expect, I disagree with his opinions about the financial crisis, too. The major rating agencies may have made an error in two in their work 4 years ago. They're not infallible. But it's silly to look for infallible oracles. There aren't any. One has to look for the least fallible ones. So what one ultimately has to do is to compare various sources of information, assign them with weights, and make up his mind. Despite the errors, it's still very reasonable that people pay more attention to Standard & Poor's, Fitch, and Moody's than they listen to a particular random pundit. In the same way, top achieved physicists who still remain active are more listened to than some other random people or grad students. It's completely reasonable. And there's no "guaranteed equality" between the Nobel prizes, either. Some of them simply say things that make sense to many more people than the proclamations by others. And the senior physicists also make their guesses about the younger colleagues who could be listened to and they sometimes inherit a part of the senior physicists' brand.

I think that a group of grad students with a website whose main "credentials" or "achievements" include the fact that they have volunteered for a project invented by an astronomer who doesn't look too credible himself are unlikely to be viewed as natural authorities by the sane people, at least in comparison with generic physics Nobel prize winners and their natural peers. And I am afraid that Loeb realizes this fact very well. He doesn't expect to convince sensible people who think rationally and carefully. Instead, he wants his gang of grad students to directly influence the people in the government.

I am not suggesting that senior physicists are always right; I am just saying that it's statistically more sensible to listen to similar people than to a random group of grad students equipped with a server running a random algorithm that was predetermined to produce a certain kind of results and that hasn't been tested when it comes to the quality of the resulting recommendations.

Fine. But let's look at Loeb's algorithm, anyway. He doesn't say much but he says enough to be sure that his proposed algorithm isn't terribly intelligent:
What if the physicists could call on a ratings agency, not unlike a lender would do before deciding whether to offer credit? I am advocating the creation of a website that is operated by graduate students and that will use various measures of publicly available data (such as the number of newly funded experiments, research grants, publications and faculty jobs) to gauge the future returns of various research frontiers.
Fine, so a group of graduate students working for Mr Loeb will be deciding on the funding for the world's science. I've already mentioned this "cute" basic goal. How will they do it? The paragraph above suggests that they will recommend those research directions that recently received lots of funding and grants and hired many people. So what the result of the advise will be, assuming that Loeb's kids really become dictators in the global science? The recently overfunded projects will probably get even more funding, more grants, and hire many more people, exponentially converging towards a singularity.

(We've seen such bubbles within science, too, but I don't want to talk about global warming all the time.)

I can't believe he doesn't see how stupid his suggestions are. He tries to create an analogy between the research funding and stock markets and it's great. Except that he obviously misunderstands the markets, too. A sensible trader usually won't buy stocks that were recently overbought and whose price has been increasing more quickly than the price of other things. A sensible investor, one of the Warren Buffer type, will obviously try to do exactly the opposite. He will try to invest into things that seem undervalued right now.

Stocks that are cheap right now may also be cheap for sensible, objective reasons. For this reason, buying "anything that went down" isn't a promising strategy, either. However, to figure out whether it is so, one must actually know something about the "beef" behind the stocks. The "beef" is necessary for predictions. A group of grad students that just uses the data from the past can't make usefully accurate or reliable predictions of the future price, whether we are talking about stocks or about the successes of a scientific subdiscipline.

(One could also talk about fast trading. It could have analogies in research funding, too. Sometimes it's useful to find two analogous investments in which one may be seen to be underpriced and buy into it. Relative comparisons of comparable things are of course the key in decisions about good investments and good grants and hires in science, too. Comparing incommensurable things is inevitably less useful and more subjective.)

Loeb is trying to hype the observational cosmology in these sentences:
The study of the cosmic microwave background provides an example of how theory and data can generate opportunities for young scientists. As soon as NASA's Cosmic Background Explorer satellite reported conclusive evidence for the cosmic microwave background temperature fluctuations across the sky in 1992, the subsequent experimental work generated many opportunities for young theorists and observers who joined this field.
Around 1992, this process began and it peaked around 1998 and a few years that followed. Amazingly enough, cosmology – a discipline that used to be a cousin of astrology and boasted order-of-magnitude errors and philosophical babbling – has become another genuinely high-precision science, pretty much on par with particle physics. But all these sentences are pretty much about the history right now. Those things belong to the past. It doesn't seem we're expecting similarly important discoveries to the discovery of the positive cosmological constant in the next decade or so. There are a few experiments we cherish but of course that it's reasonable to expect that none of them will bring us anything really new. Maybe it will but it is not terribly likely.

So by these very propositions, Loeb shows the kind of bubble-like thinking. He tries to infinitely inflate a bubble and the only justification he has involves some episode in the history of science in which this discipline was growing. But that's just not a sensible way to decide about the allocation of funding and human brains in the future!

Of course, he also offers a negative example:
By contrast, a hypothesis such as string theory, which attempts to unify quantum mechanics with Albert Einstein's general theory of relativity, has so far not been tested critically by experimental data, even over a time span equivalent to a physicist's career.
It has always been unlikely that string theory – a theory whose new phenomena mainly influence the Planck scale physics – would become a heavily experimental science anytime soon. More than 100 years ago, Max Planck calculated how high the Planck energy was. Since that time, we knew it would be extremely hard to reach those energies experimentally. I've never believed it was "more likely than not" that string theory would be tested by direct experiments; suggestions that the young string theorists were "betrayed" because there's no "experimental string theory" are just lies because no sensible person has ever sketched "experimental string theory" as a field booming in the near future. It has always looked comparably silly to various supernatural and loop-quantum-gravity-like hacks who think that they may create amazingly new God-like physical phenomena while cooking their lunch in the kitchen. The world doesn't work like that. Of course that people study string theory primarily for mathematical reasons. If you can't understand how one may think and calculate physical problems for decades without doing "new experiments", just think about string theory as a branch of maths. However, it's still a kind of maths that agrees with everything we know about the physical world. The experiments that have already been done many and many years ago, and actually just a small subset of them, are actually constraining enough so that mathematically skillful people may extract lots of conclusions out of them by "pure thought", if you wish. I understand that people who are outsiders in string theory can't possibly internalize this insight but it doesn't make it untrue.

Surprisingly enough, it was realized that I could have been wrong about the detachment from "direct" experiments, after all. At least in some sense and under certain conditions that could be true but could be untrue. String theory has led to various insights that may link the low-energy physics to its characteristic phenomena, whether I talk about the large and warped extra dimensions scenarios (whose likelihood is about 1%, in my opinion) or the AdS/CFT quantum gravity description of low-energy physics (whose experimental tests may be included to quantum gravity or not, depending on terminological preferences). Still, experiments in a near future are not why some of the talented young people become string theorists. They study string theory because they "rediscover" it and find out that it's the deepest theory that the mankind has ever found and the only theory that may reconcile our observations of all kinds of natural phenomena we know and unify them in a fully mathematically consistent framework. The fact that string theory reduces to QFTs or GR at long distances we may test experimentally – so there's no way to experimentally distinguish the pieces – isn't a disadvantage of string theory or disadvantage of QFTs or GR; it's just a demonstrable fact that only a denier of facts might deny or question.

While Mr Loeb is a completely uninformed layman when it comes to the real reasons why people study string theory, he surely seems power-thirsty and dreams about controlling a group of grad students that is deciding about the funding for disciplines for which his own IQ is 40 points too low. Of course, the bureaucratization and the ability of similar Mr Loebs to overtake the power over important issues that they're completely unable to understand is the main threat of the government as a concept.

The free markets – including the free markets that deal with the information, such as the rating agencies – are not perfect but they have the tools needed for self-regulation. The rating agencies only got famous and powerful because their consumers found their recommendations helpful in many cases. There exist mechanisms that evaluate the quality of the ratings, at least in some imperfect way, and those mechanisms are forcing the rating agencies to do a good job. On the other hand, Loeb's graduate students and their PC seem to lack this feedback completely.

I find this feedback critical in all these considerations – while Mr Loeb's main point is to deny it. Those who make the decisions about the allocation of funding must be accountable for their decisions. The results of their decisions must influence on whether or not these people maintain their ability to influence things. If these feedbacks disappear, the quality of their recommendations becomes free to plummet.

There have been many discussions about the support for risky research vs well-established research. I think that many people have prejudices of both signs. Some people just wouldn't allow science to try risky, experimental things at all. Of course, without such things, big advances would become (and would have become) impossible. On the other hand, some people want to throw billions of dollars for risky research everywhere. That's of course wrong as well. Science wouldn't be science if most of the papers and hypotheses in them were wrong. The amount of speculation that may be wrong has to be kept limited. There is probably an "ideal" distribution of the resources and human minds to research of various degrees of risk but it doesn't make much sense to think about the "optimum curve" or about social engineering tricks how to reproduce it. While I think that too many people are being pushed to becoming "original thinkers" (instead of "workers in science with a well-defined job description") although they're not up to this job – much like too many people are being pushed to earn a college degree they shouldn't possess in a natural state of affairs – it's better to leave those things to the free market of ideas.

Various people have different tolerable risk levels that may also depend on the kind of questions they're asking. Consequently, they may do different decisions. There's nothing wrong about it. The fact that people have different ideas about the value of a stock is a necessary condition for the trade with stocks, after all. It's totally healthy and, in fact, critically important that people have different opinions. However, there still has to be some natural selection.

I want to say one more thing. Loeb formulates his proposals as a "universal cure" for the questions about the allocation of grants. However, it's equally clear what his actual motivation is. As an astronomer, he probably feels that people are suggesting that his work is low-brow and intellectually undemanding relatively to fields like string theory. And make no doubts about it, Mr Loeb, it is the case. He doesn't like it so he would love to suppress those fields that are making him realize that he isn't at the top of physics.

But that would be very bad if petty people with similar totally narrow-minded, egotist motives and biases were allowed to decide about bigger issues.

In practice, different disciplines may have different waves of excitement. The excitement about string theory from the first superstring revolution (1984-85) and the second superstring revolution (1995-98) hasn't been repeated yet so one could say that string theory has been living in a slow business-as-usual era with incremental progress for 15 years or so. But the very same thing is really true for experimental cosmology, too. And most subdisciplines of physics have never gone through any wave of progress that would be as spectacular as the superstring revolutions.

One could present a story about every subdiscipline. For example, experimental particle physics lives in a new era of boom right now – something that has replaced the 20-year-long thirst for the experimental data. But this boom may be short-lived, too. The LHC works extremely well and many "natural theorists" are excited about the new observations but it's still true that the LHC has only found the 125 GeV Higgs boson, to be confirmed in this year. Everything is compatible with the possibility that the Higgs boson is the only thing that will be seen at the LHC (although I think that it's more likely than not that the LHC will see something else by 2020, too). If the Higgs boson is the only new thing to be seen, it could actually be a bad idea to start a grad school in experimental particle physics right now because the remaining tasks to be done with the colliders may be simple enough for a smaller number of physicists. There are lots of unknowns and some reasoning about these matters is more natural than another reasoning.

A fraction of the smartest young people on the planet inevitably ends up learning string theory because they are intellectually ambitious and they simply want to get to the deepest level of our understanding of Nature and string theory is the deepest level we have today, something they realize sooner rather than later if they're really smart. There are seven billion people on this planet so if a few hundred people seriously search for a "theory of everything" (and get less than $1 billion out of the world's $60 trillion annual GDP), it's not such a huge sacrifice, is it? Many of them probably know many more things about the sociology and funding related to string theory and other disciplines than what they could learn from Mr Loeb's website run by grad students. They also know something about the history of the subject, including the level of excitement in different years. More importantly, these young soon-to-be string theorists also know that they shouldn't pay much attention to these superficial sociological criteria at all because only superficial people are obsessed with this kind of matters.

The idiotic populist propaganda about the need for every scientific discipline to find implications for the observations we can make now or soon, if not for Joe Sixpack's practical life, may be attractive for the dumb people, the kind of human garbage that visits Shmoit-like websites and reads Shwolin-like trash books. But it's exactly this obsession with superficial arguments that distinguishes them from young soon-to-be string theorists; it's a reason why they're not beautiful minds but rather loud, obnoxious, and superficial crackpots. String theorists are clearly not doing string theory because it would be important for their practical lives. It's critical for the health of the cultural civilization to guarantee that much lower quality people such as Mr Loeb won't be able to dictate smarter people what they are allowed to think about. The young researchers may have a better idea about the state of the disciplines than a group of grad students focusing on a P.R. website. But even more importantly, they are going to make their decision about the investment of years of their lives and they may be expected to be much more impartial and objective than an astronomer with a website.

And that's the memo.

1 comment:

  1. Rating Agency doesn’t disclose their methodologies, watch below…