Friday, January 06, 2006

Bayesian probability II

See also a positive article about Bayesian inference...
Our fellow physics blogger Steve Luttrell believes the anthropic principle. That's not terribly unusual - roughly 20 percent of the physicists believe the same thing. But he even claims that the weak anthropic principle is science while insisting that the physics predictions should be derived from the first principles is philosophy. I am not kidding - see the end of this article.

The search for the first principles and insisting that we have a well-defined set of rules of the game that should be tested is what has defined natural science at least since the era of Galileo Galilei. I understand if someone is only interested in approximate laws that describe a certain class of phenomena. I understand that many people study so complex systems that they are forced to change their ideas about the system very frequently. I can even marginally understand if someone allows herself to "generalize" science in such a way that even the first principles are vague or environmental. But claiming that the standard rules of science - such as the assumption that there are some rules or first principles at all - are philosophy - it looks like a new level of madness to me; I apologize for this word. In some corners, the anthropic madness is evolving really quickly.

More recently, Steve Luttrell also advocated the Bayesian probability, something that we discussed recently. Frankly speaking, with all my respect to Steve Luttrell, I don't see a single argument about the essence of the question itself on Steve's page - just a lot of fog that "this is beyond my field of expertise" and all this ad hominem nonsense. So let me discuss this question at a comparable level of social science, too, because unfortunately there is no science to reply to, as far as I see.

Well, I exaggerate. I will increase the amount of essence in these comments at least by one order of magnitude compared to Steve Luttrell.

It is often said that there are two basic interpretations of probability: frequency probability (the ratio of events in a repeated experiment) and Bayesian probability (the amount of belief that a statement is correct). I am, much like an overwhelming majority of physicists, statisticians, and probability theorists (see the Wikipage about the frequency probability to verify my statement) convinced that it is only the frequency probability that has a well-defined quantitative meaning that can be studied by conventional scientific methods.

This is why Luttrell's attempts to paint my opinions as fringe opinions are completely crazy.

The Bayesian probability cannot be even defined without vague words like "belief", "plausibility", and so forth. It's just not a well-defined quantitative concept because it cannot be determined or measured with ever higher degree of accuracy. Such a kind of probability is not predicted by meaningful physical theories of physics either. The predictions of quantum mechanics are always about the frequentist probabilities. I often give numbers for "subjective probabilities", but I would never argue that such estimates are scientific. They're not scientific much like other beliefs are not scientific. Counting the angels is not scientific either.

If I say that the U.S. president is a distant relative of a particular governor with probability 20%, that's a scientifically meaningless statement with a meaningless number. The correct answer is, of course, either 0% or 100%, and the precise value 20% only reflects my psychological feelings and irrelevant subjective methods that I used to estimate the degree of "plausibility" of that statement. It's not objective, it's not measurable, it's not science. Other people can tell you a different estimate (70%) and there is no way to decide scientifically whose number is correct. If the ongoing research of this question becomes advanced enough, it may turn out that both of us were wrong, of course, because the answer is either 0% or 100%. No intermediate value can be thought of as "permanent truth". Someone may have been closer to the truth; but no one was right.

These statements about the nature of probability are completely essential for a physicist, a statistician, or a probability theorist to do her work because probabilities appear everywhere. According to quantum mechanics, everything we can predict are probabilities. I can't imagine how a physicist could study her subject if the meaning of the word "probability" were beyond her field of expertise.

What Steve Luttrell's description reminds me are the postmodern, lit crit misinterpretations of the meaning of science. They say: You know, science is just another religious ritual. The way how we do science is flexible. The natural laws are like a sneak, and we are re-adjusting them by interacting with the real world. There is no objective truth. All statements in science are just a result of social influences, discrimination and political power, including the value of PI. ;-) ... I don't need to repeat all this rubbish, you can read it in Alan Sokal's paper and all other, less perfect papers that promote the very same nonsense with a serious expression in their face. ;-)

This is not how this Universe works. The laws of Nature are like very rigid stones that are placed on other stones. Our environment is changing but the laws of Nature have always been the same. And we are learning these eternal laws with ever increasing accuracy and completeness. The more we understand how the laws of Nature work, the more they look like a hierarchy of stones placed on other stones. The less we understand them, the more they look like a shaky, flexible and sleeky snake that changes every time we try to touch it.

Robust vs. liquid science

It's pure and simple. Those who understand science prefer a quantitative, frequency-based interpretation of the probability, the existence of first principles whose validity should be tested by experiments, and other rigid scientific rules to search for the truth. Those who misunderstand and dislike science, prefer to "measure" the probability by the amount of beliefs and think about the laws of Nature as about an evolving sleeky snake.

The last category will never tell you what is exactly their prediction. They will never tell you whether their "global warming" predicts increasing or decreasing mass of ice in Antarctica. They will just try to derive a political capital from whatever answer they can get. They don't believe that there is anything such as the truth. They only believe in their personal interests. (I must sound like Ehud Barak when he explains that Arafat - and maybe most of Arabs - did not know the concept of the truth; the detector of lies would not work there.)

Positive words on Bayesian probability

While I say that the Bayesian probability is not science, it can be useful for various purposes. We may use it as a key to figure out how to behave when we don't know something - how to act in the state of ignorance. Search engines use the Bayesian formulae to organize their databases and sort the web pages. But what I want to mention is something different:

Quantitative Occam's razor

This observation makes the situation a bit more complicated because Steve Luttrell believes both Bayesian probability as well as the anthropic principle with its Rube Goldberg machines. However, the Bayesian probability offers tools to justify - and indeed, quantify - Occam's razor, a gadget to kill the anthropic principle. Simpler theories are more likely to be true.

What do I mean? A model with many parameters is almost guaranteed to fit the data better than a model with a few parameters. Which of them is more likely to be true? The Bayesian probability paradigms imply that it is the simpler model, even if its predictions are a little bit less accurate than the predictions of the model with many parameters. It is because the prior probability that a complicated model - before its parameters are determined - predicts particular results is small exactly because the probability distribution is smeared over the whole parameter space of the possible results.

I can't tell you exactly what the distribution is because the Bayesian probability is not quite quantitative. But it is enough to demonstrate a qualitative point.

The prior probability distribution for a simpler model is more concentrated. The Bayesian "level of confidence" formulae tell us to include the factor that "punishes" the model for being complicated. If you do the calculation in the Bayesian way, I believe that you will conclude that it is exactly as reasonable to believe that a model with 10^{120} vacua predicts the correct cosmological constant after one vacuum is selected (anthropically), as it is to assume that one particular single-vacuum model will miraculously give the right cosmological constant with the relative accuracy of 10^{-120}. In both cases, the psychological probability that the model is correct will be 10^{-120}. This is how the Bayesian probability works, and I fully endorse this particular conclusion.

Note that when we also consider the probability that the models predict the correct Standard Model, not just the cosmological constant, a particular heterotic model will become more likely than the whole landscape of type IIB flux vacua. This is again what the Bayesian reasoning implies. The anthropic people will deny this conclusion - except for Steve Luttrell who will struggle in between two contradictory ideas.

However, my inability to convince everyone else that the huge landscape does not make the cosmological constant more "Bayesian-probable" or "believable" or "reasonable" is evidence that the Bayesian probability is not really science. It is philosophy. And while I would agree with some of its intuitive conclusions, it's just not possible to argue that these conclusions are scientific. If we don't know how the vacuum selection really works in the world around us, we cannot calculate meaningful "probabilities" that one answer is correct. Maybe if I pretended that the Bayesian probability is science, I could convince a few hundred people that any vacuum from a large set is unlikely...


  1. Lubos:

    I thought it's 10^500 vacuas, not 10^120 vacuas. When were they able to shrink the number of possible vacuas from 10^500 to 10^120? That was a huge improvement of 10^380 times and if it was true it would have been all over the news by now.

  2. The Bayesian view of probability is not really a minority view: it is shared by many statisticians, and many learning theorists. In fact, the debate between frequentist and Bayesian views is just that; a debate. Not onesided at all.

    It is not really accurate to argue that the Bayesian view of probability is not well defined. If you look at the Cox axioms that quantify belief, you'll see that probability comes out as a natural way of measuring belief, and in fact there is no other way of doing so. Jaynes MaxEnt approach to statistical thermodynamics might be debated, but is a plausible approach to defining basic laws of thermodynamics from Bayesian principles.

    This is not to say that the physical phenomenon itself is Bayesian; that is a different kind of discussion. But Bayesian reasoning itself is cast-iron mathematics, and it is only at the level of "what do these numbers mean" are there debates.

  3. Dear Quantoken,

    no, the number of vacua has not decreased. It's more likely that a consensus would emerge that there are (discretely) infinitely many. The argument is unaffected because the argument is that the Bayesian inference would be unaffected by the number of the vacua in the case of ignorance.

    Dear Suresh,

    yes, I understand that various axioms have been defined. But they can't justify that the inference done using certain methods is scientific in nature. If you think that it is, solve of our uncertainties about the relevance of the landscape.

    Does the seemingly randomly small value of the vacuum energy imply that we should accept the existence of 10^{500} or so other Universes and things being random, or is it better to believe that one of the "very special" vacua is selected and the vacuum energy is cancelled by a process that looks like a bit of miracle according to our present knowledge?

    We will all be amazed if you give us a convincing argument. But I am afraid that until we know how something *really* works, all estimates and interferences are nothing else than random guesswork.

    If you doubt that the frequentist interference is "by far" the dominant viewpoint among the statisticians and physicists, you may try to edit the error in Wikipedia.

  4. Lubos, it is actually frequentist probabilities that are ill defined. For a short and cogent explanation of this, see quant-ph/0408058. For a mathematical proof that frequencies cannot be used to consistently define quantum probabilities, see quant-ph/0409144. For one way to think about quantum probabilities as Bayesian, see quant-ph/0501009 (and references therein for other ways).