Saturday, October 05, 2013

Superdeterminism: the ultimate conspiracy theory

The author of the blog post "Testing conspiracy theories" has informed her readers that she was just writing a paper when she was pregnant and awaiting her daughters in hospital in late 2010. The nurses were hoping that she was writing something like Harry Potter IX: Employed As Chair Umpire In the Empires of Vampires but they didn't realize that she was actually writing something much spookier.

It finally appeared as a paper called Testing super-deterministic hidden variables theories which is just a code for Trying to Beat the Big Brother and Her Ultimate Conspiracy. She tries to claim that she has a test to figure out whether we live in the so-called "superdeterministic world" which is a world in which a supervisor restricts what experimenters may think and what buttons they may press. She tries to present the idea of superdeterminism in a would-be boring, technical, innocent way,
All that superdeterminism means is that a state cannot be prepared independently of the detector settings.
but that just hides the fact that the sentence above contradicts the basic assumptions behind science and any rational thinking about the world. So what is this superdeterminism?

Bell's theorem shows that local realist theories can't predict the sufficiently strong and diverse correlations that quantum mechanics predicts and that experiments have confirmed.

Quantum mechanics circumvents these restrictions because it is not a realist theory; it postulates that there are no observables that have objectively well-defined values prior to the measurements. Most people claiming to work in the "foundations of quantum mechanics" are incapable of even daring to consider the possibility that the fundamental theory isn't realist. So they replace it by a "realist model" that is non-local, however. It's not the real thing and one can easily bring crushing scientific evidence that it's the not the real thing but when it comes to these philosophically sensitive issues, these people just don't give a damn about the scientific evidence.

John Bell wanted to prove (the invalid claim) that quantum mechanics was producing wrong predictions so he decided to isolate what these predictions could look like. His theorem was a form of an inequality – which was later followed by similar inequalities – for a quantity that actually had a value. But the "different flavor" that quantum mechanics boasts and no local realist theories may reproduce was already known to EPR (Einstein-Podolsky-Rosen) since the 1930s and the true fathers of quantum mechanics implicitly knew about the difference since the mid 1920s.

The canonical example is isomorphic to the following one.

Create a photon pair, e.g. by annihilating an electron-positron pair in the singlet state. We know that one of the photons goes in the \(+z\) direction while the other goes to the \(-z\) direction. The photons may have different polarizations. The original angular momentum \(J_z=0\) is conserved which means that the angular momenta of the two final photons must be opposite to each other. That means that either both photons are left-handed, or both of them are right-handed.

(The values of handedness are the same which translates to the opposite signs of \(J_z\) because \(J_z\) is the handedness multiplied by the sign of \(p_z\) and the photons move in the opposite directions.)

One may also argue that the final state must have the same parity \(P=\pm 1\) as the initial state. This condition determines the relative phase between the two terms in the final state vector below. I don't want to explain the detailed association of the signs because the probability of an odd number of sign errors would be comparable to 50 percent. Instead, let us assume that the final state is\[

\ket\psi = \frac{ \ket{LL}+\ket{RR} }{\sqrt{2}}

\] The sequences \(LL,RR\) describe the left/right-handed polarization of the two photons. The first letter describes the photon going in the \(+z\) direction, the second letter describes the photon going in the \(-z\) direction. You see that the probabilities are 50% for each polarization of the up-moving photon but the second photon is guaranteed to have the same circular polarization.

Using simple linear algebra, we may translate the final state above to the basis of linear polarizations. Use the convention\[

\ket L &= \frac{ \ket{x}+i\ket{y} }{\sqrt{2}}\\
\ket R &= \frac{ \ket{x}-i\ket{y} }{\sqrt{2}}

\] for both photons' polarizations. Substitute these formulae (which simply say that a circular polarization is produced as a superposition of two linear ones whose phase shift is \(\pm \pi/2\)) to the expression for \(\ket\psi\) and use the distributive law to re-express \(\ket\psi\) using \(\ket{xx}\), \(\ket{xy}\), \(\ket{yx}\), \(\ket{yy}\).

The numerical factors \((1/\sqrt{2})\times(1/\sqrt{2})\) from the definitions of \(\ket L\) and \(\ket R\) combine into \(1/2\) which cancels because all the terms from the distributive law appear twice (either with the same sign or with the opposite, canceling signs). The \(\ket{xx}\) vectors will appear twice and double; the same is true for \(\ket{yy}\) which will however have the negative sign from \(i^2=-1\). On the other hand, the mixed basis vectors \(\ket{xy}\) and \(\ket{yx}\) will cancel between \(\ket{LL}\) and \(\ket{RR}\). The final state is therefore\[

\ket\psi = \frac{ \ket{xx}-\ket{yy} }{\sqrt{2}}

\] Note that if the relative sign between \(\ket{RR}\) and \(\ket{LL}\) were negative (which would result from the opposite parity of the state), the rule would be the other way around: \(\ket{xx}\) and \(\ket{yy}\) would cancel while \(\ket{xy}\) and \(\ket{yx}\) would survive. That's actually the more relevant case in experiments but let me work with the form of \(\ket\psi\) described in the displayed equations. The other choice only differs by replacing the "perfect correlation" with the "perfect anticorrelation" but the logic is isomorphic.

You see that the mixed terms canceled even when we used the linear polarizations – with respect to two mutually orthogonal axes in the \(xy\)-plane and it's common sense that we would get the same result even if these two axes were rotated by an arbitrary angle. So the two photons' circular polarizations were guaranteed to be the same. But if we measure the linear polarizations with respect to any axis in the \(xy\)-plane, the two photons will perfectly mimic each other, too!

This seemed unacceptable to Einstein and pals who were imagining that the photons "objectively have some information about the polarization" when they're already flying away from one another. In their opinion, at most one of the unentangled (tensor-product-structured) possibilities such as \(\ket{RR}\) or \(\ket{LL}\) or \(\ket{xx}\) or \(\ket{yy}\) or \(\ket{xy}\) or \(\ket{yx}\) etc. had to be objectively realized as soon the photons left their shared birthplace. But if the state were actually \(\ket{RR}\), it would be clear that all four possibilities \(\ket{xx}\), \(\ket{xy}\), \(\ket{yx}\), \(\ket{yy}\) would be possible with the probability 25 percent. The same comment would apply if the initial state were \(\ket{LL}\); or if it were e.g. \(\ket{xx}\) and the two experimenters decided to measure the circular polarizations at the end. All four combinations would be possible, at 25 percent odds.

Needless to say, Einstein and his pals were just wrong. Quantum mechanics is flawless and the perfect correlation was experimentally confirmed. John Bell's expectation was wrong as well. All these four men, EPR+B, helped to reinforce the claim that classical physics of any sort has been permanently and irreversibly ruled out, quantum mechanics is right, and its postulates are indisputable and universally valid.

The people who like Nature the way She is – and She is quantum – like this new kind of behavior, it's cool and more sexy than what any of the classical theories could ever display. The people who are offended by Nature's quantum sexiness are trying to invent all sorts of lies and delusions why the smooth pink boobs that Nature is showing from all directions are just an optical illusion caused by an underlying classical burqa that Nature obediently wears.

But Ladies and mostly Gentlemen, Nature doesn't give a damn about your religious dogmas, burqas, philosophical and emotional preferences, holes in your scientific integrity, and intellectual limitations. The picture above isn't an appropriate picture for the 21st century anymore. She is quantum. She thinks that realism is only OK for Her classical caricatures that underestimate and, as George W. Bush would say, misunderestimate Her genuine sexiness. So no non-locality is needed; what's needed is the proposition-based, probability-laden framework that quantum mechanics indisputably demands. Locality is guaranteed to hold due to the rules of special relativity. All the individuals trying to force Nature into the burqa/straitjacket of Bohmian, GRW, "really splitting" many worlds, and many similar paradigms are bound to remain nothing more than testimonies of their proponents' dogmatism, lack of rationality, and absence of appreciation of Nature's true sexiness.

Now, let us return to the main topic advertised in the title, superdeterminism.

The "creative" forced-will loophole

Superdeterminism is a clever (at the level of science-fiction novels) alternative proposal that disrespects quantum mechanics but wants to preserve "some sort of local realism". Superdeterminism is the bold proclamation that Nature employs special KGB agents who make sure that the experimenters won't try to measure something that could be politically incorrect – something that could reveal that the perfect correlations in all kinds of polarizations aren't actually there even though it looks to us that the correlation is always perfect. (There are no serious scientific papers on superdeterminism so the original blogger's paper, which is completely illogical, as we will discuss momentarily, may still be counted as one of the most logical and scientific publications on this proposal.)

Why does it help? Well, it's easy. The photons may be objectively created in a state like \(\ket{RR}\) without any entanglement, as Einstein was imagining. How is it guaranteed that the experimenters never measure the combinations \(\ket{xy}\) and \(\ket{yx}\) of the polarizations that we have calculated to cancel? In superdeterminism, the answer is that in this case, the KGB agents prevent the experimenters from wanting to measure the linear polarizations. When the initial state is \(\ket{RR}\), the agents make sure that the experimenters only measure the circular polarizations so they always confirm the perfect correlation between the photons.

I am actually improving the situation a little; the average defenders of superdeterminism have never formulated the actual trick by which superdeterminism is supposed to "solve the problem" this clearly.

The experimenters may think that they have freely chosen what they ultimately decided to measure about their photon but this free will is an illusion, according to superdeterminism. The agents have penetrated into their brains and guaranteed that the experimenters only want what they are allowed to want. ;-) Note that the agents need to correlate what is happening in the experimenters' brains before the measurements with the properties of the measured objects. This is actually a much more brutal violation of locality – violation of the very independence of geometrically separated objects – than any other violation of locality we may be adding to a previously local theory.

Can science work like that?

The main thing you should notice is that if there are KGB agents working like that, science cannot be trusted. Indeed, it's the ultimate conspiracy because the scientists' brains may be controlled so that they always avoid learning the true essence of Nature and what is important about Her. In his 2010 book Dance of the Photons, one of the four or so quantum-foundations workers who don't suffer from a severe psychopathology wrote:
[W]e always implicitly assume the freedom of the experimentalist... This fundamental assumption is essential to doing science. If this were not true, then, I suggest, it would make no sense at all to ask nature questions in an experiment, since then nature could determine what our questions are, and that could guide our questions such that we arrive at a false picture of nature.
If an agent is controlling what a scientist wants to ask and learn according to a rule the scientist can't really decode, science is indeed impossible. This is why the author's claim that she is "testing" superdeterministic theories is an oxymoron at the very basic level. One can't really be scientifically testing any of these things because the main feature of superdeterminism is that it breaks the relationship between the results of any experiments and the information about "what is actually going on". That's why you may reliably discard the author's paper right after you finish reading its title.

A recommendation both for men and women (because I don't want to discriminate against anyone): if you're lying pregnant in the hospital, don't write physics papers because they are likely to be even stinkier bullshit than what you write most of the time. And make no doubt about it: none of the author's 10 most recent papers is correct.

But even if one tries to overlook this basic violation of logic, I don't really think that the superdeterministic theories have been shown to have any special new capability to deal with the "problem" – to succeed where the local realist theories failed and to reproduce the victories of quantum mechanics whose refusal to wear the realist burqa is what drives people to think about crackpot theories like the pilot wave theory or the hardcore conspiracies like superdeterminism.

While it's true that superdeterminism rejects some assumptions about the usual local realist theories, and by doing so, it guarantees that Bell's theorem doesn't apply to these superdeterministic theories, it's a very little step towards the scientific goal which is to accurately explain and predict the observed data. The refusal of a common assumption is a negative step. But one actually needs to do many positive steps and be "lucky" in all of them to produce successful theories.

Let me offer you an analogy. The Universe is measured to be 13.8 billion years old. That's a problem for the Young Earth interpretations of the Bible (=local realist theories) because they imply that the age is less than 10,000 years. Now, someone proposes a clever solution. Every time Moses or Jesus speaks, everything else in the world stops for a million of years while the Universe gets older (=superdeterminism). (I could have invented a better stupidity but I didn't want to spend an hour with such a task.) So it doesn't necessarily follow that the Universe is younger than 10,000 years anymore.

That's great but if you want to produce a new theory of cosmology, it is not enough to invent a new crazy reason why the old arguments that the Universe is younger than 10,000 years are circumvented. You actually need to produce a model that agrees with the age of 13.8 billion years and thousands of other pieces of the empirical data. The theory "speaking Moses/Jesus hibernate the Universe" and "superdeterminism" just can't do these things, so it is utterly ludicrous to suggest that they're competitors of cosmology and quantum mechanics (respectively) at this moment.

The theory of superdeterminism would have to make a huge progress to becoming a promising platform to explain the observations. An honest scientist should think about all likely hypotheses but I just don't think that this is one. I would bet millions that no progress in making a quantitative contact with the empirical data will be made on the basis of superdeterminism in the next 50 years. That's a reason why I won't spend any time by doing research of this remote possibility.

Are arguments against superdeterminism more than a wishful thinking?

Above, I quoted Zeilinger who said that science would be impossible if superdeterminism were right. But you might object – and I have actually repeatedly objected (in other contexts) – that Nature doesn't really care whether we can do science or whether it's easy for us. That's a valid objection. However, if you assume that science doesn't really work, you shouldn't pretend that you're the anointed exception who is able to accumulate scientific evidence in a world where it is impossible to accumulate scientific evidence.

Moreover, I think that there's overwhelming evidence that science does work and the scientific progress leads us to an increasingly meaningful picture of Nature's inner workings. If this apparent fact were an illusion, I would consider myself incapable of seeing through such illusions. I don't even understand what it could mean to make progress in understanding Nature in a world where the apparently increasing understanding of Nature may be an illusion. How would I decide that progress was made and so on?

These are the reasons why I think it's OK to assume that no cosmic-scale conspiracies exist and build science on this assumption. Even if we're controlled by a cosmic-scale conspiracy, it's interesting enough to learn what the conspirators want us to conclude – a good enough replacement for the probably hopeless wishes to learn something about the truth that only the conspirators know.

But there are much more specific, technical, and perhaps more rock-solid considerations. I think that special relativity simply does show that the experimenter, his apparatus, and the physical object studied by them may be found in pretty much independent states – and all combinations of these states must be considered possible. Indeed, the unentangled (tensor-factorized) state of the object of interest and the measurement apparatus is the assumption we make everywhere in science; it's not necessarily "perfectly precise" in all situations but it's good enough because the "correlations between the apparatus and the object of interest are unlikely to affect the measurements". Moreover, the superdeterministic loophole is made even less plausible because of the complexity of the brain processes. If we agree that the experimenter presses a button that is chosen by very complicated brain processes which depend on many details, the aforementioned KGB agent would have to be able to revert all the algorithms that take place in the experimenter's brain before he decides to press the "measure linear polarization" button.

It sounds totally inadequate to associate KGB agents with these supernatural algorithmic powers etc. if the only purpose of this hypothesis is to circumvent a conclusion that seems to directly follow from the experiments, namely that Nature guarantees the perfect correlation in the two photons' polarizations, whatever "kind" of polarization we talk about. What I really mean is that the loophole may be clever from the viewpoint of writers of B-class science fiction novels but the loophole doesn't really seem to have anything to do with the "problem" that they're trying to solve. So all these links between superdeterminism and the non-classical abilities of the quantum entanglement seem irrational to me.

There may be some conspiracy, perhaps some pattern in the randomness of the random data produced by quantum mechanics, but if that's so, the patterns are vastly more subtle than anything that any of these superdeterministic and other "pundits" has ever proposed (at least so far). I think that at the level of our current approximate description of Nature, all the additional constraints and conspiracies that have been proposed may be safely excluded.

My comments may sound mysterious – I apparently left a loophole open. What do I mean? Well, I mean ideas such as those in the paper by Papadodimas and Raju who show that too many measurements done inside the black hole may invalidate the apparent independence of the black hole interior region from the rest. This is the kind of "almost conspiracies" that are probably relevant for a theory of quantum gravity: if you ask too many questions, the apparent independence of objects that seemed independent may collapse. But the number of questions you may have to ask to reach this point may be really, really large, like the area of your spacetime region expressed in the Planck units (and the Planck area is really, really small). If someone says that the low-energy bits in a human brain aren't allowed to have independent values from the two qubits of the two measured photons, he is contradicting some results that have already been established. At least I am totally convinced it is the case.


  1. The other thing, is that it seems to me that such a theory couldn't possibly be compatible with clustering. Arbitrarily far experiments must be prohibited from falling out of causal contact since usually these people like to pretend that locality is still manifest.

  2. Fascinating. Seems to be channelling James Blish's classic SF story 'beep' in which an agency receives messages from the future and then has the job of ensuring that the events described in the message actually come to pass. They always do, of course, without any intervention by the agency. Now that's determinism!

  3. In english, what are you saying Haelfix?

    It amuses me that anyone tries to battle superdeterminism with LOGICAL stuff - are you just fucking retarded?

  4. The idea that something/somebody controls what kind of measurements experimenters want to take is not science at all, what should that entitiy be? An allmighty, omniscient God ...?

    Concerning conspiracy theories I and Young Earth crackpots, they can make Earth look younger than it is themself:

    Talking such breathtaking nonsense as (I heared) they do, everything around them loses its breath in the truest sense of the word, and gets frozen ... :-D

  5. Dear Lubos,
    thank you for yet another extremely clear and sensible article. Just one remark: Instead of poor Sabine H. you could have attacked Gerard 't Hooft who is playing with these superdeterminism ideas as part of his cellular automates program since they are the only way around Bell theorem while maintaining local realism.

  6. Wow, I honestly did not know thatit was the same kind of superdeterminism. That brings the weirdness of these 't Hooft's papers to a new level.

  7. What do you mean you did not know it? Cellular automata are surely local and real. And according to 't Hooft they can mimic quantum mechanics. So if 't Hooft is right he has a superdeterministic theory in his hands.I think he himself is even disturbed by this.

  8. I am sure that none of his models actually reproduces the results of QM. To say the least, they contradict all tests of the superposition postulate i.e. linearity. I just didn't understand that he was willing to go in the conspiracy theory direction. Written on a tablet...

  9. I'm saying super determinism likely violates another essential principle of QFT.. Namely the cluster decomposition principle, and likely is incompatible with the existence of an smatrix.

  10. Dear Haelfix, you're surely right. But it's one of the points of superdeterminism to claim that they can reproduce the observations completely differently, by ignoring all these principles that seem essential in proper QM.

    Of course that this desire is just a wishful thinking. There are no indications that the apparently cluster-decomposition-respecting predictions of QM may be shared by a completely different theory that refuses these principles.

  11. I guess we would have to ask 't Hooft himself how exactly he puts it. Maybe in a guest blog? If these model fail on such a basic level like the superposition postulate then of course the whole discussion is moot.

  12. Dear Mikael, if one adopts the linearity principle for the wave function and just a few experimental facts, the rest of quantum mechanics follows.

    Of course that people like 't Hooft who propose something fundamentally different than quantum mechanics have to reject the superposition postulate and assume/dream that they may mimic the tests of the postulates by some completely different means. Except that this hope is hopeless.

    Of course that his guest blog would be welcome but I am not going to beg him and I guess he wouldn't volunteer, anyway.

  13. I apologise for my language last night - I rarely become such an animal.

  14. So few people ever plausibly consider these classes of theories, that I feel it ought to be possible to make the constraints on them considerably sharper. The reason I brought up clustering, was that it was my impression that these people don't claim to reject the principles of qm, that like the Bohmians, they claim that the laws of qm are what they are, but it's only the interpretation that changes. Namely in this case that all experiments and observers arrange themselves just so, so as to produce the fiction that qm is valid. Still clustering seems impossible within the assumptions of super determinism...

    Anyway,, these people often try to preserve locality and causality (imagining again a deterministic classical world governed by relativity). But I feel like this is probably inconsistent. For it to work, you need to imagine a point in the distant past where all automata (or whatever) can fix their initial conditions just so. But what if there was no such point, that instead you have something more like classical cosmology. Eg the topology of an infinite flat plane with causal horizons. Then you are faced with the ultimate horizon problem, without obvious mechanisms like inflation to cure. I suspect that they would be forced to accept non locality in that case. Anyway, an amusing exercise.

  15. super-deterministic theory from Manuel S Morales

    " Spin States of Selection: Predetermined Variables of ‘bit’ "

  16. The words making up this comment seem to be quite uncorrelated, I have no idea what you are talking about ... :-D

    Did you write this with some random text generator?

  17. Speaking of the devil:-)

  18. Well, Sabine was at a disadvantage when she wrote this---I am sure that her pregnancy-enhanced hormones distorted her reason, and when this is added to the background mental fog produced by female hormones in any case, confusion is sure to follow :)

    BTW, have you picked up a case with a keyboard for your tablet? It would make typing much faster and would convert your tablet to a laptop.
    They are quite inexpensive.

  19. Dear Lubos,
    I was trying to get Prof. 't Hooft to write a guest block here but I think with your above remark I do not need to further consider this. In terms of philosphy I agree with you that the whole superdetermism road looks extremely unpleasant. But I thought with your master mind you could do something which I can't which is to look at the mathematical models of Prof. 't Hooft for a little while to confirm that they are empty of any interesting mathematical content as we all expect. This may not look to be the best use of your time but a reasonable price to pay to have Prof. 't Hooft guestblogging here.

  20. Well, he's not dumb - he's invoking the entire history of the universe to support his model - I think it's an ok argument- but I don't agree or see the point compared to just accepting randomness.

  21. I think that "shut up and let me think" is better than "shut up and calculate" Even if quantum mechanics is not ultimately deterministic (people try to make QM more compatible with their innate perception of reality which is natural but doomed to failure) scientists should be encouraged to explore every direction so I think there should be more research in the foundations of QM !

  22. "Shut up and let me think" sounds better than "shut up and calculate" to idiotic trolls like you because you're not capable of calculating anything and you may always describe the hopelessly stinking shit that you're shitting all over the Internet as a result of "thinking".

  23. Are you telling me that you're smarter than Gerard t'thooft and those guys in the perimeter institute who do research in the foundations of QM ? I'm sorry I don't buy that idea . Maybe "shut up and calculate" is better for you than "shut up and let me think" because you're not capable of thinking ? You should have learnt from your career that mindless calculations are not the way to do physics !

  24. shut up and stop upvoting your own posts.

  25. I think the obnoxious crackpot "bill" already got banned once but signed in again as another user with the same name (it is possible to do this: I did this once by accident using first a Google and then a Facebook account). To stop that sort of thing you have to black-list IP addresses and even this won't work if someone has access to many :-(

  26. LOL , old man . I just sign out from disqus , write my comment then register with a e-mail address and post . Also , I don't need to change IP . In fact , I can crack this whole shitty blog in a matter of hours and make it useless but I don't need to do so .

  27. shut up and stop adding so many guest upvotes, too .

  28. Dear Lubos,

    I think to get a very good feeling for the ideas of Prof. 't Hooft it is enough to read chapter 3 of the following paper:

    This is just two pages of elementary mathematics. It would be great to get your take on this.

  29. I looked at it and it seemed like a mathematical game without much physics. I tried to read the latest paper of prof 't Hooft about cellular automata and strings but I could not understand it, but my abilities are understandably 10 levels below prof. 't Hooft.
    Prof 't Hooft should imho try to replicate some simple quantum systems at first - for example the two slit experiment, the polarizer paradox described in the Feynman lectures or here
    Simple bound potentials and the observed quantization of energy levels.
    It shouldnt be too difficult to model the cellular automata on a computer and produce some images that would replicate the quantum probabilities.

  30. Dear Mephisto,
    this was also my impression, some apparently content free mathematical game and at the end the desired result appears. I thought the same that it would be perhaps more enlightning to see the double slit in this language. But I fear it will leave us just as clueless. The "ontological states" or at least some of them may have some very special spatial correlation built in so that can produce the interference pattern.If we cannot grasp the physics in the simplest example then perhaps it is hopeless to go for more complex examples. But all in all I share your proposal to Prof. 't Hooft to try to get more intuition into this.

  31. Lubos,
    Who are the other three or so people working in foundations who have avoided psychopathology?

  32. Anton Zeilinger has to be the luckiest person in the universe of quantum mechanics and maybe even in this quantum mechanical universe. He sits right in the middle of the sweet spot where theory -- thinking about QM entanglement/teleportation and the implications -- overlaps with practice -- designing and building experiments. No matter how smart someone is, if they never get their hands dirty and delve into the nitty gritty they are missing something. The doing provides an extra impetus to the thinking, the thinking then triggers another round of doing, and so on in a virtuous circle.

    An architect might be capable of designing a large and complex building without ever having set foot on a construction site, but she would be poorer for missing the experience. The shortcuts, the troubleshooting, the ingenious solutions compelled by practical necessity... all that would be foreign to her, and the buildings designed by her would tend to be boring.

    Now let's see if this gets a rise out of the boss... he's been rather quiet in the comments section lately :P