Sunday, September 08, 2019

Carroll vs quantum mechanics in NYT

If I could have made a bet that Sean Carroll was going to write an anti-quantum op-ed for The New York Times, I would have bet one million dollars.

It seemed clear to me because in recent years, The New York Times was increasingly connected with liars and demagogues of certain types and the quality of the content was decreasing (fortunately along with the circulation). Too bad, I didn't get an offer.

At any rate, the op-ed arrived yesterday:
Even Physicists Don’t Understand Quantum Mechanics
Worse, they don’t seem to want to understand it.
Right. The only problem is that the title and the subtitle applies to the incompetent, ideologically driven "physicists" who wrote about 28 anti-quantum-mechanics popular books for the most moronic layreaders just in recent 3 years. Actual physicists do understand quantum mechanics rather well – it has really been understood for over 90 years – and they are using it as rock-solid foundations to discover increasingly amazing things about the physical world. The op-ed clearly tries to rotate this basic truth by 180 degrees.

The first paragraph is a traditional, now mandatory, quote spreading the meme that quantum mechanics is "mysterious". The author needs to spit at the most important theory of science at the very beginning and prepare it for an even worse treatment. It's a bizarre example of chutzpah for Carroll to rely on Feynman's quote – Feynman made all these quotes in order to argue that the likes of Carroll are unimaginative, prejudiced, and wrong because there are no (classical) "pictures".

Then:
What’s surprising is that physicists seem to be O.K. with not understanding the most important theory they have.
They (the real physicists) are OK because they actually see that quantum mechanics is a complete, unified, coherent, and mathematically super-elegant scheme for providing us with (probabilistic) answers to all questions that seem to have any empirical sense.

As 19th century philosophers were already able to roughly guess, it's important for science that all of our legitimate knowledge about Nature, its laws, and its state ultimately boils down to the sensory experience. All other types of knowledge are suspect and potentially illegitimate.

Quantum mechanics has made this vague picture of science quantitative because it became a new framework for physics that directly calculates the probabilities of some sensory experience in the future – some outcomes of measurements (which are defined by a Hermitian linear operator and that have an eigenvalue of the operator as a possible result) – from the sensory experience in the past (transformed into a pure state or a mixed state).

All these probabilities are calculated as squared absolute values of some complex "probability amplitudes" (that is called Born's rule) and these "probability amplitudes" are calculated from the linear algebra of complex Hilbert spaces. The result of every sensory experience (measurement) which brings a new knowledge is mathematically translated to an update of the state vector or the density matrix (because those quantitatively encode everything that the observer knows about the world) – that is the "collapse of the wave function" that is only "more mysterious" than a step in Bayesian inference because it uses the more "elementary" complex amplitudes instead of the probabilities themselves.

Carroll admits that the discoverers of QM were briliant and QM works but
What we don’t do is claim to understand quantum mechanics.
That statement is easily proven wrong by a counterexample. I do claim that I understand it – and I do claim that many physicists from the present and from the history understood it, too. At any rate, what amounts to an "understanding" that isn't guaranteed by the complete mastery is a foggy metaphysical question. Being able to articulate the basic axioms, provide evidence why they have to be true, apply them, and verify that they work really is an understanding according to a scientist.

What I cannot do – and what we cannot do – is to visualize "what is going on" in the classical sense; see Heisenberg's Nobel Lecture that likes to repeat that "he realized that we have to throw away the desire to visualize" and that's why he really deserved this extraordinarily important Nobel Prize. And we have quite some overwhelmingly evidence that we can't do it for the same reason why we can't build a perpetual motion machine: it is impossible to do so, for very fundamental reasons. The world just doesn't obey any classical laws! Many of the "nobody understands quantum mechanics" quote really meant the same thing – that no one can visualize the phenomena as if they were classical. Nobody understands quantum mechanics in this sense – and nobody can ever understand quantum mechanics in this sense!

Any "more intimate" level of understanding is a matter of emotions or religions and – at least up to the moment when someone fills it with some new beef – it just doesn't belong to science. In particular, what is almost universally hiding behind the demands to "understand" quantum mechanics is the desire for physicists to say wrong things about Nature, especially the wrong statement that Nature fundamentally works within the framework of classical physics i.e. an objective state of affairs exists independently of any observer or sensory experience – and that the latter plays no fundamental role. But it does play a fundamental role.
Physicists don’t understand their own theory any better than a typical smartphone user understands what’s going on inside the device.
This is complete rubbish. Good physicists understand the universal rules of quantum mechanics more than all the world's engineers combined understand a smartphone. It's possible because the rules are really clear and concise.
There are two problems. One is that quantum mechanics, as it is enshrined in textbooks, seems to require separate rules for how quantum objects behave when we’re not looking at them, and how they behave when they are being observed.
The situation is surely distorted and misinterpreted but even if it were exactly as described in the second sentence, it would be absolutely unjustified to conclude that "there is a problem". A theory surely may postulate that the physical processes and/or predictions are composed of two (or more) components. To say that it is a "problem" is just breathtakingly stupid.

Darwin's theory involves reproduction of the genes, growth of organisms, their competition for resources i.e. natural selection, and then go to the beginning. It surely unifies several processes as well, doesn't it? It's normal that theories unify several processes. Almost all scientific theories have several parts that have to co-exist.

Except for quantum mechanics where this description is really wrong. The correctly interpreted "two parts" are actually the following: the actual events (measurements i.e. sensory experience); and their prediction. It's the events (measurements) that happen or not happen, according to an observer's axiomatic scheme. The second part may be consistently said not to happen at all. But for science to exist, we just want to predict what we will observe. That's where the complex numbers in calculations involving unitary operators belong. They are just tools to predict what will probably happen – what sensory experience one will have.

Despite my heroic fights, the anti-quantum zealots have proliferated and Canada's textile industry has switched to the production of T-shirts for millions of these confused people.

Again, the true division is one into the "events" and "their predictions". There is absolutely nothing shocking about that division. The prediction of phenomena in Nature was always a separate activity from the phenomena themselves. Newton's calculation of the motion of planets is something else than the motion of the planets itself, isn't it? The separation of the application of quantum mechanics to the "measurements" and "linear transformations of complex numbers done in order to predict the measurements" is exactly as innocent and self-evidently needed as the similar separation in my Newtonian example or any other example.

To say that this unavoidable separation is a "problem" is an unmistakable sign of pure idiocy.
When we’re not looking, they exist in “superpositions” of different possibilities, such as being at any one of various locations in space.
Objects are represented by wave functions which are generically superpositions but these wave functions are not the objects themselves. The wave functions are a mathematical representation of the observer's knowledge of the state of the objects or the world. The wave function $$\ket\psi$$ or the density matrix $$\rho$$ are just some complex, more algebra-based generalizations of the probabilities $$P_i$$ or probability distributions $$\rho(x_i)$$ that have always encoded someone's belief about the future events.
But when we look, they suddenly snap into just a single location, and that’s where we see them.
This "snapping" isn't mysterious because it's really a tautology. By definition, the wave function has the meaning of remembering all the probabilities of possible outcomes of the future measurement. So when the measurement is done and the result becomes known, the probability distributions and therefore the wave function must "snap" and get concentrated "within" the actual result that was seen. It's just the same statement said in two different ways – the "snapping of the wave function" is just the quantitative way of expressing the very same information, namely that the observer has looked and seen something.
We can’t predict exactly what that location will be; the best we can do is calculate the probability of different outcomes.
That's how Nature works. Good physicists are virtually 100% certain that only probabilities may be predicted. If there weren't the case, there would have to be hidden variables whose state determines the outcomes that only "look" random, and then one would be in the conflict with locality and therefore relativity via Bell's theorem or any equivalent proof.
The whole thing is preposterous.
What's actually "preposterous" is when a person who boldly calls himself "a physicist" (just one line beneath the title of this moronic NYT diatribe) denies these absolutely elementary laws and principles of modern physics a whopping 94 years after they were discovered back in the mid 1920s.
Why are observations special?
Again, observations are totally special because, as already correctly (accidentally correctly?) understood by philosophers almost a century before the birth of quantum mechanics, "the information derived from sensory experience, interpreted through reason and logic, forms the exclusive source of all certain knowledge". Observations are special because they are absolutely needed for anything and everything about the world to be known.
What counts as an “observation,” anyway?
An observer must know the answer to this question, otherwise he can't use the apparatus of quantum mechanics. For example, I have observed that the number of unbelievably stupid yet arrogant NYT op-eds was at least one on Saturday. I can give you as many examples of my observations as you want but it's clearly too much to demand to "list all of them" or "a full algorithm revealing when I know that I have observed something". The details why I know depend on the type of the observations – and it has always been the case, even before quantum mechanics. In quantum mechanics, the observer must assign the label "actual results of observations" to some data to determine the state vector (as an eigenstate of the measured operators with the measured eigenvalues). Without this input, quantum mechanics would have nothing to predict. There would be no values of observables (or truth values of propositions) that may be considered correct; the existence of all of those requires sensory experience.

Even in classical physics, the observer had to know what he considered true results of observations of the real world. For example, Kepler had to trust Brahe's observed and reconstructed 3D positions of planets to derive Kepler's laws. The difference between classical physics and quantum mechanics is that classical physics allowed one to "discuss the unambiguous, certain state of the world that exists independently of sensory experience" while quantum mechanics doesn't allow anything like that. Quantum mechanics "demands that the basic data come from sensory experience and it 'directly' relates them with others through probabilistic predictions derived from the complex linear algebra".
When exactly does it happen?
It is a physically ill-defined and inconsequential question. The observer's learning of the outcome of a measurement is a mental process, it may happen earlier or later, as long as it is done in time before the next measurements, the predictions will be right. The outcomes of the measurements are the only "real" events whose precise character (the outcome, an eigenvalue) may be unambiguously discussed and demanded to be predicted by the scientific theory. Also, what is exactly counted as a part of the observer or the apparatus is ill-defined and flexible – it's the position of the so-called "Heisenberg cut". There is no sharp location where one would have to place the "Heisenberg cut". It never hurts to include a greater portion of the world to the side that is described quantum mechanically, with all the superpositions.

On the other hand, the side of the world that is treated as "the observer side" i.e. the side knowing about the actual results of observations must also exist, otherwise quantum mechanics could make no predictions. This is how science has worked since 1925. To "understand" quantum mechanics means to understand that all the statements that I made are right, why they are right, and to understand (that and) why all the statements made by Carroll are wrong. If someone does the exact opposite, then he completely misunderstands quantum mechanics, instead of understanding it! The words "understand" and "misunderstand" are not synonymous.

Using a metaphor, Carroll demands that physicists develop an understanding how "nicely flat the Earth is", while those physicists saying that the Earth is round "refuse to work to understand physics". But they don't. They are right, the Earth is round, this is the correct understanding while the Flat Earth is a deep misunderstanding. Physicists shut up about the Flat Earth and calculate new things building on the assumption that the Earth is round.
Does it need to be performed by a person?
There is absolutely no requirement for anything related to the observer to be anthropomorphic. This clearly isn't the point at all – and everyone who tries to attack quantum mechanics by saying that "it depends on homo sapiens" is a demagogue. As Heisenberg wrote in 1958, Physics and Philosophy, page 137:

Of course the introduction of the observer must not be misunderstood to imply that some kind of subjective features are to be brought into the description of nature. The observer has, rather, only the function of registering decisions, i.e., processes in space and time, and it does not matter whether the observer is an apparatus or a human being; but the registration, i.e., the transition from the "possible" to the "actual," is absolutely necessary here and cannot be omitted from the interpretation of quantum theory.
Is consciousness somehow involved in the basic rules of reality?
Consciousness is just a grander word that makes many more people think irrationally. What is actually needed is that the truth of the propositions about the observables is extracted from sensory experience. If one postulates that the observer is "conscious" about all the sensory experiences he has, then yes, "consciousness" is an unavoidable part of any application of the laws of quantum mechanics.

An observer who has had any sensory experience knows what they mean, anyway. On the other hand, he can't operationally determine whether others have "consciousness" in the most internal sense so even the question about the existence of consciousness of other objects in the Universe may be dismissed as a scientifically meaningless ones. Neuroscience may have nice theories about how consciousness works – a part of the science how brain works – but one doesn't need any of that neuroscience to apply quantum mechanics to experiments that study other things than brains.
Together these questions are known as the “measurement problem” of quantum theory.
Right. What is called "the measurement problem" is just a bunch of loaded, meaningless, suggestive, and demagogic questions, usually posed by someone who doesn't really want to understand how quantum mechanics actually works.
The other problem is that we don’t agree on what it is that quantum theory actually describes, even when we’re not performing measurements.
Again, all knowledge comes from sensory experience so proper science – quantum mechanics – doesn't describe anything that is independent of measurements.

The fact that "we don't agree" clearly isn't a problem of a scientific theory. Every scientific theory is a source of disagreements. Even a correct theory is understood and accepted by some and misunderstood and rejected by others. That doesn't mean that there is any problem with the theory. There is only a problem with the likes of Mr Carroll who deny the foundations of physics that have been firmly in place for over 90 years.
But what is the wave function? Is it a complete and comprehensive representation of the world?
No, as explained about 50,000 times, but still not enough for the extremely slow likes of Mr Carroll to get this point after more than 90 years, the wave function isn't a set of objectively real degrees of freedom. It's a mathematical representation of the probabilistic knowledge according to an observer.
Or do we need additional physical quantities to fully capture reality, as Albert Einstein and others suspected?
No, we don't need them. In fact, we need them to be absent. This isn't just an assumption that still works. It has been basically rigorously proven – and by many independent trains of thought. The hidden variables cannot exist because that would conflict with all the no-go theorems for hidden variable theories, the free will theorem, Bell's theorem or relativity, with the low heat capacity of molecules, and more.

This question about the non-existence of hidden variables is perfectly understood, and repeating this question as if it were open – while claiming that physicists "don't understand quantum mechanics" – is pure deception. Our understanding that and why no hidden variables exist is an excellent example of a deep understanding of a problem that was achieved by physicists. The only bad news is the personal problem of some people who prefer not to understand this deep understanding obtained by physics, for ideological and related reasons.
Or does the wave function have no direct connection with reality at all, merely characterizing our personal ignorance about what we will eventually measure in our experiments?
The two sentences pretend to be equivalent but they are not. The wave function is as directly connected to reality as the laws of Nature allow. But this most direct possible connection still guarantees that the wave function is a complex generalization of subjective probability distributions. The most direct connection between "the numbers in our theories" and "reality" is still through sensory experience i.e. observations. There is no way to totally sideline observations while saying anything about reality. That is how Nature works.
Until physicists definitively answer these questions, they can’t really be said to understand quantum mechanics — thus Feynman’s lament.
They have been definitively answered in the 1920s and many times after that – and 507 times on this blog – but some people just don't want to listen and they prefer to be scientifically illiterate demagogues marketing themselves as "physicists".

In the following sentences, he writes that the "understanding" of QM – by which he means the efforts to increase the general misunderstanding among the students – should spread because it's a priority to create as many self-confident scientifically illiterate babblers similar to the author as possible.
On the contrary, students who demonstrate an interest in the topic are gently but firmly — maybe not so gently — steered away, sometimes with an admonishment to “Shut up and calculate!”
Because all these professors are well aware of the fact that everyone who has decided to turn some incoherent criticism of the foundations of quantum mechanics into a "life program" has ceased to be a scientist. We may actually say that this statement first applied to Albert Einstein himself. A difference is that Einstein had done lots of amazing things by that point – while the students who could go into "the field of interpretations" would become useless deluded crackpots right away, having achieved nothing in their lives. The professors are doing an extremely good service for the students when they try to protect them from this useless garbage.

"Shut up and calculate" simply conveys the "morale" of physics as a quantitative science where hard work, hard facts, data, and careful and precise calculations play a decisive role – while the verbal or emotional manipulation does not. Some people probably dislike this hard scientific, quantitative character of physics because they're better at manipulating masses by loaded words but physics will always be the quantitative science, not a subfield of propaganda.
Professors who become interested might see their grant money drying up, as their colleagues bemoan that they have lost interest in serious work.
Because it is what it means, indeed. If someone is satisfied with writing op-eds whining about "something wrong with quantum mechanics" without having an infinitesimal sign of a problem or an argument, and if he's satisfied with the fact that clueless laymen allow him to publish similar junk in newspapers, then he has lost the interest in serious work. Serious physicists in his environment, if they exist at all, must treat him as a counterproductive parasite because he is one.
This has been the case since the 1930s, when physicists collectively decided that what mattered was not understanding quantum mechanics itself; ...
A difference between the 1930s and the 2010s is that in the 1930s, the New York Times weren't a bullhorn for the ideologically driven anti-science demagogues.
...what mattered was using a set of ad hoc quantum rules to construct models of particles and materials.
It's a pure, 100% lie when someone says that the quantum rules to construct models of particles and materials are "ad hoc" (like written down arbitrarily and separately for each situation). All these models of particles and materials are unified and they directly and unambiguously follow from the same, concise, universal laws of quantum mechanics, perhaps with a simple supplement of a Hamiltonian. In particular, the universal postulates of quantum mechanics along with the Standard Model Hamiltonian or Lagrangian is enough to uniquely derive all the physicists' knowledge about any materials and particles that has ever been derived. String theory allows to add the gravitational phenomena into this quantum framework, too.

Carroll urges foxes to go vegan again.
Physicists brought up in the modern system will look into your eyes and explain with all sincerity that they’re not really interested in understanding how nature really works; they just want to successfully predict the outcomes of experiments.
If one can predict the results of all conceivable experiments, even all those that can be made in principle but not in practice, and if one can answer all questions about "which aspects of the theory are final and which could be replaced by 'finer' theories in the future", and if all this industry of predictions coherently follows from the same simple enough laws, then physicists understand how Nature works. In some questions, physicists think that an improved understanding may arrive in the future (like the knowledge of new particle physics phenomena at energies above the LHC scale), in others, they know it's impossible (like when it comes to the probabilistic nature of QM predictions which is demonstrably here with us to stay). The good physicists typically try to focus on "strategies to expand the scientific knowledge" where the mean expected value of the magnitude of progress is large enough.

Carroll says that physicists are doing A instead of B but a proper scientific interpretation of A and B implies that A and B are the same thing. Instead, he really wants A or B to be replaced with C which isn't good science at all.
This attitude can be traced to the dawn of modern quantum theory. In the 1920s...
As I wrote many times recently, this general positivist attitude may be traced to Auguste Comte's texts between 1830 and 1842. That's when the correct "vague philosophical ideas" that would later become important in physics were being born. The folks in Copenhagen have used the pre-existing "attitude" and they have figured out some more detailed methods how to turn the "attitude" to "quantitative predictions" (via complex matrices etc.).
Not everyone was happy that Bohr’s view prevailed, but these people typically found themselves shunned by or estranged from the field.
Because Bohr's comments were right and Einstein's were wrong – and because the people around Bohr did an amazing amount of advances while the followers of Einstein's anti-quantum school have achieved nothing whatsoever. It is absolutely right for a healthy system to expel the latter group. On the contrary, what is pathological is not to expel the likes of Carroll. The system is starting the break down now when contacts with someone in The New York Times may apparently compensate his incompetence in quantum mechanics.
In the 1950s the physicist David Bohm, egged on by Einstein, proposed an ingenious way of augmenting traditional quantum theory in order to solve the measurement problem.
The pilot wave theory isn't ingenious but a stupid yet straightforward caricature of quantum mechanics and it wasn't invented by David Bohm in the 1950s but by Prince Louis de Broglie in 1927.
Werner Heisenberg, one of the pioneers of quantum mechanics, responded by labeling the theory “a superfluous ideological superstructure,” and Bohm’s former mentor Robert Oppenheimer huffed, “If we cannot disprove Bohm, then we must agree to ignore him.”
The two men were exactly right. De Broglie's fantasies are a superfluous ideological superstructure. Even Einstein has used the term "superstructure" for these pilot wave constructions. It isn't always possible to "disprove an ideology" because its adherents will keep on pushing despite all the lethal blows to their ideas. That's why Oppenheimer's recommendation to ignore this stuff was really wise, too. It was possible and right to ignore this junk at least before the anti-physics movement hijacked some of America's most influential newspapers.
Around the same time, a graduate student named Hugh Everett invented the “many-worlds” theory, another attempt to solve the measurement problem, only to be ridiculed by Bohr’s defenders.
It's right to ridicule proponents of ridiculous ideas – and to say No when they apply for a postdoc job.
A more recent solution to the measurement problem, proposed by the physicists Giancarlo Ghirardi, Alberto Rimini and Tulio Weber, is unknown to most physicists.
I am surely a world's top expert on this GRW stuff and I can assure everybody that they're not missing anything important if they don't know what it is.
These ideas are not simply woolly-headed “interpretations” of quantum mechanics. They are legitimately distinct physical theories, with potentially new experimental consequences.
What Mr Carroll doesn't say is that all these new experimental consequences instantly and brutally clash with the known observations, even the most elementary as well as general ones, and they imply that all these woolly-headed "interpretations" and "alternative theories" are immediately disproved. For example, the GRW objective collapse theories predict new unavoidable events of ionization and radioactive decay that would have been seen a long time ago if the theory were right. They weren't seen, the GRW is dead. But some people just don't care and they keep on writing the same deceptive op-eds again and again.
The situation might be changing, albeit gradually. The current generation of philosophers of physics takes quantum mechanics very seriously, and they have done crucially important work in bringing conceptual clarity to the field.
It's existentially dangerous for the civilization to allow the proliferation of "alternative scientific departments", namely in the philosophy departments, where the scientific method isn't respected and where people aren't being fired for their persistent failures. Science depends on this treatment of wrong ideas and if one can circumvent this elimination of wrong ideas by going into another department where the scientific criteria don't matter but where they can be paid almost equally, then the institutionalized science is dead. It's dead simply because it's much easier to get incompetent, dishonest, and lazy people than the brilliant, honest, and diligent ones. The bad ones will unavoidably prevail if it is allowed at all.
The problem is that, despite the success of our current theories at fitting the data, they can’t be the final answer, because they are internally inconsistent.
The current theory is string/M-theory which includes gravity and is consistent – and it's been the case for many decades.
After almost a century of pretending that understanding quantum mechanics isn’t a crucial task for physicists, we need to take this challenge seriously.
After almost a century in which quantum mechanics has been understood and has passed every single test while its critics have failed every single test, it's already the time to treat its critics as unhinged ideologically-driven charlatans who should be relocated from the Penn State to the state penns.

And that's the memo.