Wednesday, August 17, 2016 ... / / How pop science manipulates with the status of the Schrödinger equation

In a blog post, Sean Carroll says that you should love or respect the Schrödinger equation and you should appreciate that the Schrödinger equation may be applied in a wider range of situations, not just in the non-relativistic of mechanics of point-like particles.

I agree with these two points. You could wonder why Sean Carroll hasn't written down the simple sentence in the first paragraph of this blog post and instead, added some three pages of redundant text. Well, it's because he needed to keep the percentage of misconceptions and distortions well above 50%, just like in almost all of his texts about quantum mechanics.

I will be more specific about the complaints but let me begin with his list of his "seven favorite equations":$\eq{ {\bf F} &= m{\bf a}\\ \partial L/\partial {\bf x} &= \partial_t ({\partial L}/{\partial {\dot {\bf x}}})\\ {\mathrm d}*F &= J\\ S &= k \log W\\ ds^2 &= -{\mathrm d}t^2 + {\mathrm d}{\bf x}^2 \\ G_{ab} &= 8\pi G T_{ab}\\ \hat{H}|\psi\rangle &= i\partial_t |\psi\rangle }$ OK, nice enough, especially if we can upgrade Carroll's inferior techniques to write ${\rm\LaTeX}$ equations to the full-fledged MathJax. I wouldn't write the exact same list – e.g. because there's no gauge theory, string theory, or supersymmetry in the list – but let's not discuss advanced topics.

There are at least the following problem with the presentation and "framing" of the Schrödinger equation as offered by Sean Carroll – and lots of other sources of superficial pop science out there:

1. Looks classical: the equation of quantum mechanics is written as if it were another classical equation
2. The time-dependent wave function is made to look unavoidable in QM: this is just an artifact of the formalism
3. Schrödinger's misunderstanding of the meaning of the wave function is being obscured: only Max Born did this important job and got his well-deserved Nobel prize for that
4. Schrödinger is incorrectly credited: with the generalizations of the equation for other Hamiltonians $H$
5. Sometimes there's no $H$: it's being incorrectly suggested that the Hamiltonian $H$ must be well-defined in any predictive quantum mechanical theory
OK, I don't want to degrade this blog post to a precise analysis which sources of pop science like to spread which of these misconceptions. Most of them, including Carroll, spread most of them. It's much more important for the people who are interested in physics to understand why they're misconceptions and what are the correct statements to be made about these questions.

The qualitative jump is being obscured

Carroll wrote his seven favorite equations, starting from $\vec F = m\vec a$. You may see that all of them are some mathematical equations that are relevant in physics. The first six of these equations are equations relevant in classical i.e. non-quantum physics and they're increasingly mathematically abstract, advanced, or sophisticated. One is gradually adding fields and not just positions of particles, perhaps statistical entities such as the entropy, or the metric tensor; ordinary differential equations are being superseded by partial ones, and so on.

One could be easily tempted to extrapolate this trend and say that the progress in physics is simply a path towards increasingly more abstract classical equations that define the dynamical laws of physics. Unfortunately, that's indeed how the people brainwashed by the pop science – mostly included Carroll himself – understand the role of the seventh, Schrödinger equation.

However, that's completely wrong. The step from equations such as Einstein's equations (sixth in the list of seven) to the Schrödinger equation is a step of a completely different character than any of the previous steps. Quantum mechanics is not another classical theory with somewhat more complex equations for a somewhat more complex set of degrees of freedom.

Instead, quantum mechanics is a qualitatively new theory – or a new framework – that doesn't change the dynamical equations as much as it changes the interpretation of the mathematical symbols i.e. the relationship between the mathematical objects and the empirical observations. Quantum mechanics really differs from classical physics by its observables' being non-commutative operators. For that reason, eigenvalues have to be interpreted as possible values and predictions have to be made in a probabilistic way by squaring complex probability amplitudes.

A more conceptually correct way to explain quantum mechanics involves the original, Heisenberg picture of quantum mechanics. In the Heisenberg picture, the dynamical equations such as the first one (or others that involve time derivatives of dynamical degrees of freedom) can be pretty much kept fixed. You just add hats, upgrade the coordinates and momenta to operators, and postulate that operator equations have to replace the original equations for $c$-numbers.

In particular, the equation $\vec F = m\vec a$ is just fine in quantum mechanics if you simply add the hats.

Focus on the picture, not the essence

I am basically going to make a related point which I decided to frame as an independent one. I just said that people are often made to believe that the wave function is another classical field – analogous to the electromagnetic or metric field – and the Schrödinger equation is another classical dynamical equation – analogous to Maxwell's or Einstein's equations.

In the new blog post, Carroll does nothing to counter this misconception. In many other texts, he directly helps in the propagation of this deep misunderstanding.

One reason why many laymen buy this untruth is the fact that the Schrödinger equation is still a differential equation. Differential equations are "hard enough" and once people are forced to think about the differential equation, they don't have any spare energy to think about anything else. But the other things, the meaning of the symbols and their relationships to the observations, are much more important than the mathematical form of the equation.

In fact, it is not necessary in quantum mechanics to deal with the time-dependent wave function at all. Quantum mechanics may be formulated in the Heisenberg picture – the modern reformulation of the "matrix mechanics", the first way to write down the rules of quantum mechanics as understood in the pioneering papers by Werner Heisenberg. In this picture, $\ket\psi$ is independent of time. Instead, it's the operators $\hat x(t),\hat p(t)$, and others that are time-dependent. And the equations for their time derivatives pretty much copy the corresponding equations in classical physics. You just add hats. Expectation values may still be computed in the usual way. Probabilities may be computed as the expectation values of projection operators (which are time-dependent here, much like almost all operators).

And yes, one can formulate quantum mechanics without any differential equations containing time-derivatives. Richard Feynman found the complete way to calculate all predictions in quantum mechanics "directly", without solving any differential equations (for functions of time). You just sum complex amplitudes over all histories or trajectories. The complex result is the probability amplitude for one desired evolution from a known initial state to a possible finite state. You square the absolute value and obtain the probability. Nothing else than the probabilities (or things that are obviously functions of them) can be calculated according to the general principles of quantum mechanics.

The excessive focus on the Schrödinger picture – and the fact that pop science and many textbooks suppress the other pictures and formulations – is another driver that encourages the students and laymen to strengthen their belief that the Schrödinger equation is just another classical equation in a sequence of increasingly mathematically complex equations of classical physics.

Schrödinger himself didn't know the right physical meaning of the symbols

Schrödinger himself thought that the wave function is a "dissolved electron" which is spread over the space much like butter is spread over bread. In some texts, he made this point rather explicitly. Werner Heisenberg and others already understood that the wave function was a probability amplitude wave and they were explicitly critical of Schrödinger. But he has never understood the correct physical meaning of the wave function so all his papers always remained partly wrong and partly ill-defined.

Just to be sure, the papers weren't bad. He was very good in dealing with equations of mathematical physics and he wasn't trying to spread his misconceptions everywhere. If you read the paper Schrödinger 1926c which is helpfully in English and not in German (as a modern physicist, Schrödinger sent it to The Physical Review), you will have a rather hard time to find the explicitly wrong claims about the "visualizability" of the wave function etc. But these opinions are implicitly there and he has repeatedly made the incorrect statements explicitly.

Generalizations to other Hamiltonians $H$ aren't really his invention

This is another mostly historical or sociological detail but you may view it as an important one, too. Carroll claims that the Schrödinger equation may be used even with the Hamiltonian $H$ that defines the total energy e.g. of all fields in the Standard Model. If you do so, the Schrödinger equation correctly describes the evolution of all observable things at the accuracy of the Standard Model – which, except for the absence of gravity, describes everything we have ever safely observed in fundamental physics.

Well, if you look at the Schrödinger 1926c paper, you will see that what Schrödinger has actually discovered was really just the "edition" of the equation named after him that deals with non-relativistic mechanics. Carroll complains that people, when asked to write down the Schrödinger equation, usually write$\left[-\frac{1}{\mu^2}\frac{\partial^2}{\partial x^2} + V(x)\right]|\psi\rangle = i\partial_t |\psi\rangle$ Indeed, we often like to use the term "the Schrödinger equation" more generally. I use it more generally, too. On the other hand, when you read Schrödinger's actual papers, you will see that he's never written down the general equation. He really derived the special equation above – exactly as indicated by the people whom Carroll criticizes. He deduced this equation by some more convoluted thoughts about the de Broglie wave in the presence of external potentials.

Schrödinger wasn't thinking quantum mechanically and generally about the new, quantum mechanical framework for physics. And that was the main reason why he wasn't able to go beyond the case of non-relativistic mechanics even though he arguably had the mathematical skills to do so. What happened?

Erwin Schrödinger actually did fully realize that the equation above is just a non-relativistic equation. He knew that it meant a limitation, he knew that relativity was right, he knew what relativity demanded, so he knew that a relativistic version of the equation was needed. But what he ended up with was really the Klein-Gordon equation$\left[ \frac{\partial^2}{\partial t^2} - \frac{\partial^2}{\partial x^2} - \frac{\partial^2}{\partial y^2} - \frac{\partial^2}{\partial z^2} + m^2 \right] \Psi(\vec r,t) = 0.$ He actually tried to use this equation to "solve the hydrogen atom again". He got some results that disagreed with the experiments so as a well-behaved physicist, he threw the equation to the trash bin. Later, the equation was published by Klein, Gordon, and Fock. Even though Schrödinger has beaten them as a faster mathematician, the equation never carries his name because Klein, Gordon, and Fock scooped him in the journals.

Now, if Schrödinger had known about the "general Schrödinger equation with a general Hamiltonian $H$", he wouldn't even try to apply the equation to a one-particle problem of the hydrogen atom. Why? Because he would know that the Klein-Gordon equation above simply isn't an example of the Schrödinger equation. So it cannot be considered a generalization of his non-relativistic mechanics Schrödinger equation.

Why the Klein-Gordon equation isn't a generalization of the non-relativistic Schrödinger equation? Because all general Schrödinger equations have to be equations for a complex wave function; and they have to be first-order differential equations in time. However, the minimal Klein-Gordon equation is an equation for a real, not complex field; and, more importantly, every Klein-Gordon equation is a second-order differential equation in time.

So it's simply not any kind of a Schrödinger equation or its generalization! You can't interpret it in this way.

As you know, the correct relativistic Schrödinger-like equation for the electron was found by Paul Dirac. It was the Dirac equation and immediately described the electron's spin (and antiparticles), too. The real reason why Dirac was able to do this thing wasn't that Schrödinger was incapable of learning the spinors. The main reason was that unlike Schrödinger, Dirac understood the meaning of the symbols in the non-relativistic Schrödinger equation and the new framework of quantum mechanics in general.

In particular, Dirac knew that even the "general" Schrödinger equation – even one for a relativistic hydrogen atom – has to be a first-order equation in time. That means and Dirac (but not Schrödinger) knew that it meant that the Klein-Gordon equation was at most a candidate for a new equation governing a classical field, not a direct replacement for the Schrödinger equation. He was searching for a Hamiltonian operator that would automatically obey $H^2 = (\vec p)^2 + m_0^2$ and was forced to consider the spinors and the Dirac gamma matrices. Schrödinger wasn't solving this problem at all because he wasn't fully aware of the fact that what he needed was definitely a first-order equation. In other words, Schrödinger wasn't solving the same excellent exercise as Dirac because Schrödinger didn't understand the general equation that we currently call the Schrödinger equation (for some general $H$). We credit him with something he clearly misunderstood!

It's natural to use the original name for the important generalization but it's historically misleading.

Quantum mechanical theories may exist even if there is no $H$

The last, more advanced point I want to make is that it is not even true that the existence of a well-defined Hamiltonian operator $H$ is needed for a quantum mechanical theory to exist. In the text above, I wrote that we can make quantum mechanical prediction without ever using or solving the general Schrödinger equation. We can switch to the Heisenberg picture or to Feynman's approach in terms of the path integrals.

However, in those sections, you could still assume that it's always in principle possible to identify the Hamiltonian – and to switch back to the Schrödinger equation if you want to.

That's not really the case. There may exist quantum mechanical theories that are fully predictive but they don't give you any prescription for $H$ at all. Nevertheless, they may predict what will happen.

Quantum theories including gravity and the spacetime diffeomorphisms largely fall into this category. String theory is the most well-defined example – and, for non-trivial reasons, the only fully well-defined example, but that's another discussion. If you want to keep the diffeomorphism symmetry of general relativity manifest at least to some extent, you're prevented from defining the Hamiltonian.

Note that the Hamiltonian is a generator of time translations, $t\to t+\delta t$, but such translations only make sense if you know what the corresponding time coordinate $t$ is. But a defining conceptual fact about general relativity is that there exists no preferred choice of the coordinates $t,x,y,z$. You may pick almost any function $t'(t,x,y,z)$ of your original spacetime coordinates and call $t'$ your new time coordinate! There will exist corresponding variations of $t'$ and the new Hamiltonian operator $H'$ that generates them.

In fact, if you appreciate the status of these redefinitions, you will realize that these diffeomorphisms or coordinate transformations are conceptually gauge symmetries – much like Yang-Mills symmetries in Abelian or non-Abelian gauge theories – and the physical states have to invariant under these symmetries. It also means that physical states have to be annihilated by the generators of such symmetries such as $H$. (Symmetry transformations that change the fields even in the asymptotic regions at infinity may be given an exemption; these transformations' generators such as the ADM energy may be nonzero i.e. don't have to annihilate the physical states.)

In particular, the equation $H\ket\psi = 0$ is "true" in some general formulation of quantum gravity and it is known as the Wheeler-DeWitt (WdW) equation. Only approximate ways to formulate quantum gravity in terms of this equation are known.

However, even in the absence of a fully well-defined mathematical realization of the WdW equation, it's still true that consistent quantum gravity theories generally don't want to give you a clear prescription for the Hamiltonian $H$. For example, covariant perturbative string theory directly produces the S-matrix of evolution probability amplitudes – those from $t=-\infty$ to $t=+\infty$. They're equivalent to on-shell Green's functions in QFT only. There are no nice calculable formulae for off-shell Green's functions in string theory or other gravitating quantum mechanical theories. That's related to the non-existence of completely gauge-invariant local quantities in diffeomorphism-invariant theories. Even the scalar curvature $R(x,y,z,t)$ fails to be gauge-invariant because the gauge symmetries change the values of $x,y,z,t$ which means that they map this scalar to the scalar at another point, $R(x',y',z',t')$.

This statement isn't "quite" true because if you gauge-fix the diffeomorphism symmetry, most famously by going to the light-cone gauge in string theory, it becomes possible to describe the evolution in terms of "one moment to another" steps. And the appropriate light-cone Hamiltonian, the component of the vector $P^-$ that evolves your state of strings from one null slice to another, becomes well-defined. Green and Schwarz loved to use this "light-cone gauge string field theory" in the early 1980s. It was only well-defined perturbatively but my and DVV's matrix string theory provides us with a full non-perturbative definition of this evolution of strings from one moment to another. (The BFSS model that does the same job for the stringless 11-dimensional M-theory limit was found before matrix string theory.)

But those formulations of string theory are possible only with some extra work – gauge-fixing and efforts to find new equations. If people had not found the light-cone gauge formulations of string theory, it would still be true that they can make infinitely many arbitrarily accurate predictions in string theory – the S-matrix – and the Hamiltonian and all equations dependent on it would be completely circumvented.

The absence of a Hamiltonian may be encountered in other situations, too. Quite typically, we write the Hamiltonian as a function or functional of some other observables. However, these observables are often operators obtained by "adding hats" on some classical quantities. That's possible when some classical quantities exist – i.e. when the quantum mechanical theory has a classical limit. This is not always the case, either. Sometimes, even in non-gravitational theories, the Hamiltonian may formally exist but there is no helpful "explicit formula" for such a Hamiltonian. Examples include some two-dimensional (but also six-dimensional) conformal field theories.

OK, to summarize:

There are various either explicitly incorrect misconceptions or "encouragements to think conceptually incorrectly" that are being spread in the name of the Schrödinger equation. The misleading analogies between the equation with the equations of classical physics, the focus on the particular form of the Schrödinger equation, and the emphasis on the Schrödinger picture, and the excessive promotion of the objects that the equation depends upon may be classified as important but not the only reasons why so many people end up misunderstanding quantum mechanics – what it means, what it says, and in what sense it actually differs from the old classical physics.

So even though I agree with Carroll's two well-defined assertions – that the equation is important in physics; and it is applicable outside non-relativistic mechanics of particles – I find his efforts to make the readers (and himself) more deeply immersed in the specific features of this equation counterproductive. What people need to learn quantum mechanics correctly is to suppress the dependence of their knowledge on this equation and on the Schrödinger picture.

The Schrödinger equation isn't what gives the quantum flavor to quantum mechanics. The nonzero commutators or, equivalently, the uncertainty principle is the actual spice that makes the whole difference. Promoters of superficial pop science including Sean Carroll can't "taste" this space because they mostly misunderstand quantum mechanics themselves.    