A forefinger of a person whose DNA program has mutated to include an infinite loop. Governments use the same arm when they offer a hand to the individual citizens.
I haven't changed my opinions about most of the "key philosophical questions" during the last 20 years (and, in many cases, 30 years) but the relevance of seemingly pathological mathematical structures for physics is an aspect where a gradual transformation of my opinions has become undeniable.
Discontinuous and pathological structures
Some people must have always been obsessed with discrete constructions and combinatorics. I wasn't. Once I had learned some complex analysis 30 years ago, I was convinced that the fundamental objects underlying both mathematics and the physical world have to be continuous and beautiful, in a sense that includes some smoothness.
Discrete objects are just degenerations or incomplete descriptions of the continuous ones; in other words, the discrete objects' measure is zero within the space of all objects. They're degenerated and pathological, especially if they require a procedure to be repeated infinitely many times. The (approximate) fractal above is meant to convey my feelings about some pathological mathematical structures.
This philosophical thesis may be fine as a slogan but it doesn't answer all questions in maths and physics. I am still confident that it is a very helpful guide in much of physics, up to the conventional quantum field theory and string theory, but just like any philosophy, it is misleading in many particular contexts.
As you may expect, the rest of the text will be dedicated to the counterexamples and necessary revisions of the thesis above.
Infinitely differentiable functions
Complex numbers have the advantage that each N-th order polynomial equation has N roots or solutions - some of them may coincide. Polynomials are perfectly smooth, infinitely differentiable functions that may be fully and uniquely reconstructed from their Taylor expansions. Exponentials and sines and cosines seem to have the same property despite the fact that they can only be written as polynomials of the infinite order.
So of course, my attitude used to be that we wouldn't ever need any functions that can't be defined by their Taylor expansions. Arguments that require non-smooth functions to be used are either approximate, misleading, or downright wrong. You don't need any perverse functions and the people who study them are perverts, too.
That's a great attitude that allows you - and has allowed me - to get pretty far. However, as it turned out, it has its limitations. Once upon a time, in a mathematics handbook for engineers, the sort of books that have taught me a lot, I saw some explanations of the Fourier analysis. It was very neat and at the beginning, it seemed to be compatible with my philosophy. Combinations of sines and cosines are perfectly nice functions, aren't they?
However, I was shocked to read that even non-smooth - and discontinuous - periodic functions may be written using the Fourier expansions. That's terrible! How can ugly, discontinuous, pathological discontinuous functions arise from a combination of the beautiful ones? Moreover, I was sure that I had to allow infinite sums of beautiful objects, to be able to do pretty ordinary things (e.g. Taylor expansion for the exponential).
It turned out that the beautiful objects - infinitely smooth functions - were not closed under the infinite summation (and other similar operations). One can't completely avoid the ugly objects.
Poles, non-perturbative effects, asymptotic series
Obviously, there are many other subtleties that blur the boundaries between the "beautiful" and "ugly" objects. Taylor expansions often have a finite radius of convergence; this fact is a symptom of singularities of the function in the complex plane.
Another surprise - that emerged in physics - was the existence of non-perturbative contributions. If you consider the function
f(g) = exp(-1/g2),and you define f(0)=0, it's a perfectly smooth function of the real variable g. However, its Taylor expansion is
0 + 0g + 0g2 + 0g3 + ...which obviously doesn't converge to the original (nonzero) function. All the terms in the Taylor expansion vanish because the N-th derivative of f(g) always contains the exp(-1/g^2) factor, and this factor goes so quickly to zero near g=0 that it beats any ratio of polynomial functions of g that you also produce by the differentiation. The function starts to rise from zero extremely slowly and gradually so that the Taylor expansion can't see the rise at all.
This function f(g) is not just a curiosity; if g is interpreted as a coupling constant, physics is actually full of such terms in the quantum amplitudes. They're the non-perturbative corrections - coming e.g. from instantons - because they can't be correctly expressed by the Taylor series.
Such insights contradict the naive intuition that the Taylor expansions are good to describe "any natural function". This assumption is simply showed to be wrong. It's wrong at a rigorous mathematical level. But more importantly, it's also morally wrong, i.e. wrong when it comes to its naturally guessed physical implications.
A good initial intuition is very important for a person to make some progress; however, his ability to correct mistakes and to abandon prejudices that are shown to be wrong is at least equally important. The idea that all functions that naturally occur in physics may be fully replaced by their Taylor expansions is wrong - even though one needs a high precision for the flaw to reveal itself.
Another, related surprise is that the Taylor expansions one obtains from perturbative quantum field theory (or perturbative string theory) don't converge even though they're the best perturbative approximations of answers that exist and that are finite. The perturbative expansions are the so-called asymptotic series. At the beginning, the terms are decreasing because we're adding powers of g. However, the prefactors ultimately win and the terms start to grow again. The minimum term - i.e. the uncertainty of your sum if you try to calculate it as exactly as you can (by summing a subset of the series) - happens to be of the same order as the first non-perturbative contributions.
I don't want to explain these things pedagogically here; after all, most of these topics have been covered on this blog many times in the past. Instead, the point of this essay is to emphasize that even refined, tasteful expectations about the "ethics" that all mathematical structures should obey may be shown to be wrong and one must carefully listen to Nature - and to Her native language, mathematics - if he or she wants to penetrate deeper under Her skirts.
Of course, I still believe that analytic functions are totally crucial in physics and it's always sensible to ask what happens with functions if we continue them to complex values of energies, momenta, and even complex values of distances and times. However, all the lessons about the divergences, non-existence of Taylor expansions for some functions etc. have to be respected; they cannot be denied. And they invalidate many conclusions that are based on naive reasoning.
If I return to the discontinuous functions that admit a Fourier expansion, it's another lesson. Various functions such as the wave functions in quantum mechanics have no reason to be differentiable infinitely many times (although every function that appears may be approximated by totally smooth functions arbitrarily well). Instead, we allow wave functions that are discontinuous - and even wave functions that are distributions rather than functions (distributions are even more "unsmooth" generalized functions than discontinuous functions). The distributions actually turn out to be some of the most natural bases - bases of eigenvectors of observables with a continuous spectrum.
Leibniz and Newton invented the integrals and much of the maths behind them was very pragmatic for a few centuries; top theoretical physicists would use a laissez-faire approach to the formalism. Do whatever makes sense to you. We could say that Newton's methods to deal with the integrals - and derivatives - resembled the approach of contemporary engineers. But it has simply worked, at least for the people who were competent.
Mathematics got more strict and it produced several definitions of the integral. The key physics insights of the Newtonian mechanics are, of course, independent of the type of the integral you use - the Riemann integral or the Lebesgue integral, for example. Those mathematical formalisms are just useful to make you a distinguished rigorous speaker - or a picky sourball, depending on your perspective.
People who try to use an excessively (and often unnecessarily) rigorous language or formalism often like to think of themselves as intellectually superior; they often miss the fact that the rigorous epsilon-delta gymnastics and related sports have been invented largely in order to bring the calculus and other disciplines to the people who were not competent - people who were so much less gifted than Newton et al. that they simply needed (and need) to be controlled by mechanical rules that prevent them from deriving some "really stupid things".
However, at some level, when you're trying to push the machinery of classical physics to its extremes, you may be genuinely forced to clarify all the rigorous details and use one definition or the other. The Riemann integral doesn't seem to have too many problematic features - but it's ill-defined for many functions where it's still sensible to claim that the "natural integral" should be well-defined.
The Lebesgue integral based on the measures of sets (a formalized notion of the generalized total length of intervals) seems to be more modern a way to formalize the integrals because it rarely ends with the conclusion "undefined". However, it also brings some "paradoxes". Of course, they're not real contradictions; they just contradict some naive intuition that you could consider natural for some philosophical reasons.
In particular, there exist unmeasurable sets.
How do we construct them? Take all real numbers from the interval [0,1): zero is allowed but one is not. Write this set as a union of classes C_i such that each class contains all the numbers that differ from each other by a rational difference. There are infinitely many - in fact, uncountably many - such classes; if their number were countable, the real numbers between [0,1) would have to be countable, too - but they're not. Now, take one "representative" from each class C_i to form a set M.
What is the measure of M? Well, it must be infinitely many times smaller than 1 because aside from the "representative", there are infinitely many elements in each set C_i. So the measure of M has to be zero. But then the measure of any union of the C_i sets - which can be shown to have exactly the same measure (each of them) - will also be zero and you can't ever get 1, the measure of the interval [0,1). So the measure fails to strictly additive if sets such as M exist.
Most mathematicians would tell you that the unmeasurable set M exists. However, the construction of the set M depended on the "axiom of choice" - the ability to choose one representative from each class in a set of classes, even if the set of classes is infinite. ;-) This is a very bizarre construction, especially because the set of classes is not only infinite but uncountable.
(In those constructions, "sets" and "classes" mean pretty much the same thing. I don't want to go into some even more formal aspects of set theory.)
We will surely never be able to "physically" declare who the right representatives are. In fact, researchers in "set theory" have demonstrated that the "axiom of choice" can't be proved from the other axioms, and you may live without it, too. Most mathematicians just prefer to say that the axiom of choice is true because it makes some of their proofs more straightforward.
Well, laziness shouldn't be the ultimate criterion to choose the axiomatic systems.
The price we pay for the validity of the axiom of choice is the existence of unmeasurable sets. I used to believe and I still believe that the unmeasurable set M constructed above is highly pathological and can't ever appear in meaningful applications of mathematics such as physics. From this viewpoint, it's better to deny the axiom of choice (for infinite sets of sets) even if it means that some proofs have to become more cumbersome. The advantage of this attitude is that you may adopt another axiom, namely that all subsets of the [0,1) interval have a measure and the measure is exactly additive. Isn't it pretty?
Of course, there's no "physical operational procedure" to decide whether mathematical axioms such as the axiom of choice hold.
The argument here is all about one formal axiom. There's no "pyramid of interesting mathematical structures" that would be born once you adopt the axiom of choice as one of your commandments. Instead, the axiom of choice is only good to prove the existence of some "academically real" sets of representatives that you can't ever use because they can't be specified one by one. The sets whose existence is postulated by the axiom of choice can't be "individually constructed".
I need to mention one more thing. When we were writing our textbook of linear algebra, I would consider e.g. the Jordan decomposition of matrices to be an unnecessary, pathological generalization of the diagonalization of the matrices. Well, the Jordan matrices surely do exist but they're not needed in the "healthy applications of maths", I would say.
In particular, observables in quantum mechanics are Hermitian operators and those can be diagonalized. The matrices that can't be diagonalized form a set of measure zero in the space of matrices.
However, it's true that I am much less convinced about my "segregation thesis" against the Jordan matrices today. It is true that non-diagonalizable matrices are of measure zero. But measure-zero objects are often very important even though they may look pathological at the beginning. They may be equally important simply because they're different. And their singular locus in the space of possibilities simply may be important exactly because it is qualitatively different from the generic points.
Cecotti, Cordova, Heckman, and Vafa have proposed T-branes for which the matrix of deformations is upper triangular - i.e. it only admits a Jordan block decomposition rather than diagonalization. Such objects are inequivalent to the generic branes - for which the Higgs field is diagonalizable - and they may be important for the full picture.
I don't really believe that too many students of linear algebra will need T-branes in their research. And I don't really believe that it is extremely important to teach students what the Jordan block decomposition is. But I would probably be less combative about this point than 15-18 years ago.
Another huge subtopic would be singularities such as those in general relativity. It is self-evident that many people, including Roger Penrose, used to have or still have lots of emotional prejudices about the ugliness of such objects. However, as the Penrose-Hawking singularity theorem has showed, singularities almost inevitably form under pretty generic circumstances.
The Cosmic Censorship Conjecture by Penrose tried to ban a subset of singularities - the naked ones (those that aren't dressed in an event horizon that makes them invisible). In 3+1 dimensions, the CCC is likely to hold, at least with some reasonable assumptions. In higher dimensions, it is getting increasingly likely that the CCC simply fails.
Penrose was almost certainly wrong in thinking that the existence of naked singularities automatically implies a physical inconsistency or a breakdown of predictivity. It doesn't. Quantum gravity may deal and survive with such objects. In fact, there are many important singularities in string theory - especially the time-like ones - whose fate and physics have been almost completely understood. They include orbifolds, conifolds, and others. The full physics is totally well-behaved - and, in some cases, totally equivalent to physics on smooth backgrounds - even though geometrically, the objects look singular and "pathological" as shapes.
But that's just another example of an aesthetic prejudice that could lead you to throw away or deny a theory or a possibility that is actually completely consistent - and, in some cases, may be true and profound. One must be careful about such philosophical expectations that may always turn out to be wrong.
I must mention that when I began to study string theory, I also believed that open strings, because of their singular endpoints, were pathological - and were against the "spirit" of the smooth world sheets. But we know that this opinion was just wrong, much like many other wrong negative opinions that refer to "spirits".
Perturbative string theories with open strings - e.g. type I string theory - are as well-defined as the purely closed-string ones. In modern terms, open strings end on D-branes and D-branes are topological defects - also kind of singular loci in spacetime. But that doesn't make them unphysical or inconsistent; in fact, D-branes may be physically equivalent to the fundamental strings themselves (e.g. via S-duality in type IIB string theory). The main lesson is "be careful about the spirits". In particular, the amount and the stupidity of crackpottery that various people - and I don't mean just Lee Smolin - justify by the "spirit of general relativity" is just overwhelming.
P-adic and adelic numbers, exotic differentiable structures
If we return to the unmeasurable sets for a while, I think that it will always be the case that the information about particular sets whose existence is guaranteed by the axiom of choice will remain inaccessible. However, there are many exotic mathematical structures - much more exotic than unsmooth functions or functions without Taylor expansions - that may play a much more important role in physics of the future.
The preprint that was linked to at the beginning of this blog entry talked about exotic differentiable structures on R^4. (They're exotic in the same sense as the 992 exotic 11-dimensional spheres. Note that 992 = 2x 496, 496 is the dimension of the Yang-Mills groups in 10D superstring theories, and this fact was found in the year 1984 = 2x 992. Only the latter identity is demonstrably ludicrous; all the previous ones may actually have a rational justification that goes beyond the word "coincidence".) I don't really understand how string theory may be defined on those bizarre backgrounds. But some people may understand it and they may be right.
Various exotic differential structures, much like p-adic and adelic numbers, have to perform a sequence of "infinitely many surgeries" on the usual continuous objects in order to achieve what they are all about. Such infinite surgeries may look as pathological as the hand with the fingers with their own fingers at the top.
However, the notion that a structure is pathological is often just an emotional prejudice. Such things could become viable. If your hands had those fingers with fingers with fingers, you could do many interesting things.
In physics, such fractal-like structures may conceivably offer us the same degree of predictivity as ordinary continuous structures - and non-exotic differentiable structures. After all, the predictivity of quantum field theory boils down to their scale invariance at short distances (in the ultraviolet). Fractals may possess a similar self-similarity - i.e. they may obey a discrete version of scale invariance - and such a thing may be equally relevant for producing robust predictions that don't depend on too many parameters (or they don't depend on any parameters). In fact, we know that something of the sort does replace the scale invariance in Matrix theory.
In particular, people know how to compute various things in p-adic string theory.
All such structures look very different from the phenomenological vacua of string theory - whose spacetime is continuous at long distances; and whose spacetime shouldn't be surgically manipulated at too short (sub-Planckian) distances because those ultrashort distances should keep their status of "non-existence".
But from some broader viewpoint, it's plausible that the Universe around us is just an excitation of a vacuum that has, aside from the conventional landscape of "morally analogous" vacua, cousins that are totally different, use different differentiable structures, p-adic numbers, fractals, and many other things. They may be solutions to the same underlying equations or conditions - equations that we only know from their approximations optimized for the backgrounds we consider non-pathological today.
I don't know whether it's actually true and one must realize that it is a risky business to swim in those seemingly pathological waters because you may get easily disconnected not only from all the empirical data but also from the important "mathematical data" that are related to the philosophy of state-of-the-art theories of physics. But it's totally conceivable that sometime in the future, state-of-the-art physics will be dominated by mathematical structures that we consider to be pathological curiosities today.