How the aether became dark energy

The luminiferous aether was part of ancient Greek cosmology, Maxwell's electrodynamics, and this year's Nobel prize for astronomical evidence for dark energy, as explained below. It is also an essential part of quantum field theory, where it is usually called the vacuum state or the zero-point field. The universality and uniformity of the aether is one of those bedrock scientific principles, like conservation of energy.

But if you try to learn about the history of the aether, you are likely to be fooled by the widespread misconception that Einstein abolished the aether as an Aristotelian prejudice in his 1905 paradigm shift. Thus it is hard to get a grip on how physicists started to believe in the aether again after 1905, how they came to realize that the quantum vacuum energy is the same as the cosmological aether.

Now the distinguished Danish historian Helge Kragh has filled the gap with Preludes to dark energy: Zero-point energy and vacuum speculations.

However, if one wants to point to pre-quantum and pre-relativity analogies to dark energy, a more sensible arena might be the ethereal world view of the late nineteenth century. The general idea that cosmic space is permeated by an unusual form of hidden energy - a dark energy of some sort - was popular during the Victorian era, where space was often identified with the ether. The generally accepted ethereal medium existed in many forms, some of them assuming the ether to be imponderable while others assumed that it was quasi-material and only differed in degree from ordinary matter. The ether was sometimes thought of as a very tenuous, primordial gas. According to the vortex theory, cultivated by British physicists in particular, the discreteness of matter (atoms) was epiphenomenal, derived from stable dynamic configurations of a perfect fluid. This all-pervading fluid was usually identified with the continuous ether. The highly ambitious vortex theory was not only a theory of atoms, it was a universal theory of ether (or space) and matter, indeed of everything.

The point is that by the turn of the nineteenth century few physicists thought of "empty space" as really empty, but rather as filled with an active ethereal medium. H. A. Lorentz and other physicists in the early twentieth century often spoke of the ether as equivalent to a vacuum, but it was a vacuum that was far from nothingness. Although Lorentz was careful to separate ether and matter, his ether was "the seat of an electromagnetic field with its energy and its vibrations, . [and] endowed with a certain degree of substantiality." On the other hand, the popular belief in a dynamically active ether was rarely considered in astronomical or cosmological contexts.

Einstein was at the center of this work, but he missed the boat:

Already in the fall of 1913 Einstein withdrew his support of the zero-point energy and the results reported in his paper with Stern. During the second Solvay conference in late October 1913 the question of the zero-point energy was discussed by Einstein, Wien, Nernst, and Lorentz. Einstein commented: "I no longer consider the arguments for the existence of zero-point energy that I and Mr. Stern put forward to be correct. Further pursuit of the arguments that we used in the derivation of Planck's radiation law showed that this road, based on the hypothesis of zero-point energy, leads to contradictions." In a letter to Ehrenfest a few days later he declared the zero-point energy "dead as a doornail" (Mausetot). However, the announcement of death was premature. ...

Einstein would have nothing of it. "It is well known that all theories characterized by a `zero-point energy' face great difficulties when it comes to an exact treatment," he wrote in a paper of 1915. "No theoretician," he continued, "can at present utter the word `zero-point energy' without breaking into a half- embarrassed, half-ironic smile." ...

Kragh credits Walther Nernst and Georges Lemaitre with making the link from the quantum aether to the cosmological aether. To Einstein, the big bang made the cosmological constant unnecessary, but that is not the way Lemaitre saw it. Kragh calls Lemaitre the father of the big bang, and says he foresaw the quantum vacuum as causing what is now called dark energy.

With the exception of Nernst, the zero-point energy of free space was an unwelcome concept that found no place in quantum physics until the 1930s. Moreover, it remained isolated from the vacuum energy associated with the cosmological constant also after 1934, when Lemaître clearly formulated the connection between vacuum energy, negative vacuum pressure, and the cosmological constant. This insight, which did not rely specifically on the expanding universe, could have been stated many years earlier. But it was not, and when it was stated it attracted no interest. Although the cosmological constant is mathematically equivalent to the gravitational effects of vacuum energy, conceptually the two quantities are entirely different: while the first is a property of space, the latter is a quantum effect. It was only after the establishment of modern big bang theory in the mid-1960s that Zel’dovich thought of integrating the quantum-mechanical zero-point energy with the vacuum energy of the cosmological constant, thereby starting a line of development that would lead to the famous cosmological constant problem and give the vacuum energy a central role in cosmological research.

Vacuum energy in the form of the cosmological constant appeared as a crucial element in the inflation scenarios of the early 1980s, but limited to the very early universe. Lemaître’s version of vacuum energy, on the other hand, had an effect that became relatively more important as the expansion proceeds, and in this sense it was closer to the dark energy of modern cosmology.

Someday phyaicists will be embarrassed that they every abandoned the aether.

For completeness, I should say that this paper denies that zero-point energy has been observed, and that energy density does not agree with dark energy density. Presumably there is some reason why the large scale effects are less than the naive sum of the small scale effects.