There is a myth that Einstein's discovery of general relativity was due to his following beautiful mathematics to discover new insights about nature. I argue that this is an incorrect reading of the history and that what Einstein did was to follow physical insights which arose from asking that the story we tell of how nature works be coherent.Smolin assumes that Einstein discovered general relativity, but if you study the history of general relativity, you find that most of the key ideas were developed by others.
Poincare and Minkowski developed four-dimensional spacetime, Poincare the finite propagation speed of gravity and the first relativistic field theory for gravity, Minkowski and Laue the stress-energy tensor, Ricci, Levi-Civita, and Grossmann the Ricci tensor, Poincare a partial explanation for the Mercury anomaly, Hilbert the Lagrangian approach, Schwarzschild the black hole metric.
The big equation of general relativity that was used to solve the Mercury anomaly was just Ricci = 0.
But how exactly did Einstein perform the seemingly miraculous feat of inventing a theory that correctly describes phenomena that had not yet even been observed? There is a myth which is usually trotted out to answer this query, which is that Einstein was a lone genius who followed beautiful mathematics to discover his great theory. Genius, inspired by aesthetics. Mathematics as a tool of prophecy.Einstein constructed several myths about himself.
No one was more responsible for spreading this myth than Einstein himself, who described in several essays and popular talks in the 1920's and later how he followed a trial of mathematical beauty to his discovery of general relativity. As Einstein wrote in his autobiographical notes,
``I have learned something else from the theory of gravitation: no collection of empirical facts, no matter how comprehensive, can ever lead to the formulation of such complicated equations ... [they] can only be found by the discovery of a logically simple mathematical condition that completely, or almost completely, determines the equations. Once one has those sufficiently strong formal conditions one requires only little knowledge of facts to set up a theory.
In the last twenty years historians have been doing a careful job of studying what Einstein actually did during the eight years of hard, often frustrating work ... And it was very different from the myth.
In the absence of ideas and insights about nature, Einstein fell back on mathematics as his guide. He constructed a myth about how mathematical beauty had been prophetic for his invention of general relativity and he attempted to use it to justify his forays into unified field theory.I wrote a book on this subject: How Einstein Ruined Physics.
Einstein's search for a unified field theory failed, and the roots of this failure are his embrace of mathematical beauty as a guiding principle. Over the thirty-five years between 1920 and his death in 1955 Einstein attempted many versions of a unified field theory. He tried higher, hidden dimensions, they failed. He tried more general versions of curved geometries beyond the geometry used in general relativity. They all failed to produce a useful unification.
According to popular accounts of the scientific method, such as Thomas Kuhn's The Structure of Scientific Revolutions, theories are invented to describe phenomena which experimentalists have previously discovered. ...It would be amazing if Einstein predicted all of these things out of pure theory in 1915. But Einstein did not even believe in most of them himself.
This simple schema does not apply to general relativity. All the characteristic phenomena that general relativity describes were unknown in 1915 when Einstein published his theory. These include the expanding universe, black holes, light bending in gravitational fields, gravitational lenses, time slowing down in gravitational fields, gravitational waves, dark energy. Not only were these phenomena not yet observed in 1915, most of them had not even been thought about. The fact that a century later, all of these are well confirmed is a triumph unmatched by any other theory in the history of science.
The expanding universe has little to do with general relativity. Gravity is an attractive force. If the universe were not expanding, it would have collapsed already, under either Newtonian gravity or general relativity.
Black holes and gravitational lenses are consequences of strong gravitational fields, regardless of relativity. So is bending light, if light has mass. Dark energy was not predicted or discovered until about 20 years ago.
Einstein did discover time slowing down in gravitational fields, but as a consequence of special relativity, in 1908.
Poincare discovered a relativistic theory of gravity in 1905 that predicted gravitational waves.
So Einstein's 1915 theory is not responsible for any of these things.
Special relativity was invented to explain experiments, like the Michelson-Morley experiment. General relativity was invented to reconcile special relativity with gravity, and to explain gravitational causality and the Mercury anomaly.
I wrote a series of posts on how relativity might have been invented independently of experiment. But historically, it did not happen that way. All of the worthwhile scientific theories were invented to describe phenomena which experimentalists have previously discovered.
Theoretical physicists today like to ignore experiments and try to find the great unified theory based on pure thought, so they like to promote the myth that such an approach has succeeded in the past.