This argument is sometimes used to justify string theory. But development and acceptance of general relativity was driven by experiment, and string theory has failed to even reproduce previous theories.
A couple of new papers discuss the history of general relativity: Outline of a dynamical inferential conception of the application of mathematics and Gone Till November: A disagreement in Einstein scholarship.
These explain debates about how to credit Einstein, because his notebooks are filled confusing errors, and no one can figure out how he got to his conclusions.
Peter Woit quotes a review:
Einstein employed two strategies in this search [for the GR field equations]: either starting from a mathematically attractive candidate and then checking the physics or starting from a physically sensible candidate and then checking the mathematics. Although Einstein scholars disagree about which of these two strategies brought the decisive breakthrough of November 1915, they all acknowledge that both played an essential role in the work leading up to it. In hindsight, however, Einstein maintained that his success with general relativity had been due solely to the mathematical strategy. It is no coincidence that this is the approach he adopted in his search for a unified field theory.Einstein's decisive breakthru of 1915 was discovering that Ricci = 0 could explain the unexplained portion of the precession of the perihelion of Mercury. He had rejected the Ricci tensor when Grossmann based his 1913 theory on it, but Levi-Civita and Hilbert convinced him that it was the crucial tensor.
So yes, experimental evidence was necessary for Einstein. The pure theorizing of his later unified field theory went nowhere.