The Rise of Stochasticity in Physics

New paper:

The rise of stochasticity in physics
Hans A. Weidenmüller

In the last 175 years, the physical understanding of nature has seen a revolutionary change. Until about 1850, Newton's theory and the mechanical world view derived from it provided the dominant view of the physical world, later supplemented by Maxwell's theory of the electromagnetic field. That approach was entirely deterministic and free of probabilistic concepts. In contrast to that conceptual edifice, today many fields of physics are governed by probabilistic concepts. ...

The success of the theory led physicists to adopt what became known as the mechanical world view. According to that view, all physical processes can be understood on the basis of Newton’s equations. The theory is completely deterministic. There is no room whatsoever for probabilistic concepts which play a role only in the analysis of statistical and systematic errors. Such errors were considered epistemic and, therefore, did not challenge the validity of the mechanical world view. (A notable exception is the discovery of the dwarf planet Ceres in 1801. After its discovery, the planet was lost and could be retraced only with the help of Gauss’ statistical least-squares method).

That is a pretty big exception. The Newtonian mechanics seems deterministic, but in practice it is not. Like Gauss, you have to make imperfect observations, do some statistical estimations, and make a probabilistic prediction.

The whole field of AI used to be mostly deterministic, but now they follow the same stochastic pattern. They run on deterministic computers, and use lots of deterministic formulas, but the big AI models make very heave use of statistical estimates and probabilistic predictions.

I no longer agree with saying that classical mechanics is deterministic. All of science is inherently stochastic.

Historically, probability was developed after calculus. I thought that calculus would have been the conceptually more difficult subject.

Advocates of many-worlds theory reject probability. The theory does not make some worlds more likely than others, as some assume. I think that the followers must have some fundamental misunderstanding of what probability in math and science is all about.

Today, use of probability and statistics is pervasive in all of science. Every prediction is made with some probability, and every test is analyzed with statistics. So I do not think that there is any such thing as deterministic science. Classical physics is not, and neither is biology, chemistry, medicine, or anything else.

In summary, in the last 175 years physicists have been led or been forced to ascribe an ever increasing role to probability in the description of nature. I have listed four causes for that development: Loschmidt’s number supporting Maxwell’s theory, irreversibility leading to Boltzmann’s approach, Bequerel’s discovery, spectral lines and black-body radiation leading to quantum theory, and Poincare’s discovery of classical chaos. Random-matrix theory is different. It was not imposed on physicists by an experimental or theoretical discovery but was introduced to compensate for the incomplete knowledge of the Hamiltonian.

He is right about those trends, but probability would have become essential in all of science even without those four trends.