His 1996 Probability in Quantum Theory says:
For some sixty years it has appeared to many physicists that probability plays a fundamentally different role in quantum theory than it does in statistical mechanics and analysis of measurement errors. It is a commonly heard statement that probabilities calculated within a pure state have a different character than the probabilities with which different pure states appear in a mixture, or density matrix. As Pauli put it, the former represents "Eine prinzipielle Unbestimmtheit, nicht nur Unbekanntheit". But this viewpoint leads to so many paradoxes and mysteries that we explore the consequences of the unified view, that all probability signifies only incomplete human information. We examine in detail only one of the issues this raises: the reality of zero point energy.The German means "A fundamental uncertainty, not only obscurity", but probably sounds better when Pauli says it.
A lot of other smart physicists have said that quantum mechanics shows that nature is intrinsically probabilistic, such as R.P. Feynman.
These physicists are smarter than I am, but I say that they are wrong about this. It is crazy to say that probability is a physically real thing, and quantum mechanics does not require such a view. I detailed my opinion in this post on probability last year, and in many other postings and essays on this blog.
Probability is mathematics, not physics. It is essential to nearly all empirical quantitative science, because it gives the tools for comparing theory to experiment. As Jaynes explains, probability gets used in statistical mechanics and analysis of measurement errors, and the use in quantum mechanics is not much different.
Pauli's phrase is reasonable if interpreted to just mean that quantum uncertainty is not just the obscurity of not knowing the values of hidden variables. The hidden variable theories have all failed, as best demonstrated by the Bell test experiments.
Most physicists go farther, and argue that quantum uncertainty is some sort of physical thing that is fundamental to the theory, that proves indeterminism, that can be used for super-Turing computation, and that is even conserved in black holes.
Probability does none of those things. (The world may be indeterministic, but for other reasons.) It is just a mathematical construct that tells us what to expect.
Just look at the picture of Jaynes, and compare it to the wild-haired physicists! Who is more likely to give you the straight truth about the nature of reality?
Here is a statistician squirming about a blown prediction:
Data guru Nate Silver of FiveThirtyEight.com tells NPR's Scott Simon how all the forecasts, including his own, were so far off in predicting the results of this week's British election. ...Probability is what allows him to explain away his failure.
SIMON: Yeah. I've got to tell you, Mr. Silver, you're not giving people much of an incentive to read FiveThirtyEight seriously if you're essentially backing away from the idea that you can reach any conclusions. That's why people read you.
SILVER: Sometimes the right conclusion is to say that people are too sure of themselves, right?
SIMON: Yeah.
SILVER: Sometimes it's to anticipate uncertainties in your environment. And over the whole course of our life span at FiveThirtyEight - and we've predicted many things in politics and sports and other events for many, many years - historically, one thing that's distinguished us is that our probabilities have been right over the long run. That when we say something has an 80 percent of happening, it happens about 80 percent of the time and it doesn't happen about 20 percent of the time.
Quantum mechanics is similar. If an atom is in a superposition making one energy value 80% likely, then probability is what lets you explain away a measurement that gives another energy value.
The election is analogous to the atom energy measurement. Silver's view is that elections have fundamental uncertainties, so he can just give probabilities. Just like the quantum mechanics professors.
Roger,
ReplyDeleteYou're right about probability, but the idea of probabilities as modeling a physical "randomness" in the universe is so strong in most people that it requires some kind of explanation. Why do people keep being drawn to this false conception of probability?
Well, there's an interesting story there. The first step is to always carefully distinguish between frequencies and probabilities. They may both be positive and sum to 1, but they are different in nature. Real physical frequencies are no different than any other fact of nature. They can measured, estimated, predicted, and so on. They have a physical reality independent of our state of information. Probabilities on the other hand measure uncertainty implied by a state of information and can change dramatically when our state of information changes.
The confusion is severe because we can model the uncertainty of any physical thing, including frequencies. Thus given a frequency distribution f_1,...,f_n we can talk about the probability of it P(f_1,...,f_n). Again note that P() is describing the uncertainty in our knowledge about a real frequency distribution and is not itself the frequency of anything.
What often happens in statistics (and statistical mechanics) is that for large n, these probabilities distributions P() become sharply peaked about single distribution F_1,...,F_n. This distribution is really the mode or “mostly likely” value of P(f_1,...,f_n).
Because of that, if real observed frequencies distributions f_1,...,f_n lie in the high probability region of P() they will be approximately similar to F_1,...,F_n. It will appear in fact as if the f_1,...,f_n were a “random draw” from F_1,...,F_n.
When people conceive of a probability distribution as a modeling a physical randomness, or talk about “estimating a probability” distribution, and so on, they're really talking about F_1,...,F_n. But instead of calling it the “most likely frequency distribution” they very confusingly call it a “probability distribution”. It's amazing how much confusion can be swept away by always calling frequencies “frequencies” and reserving probability for modeling uncertainty.
As a side note, finding the mode of P(f_1,...,f_n) involves a maximizing procedure. To cut a long story short, in many instances this maximizing condition is equivalent to finding a F_1,...,F_n which maximizes the “entropy” subject to constraints. That's why you get the bizarre unexplained historical fact that almost all the distributions (normal, binomial, Poisson,....) of regular classical statistics are in fact maximum entropy distributions (bizarre because classical statistics had nothing to do with statistical mechanics historically).
Incidentally, one consequence of this perhaps relevant to Quantum Mechanics, is that all probability distributions satisfy the same formulas (sum and product rule). However, the F_1,...,F_n can potentially follow any rule or transform in any way, just like any physical value potentially can. So whenever you have something that looks like a "probability distribution" but doesn't follow the usual rule of probability theory, then maybe you're actually talking about a F_1,...,F_n.
At the end of his life, Jaynes saw his work in probability theory as a major proven success, but had nothing like the same confidence about his work in QED.
ReplyDeleteOne gets the feeling he strongly felt that Bayesian statistics had a huge unrealized role to play in understanding Quantum Mechanics, but he likely thought his own speculations would have to change a lot before it was all said and done.
Still, it's a pleasure to read a first rate physicist thinking critically about Quantum Mechanics in a very down to the earth way, without any of the usual mysticism and without any of the usual "I'm the next quirky lone wolf super genius who's going to revolutionize our understanding of the world" style of self promotion and inflated self regard.
Then we are led to ask what distributions are more real than others. The game is endless when it comes to essentialisms. Feynman basically conceded that "negative probabilities" were just convenient: http://cds.cern.ch/record/154856/files/pre-27827.pdf
ReplyDeleteI see the use of negative probabilities in front-running the options market. Seems that's all "smart" people are good for these day. Financial parasitism by means of central banker ZIRP.
I don't know why people still talk about Quantum Mechanics as if Quantum Field Theory does not even exist. The two are very different. All the paradoxes of QM disappear in QFT and its conceptual framework is necessary for building intuition for doing engineering. I think its just too hard for literally everyone. Ever since the days of Schwinger when he was ignored for being too formal (too many equations), its been a descent into anti-science behaviour.
ReplyDeleteWhen I say quantum mechanics, I mean to include quantum field theory. Yes, QFT is a local and relativistic theory, and I would not say it has problems, but those who complain about EPR paradoxes and other such matters have the same objections to QFT.
DeleteRoger, my comment was not entirely serious and was intended to mock the true believers of 'Symmetry Principles Forever!!!' who are especially fanatical working in QFT. The reason being symmetry arguments and group theoretical reasoning are strictly speaking unnecessary in non-relativistic quantum theory. Its not really well known today but in the heyday (20's and 30's) of quantum theory, group theory was widely considered a plague of useless abstraction (Gruppenpest in German). I
DeleteWell I can't answer that question for everyone, but I can give you Jaynes's reason. Jaynes was there at the birth of QFT, knew or studied from most of the key players, and in his day job was a QFT ninja. He also seems to be have been a big fan of Schwinger.
ReplyDeleteThe reason he persisted in looking at QM seems to have been that he thought all the major success attributed to QFT didn't actually use QFT and could be derived from conceptually simpler theories that might serve as a better jumping off point to discover new physics. Or could at least suggest interesting experiments which might yield which stood some kind of chance of discovering something useful.
>> Just look at the picture of Jaynes, and compare it to the wild-haired physicists! Who is more likely to give you the straight truth about the nature of reality?
ReplyDeleteWow, I've seen some probabilistic extrapolations from small data sets before.