The Invention of Large Language Models

Most people think that LLMs were invented by Google in 2016, or by OpenAI several years later.

AI expert Andrej Karpathy wrote on his blog:

The Unreasonable Effectiveness of Recurrent Neural Networks

May 21, 2015

The concept of attention is the most interesting recent architectural innovation in neural networks.

He constructs some LLMs, and his results seem pitiful compared to what is done today, but I would say he has a proof of concept.

Google introduced the transformer in Attention Is All You Need, a 2017 paper. As you can see, attention was already a hot idea at the time.

I am not sure who should get credit for inventing LLM. The basic ideas of neural nets go back decades. They got a whole lot smart when gaming GPU chips became fast and widely available, and AI researchers figured out how to use them efficiently.

No Quantum Nonlocal Effects

This is an elementary fact about quantum mechanics.

Dr. Bee explains:

Most importantly, if you do something to one of the pair of entangled particles, 4:46 that does nothing to the other. Say you turn the spin of the particle on the right upside-down 4:53 even though you don’t know what it is. Then the spin of the other particle doesn’t change at all. 4:59 No, it doesn’t. You wouldn’t believe how often I see even physicists get this wrong. I just 5:05 the other day heard a talk from someone who works on quantum computing for heaven’s sake 5:10 who said that if you manipulate one of a pair of entangled particles then that will non-locally 5:16 affect the other. It will not. It’s just that if you make a measurement on one of the particles, 5:23 then that will tell you something about the other. Because they’re correlated. ...

The person who 5:50 makes a measurement on one end can’t tell that a measurement was even done on the other end. 5:56 So: Entanglement is real, we know that. Whether spooky action is real is still a matter of debate, 6:04 but you can’t send information faster than light with either.

She is correct. No one has ever found any nonlocal quantum effects. Just correlations.

It should not be complicated. The same happens classically.

What she says about "spooky action" is a little confusing, so here is her explanation.

2:36 Strange or not, Bohr said that when we measure a particle, 2:40 this superposition “collapses” and suddenly the particle is in only one place. It’s this collapse 2:47 that Einstein referred to as spooky action. Because it would indeed be faster than light. 2:53 The moment you find the particle in one place, you instantaneously know it can’t be elsewhere. 3:00 Einstein disagreed with Bohr. Einstein thought that quantum particles are really only in one 3:06 place and that the sudden update of the wave-function just means that you 3:10 have learned the particle isn’t elsewhere. And his main argument, here it comes, was that by 3:17 claiming the collapse is a physical process, Bohr was introducing a “spooky action at a distance”. 3:26 Even if this spooky action existed though, it couldn’t transfer information. Just because you 3:32 find out what’s going on elsewhere doesn’t mean you sent information there.

When you find a classical particle, you immediately know it cannot be elsewhere. If that is spooky, then classical mechanics is spooky. Regardless, no information or anything else goes faster than light.

Usually she branches into a plug for superdeterminism. Mercifully, she did not this time.

Five Ways to Think About Quantum Supremacy

Aventine reports:

When Google announced that it had achieved quantum supremacy in 2019, the headlines were thrilling.

The world of quantum computing had taken a remarkable step. Google, with its Sycamore quantum processor, had performed a calculation in 200 seconds that, the company claimed in the journal Nature, would take a supercomputer 10,000 years.

This feat, named quantum supremacy by John Preskill, a theoretical physicist, back in 2012, promised to usher in a new world of computing performance. ...

Only it didn’t play out the way Google hoped or expected. ...

And then, earlier this summer, researchers from Shanghai Artificial Intelligence Laboratory in China completed the same task in just 14.22 seconds, driving a final stake through the heart of the Google quantum supremacy claim.

It’s not the only warning sign for the industry. Venture capital investment in the sector has fallen off a cliff, from $2.2 billion globally in 2022 to about $1.2 billion in 2023.

It then reports the opinions of five experts, but none of them say that quantum supremacy has been achieved. The closest is Scott Aaronson who says:

Quantum supremacy can be achieved and then unachieved later. It’s a little bit of a moving target in that sense. But all expect that we’ll eventually get to a place where quantum computers are just routinely doing things that classical computers cannot replicate within thousands of years or millions of years, and at that point there’s no more arguing about it.

Achieved and then unachieved? This is a bit like a mathematician saying something was proved, and then disproved. If it was later disproved, then it was never really proved.

Gil Kalai argues that quantum supremacy is impossible, and explains further here.

ChatGPT is better at English than Tigrinya

From a recent Nature magazine podcast:

0:00 chat GPT has a language problem so my language is called T it's spoken in in 0:07 Ethiopia and also in a Syria ...

how we design 3:38 this technology how it could be used and the impacts it could have in this podcast we are going to 3:45 explore the relationship between llms and languages and ask what must be done 3:50 to ensure that AIs work for 3:56 everyone. I think everybody in the community agrees that we need to democratize AI there should not be 4:02 disproportionate benefit in one language versus the other, right? so we want Fair 4:08 access you know and we want to empower communities in different 4:13 [Music] 4:24 languages

I never heard of this language, so I google it:

Tigrinya is a Semitic language spoken in the Tigray Region of Ethiopia and central Eritrea1.

Actually, ChatGPT does surprisingly well in this language.

No, nobody agrees that ChatGPT should be equally beneficial in all languages. It is impossible. Most of the worthwhile training materials are in English, and English is well-suited to being the world's language of choice.

I post this as an example of how the leading science publication has gone Woke.

Albert explains Weirdness of Quantum Mechanics

David Z. Albert is one of the leading popularizers of quantum mechanics, and is on this podcast:

David Albert is the Frederick E. Woodbridge Professor of Philosophy at Columbia University, director of the Philosophical Foundations of Physics program at Columbia, and a faculty member of the John Bell Institute for the Foundations of Physics. This is David’s eighth appearance on Robinson’s Podcast.

I think he is mostly known for trying to give a philosophical defense of string theory, in the absense of any evidence.

He tries to explain what is weird about QM

24:01 what's often considered striking and unsettling about quantum 24:08 mechanics is that at the beginning of the 20th century people start doing all 24:15 kinds of experiments where unlike in the Newtonian case where we're just taking 24:20 it for granted that yeah there're you know you look at these little dust Moes 24:26 or something like that how they're moving around they themselves presumably consist of billions upon billions of 24:33 these Elementary Point particles which we can't actually see we're sort of taking it for granted that things are 24:38 going to work out in the beginning of the 20th century people fooling around with cathode ray tubes and and stuff 24:46 like that begin to be able to keep track 24:51 of the Motions of individual Elementary particles 24:58 and these particles um um are behaving in ways that are almost 25:08 inconceivably bizarre okay in 25:13 particular people manag to convince themselves by doing lots of experiments 25:19 with these Elementary particles that things like electrons for 25:26 example could be in you you know um it's possible for an electron to be located 25:31 at this point in space and possible for an electron to be located at that point 25:37 in space those are the familiar Newtonian possibilities here or here or 25:42 here or here or here what these experiments at the beginning of the 20th century suggested 25:49 to people and I'm condensing here 30 years or so of wrestling with 25:56 the with the results of these experiments in the beginning of the 20th century there are certain sets of 26:03 experiments so-called interference experiments um double slit experiments 26:10 if you can read about this want to read about this in the literature Neutron interferometry experiments stuff like 26:16 that there's a whole um family of of experiments which 26:23 slowly persuades people that um 26:29 um that apparently there are certain states that 26:34 electrons can be in electrons and neutrons and all elementary particles 26:41 can be in um once again there's a possible state where the electron is at 26:46 Point a there's a possible state where the electron is at point B what people became convinced of is that in addition 26:54 there are possible states of electrons such that the very question is it 27:01 located at Point a or is it located at point B or is it located at neither of 27:07 them fails to make sense okay that um that the question that there are certain 27:15 situations in which the question is the electron in box a or in box b or in 27:22 neither of the boxes is like um is is is is like a 27:32 question of the form um um is the number five Married 27:38 okay or or what is the weight in grams of Catholicism okay or something like 27:45 that um questions that philosophers often refer to as category mistakes okay 27:51 um that there can be situations of a material particle an electron okay where 27:59 asking whether or not it's in box a is somehow like asking what the marital 28:06 status of the number five is or what the weight and grams of Catholicism is okay 28:12 um this strikes people is absolutely bizarre people felt forced to 28:18 conclusions like this because if you tried to tell the story of these 28:23 experiments in a way that used locutions like well at this this point in the story the electron I suppose must have 28:30 been here or at this point in the story the electron must have been there or even if you just insist that at this 28:37 point in the story there must be some place where the electron was because after all it got from here in the 28:43 beginning of the experiment to there at the end of the experiment it must have gone by some particular route either 28:51 through this route or through that route you try to insist on that you try to tell yourself a story about how these 28:59 experiments went okay that's consistent with the results you got you find that 29:05 every particular Claim about which route the electron might have taken somehow collapses into nonsense becomes 29:13 inconsistent with certain of the results of these experiments that you actually did okay so an idea grows 29:21 up um um that it appears to be a feature 29:28 of the way these fundamental particles behave that for every pair of states 29:34 that such a particle could be in that is the state of being located at a and the state of being located at B there's also 29:42 another radically unfamiliar but physically possible state which is 29:47 referred to as the superposition of being located at a and being located at b or the quantum 29:54 mechanical superposition of being located at a or being being located at and being located at B which is which 30:03 which we can we can argue from our experiments is not a case of being 30:08 located at a and not a case of being located at B and not a case of being 30:14 located at both A and B as you often find in the popular literature when they 30:19 try to talk about superposition and that's very bad too and that would reduce the mysteriousness of this way 30:27 below what it actually is what these experiments suggest is that it is wrong 30:33 to say under those circumstances that the electron is located at a and that it's wrong to say that it's located at B 30:40 and that it's wrong to say that it's both at A and B and that it's also wrong 30:45 to say that it is neither at a nor is it at B okay if you think there must always be a 30:53 fact of the matter about where the electron is that exhausts the line iCal 30:58 possibilities okay um um and so people felt 31:04 forced um to acknowledge that what these experiments seem to be screaming at us 31:12 okay is that for any two states that an electron could be in A and B there are 31:18 other physically possible States in in in you know which can't be rightly 31:25 characterized by saying that they're at a can't be rightly characterized by saying it's a b can't be rightly 31:31 characterized by saying it's a both and can't be rightly characterized by saying it's a neither okay and rather you seem 31:41 to be confronted with a situation in which um asking whether the 31:47 electron is at A or B is like asking about the marital status of the number five okay it's just a 31:55 nonsensical question good um

No, this is not weird. The same thing happens in classical mechanics. If you put a ball in a box and shake it up, you do not know where it is. When you open the box, you find it in one position, and not others.

Why is it surprising that electrons behave this way, when classical balls do also?

He goes on to strongly attack Bohr for saying that it does not make sense to talk about the exact location of an election in a box, until you open the box and look at it. Then he praises various alternatives, such as hidden variables and many-worlds.

Bohr was right, and Albert is wrong. The electron is not a particle. If you ask for the exact location of the election, you are basing the question on the faulty premise that the electron is a particle, and it has no answer because it is a meaningless question. As Albert complains, the Physics community accepted Bohr's philosophy about a century ago.

Albert is one of many QM expositors who argue that the textbooks do not make sense so there must be something better. They are just wrong. The textbooks present a perfectly good theory, and all the alternatives have horrible defects.

Sabine Hossenfelder is another one. She just posted a podcast on Can Quantum Physics Explain Consciousness After All? Her answer is No, of course, as she ends up saying Bell's Theorem requires superdeterminism.

She is a free will denier, but see the new paper Decision theory presupposes free will. You have to reject a lot of good science, if you reject free will.

Peebles on Philosophy of Physics

Astrophysicist P. J. E. Peebles writes:

I argue that research in physics operates under an implicit community philosophy, and I offer a definition I think physicists would accept, by and large. I compare this definition to what philosophers, sociologists, and historians of science, with physicists, say we are doing.

He argues that there is an unspoken phylosophy of Physics that most physicists agree with, and they do not necessarily agree with what philosophers say.

Recall Weinberg’s remark that “no one has been able to think of any way to change quantum mechanics in any way that would preserve its successes without leading to logical absurdities.”

I agree with that. People complain about quantum mechanics a lot, but all their alternatives are absurd.

Einstein offered the elegant thought that (in the English translation by Sonja Bargmann 1954)

The supreme task of the physicist is to arrive at those universal elementary laws from which the cosmos can be built up by pure deduction.

I wrote a whole book on how Einstein ruined Physics with this sort of thinking.

People think that Einstein's greatest accomplishment was discovering relativity in 1905 from universal laws. My book shows that relativity was actually developed by others who worked directly from experimental results.

The above essay is a good summary of mainstream physicist thinking.

Cantor and the Theology of Infinity

Kateřina Trlifajová wrote a new paper on infinity:

Discussions surrounding the nature of the infinite in mathematics have been under way for two millennia. Mathematicians, philosophers, and the- ologians have all taken part. The basic question has been whether the infinite exists only in potential or whether it exists in actuality. Only at the end of the 19th century a set theory was created that works with the actual infinite. Ini- tially, this theory was rejected by other mathematicians. The creator behind the theory, the German mathematician Georg Cantor, felt all the more the need to challenge the long tradition that only recognised potential infinite.

The issue turns surprisingly theological, with St. Augustine, St. Thomas Aquinas, and the Pope on the side of Cantor.

Cantor was convinced that the knowledge of infinite numbers had been revealed to him by God, who guided his steps from pure mathematics to an interest in theol- ogy and philosophy so that he could improved a proper understanding of God and nature.

Cantor's papers from 1883 to 1895 were met with skepticism, but by 1897 Mathematicians were on board with his theory of infinities.

Xkcd just had a cartoon about large numbers, explained here.