The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.
Claude code was used to organize the material and to run simulations. The simulations were to determine the likelihood that the text was Semitic vs Tom got lucky. Tom has assigned probabilities to each of the syllables he has proposed sound values for.
I think I caught this guy's reddit posts on the subject. Someone was playing around with statistical analyses of a big Linear A corpus + some other corpora. There was an extremely clear signal that Linear A seemed to be much more similar to one other corpus than to the others. This was the first time I've ever heard of something that might* have been a good hint for decipherment. There's a Dutch professor emeritus (in linguistics) who claims it is Hurrian-Urartian and he's been posting youtube videos about his "decipherment" but he didn't seem too convincing to me.
Claude helped write code to read and parse the corpora and to do some fairly basic statistical analysis along the lines of "which Linear A symbols most often occur together" and "if we use known Linear B sound values, which of the other corpora most often have vowel similarities with the Linear A corpus".
You can write that code yourself or you can ask an LLVM to write it for you. The provenience of the code isn't important.
*) He later deleted some of them, I think. What was still there on reddit a few weeks ago had dead links to a web site of his with statistical tables and I believe also code.
There are two relevant decipherment ideas: the new one that is the first one sufficiently "un-bad" to be taken seriously and the one by the Dutch emeritus.
The Dutch guy thinks he can recognize some names (personal and toponyms) and that seems to be his main evidence for Linear A being Hurrian/Urartian. His theory doesn't seem very convincing (and his youtube videos are incredibly booooring).
Sure it is. We're humans, not robots (well, I think I am, and I presume you are as well, but for all we know, we could be living in a simulation), so if the non-deterministic system decides to generate code that calls the variable foo one day and bar the next, as long as the code still does what's being asked of it, why do I care that the non deterministic system chose to call the variable something different when run on Tuesday? There's the computer science definition of determinism and the engineering result of "does it work", which are at odds. It's like the halting problem. We haven't solved the computer science definition of the halting problem, but give some C code with a loop that won't terminate to Claude, and it'll call that out as not halting.
All things aside, I think this misses the forest for the trees on the halting problem.
It's not about being able to throw claude or codex at a loop and having it evaluate it for halting, it's about being able to do this for arbitrary code. Computer science rigourously defines the halting problem as not computable and undecidable. within the framework of using something akin to static analysis using any deterministic Turing machine.
There's not really a question of "solving" the halting problem like there's some as-yet unknown way of generally figuring out if arbitraty code halts. Turing proposed a proof in 1937 in favour of undecidability of what we now know as the halting problem, building on ideas first articulated by Church a few years prior.
Frankly, if anything, it's reasonable to say that the halting problem's been solved, just in the direction of undecidability rather than decidability.
Anyway, back to LLMs; as code gets more complex, the robot will need a bigger context window, more hardware resources, and more time, all of which will be variable due to the noise inherent in the system. It'll be difficult to put a useful upper and lower bound on how much computing power and time it'll take to figure out if a program ever halts. Which is all a bit moot, frankly, in the context of halting, but useful to keep in mind in the more general context of using these things as analysis tools.
> even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.
If you had the other things, being "stochastic" is not even remotely a show-stopper. Stochastic processes abound and are the reason the mathematics of statistics was developed in the first place, ultimately allowing us to create such things as LLMs.
When all the relevant steps gets published, I absolutely expect a lot of people to (attempt to) reproduce this work even though LLMs are stochastic.
My issue with this is that it's a form of "soft" reproducibility, where it'll work for many (maybe even most!) people, but that depends on the way the original prompt was formulated (read on) and the state of the random noise in the system.
On the prompt formulation; prompts with very similar formulations (in terms of both semantics, hamming distance, or both) can lead to _wildly divergent_ outputs in my experience. It's not rigourous, and when that divergence happens, it's extremely difficult (arguably impossible, by nature of the architecture of transformers) to identify why the divergence happened and where.
Every day when you lower your butt onto your chair, you trust a stochastic system enough to assume you'll rest on the chair safely and not spontaneously phase through, which would lead to rather gory and painful terminal experience.
Physics at macro scale is stochastic, which is a good reminder that stochastic != uniformly random. Expected distributions matter.
> QM has such small standard deviations as to be irrelevant on the macro for things like bums and chairs
I was going to segue into thermodynamics as a backup example, but you made me think of something better.
> IMO a better example would be the stochastic nature of quality control in manufacturing.
How about, more specifically, food manufacturing? Or maybe, let's talk about cooking?
Cooking is as stochastic as it gets, and we handle it fine. It could be better - the better version is called "chemical process engineering", it's what cooking looks like when you care about quality and consistency of output, and can afford the equipment and process actually necessary for it. Regular people don't (i.e. neither care, nor can afford) - we call this cooking. It's an art, not a science, and people not only do it, but love it, and tie their identities to it, and build businesses around it, and a culture that embraces all the compromises (and calls the more serious approach "unhealthy").
> Cooking is as stochastic as it gets, and we handle it fine. It could be better
My attempts at making bread have been too stochastic, in that it hardly ever produces nice results.
But yes. Eyeballing how much dried herbs to put in my dishes because I like what 2-isopropyl-5-methylphenol does for them. Usually it works, sometimes it's just a bit too Italian.
Might not be the amount, you may have not controlled for humidity or temperature (wink wink), or just that the timer on your oven is off by one minute per every ten minutes, and its bang-bang thermostat never actually reaches the temperature you set on the panel, and...
... in some sense, it's a miracle most people deal with this kind of bullshit without complaining much.
(Probably because they don't realize it's something to complain about. It's just how things are.)
There are too many value judgments in this post. You can "cook" like "regular people" do, and be completely serious, and apply chemical and physical knowledge in doing so, and test the output for quality; generally that's what restaurant chefs do. It doesn't make sense to cook like you're tooling an assembly line, because you aren't cost-optimizing and packaging a product that needs to sit on a store shelf for weeks, months, or years while maintaining its desired qualities.
Speaking generally, food produced though "chemical process engineering" (a.k.a. factories) must compromise on many axes, one of them being nutritional content. We intuitively do not care about several of these dimensions when cooking food with fresh ingredients, at least not at the scale of, say, Kellogg's or General Mills.
Maybe that's evidence of accepting a stochastic process in our daily lives, but you're kind of selling the tradition and science of cooking short when you argue that factory-produced food is a "more serious approach".
The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.