JonathanBaxter

@wikiemol Rice says you can’t have a perfect always correct decider for any nontrivial semantic property of arbitrary programs. But humans can’t do that either. So Rice doesn’t show that machines “can’t do semantic reasoning”; it shows there’s no complete procedure. I agree there’s a real groundedness issue for LLMs: they’re not situated in the world the way we are, so their semantics can be quirky and sometimes unmoored. But ours are also “quirky” in their own way; it’s not obvious that this is a clean human vs machine line, as opposed to different kinds of embodiment, training signal, and verification.

@tao At present, to a highly capable expert such as yourself, the AI looks like stochastic cleverness. But to someone operating a step or two below, it looks like genuine intelligence. So is its just a question of scale or is it a fundamental deficit? I'd agree that it's probably fundamental, since it's exactly the experts working at the frontier that are generating the truly new ideas, and that's why they notice the AI is not. We plebs that are just following along can't distinguish. Until AI becomes creative, it will remain this way. But then, before the reasoning models came out you could have said the same thing in respect of reasoning. I did, and I was proven wrong almost immediately. Turns out you can simulate reasoning pretty effectively. Can we do the same for creativity? I wouldn't bet against it.

@wikiemol Rice doesn’t say “no semantic property is decidable by a computer” in general. It says there’s no total, always-correct algorithm that decides any nontrivial semantic property for arbitrary programs. That still leaves plenty of semantic questions decidable for restricted program classes, restricted languages, or fixed systems. If by “decide” you mean the everyday epistemic sense (often tell, with fallibility, in many cases), that’s a different notion than “decidable” in Rice’s sense, and Rice doesn’t rule that out for computers (or humans).

@wikiemol This is basically Church’s theorem / undecidability of theoremhood for sufficiently strong r.e. theories, not Rice. But the same comment I made above applies: it shows limits on universal, infallible decision procedures (deciding theoremhood / non-theoremhood in general), not an impossibility of semantic reasoning in practice. You’re in good company: it’s the same (incorrect in my opinion) move Penrose makes, i.e. taking an undecidability/incompleteness result about perfect deciders and turning it into a sweeping "in principle" claim about minds or machines. So if you want to argue from authority, you certainly have me beat :)

@wikiemol You’ve definitely weakened the claim by dropping the "humans can do what AIs can't" conclusion, but the remaining statements are still pretty slippery. "The set of questions an AI can't answer is not knowable by the AI" is only a theorem-like claim if "knowable" means a universal, always-correct internal decider over all possible questions. In that strong sense, self-reference/undecidability is the real issue, and introspection doesn’t evade it. If "knowable" is meant in the everyday sense (often able to tell when it will fail), then it's not a computability barrier. It's an empirical calibration/engineering question. "It's possible a human can know something an AI can't" is doing almost no work if "possible" just means "not logically impossible." Unless you're asserting some principled asymmetry like non-computability in humans, the default working assumption cuts the other way: whatever humans can do, a sufficiently capable AI could in-principle do too, just with different failure modes.