Fabian Giesen

@ubernostrum @bert_hubert But mostly, I don't actually care about the productivity. And even if I've seen enough net-positive productivity uses by now to believe it's possible in principle to get significant wins for certain (many?) tasks out of LLMs, I don't think it matters. Because independent of what the tech can or can't do, the "everything else" about it is such a trash fire that I don't want to touch it with a ten-foot pole.

@aeva@mastodon.gamedev.place I, for one, think that if a project's interactions with the wider AI ecosystem are limited to "take their money and run", I can live with it I don't want them on any boards or steering committees but sure they can be paypigs

@aeva @lritter maybe the real piracy is the friends we made along the way? ❤️ Oh sorry. That was One Piece.

@bert_hubert @danluu specifically, strip away the ginormous models, the training data of questionable provenance, the hype men pushing it etc., and you get, at the very least, a technology that seems to be genuinely good at let's call it "fuzzy data mining" on unstructured text. Good enough to convince a lot of people it's way more than that, in fact. And I can definitely think of many use cases for that.

@bert_hubert @danluu That said, right now, you just _can't_ use any of these models without funneling more money and collective mind share at what I think are terrible businesses that are IMO morally (if not necessarily legally) criminal, and who are actively pushing their users to use their products in ways that are tremendously destructive.

@aburka@hachyderm.io @aeva@mastodon.gamedev.place hence "take the money and run" and not "shut up and take the money"

@nothings There's nothing about the tech that makes it so you couldn't use it for the things where it works well enough, and drop it for the parts where it doesn't. That's not at all where we are right now, but it's definitely not _impossible_ (nor would it even take _any_ improvement of the current tech, this is a social problem), which is why I think arguments along the lines that there's no evidence for realistic wins are weak.

@nothings I still think the companies and their business models are a trash fire. I also still think that esp. the cumulative effect of LLM spambots, LLM junk PRs etc. has been a massively negative impact on my life, and is likely to create a "subprime tech debt crisis" on projects that went all in on it very soon, if it's not already happening, which seems possible. On balance, I'd much rather they didn't exist. But they do, and "there's no wins to be had here" just doesn't seem accurate.

@jplebreton@mastodon.social although to my shame I must admit that, owning as I do only volumes 1 through 3, I have yet to reach the parts where anything is actually drawn

@jplebreton@mastodon.social ah, good old "A most excellent machinery for the facilitation of drawings posthaste, comprising methods of both describing the rendering of a perspective image and executing said rendering in the shortest amount of time that modern inventions afford us, including a small treatise on a novel formalism describing mathematically how to put Cartesian points into perspective."

@Doomed_Daniel@mastodon.gamedev.place @zeux@mastodon.gamedev.place @lisyarus@mastodon.gamedev.place This is not "fast math" level shenanigans where the semantics are extremely fuzzy and the perf win often minor; this one is _usually_ fine (except when it isn't, of course, but this is a lot more rare than say fast-math screw-ups), the perf difference was quite substantial, and it was a very niche thing to want to turn off.

@nothings@mastodon.gamedev.place It's more common than you think. Some automatic UV unwrapping methods guarantee perfectly orthogonal (e.g. LSCM, although that's now a bit long in the tooth). Others don't guarantee it exactly but try to be close. And many standard methods (e.g. the cylindrical projection commonly used for heads/faces) also guarantee it.

@nothings I don't know what the !/$ for that looks like since I don't know what we pay for the tokens it uses, but as a coder this is functionally just another set of compiler warnings that I get as part of the review/CI flow, and likewise the friction from BugHawk false positives is not any worse than the friction from false positive C++ compiler or other static analyzer warnings.

@nothings no particular conclusion here, other than the thing I already said waaay up in the thread, namely, secondary tooling like that can be a significant force multiplier for at least _some_ tasks, it doesn't need to be perfect (or even particularly good) to be useful, and some it is definitely within reach of current LLMs. E.g. BugHawk has been part of Epics checkin flow for what, like, a year now? I don't know what the potential overall upside for stuff like this is, but it's nonzero.

@bert_hubert @danluu e.g. recently at work, somebody debugged an issue by, essentially, enabling a shit-ton of verbose logging, getting 500MB of logs from a good run and 500MB of logs from a bad run, pointing Claude at both log files and the code base and essentially prompting " is happening in subsystem X, this one log is good and this one is bad, find out where funny business happens first and point at the originating source line" and it did manage to do so.

@Doomed_Daniel @lritter 44 Jahre alt und immer noch aktuell...

@dngrs nope. I have, however, historically confused uninterested and disinterested

@Doomed_Daniel @dotstdy @nothings definitely! That's also what I went into with the "shitty prototype to show the vision" example. Even if you later Ship of Theseus the whole thing, if it did nothing other than convince people that there's a tool-shaped hole here that you can plausibly fill with something, for many problems, _that's still a win_.

@nothings There definitely are things you can use them for that where their shortcomings don't particularly matter. (But still don't justify the general trash fire situation.) Whether it's worth it financially in a non-subsidized market is also unclear. But "it doesn't even work" does not jive with my experience. Some of it does. A lot of it doesn't. Everybody LLM'ing it up gives you a ton of the bad along with the little good.

@nothings and "it's only approximately right and will keel over if you hold it wrong" is already true for most of the non-vibe-coded tools of that type that I use on the regular. including things like CDep (and most other build systems) etc.

@nothings There is not an infinite supply of such tool-shaped holes, but all projects I've been on for more than a few weeks had a few of those. And some of those definitely are a net positive. E.g. BugHawk at Epic (LLM-based static analyzer) is about the same as the other static analyzers we were already using in terms of false positive rate, but it finds different bugs and now has, on multiple occasions for me, found subtle bugs that slipped through code review and would've sucked to debug.

@nothings I definitely don't feel comfortable with any statements along the lines of "there's nothing here". Tools like BugHawk or PatchSpace (the vibecoded GUI patch diff analyzer I alluded to way early in the thread; didn't get into specifics because it didn't matter) have reduced turnaround time for tracking down certain kinds of bugs massively, and they wouldn't exist without LLMs.

@ubernostrum @bert_hubert Anyway. I don't think what's happening here is "there's no way for these tools to increase productivity significantly". AFAICT, what's actually happening is that they enable some uses that are productive (and reduce shit work), some that sap everyone's will to live, many that erode the codebase in the long term, and a lot of ~no-ops, and since they're used so indiscriminately, mostly projects get the geomean of all of these which usually turns out to be ~1.

@ubernostrum @bert_hubert Mind, the same thing has also happened in the other direction, where on one memorable occasion, I got cc'ed in to review a vibe-coded bullshit PR that didn't even do what it was supposed to do, turned out to be untested, and wasted cumulative hours of the reviewers' time in the process, being a significant net negative force multiplier, because not only didn't it do anything (and had to path to getting committed), it kept multiple other people from doing actual work.

@ubernostrum @bert_hubert The whole thing is hardly rocket science but as it is the thing was online and mostly working (well enough to be worth using anyway) within something like a day and added significant momentum to the underlying effort going forward. Not least, because it instantly freed up a lot of time for those 10 people who were previously getting by with a bunch of less-structured scripts and manual steps. (And then turned around and added more functionality to the new app.)

@ubernostrum @bert_hubert I'm still solidly in the LLM hater camp myself (for other reasons), but "there's no significant productivity improvements for devs that can be unlocked via LLMs" is a claim that I've now seen several counterexamples to and that I can't in good faith keep making. I don't think it's a 10x for SW in general, but also, "SW in general" is a uselessly broad category for any such argument to begin with.

@lritter ich bitte sie, herr ritter.

@nothings See also things like oodas/oodis and the spec that I now routinely use during Oodle debugging. None of this vibecoded, I mention because it's along the same lines as dev-specific secondary tooling that wouldn't have existed if not for the external pressures that created it, and that turned out to be useful for way more things than we initially thought. (But only for me, since AFAICT I'm the only one to use it.)

@bert_hubert @danluu The key problem with the accidental/essential complexity distinction is that it's ridiculously sensitive to framing and non-functional requirements. For a concrete example, take something like say a simple spell checker like classic Unix spell that checks each word in one file against a dictionary of known words (say an ASCII file, one line per word).

@bert_hubert @danluu On late 70s HW, that doesn't nearly fit into memory. You could have a tool that reads one file word by word and opens the dict, keeps re-reading it for every word, doing a brain-dead linear search, and solve it that way. On that HW you wouldn't consider this a solution, because it doesn't actually work for the workflow in mind; it would be slow enough to be essentially useless.

@ubernostrum @bert_hubert This is definitely not a case of "adding a lot of code to an existing complicated system", but it is a case that is somewhere between "have a LLM write a throwaway script" and "LLM code going into the main code base". Arguably the real thing here is more of a project management failure because this kind of tooling should've been existed _before_ there was a major emergency that made better tooling a necessity, but it didn't, and the vibe-coded thing certainly helped.

@nothings@mastodon.gamedev.place @zeux@mastodon.gamedev.place but yes, non-orthogonal T/B are commonly skipped, same as e.g. non-uniform scaling on object->world transforms (or even just scaling entirely), and for largely the same reasons - it is only sporadically useful and makes everything a bit harder and more expensive

@dotstdy @nothings For sure.

@ubernostrum @bert_hubert Mind, my personal experience with LLMs has been terrible, but as it is, something like three fifths of most repositories I work on are code in that liminal space between one-off tool and shipping product (internal dev tools that are incredibly useful but fail spectacularly if you hold them wrong). By last Dec I was kinda where you are right now. What's happened since at work has made it ~impossible for me to maintain that position.

@ubernostrum @bert_hubert Mind, I'm not at all sold on the way a lot of this went down (and was arguing for better tooling for this task for years), but I'm definitely not comfortable saying "I see no way this ends up game changing" because I've now, several times, seen massive impacts from special-purpose tooling that did a pretty obvious thing but didn't get made for a long while just because there was never time.

@ubernostrum @bert_hubert The reality of the matter is that most of us can just fuck off for an afternoon and try a long shot without anyone noticing, and most of us can't fuck off for a week and try a long shot in the same way, so something that allows you to get a shitty prototype of a tool that would take a week within an afternoon can be good enough to sell people on the vision if nothing else, and gain momentum on tooling that ends up transformative.

@nothings@mastodon.gamedev.place @zeux@mastodon.gamedev.place you pretty much always need N for lighting but you might not need T/B, for example for distant LODs where you might skip normal mapping entirely e.g. for a character that's 30 pixels tall, pretty much the only thing you get out of normal maps is more specular aliasing. :P

@nothings @pervognsen point being, MS especially in the 90s was a fucking nightmare, but this case in particular hinged on a very bullshit-y software patent

@ubernostrum @bert_hubert I don't want LLM-generated code in anything I anticipate being maintained over long stretches of time (I consider not understanding key parts of your own project to be a liability), but I've definitely had several cases in my career where I had very slapdash tooling+manual intervention but it was very hard to argue for the ~week it would've taken to do a good version, because without at least a viable prototype it was hard to estimate how much it'd save in the long run.

@nothings Very hard for me to quantify what the ROI for that was and if it's been worth the extra time I spent on it, but I mention all this because there's a lot of specialized tooling like that which is potentially very useful, some of it (like GPlayer) IMO plausibly within the reach of current LLMs, and getting more "secondary tooling" like that (at least some of which otherwise never exist) is definitely non-zero impact.

@bert_hubert @danluu this is the same kind of "solving the problem by brute-force shit work" as my toy "spell checker by re-reading the dict every time" example, and it's certainly not useful for every problem, but at the same time it doesn't seem implausible to me that there are entire categories of bugs amenable to this kind of thing if you have tooling that does it, and that a human would just never do manually if there was any other way

@nothings @pervognsen OK, didn't know about that part.

@bert_hubert @danluu On a modern machine, that approach is still dumb, but it would totally work at more than acceptable speeds for small ASCII documents. Moreover, with newer langs/libs, you could read the whole dict into a hash table in RAM in something like 10 lines of Python or whatever, and get a less comical sln. None of these were viable in the 70s, but essentially all the complexity of a 70s practical spell checker turned out to be accidental after all.