Replying to
@zhenyi@mastodon.social in case anyone else is wondering, the original comic is Buttersafe's "Traps":
https://www.buttersafe.com/2011/01/27/traps/
Remote
ansuz
@ansuz@gts.cryptography.dog
Formerly ansuz@social.cryptography.dog, but I've finally switched from hosting my own mastodon instance to hosting a gotosocial.
See my pinned post(s) for an introduction. I manually approve follow-requests, so put something in your bio or say hi so I know you're a real person.
0 Followers
0 Following
49 Posts
Joined January 11, 2026
profile picture by:
website:
Replying to
@jonny@neuromatch.social it's astonishing how much of software security boils down to
A: we implemented a reliable sandboxing methodB: actually, no you didn'tGOTO 1
containerization, restricted root permissions, hardware attestation... I seriously don't understand why anyone buys such claims.
Replying to
@jacqueline@chaos.social I understand that this isn't for everyone, but it's amazing how many problems I've managed to avoid by simply pushing my code to manually configured git remotes over ssh.
Replying to
@christmastree@mastodon.social oof, that's terrible 😬
I'm happy to have helped clear that up!
Replying to
@jonny@neuromatch.social @ehproque@neopaquita.es it'll be cheaper once Claude figures out how to rig up a cold fusion reactor /s
Replying to
@jonny@neuromatch.social I realize I'm probably going to be saying this until I die, but their first mistake was thinking of that freely available software as their "supply chain".
I was today years old when I learned that the perpetrators of the Jonestown massacre used "Flavor Aid" to poison their followers. It was misidentified as Kool-Aid by police.
Nearly 50 years later we're still using Drinking the Kool-Aid to suggest cult-like behaviour.
Replying to
@mark@tacobelllabs.net @burnitdown@beige.party @srtcd424@mas.to @evacide@hachyderm.io I've seen a number of developer-focused IRC channels struggle with the same problem as every other platform. People whose first stage of research was asking a chatbot follow up by running their half-baked ideas by some actual experts without disclosing that at least half the thoughts aren't their own.
These exchanges often go on for a while before people figure out that they're dealing with slop, at which point they get annoyed and reinforce the trope of unpleasant IRC veterans yelling at beginners, who I assume only end up retreating back to their sycophantic LLM "tutors".
Replying to
@hackbod@mastodon.social @cwebber@social.coop to be clear, neither I nor @joeyh@sunbeam.city suggested that it was at all a good idea to consider LLMs as a type of compiler. Speaking just for myself, I think it's a horrendous idea.
I was simply agreeing that an LLM could technically be made to be deterministic, so the issue is not strictly about their current lack of determinism, but our inability to reason about what their outputs will be for a given input.
Replying to
@remixtures@tldr.nettime.org in addition to the obvious risk that Google will hold on to the biometric data they will collect through this, a few seconds of video should be sufficient to thoroughly profile imperfections in your camera's lens(es), which can be used to identify photos or video taken with the same camera.
This BBC article quotes Jessica Fridrich, who I believe pioneered use of the technique and its applications in digital forensics:
The hidden fingerprint inside your photos
Replying to
@otfrom@functional.cafe I haven't gotten around to reading that, but if I ever do I'll keep that headcanon in mind
Replying to
@maklem@social.leuchtstark.dev indeed. I don't think @david_chisnall@infosec.exchange , myself, or Alan Kay meant to suggest that "problem recognition" alone was sufficient for an org to avoid dysfunction. I agree, though.
Replying to
@siguza@infosec.space @Kroc@oldbytes.space @jacqueline@chaos.social calling it a fear or phobia seems like exactly the sort of thing the AI hype crew would like
Replying to
@pluralistic@mamot.fr I've jammed with Adam and can confirm he is the real deal, but certainly did not expect to see him popping up on fedi 😅
Replying to
@mttaggart@infosec.exchange is this about the Low Level Learning Youtube channel? Because it sure sounds like it could be about him
Replying to
@thomasjwebb@mastodon.social @cwebber@social.coop @joeyh@sunbeam.city 💯
Local models like llama could be reworked to accept a seed for their RNG. There'd be less risk of them becoming unavailable, and they'd be both deterministic and reproducible, but they'd still be terrible for all the other reasons that LLMs are terrible .
"Sovereign" and reproducible slop is still just slop 🤷
Replying to
@david_chisnall@infosec.exchange Alan Kay has (re)framed things this way as well. I believe he described the important skill as "Problem recognition"
Replying to
@kbal@fedia.io @mullvadnet@mastodon.online @crazyeddie@mastodon.social this. If it was anonymous then it would also be transferrable, which would defeat the supposed purpose. It's almost as if the actual purpose is increasing surveillance capabilities 🤔
Replying to
@ricci@discuss.systems I imagine their backroom conversation went something like
...but if we had to get participant consent we wouldn't be able to do a study like this...
🙄
A chat with @davidbenque@mastodon.ie about the modern use of the sparkle emoji to signify #AI led me to wonder if there was a good way to track exactly when this trend started.
It then occurred to me that because emojis are just a type of text, usage of sparkles should show up in google trends. Sure enough, it did.
At first I thought we might have passed "peak sparkle", because interest appeared to decline after February, but that seems to be an artifact of having generated the graph halfway through the month of March.
Once more, for the people who missed my previous rants on the topic: It's okay to adopt the stance that age verification is a bad idea.
You don't need to weaken your position by saying it might be okay if they eventually used Zero-Knowledge Proofs or pseudonymous systems.
First, such schemes aren't suitable for this domain even at a theoretical level. If you think they might then you don't understand the problem.
Second, even if ZKP was an applicable solution, it would still prevent undocumented people from accessing the internet like those that various governments choose to recognize as human.
Third, one goal of a functioning society ought to be producing responsible adults, and treating them as complete children until they suddenly turn 18 is not a reasonable way to accomplish that.
Government-mandated age verification for the internet is bad public policy, and it's 100% okay to say as much.
#AgeVerification #surveillance
At some point in the past I put World Frog Day on my calendar because silly events like that are an easy way to make my future self laugh.
Unfortunately, it seems that the guy who runs the World Frog Day website has gone all-in on "AI art", aka #slop.
His "art" page is a bunch of frog images generated with MidJourney, and I guess he's already gotten so much hate mail about it that he wrote up a page explaining "The Environmental Benefits of AI-Generated Art" which is just painfully wrong 😬
I like the idea of raising awareness of the importance of frogs in ecosystems, but uhhhh, fuck this guy.
The extent to which slop has permeated everything tech-related has made a domain that is both my profession and hobby pretty unbearable for a while now.
I felt like I was reclaiming some space for myself by figuring out effective ways to detect, ban, or poison scrapers that requested resources from my server, but that too became unrewarding after a while.
I still believe that it's important to resist these technologies, but I'd prefer to direct my energy towards rather than against something. So, a few weeks ago I started playing with the godot game engine, experimenting with small projects and gradually progressing towards some more ambitious game ideas that I've had in my head for many years.
I don't have too much to show for it just yet, but the whole process has greatly improved my mental health, which is something.
Replying to
@xrisk@brain.worm.pink I can see how it could come across like that, but you might be projecting a bit. That wasn't my intention at all.
I only meant to point it out because most people (understandably) don't realize that this is true of binary classifiers (if they even know what a binary classifier is).
For example, for a while the EU was pushing really hard to implement mandatory client-side scanning for CSAM in the context of end-to-end-encrypted messaging apps. The model would run on your device and attempt to flag any offending media that they couldn't otherwise detect in public posts or unencrypted cloud backups.
The problem is that if anyone extricated that model from the app, they could do what's called an inversion attack, and produce a new model capable of generating content that the classifier would have flagged. In some cases this is sufficient to closely approximate samples from their training set.
In this example there's no ML needed, as the software works purely mechanistically, but the principle is the same. There's nothing the author can do to address it, as the problem is fundamentally unsolvable.
Replying to
@joeyh@sunbeam.city I'm glad to see that someone else has considered this angle. It's always bugged me a little when I see the "they aren't deterministic" argument, but I've kept it to myself because nobody likes a pedant and of course @cwebber@social.coop already understands as much.
I just worry that if this critique were to become more popular then the LLM makers would just implement the ability to specify a seed, then sit back and play the game where they say
we heard your criticism and have addressed it
Most people have no reason to have developed an advanced reasoning capacity about randomness, and I dread having to explain to them how something can be both deterministic and stochastic in nature 😣
Happy Stanislav Petrov day!
Petrov was responsible for overseeing an early warning system and judging whether a retaliatory nuclear strike should be launched in the event of an attack against the Soviet Union.
The system indicated an attack was inbound. He judged it to be a false alarm, and disobeyed his orders.
There's a lot more to the story, but the part I find most interesting is how it was suppressed. To quote the wiki article:
According to Petrov, he received no reward because the incident and other bugs found in the missile detection system embarrassed his superiors and the scientists who were responsible for it, so that if he had been officially rewarded, they would have had to be punished.
A friend sent me a link to some music on youtube ("1 hour Iranian Jazz").
It was just a still image with some music in the background, a classic pattern for slop music, so I was suspicious.
I looked in the description and saw some jazz musicians I recognized, so I figured maybe it was just an audio recording from some live session that got thrown on youtube.
After listening for a bit, I started to get the sense that it wasn't who it claimed to be. One of the listed musicians was Vijay Iyer, who has a fairly recognizable style on piano, and this didn't seem like him.
I scrolled through the comments for bit, and eventually stumbled across a comment from Vijay Iyer himself asking politely to have his name removed.
I am so incredibly tired of this bullshit.
#music #slop #ai
I spent a few hours yesterday trying to do all the KYC stuff with an online payment processor (I'll share the reason why in a while 😉).
Website verification was one of their required steps, but their crawler got caught and banned by my anti-scraper defenses, and I couldn't figure out which bot was theirs, so I temporarily disabled it that system and unbanned all the IPs that might have been theirs.
I forgot to reenable it after verification was complete. Traffic to my webserver spiked 3000% overnight.
EDIT: typo fix
Replying to
@rachel@transitory.social @cstross@wandering.shop @mwl@io.mwl.io @pwassonchat@eldritch.cafe yes! I used to work on collaborative editing software in France, and heard "edition" quite often to describe the process of editing.
It still sounds a little weird to me, but it's perfectly logical.
Replying to
If we include some recent estimates of private debt that the major LLM companies had tried to keep off of their books, then we get around 3 Trillion USD sunk into this bubble.
According to this one study mentioned in an article by The Guardian, 330 Billion USD might be sufficient to permanently end world hunger. That's from almost six years ago, so maybe we need to adjust a bit for inflation, but either way we're uncomfortably close to the point of having been able to end hunger ten times over.
Replying to
@neil@mastodon.neilzone.co.uk the mesh networking field/community/hobby has had serious issues for a long time.
Routing is a deep and fascinating topic, so you can always find academic types who want to engage with things on a purely intellectual level.
Then you get proto-fascist prepper types because decentralized comms are good to have in an emergency.
Then you get grifter types because there's money to be made doing bulk orders on cheap hardware and flashing some FOSS stuff on it.
There's always some folks promoting themselves as an alternative to the status quo of bad ISPs, but who secretly want to become the next generation of landlords.
None of these people can build a network on their own, or if they could it wouldn't be very useful. So, in practice the people who are still involved after a number of years tend to be those who are willing to look the other way while collaborating with fascists and grifters. _Mess_ hardly even begins to describe the situation 🙈
Replying to
@engineer@metalhead.club @danielquinn@mastodon.social @HennaVirkkunen@ec.social-network.europa.eu Historically, that has not prevented the projects with signed contracts from going forward. A lot of (EU-funded) Blockchain/Web3/NFT research projects were carried out long after their valuations had collapsed.
Boosted by @jwz@mastodon.social
Today I learned that there is an AI data center being built in the Indian city of Raipur, where I used to live.
Raipur is very hot (Wikipidia says the record high is 47.9C), and I'm pretty sure that doesn't take the effect of humidity into account. It wasn't always this hot, but it gets a little worse each year and that has added up.
The majority of the city's electricity is sourced from coal power plants, and for a variety of reasons its grid is not especially stable, with particularly frequent blackouts during the rainy season. That often means that there's no AC during the most humid and second-hottest time of the year.
It is dry year-round except for those few months of monsoons, when the local groundwater reserves are replenished. Despite the intensity of those storms, there have been progressively worse water shortages every year. I've done a lot of rooftop gardening there to reduce the urban heat island effect, and I've watched all but the most heat and drought-resistant plants wither and die. At times water was being brought in by tanker trucks and there just wasn't enough for both human use and the whole garden.
I find it difficult to imagine a worse place to build an AI data center, and yet here we are. They are going to burn coal to power the AI chips and cooling rigs during heat waves and water shortages that are already literally deadly to the people living there.
This is the sort of thing people are justifying when they talk about how much more productive they are thanks to their spicy corporate autocomplete. The anger that I feel is not some abstract moral high ground, but a visceral reaction to having gone outside in deadly climate conditions to spread an insufficient amount of water on my dying plants.
When they say that AI is the future, this is what that actually means.
I find it very distressing to see that ImageMagick is included in the "open-slopware" list, not due to "purity culture" as boosters propose, but because of good old-fashioned threat modeling.
It is incredibly difficult (bordering on impossible) to write large projects in the C language without introducing memory-safety issues that can introduce Remote Code Execution (RCE) vulnerabilities.
Code for converting between exotic media formats is particularly prone to bugs which enable RCE (see this list of fairly recent RCE vulns in Imagemagick).
The more popular a particular software dependency is, the more valuable it is for attackers who want to introduce a backdoor. We were very lucky that a backdoor in xz was discovered before it could be meaningfully exploited, but we honestly have no idea just how many instances of such a social engineering attack have gone undiscovered.
It isn't particularly difficult to poison LLMs with information related to specific domains. BBC reporter Thomas Germain recently manipulated ChatGPT into returning results confirming that he he ranked 1st place in a non-existent hot-dog eating contest.
- Imagemagick is written in C
- it handles basically every exotic image format you can think of
- it's used by approximately everyone, from big tech to the mastodon server on which you are probably reading this post
- they're now using LLMs trained on data that anyone could have poisoned to develop new features
I sincerely doubt this will end well.
I just sent my first-ever DMCA takedown request because I discovered that some slop/aggregator website is hosting a complete copy of one of my blog articles but with tons of ads/tracking scripts 😠
Replying to
@cwebber@social.coop @joeyh@sunbeam.city agreed!
Using a deterministic PRNG for an LLM would only mean that it could be relied on to behave consistently under identical circumstances. It might be fair to call something like this reproducible, but I expect compilers to be predictable, which is a different thing entirely.
Replying to
@crazyeddie@mastodon.social @kbal@fedia.io @mullvadnet@mastodon.online It should be possible to prove that you are over 18 without revealing your exact age. There are systems for proving arbitrarily complex statements, though I'm not sure whether those are the ones the EU system has adopted. In any case, a simple approach would be to use an interactive ZKP, so both participants (the prover and the verifier) would need to exchange information for the verifier to be convinced of anything. In such a situation the prover would need to assist with any sort of brute force attempt, which they would presumably avoid.
My point above about non-anonymous systems being transferable is basically saying that there's nothing stopping a person who is over a certain age from providing proofs for other people who are below that age. It's comparable to asking your friend to share their Netflix password and relaying the OTP that gets sent to their phone number, email, or whatever.
Pseudonymous systems as you describe can't prevent that unless they force you to provide enough information for it to not even really be pseudonymous anymore.
Without tying a proof to a specific device (a problematic concept itself that effectively relies on DRM) or otherwise forcing them the proof to reveal uniquely identifying information there is nothing to stop an older sibling or friend from providing proofs to others under 18. They could try to make that illegal, but they'd need to somehow catch them in the act (as with the crime of providing alcohol to minors), and the technology involved would make that very difficult.
tl;dr the proposed systems cannot do the things politicians and lobbyists have claimed.
Replying to
@stephaniewalter@front-end.social I saw this WHATWG github issue today:
Proposal: agent-discoverable page metadata #12683
Evidently the site-wide LLM policies are insufficiently granular and they want a standard for per-page AI agent-specific instructions 🙄
Replying to
@rachel@transitory.social @cstross@wandering.shop @mwl@io.mwl.io @pwassonchat@eldritch.cafe "prepone" as an opposite of "postpone" is one of my favourite quirks of Indian English
Lots of people have compared today's shitty AI to radium, and asbestos, and lead. Those are fair comparisons, but hear me out...
What if LLMs are more like homeopathy? We've known that homeopathy doesn't work any better than a placebo for decades, and yet here we are.
People still study to become homeopathic practitioners. People still choose it over treatments which have been validated by rigorous scientific studies.
People still get rich selling sugar pills, and spend a portion of those profits to prevent their sugar pill industry from being effectively regulated.
I'm not saying the technology is inevitable. At the same time I think it's important to acknowledge how common it is for discredited practices to retreat to the fringes rather than fade away entirely.
#AI
happy Suez Canal Obstruction day to those who celebrate
https://en.wikipedia.org/wiki/2021_Suez_Canal_obstruction
#boatStuck
Replying to
John Gruber's article responding to Anthropic's watermarking is silly, and he deserves to be dunked on. I just hope people stay vigilant against assuming that the system is good because LLM users are against it. I don't use LLMs, and I'd like to have a reliable system for detecting their output, but unfortunately I don't think that's a realistic possibility.
Remember that the claims about the efficacy of this approach are coming from a company run by people who believe their statistical model has emotions. Their core product is held together with successive layers of duct tape and band-aids, and we know this because its source code was leaked due to poor security practices. Oh, and let's not forget that these people lie constantly in addition to being incompetent.
The watermarking approach they intend to use is a lot like mandatory online age verification in that:
it is unreliableit has negative externalities
It's also a bit like the key escrow systems proposed in the 1990s in that they will need to safely distribute a single shared key to every one of their datacenters. If (when) a key leaks they will need to switch to using a new key for generation, and check against all previous keys at the detection phase.
The paper this is based on has "scalable watermarking" in the title, but that's only in comparison to the approach of checking text against a giant database of all previously generated text. More realistically, it scales very poorly in a bunch of ways, and will only really serve to create additional demand for datacenters.
Our environment and electricity bills are already bearing the costs of generation, but now we'll have people introducing an extra watermark-stripping step, and then more detection steps which (I can't stress enough) are not going to work.