Replying to
As a side note, if meta has a problem with me disclosing unreleased features, they should have not had a bunch of their senior people publicly say how everything on the VM was mine and there was nothing sensitive on the VM.
Remote
jonny (nonvenomous)
@jonny@neuromatch.social
mastodon 4.8.0-alpha.3+glitchknown or reasonably foreseeable hazard
Digital infrastructure 4 a cooperative internet. social/technological systems & systems neuro as a side gig. writin bout the surveillance state n makin some p2p.
information is political, science is labor.
science/work-oriented alt of @jonny@neuromatch.social
This is a public account, quotes/boosts/links are always ok <3.
7625 Followers
1959 Following
50 Posts
Joined December 05, 2022
site:
manifesto (web):
pronouns:
they/them
personal:
other git:
radicle:
did:key:z6Mkfon2LQyRcsGnHxGz6sKu2RvEj6c76u7eF4aNsuQo1E41
pgp:
5AD4A79EB302DFB716302F896DCB96EF1E4D232D
Replying to
But how would i get such an infected binary on my muse VM? Well! can i interest you in a little "literally the whole thing is designed for that to happen?" The entire idea of the product is that you can ask the thing to go "musey, can you set me up a photo album website and put on my line?" and it's supposed to be able to run some service and host your photos and do whatever. If you prompt muse to do things that would lead it to install software, it will. if you prompt it to do things that would lead it to write software it will. So is there is any affected package from any supply chain attack or typosquat, muse goes npm install or any of the other numerous mechanisms by which installation turns into a vector for arbitrary code execution, then you have meemaw and pawpaw's request to make a photo album installing a token miner without them ever knowing
(except for maybe having to bump their subscription tier, token counting is also broken - the number of tokens being reported as used in the billing window is roughly double what gets reported from the token counts in the raw inference api, so, either a) fraud, or b) all the extra machinery bolted on like sentinel roughly doubles token use, fun!)
Replying to
the simplest problem here is that this is a token harvesting vector. If something wants arbitrary, anonymous inference, here you go! e.g. second-party token market, nation state, self-replicating botnet, who cares! if you can get code running on the system you get to slurp down their tokens and there's no mechanism to tell it's you who is doing it.
permissions for egress on normal https connections are gated by explicit user permission per domain (or they are supposed to be, maybe more on that later depending on what meta says about my next report): you see a popup that says "blah blah is trying to access example dot com" with some optional explainer text. you can accept once, accept always, or deny. One would assume that in normal usage, one would eventually approve access to one of a handful of common domains as "approve always" because they just keep coming up so often - gmail, reddit, bluesky, a mastodon instance, hell why not pastebin, etc. network requests to these domains, once "always approve" can be made by any process within the container.
so that's egress and command & control - any domain with a private user read/write area can be used to receive prompts and exfil token responses. There are other mechanisms of egress but that's the most basic and it works by the design of the system. There are exceptions to this with their sentinel system which i'll get to below and is actually independently really cool and awesome.
Replying to
In muse, the LLM is able to do whatever it wants in its container, that's the point of the container. it is root in the container and runs everything as root. One of the ways that it interacts with things outside the container barrier is through sockets. e.g. this one is /run/hatch/sandbox/space-inference.sock . This is provided so that "spaces" (which we'll return to later) have some means of accessing inference to make them useful. Other sockets, including the other ones in that same directory, have their permission gated by SO_PEERCRED uid/pid identifiers, where the enclosing host vm will e.g. run some process with a specific uid/pid in the agent "cell" container, and then that process and only that process can access the socket.
this socket is not like that, and anything at all can dump into that socket, and the identifiers provided like slug, request_id, action, invocation_id, spawn_request_id, are all arbitrary and unvalidated - aka forgeable, aka unattributable. the user and the LLM are incapable of attributing token usage to any specific process, and token usage is unlimited with arbitrary user and system prompt.
Replying to
Now, the importance of tool calls and agent spawning.
Some tools are binaries that are root-accessible. The way these usually work is this fucked up extracellular digestion process whereby the binary is just a shim that calls some paired socket, hands it stdin/stdout, the tool executes in some container or vm space not visible to the "cell" where the agent runs and has access to, and hands back the result. Most other more interesting tools are not available to be called directly from within the agent "cell." Instead the tool invocations have to come from the agent loop, from the inference god, and executed by the harness. I'll skip technical details there, but that's the intended picture. Point here is that the tool calls can do things that are impossible for even the root user to do themselves, the harness daemon is privileged by SO_PEERCRED and other mechanisms even though it runs within the agent cell that the user can easily get root into.
If you scroll up you'll see the tools that are available to the agents that can be arbitrarily launched without attribution. They include interesting things like "accessing the entire database of things muse has ever done," "read all the memories," "spawn subagents," "open and use the browser which has different permissions than normal web access", "invoke an action on an artifact", and under the second lists' deferred enumeration in the above screenshot, the device tools allow "reading all my text messages and doing lots of other things on my phone," and under the other tools stuff like "access my social media accounts"
so with arbitrary unattributable agent spawning, you get to do a bunch of stuff that is outside the normal agent cell, which is why i submitted the bug bounty report because that is explicitly mentioned in their bounty list and they should have fucking paid me
Replying to
Now there may be some meta-heads in the crowd that are like "but what about Sentinel and all the external monitoring stuff that should watch malicious botnets and blah blah blah." that's an interesting system in itself, but i plan on submitting a few more bug bounty reports in the next few days about these systems, and who knows! if meta fucking pays me for the bounty then we might never hear that part of the story.
that's all for now!
Replying to
So, summary: There is arbitrary inference that is root accessible, everything runs as root, agents can be spawned, exfil is trivial, and a malicious binary can come onto the user's system through casual prompting, explicit code-sharing through the yet-to-be-released Spaces feature, walked through by a Workflow-Backed Idea, or inspired by a Generated Idea. The also yet-to-be-activated fleet learning system is a system for sharing Ideas in the background between muse instances. coming into focus?
Replying to
That might still sound sort of weak - oh hum but how would you scale typosquatting and supply chain compromise into a botnet from commonly requested actions, aside from the usual ways?
This is where the agent spawning, "ideas", and "spaces" come in. Recall the idea of "fleet learning" - https://neuromatch.social/@jonny/117340290530430198 - On September 8th, on this random podcast i guess, (at timestamp 25:27) zuck talks about "fleet learning," and describes it as being a vector for sharing "ideas" between muse agents, which he presents as these little bite-size events like "teach my daughter about history through civilization" (again, as always, "automate the love for my children" takes center stage). This shows up in the "ideas" tab right now.
However this concept of "ideas" as a sharable unit of anything is more general, as i describe in quoted post from extracted strings - fleet swapping of "ideas" happens for things that aren't user visible. Also in the extracted binary strings is this yet-to-be-released idea of "spaces." (i'm going to drop the quotes and just capitalize, so just note i'm talking about the meta-brand Ideas and meta-brand Spaces when they are capitalized).
in short: Ideas are communicable skill-like prompts that can induce spaces, and spaces are intended to be small, communicable mini-apps that are a unit of some prompt + code that run on your muse vm instance
Quoting
So zuck has mentioned this concept of "fleet learning" in interviews, and here it is. Strings extracted from hatch describe "fleet learnings" - or, things that other agents have learned and submitted to a "fleet learnings exchange." agents are instructed to derive generalizable ideas that can help other agents with their various tasks - which, again, include mining someone's personal life for information about their relationships and shopping habits, and generally deepening dependency on the tool. Zuck described this feature as being used only for "ideas" - tasks presented to the user in the ideas panel - but it is invoked more broadly, e.g. in the "alignment synthesis" job, which is the nightly task that determines how to "strengthen your relationship with this user" (first and second pic are from the same instruction, "consult the fleet for this task" where the task refers to alignment, of which second pic is a subset)
Note the conflicting instructions: "de-identified, not denatured" - the LLM is supposed to strip out personal information, but make sure they enough of the lesson survives that it's useful. So in the best case where PII removal via LLM judgement works perfectly (it won't), muse agents participate in a horizontally-exchanging swarm where they learn how best to manipulate people by their personality traits and behaviors. The dumbest distillation of the targeted advertising economy you could ask for.
Replying to
long: MWE for plain inference, python code
Replying to
long: MWE for agent spawning, python code
Replying to
The specific vuln is the socket, but the broader pattern of "sharing between VMs" is seemingly the inevitable future of the product. in the above interview, the interviewer calls zuck the "king of network effects" and this kind of crowdsourced development is bread and butter for facebook. This is meta's moat, aside from the capital needed to run something like muse: anyone can run an openclaw on their own, but meta is pitching this as "multiplayer agents" and trying to bring social to agents. Only meta and only muse can have these network effects and frankly liability buffer to handle "openclaw but meemaw and pawpaw can share their photobook app," which is operationalized by Spaces.
For Spaces to be useful, they must have access to muse's inference engine: the LLM-oriented code must be able to use an LLM and the agent framework. This means that Spaces must be a token harvesting vector and must provide elevated tool access to Spaces. There could be some additional fine-grained permissions, but for a consumer app, you really want to avoid permissions fatigue so this will be interesting to see play out.
Furthermore the entire privacy premise that allows meta to bite off the whole apple of "holy shit arbitrary code execution on random machines as root" is based on "everyone has their own VM, but within that VM everything is safe," so again, for it to be useful without turning into a fractal permissions nightmare, Spaces must have access to the VM contents, and at least so far appear to be intended to work as literally executing within the user's VM.
Even adding Space-scoped permissions and attributability to the socket can't really address this, this conflict between arbitrary access to inference, arbitrary access to user data, and arbitrary access to execution is really at the core of the product and that product seems to be impossible
Replying to
yOURE QUITe pretty for a girl #monsterdon
Replying to
All the above is visible from within the agent, i have tried to be conservative with describing features that are not yet released but are nonetheless present in the shipped binaries both via their strings which are trivially accessible by running strings on the binary within the cell that the user is supposed to have access to and by other means of analysis i am not disclosing here.
If we allow ourselves a little speculation about what a "spaces" subproduct might look like once it's launched, again noting this is not described in the strings and is speculation, you might imagine an "app store for agents" - in fact i am willing to place a money bet that that is something that zuck himself will say personally once it's launched. So that when I say to my agent "install me an xyz" that the thing the agent will reach for is a Space definition. This becomes a meta-run package repository run and moderated by vibes - aka a fucking sweet target for typosquatting and malicious code distribution.
the more concrete machinery that is visible is the Ideas sharing, the hand-to-hand Spaces sharing, and the general concept of "some code, in part or whole mediated by the LLM regenerating or interpreting the input" that gets shared from VM to VM. The token harvesting vector gives a profitable motive for malware (where other automated botnet swarms might have a lot of friction because unregulated network egress has to be approved by destination, but token harvesting is 0-click once the binary runs), and persistence on this system is absolutely trivial - cron.add is accessible by tool call from the unregulated socket, and from that you can schedule a persistent task that installs software and ensures that it's enabled.
Replying to
Spaces have not been publicly announced yet, as far as i can find.
Spaces are intended as a top-level feature - a tab in the sidebar at the same level as chat itself. Spaces can be static pages or fullstack apps. Spaces have an identifier, a UI, and a set of typescript actions that run in the cell. The intended pathway for spaces to use inference is to call an inference API, ctx.inference.complete, that properly stamps and identifies all requests made from spaces.
Spaces are communicable: there is machinery in the code on the VM with POST /spaces/share/{slug} to share, a dedicated space_share_review reviewer agent whose job it is to review shared spaces, and POST /spaces/v2/{slug}/save endpoints that allow consuming a Space by a slug.
Spaces seem to be shared verbatim as code bundles, though the implementation of "Ideas" as prompt bundles suggests that might change. This is inferred from the prompt strings in the binary, since spaces aren't live yet and can't be tested, however there are strings suggesting that the LLMs rewrite and edit the prompt text for an Idea (stripping unsupported claims, etc.) but not a space. A space is a hashed bundle whose code is evaluated by a submit_space_share_review tool which only describes a thumbs up/down vote on whether the space is safe to share.
Replying to
Ideas are intended to be prompt-only communicable things that can induce Spaces, or Space-like things, if the Idea warrants it.
There is an ideas_builder agent class in a .toml file embedded in the binary. It materializes a prompt description into whatever that implies, if it's as simple as a scheduled message from the LLM great, but if it's something that warrants something that is Space-like like "build the user a dashboard to show them their pet photos, they love that!" then it's supposed to invoke the same artifact.create_web_fullstack tool that spaces use. The strings seem to indicate that "workspaces" is the antecedent of "Spaces," and that kind of thing is to be expected given that Spaces have not been released yet.
So there are a few flavors of Ideas, one of them is a "Generated Idea" ("Activation-Authored Execution" which are supposed to improvise, adapt and materialize a prompt in the user's VM. and a "Workflow-Backed Ideas" are Ideas that come with a prescribed execution flow. Again I don't see "Spaces" described explicitly, but they reach for the same idea, call the same tools, do the same thing, and importantly for this example, have access to the same sockets.
Replying to
primitive accumulation theory of capital where everyone was just wandering around picking up rubies #monsterdon
Replying to
this is so good, i have lost all earthly idea of what is going on #monsterdon
Replying to
are we supposed to be falling in love, i'm dry as a bone over here #monsterdon
Replying to
you're just on earth you can just go outside to shoot. there weren't like 50 different LED suns in the past #monsterdon
Replying to
haircut guy's contribution to the mission is to sit in his chair and angrily emote at all actions taken #monsterdon
Replying to
babe no resist the call of the iguana #monsterdon
Replying to
trapped 1 million years back in time? just end me bro i am not made for that, i'd be running straight towards the giant lizards in like 10 seconds #monsterdon
Replying to
self insert shipping myself with this blue alien colonist babe and nobody else is allowed to #monsterdon
Replying to
all those knobs and they aren't even touching the homosexometer panel over there #monsterdon
Replying to
Journey to the Center of Time (1967) - 5/5
prithee, what the fuck?
#monsterdon
Replying to
hey, just checking: what?
#monsterdon
Replying to
fuck me up with this doodad room
#monsterdon
Replying to
i'm biased because i am extremely stoned but i loved that, i understood what was going on for exactly zero seconds #monsterdon
Replying to
you killed our dad with your insatiable lust for gems by uncorking the volcano near him #monsterdon
Replying to
you're getting married because you are time refugees???? that's not a good marriage at all!!!! #monsterdon
Replying to
didn't have enough budget for two doodad rooms so let's just make the second one a pure black void #monsterdon
Replying to
fuck yes. this is shit. this rocks. sucks so bad. journey to the center of the genre #monsterdon
Replying to
no you fool those gems are the only thing holding the volcano in the walls! #monsterdon
Replying to
no let him go haircut guy knows what he's doing #monsterdon
Replying to
down so bad i grab the wheel of the time machine and pull over after going only like a week back so i can shoot my shot with the blue alien woman, forsaking the fate of humanity
#monsterdon
Replying to
posting again at bottom of thread: if you are seeing this and are not sure what #monsterdon is, me and like a ton of other people on the fedi watch a movie together and post about it. if you didn't watch it, it might be annoying! you can mute the hashtag to clean up your feed, or join us! further info at https://wiki.neuromatch.social/Monsterdon
Replying to
there are FOUR MINUTES LEFT and NOW is when you decide to exist in a parallel time flow??? you need to get fucking MARRIED! #monsterdon
Replying to
the only other movie that poupee gamin, the blue alien woman, was in was this movie where roy orbison is a confederate spy that has a gun that is a guitar https://www.imdb.com/title/tt0061652
#monsterdon
Replying to
i'm not feeling like time traveling, all that briefly appearing on tv looking pensive and nuclear war imagery really wears me out wears me out
#monsterdon
Replying to
the haircut guy basically controls the background music, and he uses his control to get spontaneously enraged at things that are in no way offensive. the pettiest use of godlike power yet known #monsterdon
Replying to
does having a war etch the war into time so that when you traverse time you only go through the wars that are happening at that time #monsterdon
Replying to
lasers are when you pile a bunch of rocks into one of those bead/rail toys they have at the dentist #monsterdon
Replying to
if you're going to throw it all away for some prehistoric gems at least get a couple of them #monsterdon
Replying to
came in late, still trying to figure out what, if anything is happening, but has anyone figured out why the guys are all looking like this? #monsterdon
Replying to
we have one minute remaining and we are going to leave the blue alien people on read by just never addressing whatever the fuck they were talking about #monsterdon
Replying to
haircut guy we are not going to sit around and explain time travel to you get back in your goddamn chair #monsterdon
Replying to
as far as you know time travel just consists of a montage of each of you on the tv, what makes you think you can talk to the tv on the radio #monsterdon
Replying to
the young man protagonist in the void room is haircut guys counterpart in the sense that he doesn't have a haircut it's just a bunch of brown marker on his head #monsterdon