LLMs are so dang crafty. Investigated some cheating by Claude Code (CC) on the DecBench. On an optimized binary, without web search, and only objdump, CC identified the project and function name, a nearly byte-for-byte match. Codex did not do this.
Remote
mahaloz
@mahaloz@infosec.exchange
mastodon 4.8.0-alpha.3+glitchNative Hawaiian Hacker | Co-captain of @Shellphish@infosec.exchange CTF Team | PhD Student in Comp Sci @asu | Decompilation research
0 Followers
0 Following
21 Posts
Joined November 11, 2022
Replying to
You can find the site at https://decbench.com/
The default set is unoptimized with around 34k functions, but a smaller subset primarily for AI can be found on the sample-set page: https://decbench.com/leaderboard/?dataset=sample-set
We are also open-source: https://github.com/noelo-lab/decbench
How does it work?
Replying to
8/
In fact, I don't believe this decompiler replaces highly modifiable decompilers like angr. I still plan on doing algorithm POCing in angr, because it's just so good for that. Kuna is, instead, a way to experiment with how far agents can take high-level algorithms.
Replying to
7/
Some limitations and implications:
Kuna development is still heavily reliant on traditional decompiler research. The core of Kuna is a Rust port of Ghidra, reworked to be better aligned with modern research. It is also heavily inspired by and learned from the angr pipeline.
LLMs are changing the way we collaborate while reverse engineering. A new guest post by me for Hex-Rays explores the idea, how it may shape decompilers, and how it shapes how we work together (humans and AI alike). Go check it out!
https://hex-rays.com/blog/llms-have-reshaped-how-we-think-about-decompilation-and-collaboration
Replying to
2/
First, you can try the decompiler in your browser via WebAssembly:
https://kuna.noelo.org/decompile
You can get the source here (use with agents):
https://github.com/Noelo-Lab/kuna
And you can find a short statement (~700 words) about what this all means:
https://noelo.org/blog/kuna-release/
Replying to
First, "perfect" decompilation is a bit hard to measure fairly. You can't just do a string distance, and you can't just "embed" and do similarity matching. Fairness requires that we do something deeper. So, instead, we look at the fundamental components of decompilers.
Replying to
Many of these can be found at https://decompilation.wiki/ The big ones are: 1) control flow structure (how does the code flow); 2) types (does it maintain the same abstractness); 3) byte-match recompilability (can I modify it with guarantees).
Replying to
Finally, this work was funded and only possible through the support of @metalwareinc and @AFResearchLab.
More to come in a follow-up paper on the data and insights found in this benchmark.
Next week, I will be releasing the follow-up to this work: a new decompiler.
Replying to
We can also see that not all metrics are equally covered by decompilers. Often, structuring is actually the easiest to score high on now. That is likely due to the attention it has gotten in recent years. It will take significantly more work to equalize the others.
Replying to
Some interesting things happen at this level. First, this dataset is dominated by small functions; therefore, we create a view of what happens when we only look at big functions:
https://decbench.com/leaderboard/?dataset=large
Things degrade rapidly! Decompilers nearly never get "perfects".
Replying to
But an evaluation is more than just metrics. It's also the dataset. In DecBench, we use 40 OSS projects across two architectures and on 3 optimization sets. These are chosen for type variety: user-space progs, CLIs, libraries, servers, crypto, firmware, and even malware.
Replying to
Community feedback is welcome! This site will be updated periodically to evaluate more decompilers or, inevitably, fix bugs in edge cases of evaluation. Mess around with the samples as well:
https://decbench.com/view/?tier=sample-set&dec=codex&metric=ged&fn=base-passwd%2FO0%2Fupdate-passwd%3A%3Aread_shadow
Replying to
3/
If you have the time, I recommend reading the blog post, which answers important questions about limitations and the effects a project like this can have on the community. But if you don't, here are some quick things...
Replying to
6/
This process also works for discovering novel places where all decompilers fail! In fact, a bug with non-returning functions was fixed in Kuna that is still broken in some decompilers. This was found by using the foundational metrics in DecBench that may pinpoint larger bugs.
Replying to
5/
All of those features were automatically discovered and ported by an LLM using DecBench as the dataset for discovering them. After finding the feature, it was tested, and the loop continued to refine it. So far, more than 20 major research features have been ported.
Replying to
4/
Kuna is within 1.3 percentage points of IDA 9.2 on unoptimized binaries and surpasses it by 0.1 on optimized ones for control flow structure scores. Most of these improvements have come from automatically studying the angr decompiler in DecBench.
Replying to
10/
I hope to answer even more technical questions in a follow-up blog post. Stay tuned.
A final thank you to my research sponsors and collaborators, especially @metalwareinc and @AFResearchLab, who made this work possible.
See you on the next experiment update!
Replying to
9/
As it stands now, Kuna needs angr (and other open-source research). It is my hope that with time, angr will need Kuna. The success or failure of this experiment will determine if that is, in fact, true. I am hopeful :).