this post was submitted on 12 May 2026
24 points (96.2% liked)

Technology

42922 readers
236 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 4 years ago
MODERATORS
remington@beehaw.org
alyaza@beehaw.org
TheRtRevKaiser@beehaw.org
gyrfalcon@beehaw.org
rs5th@beehaw.org
coldredlight@beehaw.org
SemioticStandard@beehaw.org
24
ProgramBench, a new benchmark from Facebook/Meta(by SWE-Bench creators) to see if LLMs can recreate real executable programs (ffmpeg, SQLite) from scratch with no internet access- They all score 0%. (i.redd.it)
submitted 3 hours ago by beep@piefed.world to c/technology@beehaw.org
4 comments fedilink hide all child comments
 

cross-posted from: https://piefed.world/c/tech/p/1101720/programbench-a-new-benchmark-by-swe-bench-creators-from-facebook-meta-to-see-if-llms-can

Comments

Benchmark.

Links

./ProgramBench

Can language models rebuild programs from scratch?

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior.

In each task, the agent receives an executable and its documentation, and it must re-implement the given executable. It does not get access to any of the executable's source code, it cannot de-compile the executable, and cannot use the internet. There are 200 tasks in total covering different program complexities, ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler, FFmpeg, and SQLite.

The agent must choose a language, design the architecture, write all source code, and produce a build script. Every design decision is the model's to make.

Once the agent submits a program, our test suite compares the candidate program's behavior against the original program. A candidate program passes only if all tests for that task pass.

Photo source: X/Twitter.

you are viewing a single comment's thread
view the rest of the comments
[–] rozodru@piefed.world 2 points 1 hour ago* (last edited 1 hour ago)

well duh. You can test this yourself and even ALLOW the agent to access the internet. if you asks it to build something let it pick it's own language. 9 times out of 10 it's going to pick a language that tech bros and tech blogs have been raving about on linkedin and other sites. AND then it generally wont' utilize the latest API or update to said language if no one has written about said latest API/update. months ago I tried this as a test. Claude selected Smithy and Rust I believe (can't remember exactly) but the problem was it was utilizing an out of date Smithy version to try and get it to work with an up to date Rust version even though the latest Smithy was available it refused to utilize it. Why? well no one wrote about Smithy. So it would use the version it could find blog posts and articles written about it.

Now if you take internet access away from the agent, it's not going to know what to use.