Pengujian Replit Menunjukkan Bot Pengodean AI Gagal Membuat Hingga Setengah dari Aplikasi Baru

WHALE WATCHER · 2026-10-29T02:32:39.000Z

Situs pengodean berbasis peramban Replit mengatakan bahwa Replit ViBench 2.0 menemukan alat pengodean AI hanya mampu menyelesaikan 49% hingga 65% pengujian pembuatan aplikasi, sehingga manusia masih perlu memeriksa hasilnya. Poin-Poin Utama Replit merilis ViBench 2.0, sebuah tolok ukur yang mencakup 17 produk, 241 permintaan fitur, dan 544 rencana pengujian Di Replit ViBench 2.0, agen pengodean teratas hanya berhasil melewati 49% hingga 65% build. di ViBench 2.0 Agen memulai dari proyek kosong, membangun produk secara bertahap, dan menambahkan fitur tanpa melihat seluruh peta jalan Delapan produk tersedia untuk umum di GitHub, sementara sembilan lainnya tetap tersembunyi di ViBench 2.0

Browser coding site Replit said Replit ViBench 2.0 found AI coding tools could finish only 49% to 65% of app-building tests, so humans still need to check the work.
Key Takeaways
Replit released ViBench 2.0, a benchmark of 17 products, 241 feature requests and 544 test plans
In Replit ViBench 2.0, top coding agents pass only 49% to 65% of builds. in ViBench 2.0
The agent starts from an empty project, builds a product in stages and adds features without seeing the whole roadmap
Eight products are public on GitHub while nine remain hidden in ViBench 2.0
The company posted the release on X. It makes browser-based coding tools and agent workflows for developers and nontechnical users.
Replit ViBench 2.0 Starts With Nothing
Most coding tests hand an AI agent a bug or task inside an existing codebase, but ViBench 2.0 does the opposite.
The agent starts from an empty project, must build a product in stages and then keep adding features without seeing the whole roadmap first. That setup matters because later requests can break earlier design choices.
The benchmark models products like Uber, GitHub and Google Docs, where schema, routes and deployment decisions made on day one can constrain every later change.
The held-out structure is part of the point. Eight products are public on GitHub, while nine remain hidden, which makes leaderboard chasing harder and rewards systems that generalize across long development chains.
From Vibe-on-Vibe To Full Product Work
ViBench 1.0 focused on products users were building at the time, including collaborative whiteboards and resume builders.
Wednesday’s release broadens that frame to “entire businesses,” not one-off apps, and extends the older “Vibe-on-Vibe” idea into a multi-step product arc.
That is a real shift in what AI coding is being asked to prove. It shows a coding copilot can look strong when it edits one file or fixes one test, but it looks much weaker when it has to keep its own earlier choices alive through repeated feature additions, clean deployment and empty-database launches.
Also Read: Meta Hands 5 US Universities AI Tools To Speed Medical Research
The firm also said every result can be rerun from release v2.0.6 on its OpenHands-based harness, underscoring that reproducibility has become one of the main fights in AI evaluation because vendors often post high scores on private setups other developers cannot check.
Why The Platform Is Framing The Next AI Coding Fight
ViBench 2.0 is also a strategic move by Replit, which has been pushing further from code completion toward full software creation.
Replit makes browser-based coding tools and agent workflows, and framing benchmarks around long-horizon product work could advantage platforms that control the full build loop.
That framing matters for enterprise buyers deciding whether an agent can ship real internal tools. A failed build in this benchmark does not mean a syntax error, it can mean the system made an early architecture choice that later blocked a working feature or shipped a build that fails on a clean database.
How The Test Chain Exposes Real Failure Modes
ViBench 2.0 works because it measures an agent across a sequence, not a single answer.
The agent gets a short spec, builds version one, then receives as many as 18 later feature requests, each depending on earlier code still working, which makes planning part of the score.
The clean-deployment rule is one of the hardest parts. Tests launch on an empty database, so a build that quietly depends on leftover local state fails, punishing demo success that looks fine in a sandbox but breaks in production.
For developer teams using Codex, Cursor or Replit-style agents, this is closer to actual software work than bug-fix benchmarks.
The 49% to 65% pass range also changes how to read coding agent claims. A model that solves half of long product builds is useful, but it still needs supervision, rollback and human architecture review.
Watch whether rivals adopt ViBench 2.0 or ship their own long-horizon suites over the next few weeks.
That will show whether the field accepts full-product durability as the next score that matters.
Read Next: Invesco Hands Advisers 2 Claude Tools To Build Portfolios Faster