Android Bench 2.0 introduces long-horizon tasks and continuous scoring
Public- 18 Sep, 2026

Google has released Android Bench 2.0, a major upgrade to its benchmark that evaluates how well large language models help developers with real-world Android work. The headline addition is the first set of long-horizon tasks (LHT): complex jobs that typically take an engineer several days or, in some cases, up to a week to finish. The release also brings agentic evaluation, beginning with agents from the corresponding model providers, and aligns the framework with the Harbor framework. Early versions of the benchmark focused on incremental changes, mostly bug fixes and small feature requests, which reflected the limitations of AI assistance at the time.
The new long-horizon tasks are designed to mirror the ambitious work developers actually delegate to AI. They include upgrading dependencies, adding new features, building apps from scratch, and converting a cross-platform app to Android. To grade these multi-day efforts, Google is moving away from binary pass/fail scoring, which it argues obscures real capability. A run that refactors 40 screens to Jetpack Compose, sets up database tables, and meets 90% of requirements but fails a single edge-case assertion would have scored 0% under the old system. Continuous scoring now yields a completion rate, calculated from factors like functionality, visual fidelity, and avoiding regressions, with objective penalties for deviating from evaluation instructions or structural constraints.

The results reveal clear strengths and weaknesses. The highest pass rate on long-horizon tasks is around 28%, far below the roughly 91% seen on the benchmark's original tasks. Across model tiers, AI writes new code better than it refactors existing code, since refactors and migrations hinge on architectural complexity rather than code volume. Models handle well-established, deterministic transformations reliably, such as converting Java to Kotlin, swapping Retrofit for Ktor, or introducing a ViewModel layer, applying these patterns consistently even across 125+ files and 8,000+ lines of code. They struggle, however, when a task requires runtime validation (like missing dependency-injection graphs), involves breaking framework changes, or runs into knowledge gaps with unreleased libraries.

Porting cross-platform apps to Android remains an open challenge: no model reaches a 100% pass rate, and frontier models top out at roughly an 80% completion rate. Agentic evaluation is now part of the benchmark, starting with models paired with their provider's agent, for example GPT 5.6 Sol on Codex and Gemini 3.8 Flash on Google Antigravity. This pairing shows that harness design matters, with prompt caching and compact tool windowing reducing token usage. The leaderboard has also expanded, adding Gemini 3.8 Flash, Gemini 3.7 Flash, GPT-6, Fable 5.1, Kimi K3, and Qwen 3.8 Max; OpenAI's GPT-6 Astra leads with a 28% pass rate.