A music app can offer a thousand playlists and still miss what one person wants. One listener wants a single button for the morning commute. Another wants to compare candidates, keep notes, and revise yesterday’s choice. A product team can anticipate some of these needs, but every additional preference becomes another decision about what a common interface should contain.
Personal agents suggest a different way to produce software: start with a person’s requirement, create an application around it, and let the agent continue to supply its behavior. The application becomes a place to return to, with controls, records, and a history of decisions. Software creation is distributed across users and moments of use instead of ending with one centrally designed interface.
LiveUIBench asks what it takes to evaluate this idea. A convincing screenshot is only the beginning. The application must carry out a submission, understand a revision in the context of what happened earlier, and retain the right information when the user returns.
From an answer to an application that stays with the user
A conversational answer can be useful once. An application makes a stronger commitment: it gives the user a stable place to act again. If a household restocking list already contains several products, changing the quantity of one product should preserve the others. If a user returns to a gift planner, the recipient and the selected gift should still refer to the same saved record.
We call the task Agent as App. The agent creates a personalized application from a natural-language requirement and remains responsible for interpreting its later inputs. The generated object includes an interface and a task-specific orchestration; a shared runtime carries out operations, retains information, and renders the result.
interface and behavior + saved state
This representation separates two kinds of change. An ordinary input can use the existing plan while changing its state. A new requirement can change the plan and the interface while preserving the application’s identity and relevant records. A useful personal app needs both.
Following one record through a whole workflow
Consider a gift planner. The application must work beyond its initial view. The user must be able to add a recipient and a gift, revise the intended entry, and find the revised result on return. Each stage depends on the one before it.
The recorded example below follows this progression through the visible interface. It illustrates what an evaluator needs to preserve: the same application, the intended record, the requested change, and a result that remains available. These are application examples from the paper, rather than a claim that every benchmark generation behaves this way.
Evaluating the lifecycle, not just the first screen
LiveUIBench supplies an initial requirement, an executable service world, and ordered episode requests. The generator receives the initial requirement and permitted service context. Later requests arrive at their assigned stages. A revision can refer to an earlier result; in shared-world tasks, a later application can depend on records written by another application.
The paper’s evaluated collection contains 500 tasks, 2,408 episodes, and 10,590 assertions. It includes 302 single-application tasks, 130 multiple-tool tasks, and 68 shared-world tasks. These are the paper’s survey counts; subsequent code and dataset releases have their own provenance and should be identified separately from this reported evaluation.
The services include Spotify, Amazon, Gmail, Phone, Venmo, Splitwise, Todoist, Simple Note, and File System, with AppWorld supplying the executable environment. The task families describe the organization of the workflow. They are not interchangeable with a count of distinct service dependencies.
Two metrics answer different questions
Assertion pass rate measures how many assigned checks succeed. Full-workflow rate asks whether every assigned check succeeds together for the same task. The distinction matters because a user experiences an entire workflow, including its weakest required step.
A model that succeeds on different pieces of many tasks can have high assertion coverage while completing relatively few applications. Full-workflow success therefore cannot be reconstructed from an attractive interface or an average of isolated local accomplishments.
The model survey invokes the applications’ declared actions with episode requests. The recorded application examples additionally follow their visible interfaces. These are different observation channels: the reported survey is not an end-to-end GUI navigation contest. Record-centered examples add independent service reads to distinguish an acknowledgment from a saved change.
Experimental results
Opening is easy; completing the user’s task is hard
The opening-to-completion gap is the central empirical result. In the paper’s table, Grok 4.6 passes opening checks on 491 of 500 tasks, yet its full-workflow rate is 19.60%. GPT-5.6-sol passes opening checks on 483 tasks and reaches 71.09% assertion coverage, but completes 11.40% of workflows. An application can expose the expected controls without completing what the user asked it to do.
Ranking models by partial coverage also changes the story. Kimi K3 has lower assertion coverage than GPT-5.6-sol, 61.11% versus 71.09%, while its full-workflow rate is higher, 14.40% versus 11.40%. Local successes need to occur together in the same retained application. Broader partial progress alone does not guarantee that concentration.
Task families expose further differences. DeepSeek V4 Pro leads the reported multiple-tool column at 5.38%, while completing no shared-world workflow in that table. The table compares systems using their recorded generation procedures.
Submission and revision account for much of the loss after opening. Reopening appears less often as the first failure, but an earlier failed submission can hide a later persistence problem in that statistic. First-failure stages locate the earliest unmet check; they do not diagnose the model’s internal cause.
Service activity and the visible answer must agree
A successful service call and a supported visible result are separate requirements. The service world may change while the interface shows stale information. A confident confirmation can appear without the intended service effect. A revised entry may be saved as a replacement rather than as an edit to the original record.
Record-centered checks follow a stable entity identifier through creation, revision, retrieval, and return. This makes the difference concrete: did the requested priority change on the intended task, and did an independent read recover it? Did the interface show the current value rather than an earlier stored view?
Operational checks follow concrete actions and records. Whether selected music matches a desired listening experience is a different question: it requires checking the music itself. The paper separates these semantic requirements from its recorded survey measures.
What changes when software is personal?
Agent as App moves the evaluation target from a one-time output to an ongoing relationship among requirements, interface, behavior, and state. The hard problem is maintaining that relationship when a person changes their mind. A revision should alter the relevant behavior while preserving information that remains useful.
This gives system builders a concrete set of design pressures. Interface controls need to bind to the intended operation. Service outcomes need to be read back and presented accurately. Saved state needs to remain connected to the right entity. Failures need to remain visible rather than being converted into an unsupported success message.
The benchmark follows ordered interactions within a retained application. Its stage index tracks episodes. Learning preferences over months and evolving an interface across open-ended use remain further research questions.
A personal agent can become the mechanism through which software adapts to individual needs during use. LiveUIBench makes that promise testable: can the application finish the workflow, retain the right information, and still make sense when the user comes back?
BibTeX
@misc{zhang2026liveuibench,
title = {LiveUIBench: Evaluating Agent as App Workflows},
author = {Zhang, Tong and Cheng, Yuxuan and Zhang, Ruimin and Peng, Yun and Xie, Tao},
year = {2026},
eprint = {2610.27977},
archivePrefix = {arXiv},
primaryClass = {cs.SE},
url = {https://arxiv.org/abs/2610.27977}
}