This workflow has democratized quality at LinkedIn, creating a discipline where non-technical stakeholders can author resilient, agent-driven tests.
Rigorous evaluation
An autonomous agent that reports phantom bugs is worse than useless; it erodes trust. We optimize QA Agent for precision over recall, accepting that we might miss some issues in exchange for confidence that every reported bug is real.
Building a rigorous evaluation framework turned out to be one of the harder problems we solved, and one of the more important ones.
Golden dataset framework
Early on, we realized we couldn't evaluate the agent against the live app. The real environment changes constantly: new deployments land, experiments toggle, server-side content shifts. Running the same test twice might produce different results for reasons completely unrelated to the agent's behavior. This makes evaluation results noisy and unreproducible.
So we built a golden dataset framework. The workflow looks like this: a human reviewer annotates an agent run, marking which actions were correct and which tasks completed successfully. This feedback is captured and stored as a golden dataset, a frozen snapshot of the application state at each step, including screenshots, view trees, and the expected action at each point.
The key insight is that evaluations replay against this captured state, not the live app. When we run an evaluation, the agent executes normally, but every device command is intercepted. Screenshot requests return the pre-recorded image. View tree requests return the pre-recorded hierarchy. When the agent takes an action, instead of executing it on a real device, we compare it against the recorded action signature. Did the agent tap within the correct element's bounding box? Did it scroll in the right direction? Did it enter the right text?
This gives us deterministic, reproducible evaluation. We can run the same golden dataset a hundred times and get consistent results, which is essential for measuring whether a change actually improved the agent or just got lucky.
We measure accuracy at three levels: action-level (did each individual action match?), task-level (did all actions in a task match?), and replay-level (did the entire test pass?). Golden datasets can be tagged and filtered, so we can evaluate against specific subsets: negative examples, feed-related failures, or platform-specific cases.
Experiment framework
The golden dataset framework becomes truly powerful when combined with our experiment system.
During production runs, experiments execute in parallel with the production planner. The agent uses the production action, but the experimental action is logged alongside it. We compare whether the experimental planner would have tapped the same element, scrolled the same direction, or reached the same completion judgment. These comparisons are logged with full observability, including annotated screenshots showing where each planner would have tapped.
When we want to evaluate a new approach more rigorously, we replay it against our golden datasets. Because golden datasets capture the full application state, we can test a new planner variant against dozens of real test scenarios without needing devices, without needing the live app, and with fully reproducible results.
This is how we validate planner changes before promoting them. We run candidates against the same golden datasets as the production planner, measure action accuracy and task completion rates, and compare the results side by side. No guesswork, no "it seemed to work better in manual testing." Quantified improvement or quantified regression.
When tests fail in production, we triage them to understand what went wrong. Did the agent misnavigate? Did it hallucinate a bug? Did it miss a real issue? These failures become negative examples that we incorporate back into the golden datasets, continuously improving accuracy. Today this triage is manual; in the future, we hope to automate it.
QA agent in action
We have seen QA Agent perform feats that would be impossible for standard automation frameworks.
Self-healing through a major migration
On any given day, hundreds of changes land in the main LinkedIn codebase. Test IDs have changed. UI treatments have been modified. View hierarchies have been restructured. Traditional test automation would require constant maintenance and alignment to keep up.
QA Agent has continued executing core flows throughout this migration. Because it navigates visually rather than relying on implementation details, it adapts as long as the user-facing experience remains recognizable. The button still says "Apply"; the agent finds it and clicks it, regardless of what changed underneath. When the UI does change enough to break a System 1 path, the agent falls back to System 2, reasons through the new layout, and updates its memory for the next run.