Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging.
This paper introduces Work2Skill, a framework for capturing human graphical user interface (GUI) interactions to enable reusable agent skills, addressing challenges in transferring human domain knowledge to computer-use agents.
What happened
The article presents Work2Skill, a system that records human GUI interactions and converts them into reusable agent skills. It describes the creation of the WorkDemo-350 dataset and demonstrates the framework's effectiveness through benchmark evaluations.
Why it matters
This research advances agent systems by enabling them to learn from human interactions, which is critical for automating complex tasks in unfamiliar software environments.
- Human GUI interactions contain reusable skills that can be extracted for agent training
- Multimodal datasets are critical for evaluating agent learning systems
- Context-aware skill induction improves agent performance in complex tasks
The bigger picture
The framework provides a foundation for improving agent autonomy in GUI-based tasks, with potential applications in software automation and human-AI collaboration.
Work2Skill advances AI agent capabilities by systematically capturing and reusing human GUI expertise, demonstrating measurable improvements in task execution through its novel framework and dataset.
Technical details
- WorkRecorder infrastructure for aligning screen recordings, input events, and accessibility data
- Skill-induction method that identifies workflow rules and dependencies from human demonstrations
- Evaluation metrics including MyPCBench Bounded Action and OSWorld-Verified benchmarks
FAQ
What is the primary purpose of Work2Skill?
To capture human GUI interactions and convert them into reusable agent skills for complex task execution.
What is WorkDemo-350?
The first synchronized multimodal dataset of human desktop GUI demonstrations for agent skill induction and evaluation.
How does the framework improve agent performance?
By identifying workflow rules, step dependencies, and completion checks from human demonstrations to create context-aware skills.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.AI on Oct 12, 2026.
Comments
Sign in to join the discussion.