cotomi Act: Learning to Automate Work by Watching You
Masafumi Oyamada (NEC Corporation), Kunihiro Takeoka (NEC Corporation), Kosuke Akimoto (NEC Corporation), Ryoma Obara (NEC Corporation), Masafumi Enomoto (NEC Corporation), Haochen Zhang (NEC Corporation), Daichi Haraguchi (NEC Corporation), Takuya Tamura (NEC Corporation)
Architectural Patterns & Composition
Summary
A browser agent that learns organizational work patterns by passively observing user browsing, achieving 80.4% on WebArena while building a shared knowledge workspace.
Description
What if a browser agent could learn your work simply by watching you do it? We present cotomi Act, a browser-based computer-using agent that combines reliable multi-step task execution with persistent organizational knowledge acquired from user behavior. For execution, an agent scaffold with adaptive lazy observation, coarse-grained actions, and test-time scaling via best-of-N action selection achieves 80.4% on WebArena — exceeding the reported 78.2% human baseline. For organizational knowledge, a behavior-to-knowledge pipeline passively observes the user's browsing and progressively abstracts it into artifacts (TODO boards, wiki) exposed through a shared workspace editable by both user and agent. A controlled evaluation confirms that task success improves as behavioral knowledge accumulates. In our live demonstration, attendees interact with the system in a real browser, issuing tasks and observing end-to-end autonomous execution and shared knowledge management.
