Skip animation and read the results

“What Are You Really Trying to Do?”

Co-Creating Life Goals from Everyday Computer Use

UIST 2026

Abstract

Recent advances in user modeling make it feasible to conduct open-ended inference over a person's everyday computer use. Despite longstanding visions of systems that deeply understand our actions and the purposes they serve in our lives, existing systems only capture what a person is doing in the moment, not why they are doing it, limiting these systems to surface-level support. We introduce striving co-creation, a process for inferring broader life goals from unstructured observations of computer use. Grounded in Activity Theory and Emmons' personal strivings framework, our system progressively constructs a hierarchical representation of a person's activities. Strivings are, however, difficult to fully resolve from observation alone, as the same action can be driven by many different goals. Our system therefore supports an editing interface that gives people agency over how they are understood by the system, feeding their corrections back into subsequent rounds of striving induction. In a week-long field deployment (N=14), we find that our co-creation process produces strivings that participants recognize as representative of their long-term goals and gives them greater agency than baseline methods.

Observations are stored as a hierarchy in an annotated graph.

What Are You Really Trying to Do?

Dedicated to preserving cultural identity and connection while mitigating the isolation of life abroad
Being a reliable source of care and support for aging family, even from afar

Results

We ran a week-long study with 14 participants.

Onboarding context improves individual striving quality.

Information people shared about their lives helped Tempo interpret their computer use and infer better strivings.

Individual striving quality: estimated marginal means 1.43 for Tempo, 1.14 without user context, and 1.09 without hierarchy or user context. Adding user context was significant, p less than .05; hierarchy alone was not.

Hierarchy produces a more representative set of strivings.

Grouping observations into intermediate levels helped the overall set reflect what participants were working toward.

Set-level striving representativeness: estimated marginal means 1.04 for Tempo, 0.80 without user context, and 0.08 without hierarchy or user context. Adding hierarchy was significant, p less than .05; additional user context was not.

Hierarchy supports a better editing experience.

Participants rated editing with the hierarchy more highly than editing with the screenshot view.

Editing experience: estimated marginal mean 1.82 for the hierarchical view and 0.85 for the screenshot view on a −3 to +3 scale, p less than .001, with ratings pooled across seven dimensions.

Estimated marginal means on a −3 to +3 scale; error bars show ±1.96 SE.

How Tempo works

Screen observations become operations, actions, activities, and strivings.

Tempo architecture: observe and transcribe screenshots, audit sensitive frames, extract operations, group actions, propose and reconcile activities, then synthesize and refine strivings in a property graph.

Editing the hierarchy

People can edit, reassign, remove, and merge the system’s interpretations.

Four editing operations from the paper: A, edit an activity inline; B, reassign it to a different striving with an explanation; C, remove an incorrect striving; D, merge related activities while preserving their evidence.

Citation

Shardul Sapkota, Matthew Jörke, Zane Sabbagh, Omar Shaikh, Grace Wang, and James A. Landay. 2026. “What Are You Really Trying to Do?”: Co-Creating Life Goals from Everyday Computer Use. In The 39th Annual ACM Symposium on User Interface Software and Technology (Detroit, MI, USA) (UIST ’26). Association for Computing Machinery, New York, NY, USA, 20 pages. doi:10.1145/3830398.3830630

BibTeX
@inproceedings{sapkota2026tempo,
  author = {Sapkota, Shardul and J{\"o}rke, Matthew and Sabbagh, Zane and
            Shaikh, Omar and Wang, Grace and Landay, James A.},
  title = {``What Are You Really Trying to Do?'':
           Co-Creating Life Goals from Everyday Computer Use},
  booktitle = {The 39th Annual ACM Symposium on
               User Interface Software and Technology},
  year = {2026},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  location = {Detroit, MI, USA},
  numpages = {20},
  doi = {10.1145/3830398.3830630},
  url = {https://doi.org/10.1145/3830398.3830630},
  series = {UIST '26}
}