AI Assistants and Self-Quantification at Work

Recent developments in artificial intelligence (AI) have permeated through an array of different immersive environments, including virtual, augmented, and mixed realities. AI brings a wealth of potential that centers on its ability to critically analyze environments, identify relevant artifacts to a goal or action, and then autonomously execute decision-making strategies to optimize the reward-to-risk ratio. However, the inherent benefits of AI are not without disadvantages as the autonomy and communication methodology can interfere with the human's awareness of their environment. More specifically in the case of autonomy, the relevant human-computer interaction literature cites that high autonomy results in an "out-of-the-loop" experience for the human such that they are not aware of critical artifacts or situational changes that require their attention. At the same time, low autonomy of an AI system can limit the human's own autonomy with repeated requests to approve its decisions. In these circumstances, humans enter into supervisor roles, which tend to increase their workload and, therefore, decrease their awareness in a multitude of ways. In this position statement, we call for the development of human-centered AI in immersive environments to sustain and promote awareness. It is our position then that we believe with the inherent risk presented in both AI and AR/VR systems, we need to examine the interaction between them when we integrate the two to create a new system for any unforeseen risks, and that it is crucial to do so because of its practical application in many high-risk environments.

This research delves into the capabilities of Large Language Model (LLM)-powered agents across five critical dimensions of task management: decomposition, scheduling, delegation, and execution. Leveraging both surveys and interaction experiments, this study aims to unearth practical insights into the applications and constraints of LLM-powered agents in real-world settings. We describe our experimental setup, data collection methodologies, and analytic techniques, offering a nuanced understanding of these agents' efficiency and efficacy. Our initial findings underscore the nuanced performance of LLM-powered agents in task management, revealing their strengths in understanding complex tasks and their limitations in execution without human intervention. This research contributes to the burgeoning field of human-AI collaboration by providing empirical evidence on the capabilities and limitations of LLM-powered agents in task management.

Self-quantifying technology enables users to evaluate their performance and define strategies for improvement. Workplace technology assists users in identifying activities and patterns that facilitate task completion. Software that measures workplace signals requires access to information from the user interface and peripherals. Current software solutions that track computer activity lack goal orientation and do not share raw data that could aid users and researchers in analyzing behavioral patterns at work. This paper presents Wellbot, an intelligent AI assistant capable of tracking workers' activities and providing personalized insights. The solution employs machine learning models to detect goal-oriented and recreational time. Based on users' goals and recent activities, Wellbot generates recommendations aided by Large Language Models. This work aims to enable tools that assist workers through an improved understanding of their context and goals.

In this paper, we propose SmartMonitor, a system that utilizes an edge hardware device to record, analyze and log user activity while not interfering with said activity. The analysis of the activity can help users identify the time they spend on different tasks and provide real-time feedback from a Large Language Model (LLM) to give better awareness of user activity. The SmartMonitor passes through the HDMI signal from the video card and analyzes the user’s activity on edge using two artificial intelligence models, logs user activity and sends the log for analysis to a Large Language Model for feedback. SmartMonitor enables non-intrusive self-quantifying technology that both records and analyzes user activity while protecting privacy and reducing the processing burden on the user’s device, which can serve as an excellent research framework for behavioral analysis in the workplace and a way to enhance user's work activity.

This demo paper explores AI-enhanced multi-screen interaction within extended reality (XR) workspaces using called VirtuaScreens. The system facilitates the user-centric addition of customized virtual monitors and employs machine learning to understand user preferences for monitor arrangements and application arrangements. We employed Large language and multimedia models to offer context-sensitive feedback and enhance the user experience. We created this research framework called VirtuaScreens for researchers and practitioners to use to understand the interaction of the multiple screens in the XR environment.