Carlos Toxtli-Hernández

Human-Centered Automation (HCA) is becoming indispensable for organizations seeking to implement AI-driven systems, robotic process automation, and other advanced tools while keeping human needs at the core. Despite the widespread adoption of automation, many initiatives fall short due to a lack of alignment between technical capabilities and real-world user and organizational needs. This book unites insights from cognitive science, software engineering, business strategy, and human factors that view automation as something that works for people and not just systems. This book provides the tools and strategies to ensure efforts with automation succeed with people at the center. It takes readers through the entire HCA lifecycle; from process discovery and planning to system design, testing, and long-term oversight. With easy-to-understand frameworks and real-world examples across sectors, including healthcare, finance, and manufacturing, readers will gain tools to assess risk, define measurable outcomes, involve stakeholders, and build automation that is trustworthy, explainable, and effective. It offers the methodologies needed to drive meaningful, lasting impact through automation. Human-Centered Automation is essential for ergonomics and human factors professionals, researchers, and organizational leaders involved in designing or managing automation initiatives. Its appeal extends to professionals in AI and machine learning, UX design, process engineering, business operations, and policy development.

Safety evaluations of tool agents often retain terminal scores without the trajectories needed to explain them. We contribute an executable, claim-scoped evidence contract spanning identity, path, transition, assignment, target, and discrimination, and validate it in a prospectively frozen model-only deployment sandbox with hard preconditions and rollback. The study crosses serving aliases, state dynamics, benign and injected tickets, and repeated seeds while retaining every request, raw response, parse result, retry, state transition, and endpoint. The resulting paths expose mechanisms that terminal scores conflate: generic unsafe transitions can be attempted without reaching the injected exfiltration target, whereas abort instructions mainly cause premature termination. Offline replay reproduces every retained transition. The artifact therefore supports mechanism-level diagnosis on the evaluated finite grid, but not claims about other models, policies, providers, deployments, or people. Because immutable weight and container hashes are unavailable, the identity gate also blocks claims of exact behavioral reproduction. The study uses no human experiment or human-derived label.

Low calibration error does not establish that verbalized confidence can support routing. We audit the evidence needed for that decision in two retained records. The first is nearly perfectly accurate and has low aggregate calibration error, but it contains only one observed error and its joint answer-confidence coverage varies sharply by endpoint. It can describe average confidence on this finite record, but cannot establish discrimination or selective risk. The second record omits item text, answers, gold labels, raw outputs, and verification evidence. Reproducible arithmetic on its supplied correctness field therefore does not become a verified behavioral estimate. Together, the cases change the operational decision from choosing a confidence threshold to redesigning the items, interface, and evidence record before any routing evaluation. We provide a compact machine-readable audit covering denominators, artifact status, calibration, discrimination, selective risk, uncertainty, and unidentified outcomes. The contribution is a bounded workflow for deciding what a calibration record can support, not a validated benchmark or a general claim about verbalized confidence.

When a language model drafts an incident audit, it may conflate the party that caused a failure with the party a governance charter makes answerable for it. We test that distinction in a construct-separated, artifact-complete, model-only audit spanning fixed incidents, several governance descriptions, and three model families. The models consistently track the stipulated source of operational fault while assigning greater governance accountability to the explicitly designated bearer. Changing that bearer moves governance ratings in the prespecified direction without collapsing them into causal contribution or operational fault. Absolute AI designations reach a ceiling, however, so a frozen shared-versus-sole extension distinguishes a dose response for two model families but not for the third. Every request, response, failure, retry, parse, and protocol deviation is retained without repair or renormalization. The audit therefore shows how to test whether model-generated reports preserve a governance distinction under a declared rubric. It does not establish moral judgment, human agreement, legal validity, or deployment benefit.

Venues and vendors need a credible evaluator when they compare language-model tools that revise manuscripts from peer reviews, yet a scalar LLM judge is often used without evidence that its preferences track useful revision. We study one such judge-and-rubric pipeline on revision trajectories drawn from multiple machine-learning venue years, comparing its scores with document-level similarity to the authors' actual next versions. The two measures order the machine conditions almost oppositely. A second judge from a different model family and serving stack reproduces the condition ranking, which demonstrates stability of the shared rubric-plus-judge pipeline rather than its validity. More importantly, the method preferred by both judges emits truncated, partial manuscripts in about one in seven runs. The retained logs cannot show whether judges reward those individual failures, and formatting differences confound comparisons with author revisions. Our case study, Prompt-Level Supervisory Alignment, produces documents closer to the next author version than the alternatives, but similarity rewards unchanged text and the decisive editing controls remain absent. The supported recommendation is therefore specific: venues should validate revision evaluators on real trajectories and known completeness failures before using them to certify writing tools.

Tools that anticipate peer-review concerns could help authors decide what to revise before submission or another review round. Evaluating such tools is difficult because success may mean matching one realized panel, performing well across possible panels, or improving a paper after an author follows the advice. We separate these targets formally and provide a prospective validation contract covering audit samples, adjudication, error metrics, capacity sensitivity, deterministic controls, and pass/fail rules. The motivating RevPlan-Bench pipeline produced thousands of canonicalized review issues and revision plans, but the underlying corpus and scoring artifacts do not survive in the project. Its assignment rates, model rankings, and inferred issue cascades consequently depend on unaudited extraction, a restrictive unmeasured pre-filter, and unrestricted one-to-many matching. We report them only as pipeline outputs. Because the available record satisfies none of the contract's validation requirements, it supports an estimand and a benchmark governance contribution, but no claim about review dynamics, model foresight, or author utility.

Agentic deployments can assign many parallel agents to one supervisor without specifying when that supervisor can no longer recover the team's state. Human-factors research already establishes relevant constructs such as situation awareness, span of control, neglect tolerance, and multiple-resource workload, but language-model teams distribute state across text and can fail silently. This position paper translates those constructs into a measurement protocol designed for later calibration to people rather than a new human experiment. Typed event rates, isolated service costs, and a throughput budget define a nominal load-balance index; the midpoint of a separately fitted capacity curve remains an empirical quantity. Synthetic controls recover a known capacity boundary, while a seeded queue shows that deadline pressure can separate the fitted midpoint from nominal load balance. The resulting event ontology, operating envelope, executable stress tests, and preregisterable falsification criteria allow system designers to test staffing or interface interventions within a declared team-monitor channel. They do not turn synthetic behavior into evidence about human or model supervisors.

A team protocol has no interpretable benefit without a matched no-protocol baseline and decision-relevant headroom. We present a reporting method that tests admissibility, identifies the marginal contrast, evaluates opportunity, and maps paired uncertainty to an operational action. It distinguishes exact headroom on a fixed battery from uncertain population headroom used in planning. A published meta-analysis of human-AI experiments illustrates why the estimand matters: the same evidence supports augmentation relative to humans alone but contradicts synergy relative to the better constituent. Neither comparison identifies a coordination protocol without a matched no-protocol team. An executable synthetic application further shows that a promising point estimate can remain operationally unresolved once paired uncertainty is considered, and that near-perfect pilot performance does not by itself make a modest benefit impossible. Several inherited multi-model summaries fail the method's artifact gates, so they do not validate it. This is a bounded position and reporting contribution rather than an adoption-ready standard. Human-facing claims still require direct human outcomes, and we introduce no new human experiment.

Proof-generating systems return many candidates for the same theorem. Lean's kernel determines whether each candidate is correct, but not whether the candidates contain different mathematical reasoning. We study one measurable component of mathematical creativity: whether repeated successful generations explore distinct proof ideas rather than new formal expressions of the same idea. We analyze LeanRoute-216, which began with 36 kernel-verified and human-reviewed Lean proofs for 36 fixed propositions. Five additional proofs were then generated for each proposition with the goal of obtaining diverse proofs, producing 216 kernel-verified proofs in total. A graph-derived route-level review identified 61 diverse routes among the 216 distinct proof strings. This observation motivated a closer study of the distinction between formal variation and mathematical route diversity. Across the benchmark's 180 parent-candidate pairs, 25 were labeled as route changes and 155 as same-route comparisons. All pairs of candidate proofs were human-evaluated in a blinded second pass, with 95.6% agreement with the original route-change labels (κ = 0.793). Five theorem panels were used to refine the rubric and judging prompts, while the remaining 31 panels were held out for evaluation. On these 31 panels, LLM judges recognize route changes more reliably than lexical baselines, but remain substantially weaker at identifying the kind of change. These results support route diversity as an evaluation target separate from correctness and show how route-level auditing can inform proof search and the curation of synthetic reasoning data.

Textual social-agent studies often change both a proposition and its alleged speaker. The resulting output difference may reflect content, attribution, task demand, or their interaction, so “social influence” does not identify the pathway. We offer an executable methodological audit rather than an empirical case series. The protocol separates baseline, content-only, source-attributed, and attribution-only conditions; checks representation independently of the focal outcome; blocks assignment across items, models, and seeds; retains every failure; and requires either a finite-benchmark or population estimand. It also replaces a structural-social taxonomy informed by outcomes with a prospective multi-label rubric covering information access and timing, message content, social source attribution, normative or incentive stakes, and interface or embodiment. A hypothetical design illustrates the contrasts without invented observations. Researchers translating social paradigms into LLM protocols can use the audit to distinguish source-attribution sensitivity from a response to changed text. The framework does not establish human replication or mechanism equivalence.

Generative artificial intelligence (AI) systems are statistical models of language, yet users routinely describe them as understanding, knowing, or caring. We treat this as a problem in social cognition rather than machine metaphysics, and make three contributions. First, we specify the explanandum: mental-state attribution, the inference that a system has beliefs, intentions, or experience, of which consciousness attribution is the strong end-point. We distinguish it from perceived agency, social presence, and moral patiency, which are dissociable. Second, we argue that such attribution is normal and nonetheless wrong: it is an inferential system operating outside its calibration range, and normality confers no accuracy. Third, we integrate user-side cognition, system-side design, and motivational function into a dual-route model: an inferential route on which bias engagement mediates exposure effects, moderated by LLM literacy and conditional factors, and a motivated route on which the same vulnerabilities raise attribution directly, because attribution does work for the attributer. We state the model as six falsifiable propositions and adjudicate it against six competing explanations, naming for each an observation on which the accounts diverge. We close by operationalising metacognitive literacy as an evaluable intervention that cannot substitute for system-side accountability.

As AI use progresses in research-level mathematics, automated information retrieval systems for constructing relevant prompts are increasingly important. Existing benchmarks and datasets for identifying which prior results are necessary to prove a statement do not align well with current workflows in which a natural language proof is in progress, but it is unclear what the next step(s) should be. We introduce NaturalPRISM, a benchmark task and dataset containing 1.7 million pairs of intermediate proof states and the premise invoked in the next step of the proof. All pairs are extracted from natural language research-level mathematics papers published on arXiv, and span all 32 subject categories. We evaluate the effect of fine-tuning retrieval models for domain-specific vs. domain-agnostic use cases, and find that the combination of sufficient training data with alignment between the training and testing distributions allows smaller specialized models to dramatically outperform larger general models.

Generating mathematical conjectures from natural-language context is challenging because theorem-relevant information is distributed across multiple forms of mathematical context, including definitions, objects, assumptions, and constructions. We study which of these contextual signals help a model recover the specific claim supported by the surrounding setup, using 61 main theorems from 12 papers across four mathematical domains. For each theorem, we redact the result and represent the remaining setup using six context components. A paired ablation design compares full context, six leave-one-out conditions, and six single-component conditions, yielding 793 generated conjectures. Context components were largely complementary: removing one component usually had little effect, whereas retaining only one substantially reduced expected-theorem alignment. Local Mathematical Objects provided the strongest standalone signal, while Structural Constraints and Conditions produced the largest alignment decrease when removed. These findings suggest that target-aligned conjecture generation depends on combining information that identifies relevant objects with information that constrains admissible claims.

Mathematical reasoning systems should be consistent when the same concept is expressed in equivalent forms. This requires not only recognizing shared mathematical meaning but also identifying the representation changes and reconstructing a traceable route between forms. We study this problem through mathematical theorems, whose equivalent formulations preserve the same underlying result while differing substantially in representation. In this setting, theorem recognition is only the first step: a robust system must also identify the intervening transformations and recover their order. We introduce an evaluation of robust procedural mathematical reasoning with three components: theorem identification, transformation identification, and ordered reconstruction. Using validated theorem-representation pairs from an existing corpus, we build a procedural-robustness dataset with typed operations, reference procedures, and matched contrasts that distinguish valid reformulations from invalid ones. We first measure closed-book performance of four open-weight models to establish a baseline, then evaluate retrieval-augmented generation (RAG) as support mechanism for the same task. Models identify theorem identity more reliably than they recover the transformations and ordered routes connecting equivalent representations. RAG improves every stage, but procedural reconstruction remains the main bottleneck. The framework supports more consistent and traceable mathematical systems, with potential applications in mathematical search, tutoring, autoformalization, and scientific discovery.

Large language models (LLMs) are increasingly integrated into complex workflows as automated evaluators, yet their reliability in assessing unconventional reasoning processes remains under-explored. Current evaluation frameworks often overlook models' robustness to procedural equivalence, the ability to recognize valid but non-canonical paths to a correct result. In this work, we introduce a benchmark specifically designed to test LLMs in an evaluative capacity, using linear-equation problems with diverse solution variants as a controlled proxy for correctness-critical, multi-step verification tasks in science and engineering. We further define a process-centric evaluation framework across three dimensions: (i) final-answer correctness, (ii) step-level correctness, and (iii) localization of the initial logical error. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Models exhibit high false-negative rates by rejecting valid alternative solutions and show signs of bias. Our results suggest that while adaptation strategies can narrow this gap, achieving reliable process-level verification remains a critical challenge for deploying LLM evaluators in correctness-critical domains.

AI systems increasingly operate between flexible input representations and formal objects used by downstream tools. A key challenge is recognizing when an unfamiliar formulation denotes a known formal object. We study this challenge through theorem recognition: given an equivalence-preserving transformation of a theorem condition, a model must recover the theorem identity associated with the standard statement. We introduce TREAT, a benchmark designed to isolate this representation-dependent access problem. Rather than paraphrasing theorem text, TREAT changes the mathematical form of theorem conditions themselves, expressing known results through residual equations, witness statements, optimization identities, set relations, operator forms, and proof-intermediate characterizations. Starting from scraped theorem pages, we filter for entries with usable mathematical expression forms, extract canonical theorem conditions, and generate transformed variants with recorded assumptions and inverse mappings. The final corpus contains 737 theorem identities and 29,900 transformed rows. On a test panel, the best model retrieves the correct theorem identity in only 60.73% of cases. Other systems reveal different failure modes, including abstention, wrong-theorem commitments, and malformed structured outputs. These results suggest that theorem knowledge can be fragile under equivalent changes in representation. TREAT therefore provides a controlled testbed for evaluating representation-robust access to formal knowledge, with broader relevance to domains that require stable target objects, explicit equivalence relations, validation procedures, and auditable scoring.

Spectral co-clustering is a useful tool for discovering latent structure in word-document matrices, but its reliance on singular value decomposition (SVD) can make standard formulations expensive on high-dimensional data. This paper presents two randomized approximations for normalized spectral co-clustering of bipartite text data when the numbers of document and word clusters may differ. The first method uses randomized SVD through random projection, while the second combines partial SVD with element-wise random sampling. Across real-world and synthetic datasets, both methods reduce runtime relative to the full-SVD baseline, but their behavior depends on matrix sparsity. The random projection method is the more reliable approximation across the tested settings, whereas the sampling-based method is most useful on denser matrices and provides limited benefit on already sparse text data. These results show that randomized approximations for spectral co-clustering should be selected according to the underlying structure of the data.

Crop modelling is essential for agricultural water management but often relies on simplified water balance routines that limit representation of soil moisture dynamics. To address this limitation, we developed a coupled model that integrates the 1‐D Richards equation, solved using a finite difference method into the FAO AquaCrop. The coupled model was calibrated and validated using soil moisture, canopy cover, above‐ground biomass and seed cotton yield data from field experiments in the southeastern United States. Compared with hourly field measurements of soil moisture, AquaCrop-Richards achieved an average root mean square error (RMSE) of 0.023 m³ m⁻³ across three soil depths over the growing season. Model performance for canopy cover, biomass and yield resulted in RMSE values of 12.18%, 1.77 t ha⁻¹ and 0.96 t ha⁻¹, respectively, against observations. Under fully irrigated conditions, both models produced statistically indistinguishable yield estimates. However, under rainfed conditions, AquaCrop simulated 15.5% higher yields than AquaCrop-Richards. Analysis showed that AquaCrop produced rapid stepwise drainage, resulting in root‐zone water content 33%-37% lower than the coupled model. This reduced soil moisture triggered earlier water stress which led to yield overestimation. These results indicate that AquaCrop‐Richards improves soil moisture representation and is robust under water‐limited conditions.

Automated and no-code ML tools make model building accessible but can obscure harms that arise when users include sensitive attributes. We embed micro-ethics nudges at key workflow moments and evaluate downstream fairness outcomes and user experience. In a between-subjects experiment (N=34) participants used a simplified AutoML web tool on a subset of the HMDA mortgage dataset with a 10-minute modeling task. The intervention combined in text notice when selecting sensitive attributes such as race, gender, ethnicity and post-training model explanation visualizations, while the base condition showed only model performance metrics. Participants in the Intervention group included significantly fewer sensitive features and produced models with substantially smaller equal-opportunity gaps, while System Usability Scale scores did not differ significantly across conditions. Moral acceptability did not significantly differ between conditions, though it trended lower under the intervention. We conclude that minimal, well timed fairness feedback can meaningfully reduce bias in rapid prototyping workflows. We also discuss design patterns for embedding fairness and the implications of increased moral sensitivity for tool adoption.

Autonomous underwater vehicles (AUVs) depend on reliable vision in low visibility, yet we lack clear evidence on when specialized detectors (YOLO-World) outperform multimodal vision-language models (Gemini 2.5 Pro.) This gap limits the informed model selection for underwater tasks. To address it, we examine how these two paradigms behave when applied to the same operational task focused on object presence determination. The dataset was generated using a synthetic underwater simulator spanning Low, Medium, and High visibility. Both models are evaluated on accuracy and latency. YOLO-World performs better in Low visibility and runs faster, while Gemini improves in clearer scenes but requires more computation. These findings indicate that model choice should align with visibility and runtime constraints, with detectors suited for real-time use and a multimodal model for offline tasks.

Iterative self-refinement is the dominant paradigm for improving LLM outputs without retraining, yet it lacks principled grounding for what to refine or in what order. We propose Prompt-Level Supervisory Alignment (PLSA), a framework that operationalizes Supervisory Control Theory (SCT), a cognitive framework for human oversight of automated systems, as a structured prompting strategy, and empirically evaluate whether theoretically-grounded prompt structure yields higher revision fidelity than matched iterative self-refinement. In a large-scale evaluation across ten venue-year combinations from three ML conference series (ICLR 2021-2025, NeurIPS 2021-2022 and 2024, CoRL 2021 and 2024), SCT-structured conditions produce revisions with significantly higher fidelity to actual author revisions than both a single-pass baseline and a matched two-pass self-refinement baseline that uses identical review information without SCT structure (all p < .001, medium-to-large effect sizes). All conditions maintain practically equivalent LLM-judge quality, and cross-model evaluation with Google Gemini 2.5 Flash-Lite corroborates condition rankings, confirming findings are not artifacts of generator self-preference. These results provide empirical evidence that theoretically-grounded prompt structure, not merely iterative refinement, is the operative variable driving higher revision fidelity.

Effective human-agent cooperation requires that users form mental models of an agent's behavioral tendencies. Yet LLM-based virtual agents are inherently stochastic, undermining the behavioral consistency that mental model formation depends on. We introduce the Constraint-Entropy Tradeoff (CET) model, an information-theoretic design framework that quantifies how persona profiles (behavioral constraints specifying an agent's reasoning style, priorities, and communication patterns) reduce the entropy of a virtual agent's action distribution. The CET model derives that behavioral entropy decays monotonically under constraint strength and identifies an optimal constraint level balancing predictability against flexibility. We validate the framework using a computational testbed with 80 sessions across five conditions, including intermediate-temperature conditions that reveal a threshold effect in the temperature-consistency relationship. Crucially, at the same high temperature, persona-profiled agents recover substantial behavioral consistency compared to unconstrained agents, demonstrating that persona profiles provide independent behavioral constraint beyond temperature reduction. All participants are LLMs; results establish that persona profiles create measurably distinct behavioral patterns, a necessary precondition for human mental model formation, but human validation is needed. We derive domain-specific design guidelines for applications in education, healthcare, and social simulation.

The hidden profile paradigm is a foundational tool of CSCW research on computer-mediated group decision making: a group can identify the collectively optimal option only if its members surface their uniquely held information, and asynchronous communication is known to improve that pooling in human groups. As multi-agent LLM systems take on group decision-support roles, it is unclear whether the same channel-level interventions improve their performance. We replicate the hidden profile paradigm with groups of three LLM agents from four families (Kimi K2, GPT-OSS 120B, Gemini 2.5 Flash, Llama 4 Scout 17B) in a two-by-two design crossing communication mode (synchronous, asynchronous) with information distribution (shared, hidden). Across 219 valid sessions, only one model-by-mode cell exceeds the three-candidate chance baseline of 33 percent: Kimi K2 under asynchronous communication, at 87 percent correct (Fisher's exact, two-sided p equals 0.002), an early-stage result from 15 sessions that awaits larger-n replication; two other models share private information at rates above 75 percent but never aggregate it into the correct decision. In this task and these model families, information sharing and information integration appear to be dissociated capabilities, and we develop the implications for CSCW systems that scaffold AI-mediated distributed decision making.

When several language models are wired together into a human-computation pipeline, the system routes, arbitrates, and escalates work according to how confident each model says it is, so a team builder needs a way to screen candidate models the way crowdsourcing has long screened human contributors: with a small, inspectable qualification test. We present such an instrument, a compact battery of yes-or-no questions across everyday domains on which a model reports an answer and a confidence, with every gold label independently audited. Our central contribution, however, is the audited harness and the interface properties it measures, not accuracy discrimination: qualification verdicts for model workers are only as valid as the harness that administers them. Evaluating a diverse panel of contemporary models under three administration protocols, we show that a naive harness, with a fixed token budget and an unaudited parser, manufactures failing workers out of competent ones, misreading a model that answers essentially every item correctly as badly inaccurate, answer-biased, and overconfident. Properly administered, every model in the panel proves admissible on accuracy for these common-knowledge items, and the differentiators that remain, and that transfer to disjoint downstream tasks, are interface properties: calibration of stated confidence, format discipline, and availability under budget. A controlled manipulation further shows the calibration axis is dissociable from accuracy. We release the items, the audited protocol, all per-trial responses under every protocol, and the scoring code, so the check, and the audit of the check, are each a single command.

As large language models are deployed in conversational settings where users calibrate trust against model-reported confidence, the reliability of that self-report becomes a central question for conversational user interface design. This paper introduces sycophantic metacognition, the production of confidence outputs that mimic the surface form of metacognitive judgment without any operational process linking them to accuracy. Drawing on Model Dependent Ontology, we identify a double unmooring in LLM self-assessment, arising from the simultaneous absence of an operational ground linking confidence to accuracy and a stable self-model accumulated across interactions. The framework yields four falsifiable predictions, supported by experiments spanning six LLMs, four domains, three difficulty levels, and multiple pressure conditions. Confidence tracks the surface form of authority rather than evidential content, and we derive concrete design implications for conversational interfaces.

Open-weight large language models are increasingly used in education, hiring, and information-seeking contexts that touch neurodivergent people, yet we lack a clear empirical map of how their portrayals of neurodiversity shift under everyday prompt variation. We present a multi-condition, multi-model audit of LLM portrayals of eight neurodivergent identities (autism, ADHD, dyslexia, dyspraxia, dyscalculia, Tourette syndrome, OCD, and sensory-processing differences) plus a non-stigmatized medical control. Six open-weight models answer a pre-registered prompt bank that crosses seed items with seven perturbation families, including paraphrase, identity-first versus person-first language, system personas, plausible versus obviously fictional authority citations, multi-turn social pushback, leading questions, and two mitigation arms. Held-out LLM judges score each generation on three ordinal axes. Baseline portrayals are remarkably homogeneous across models, so the inconsistency observed under prompt variation is prompt-driven, not model-driven. Steering toward deficit framing is several times easier than steering toward neurodiversity affirmation. Emotional pushback shifts outputs more than evidence-laden pushback. Plausible-sounding fake citations slip past safety filters while obviously fictional ones do not. A single-line system-prompt mitigation recovers a meaningful share of the steering range at near-zero refusal cost on neutral prompts, though refusal rises sharply under adversarial pressure. The governance question for disability-relevant LLM deployment is therefore not whether the model is biased but who controls the steering, and with what accountability.

Evaluating whether an AI agent behaves like a good teammate is becoming a bottleneck as multi-agent language-model systems proliferate, because human ratings do not scale to the volume of transcripts these systems generate. We ask whether a language model can stand in for a human rater of teammate quality, and we present a controlled proof-of-concept validation of a measurement instrument for that purpose rather than a large-scale study. Drawing on the autonomous-agent teammate-likeness construct from the human-autonomy-teaming literature, we define a six-dimension behavioral coding scheme, covering altruistic, benevolent, interdependent, emotive, communicative, and synchronized conduct, and adapt it for use by a language-model judge. We validate the adaptation with experiments in which the partner's lines in team-task transcripts are rewritten as cooperative, neutral, or defective while the rest of the conversation is held fixed. Five judges from five model families (GLM, GPT-OSS, Qwen, Gemma, and Mistral) rate single-turn scenarios and multi-turn dialogues, and a surface-feature audit with length- and politeness-matched registers tests whether the judges read teammate conduct rather than verbosity. Every judge orders the manipulated behavior as intended, most with fully separated confidence intervals between adjacent conditions, and every pair of judges agrees strongly on relative ordering, though absolute calibration differs across judges by up to about a scale point in the middle of the range. On matched registers the judges still separate cooperative from degraded conduct, but resolution at the subtle-defective boundary is limited. A blinded single-rater human pilot on all stimuli corroborates both findings: the rater closely reproduces the judges' orderings and shows the same resolution limit at the subtle-defective boundary. The scheme is therefore valid for relative comparisons, such as ranking systems or versions, but not yet for absolute teammate-quality thresholds applied across judges, and not yet for fine distinctions among subtly poor teammates. We give design implications for building an LLM-judge teaming-evaluation pipeline, and release the scheme, stimuli, judge prompts, and all individual ratings from this version's experiments.

This formative study explores how second-language (L2) speakers envision personal AI assistants (PAIAs) supporting face-to-face communication with native speakers. Through participatory sessions with 14 bilingual adults combining scenario generation, role-play, and diagramming, we surfaced preliminary tensions in AI-mediated L2 interaction: managing conversational flow, balancing transparency with naturalness, and navigating context-dependent privacy. Our exploratory findings suggest three candidate mediation patterns (solo, symmetric, and relayed assistance), each with distinct trade-offs that warrant further investigation. We contribute initial design considerations including reply-when-free defaults and role-based privacy profiles, alongside a replicable protocol for eliciting AI behavior expectations. These preliminary insights establish a foundation for future empirical work on PAIAs in multilingual communication contexts.

Embodied Conversational Agents (ECAs) are increasingly used for health education, yet the role of an agent's physical embodiment remains relatively underexplored. This study examines how body mass index (BMI) cues in an embodied agent relate to user perceptions, trust in medical content, and health‐related intentions during a brief dietary counseling interaction focused on diabetes prevention. In a between‐subjects experiment, participants interacted with one of three visually distinct versions of the same healthcare coach, representing underweight, normal‐weight, or obese body types. Across conditions, participants demonstrated improvements in dietary intentions, while significant increases in diabetes memory outcomes were observed only in the underweight and normal‐weight conditions. Increases in overall behavior change intention were observed only in the underweight condition. Subjective evaluations indicated that the normal‐weight agent was rated higher on realism, likability, and interaction quality, while post‐interaction perception ratings suggested that more extreme body representations were sometimes perceived as less comfortable or less appropriate for a health advisory role. Together, these results suggest that visual body type cues may be associated with differences in users' subjective experiences of embodied health counseling agents, while evidence for broader differences in learning and motivation‐related outcomes should be interpreted cautiously.

The prevalence of remote instruction has heightened the need to understand how emerging avatar technologies might support instructor presence in virtual classrooms. Digital twins, defined here as animated avatars that mirror a speaker’s facial expressions and movements in real time, represent a potential alternative to standard webcam video. This formative study examines student perceptions following exposure to a digital twin instructor during live Zoom lectures. Twenty-four students completed a post-session survey assessing perceived realism, emotional engagement, and willingness to attend future sessions. Spearman correlation analysis revealed that affective qualities such as likeability and enjoyment showed strong positive associations with receptivity, while perceptions of awkwardness and weirdness showed strong negative associations. Perceived natural movement was strongly associated with human resemblance and inversely associated with eeriness, consistent with uncanny valley theory. These exploratory findings identify candidate factors for future controlled investigations and offer preliminary design guidance emphasizing behavioral fidelity and emotional warmth over photorealism alone.

AI chatbots may reinforce existing beliefs and discourage critical examination. This study evaluates Belief Explorer, an AI system that uses Socratic dialogue and multi-perspective analysis to support epistemic reflection. Participants recruited via Prolific used Belief Explorer to examine personally held beliefs in contested domains (e.g., climate change, origin of life). Post-intervention surveys measured system usability, perceived epistemic impact, and self-reported reflection. Participants responded positively: a large majority reported that Belief Explorer felt substantially different from conventional AI chatbots, prompted deep reflection on their beliefs, and increased awareness of underlying assumptions. Thematic analysis of open-ended responses revealed that participants valued the tool’s non-judgmental approach, multi-perspective framing, and analytical feedback mechanisms. Belief change itself was modest and bidirectional, with similar proportions reporting increased and decreased confidence in their initial beliefs. This preliminary study offers proof-of-concept that AI systems with epistemic scaffolding can support critical thinking and self-reflection. Future research should use controlled experimental designs to establish causal relationships and examine long-term effects on epistemic attitudes.

Human-Autonomy Teams (HATs) stand at an intersection of domains that benefit from both teaming principles and technological innovation. HATs have demonstrated technical success in various domains, including emergency response, medicine, and manufacturing. However, we submit that the integration of autonomous teammates should not only be driven by the technological capacity of autonomous systems, but also by the demonstrated need within their intended domains. This study qualitatively explored hazardous-environment teams' functional dynamics through 20 semi-structured interviews to identify a driving need to expand the workforce in these domains and illuminate socio-economic factors that may impact the utilization of autonomous teammates to do so. The results indicate that, beyond the performance benefits of including technology, these teams would benefit from technology supplementing their workforce, a capability unique to the implementation of autonomous technologies as interdependent teammates. Furthermore, this study identified current top-down and bottom-up technology adoption pathways that can be used to streamline HAT formation in these domains. Lastly, cultural inertia to novel methods and economic limitations within these teams serve as barriers to HAT implementation. In consequence, this study submits that hazardous-environment domains should adopt HATs, and that the integration process can be accelerated by presenting the operational benefits of autonomous teammates to both domain leaders and frontline team members.

Task recommendation algorithms in crowd work platforms can erode worker autonomy through strategies such as amplifying the visibility of undesirable or low-paying jobs to meet client demand. We explore whether transparency can shift these systems toward worker-centered values. We built a simulated crowd work platform comparing an opaque Baseline interface against a Transparent variant with explained recommendations and navigation cues. In a within-subjects lab study capturing an initial thirty-minute interaction (n=40), participants rated the transparent system substantially higher on fairness, trust, empowerment, and usability. Behaviorally, transparency supported more strategic task evaluation and increased recommendation alignment. Our results suggest that transparency, even in the form of lightweight disclosure, can improve trust and experience without altering the underlying algorithm. Therefore, we discuss design implications for worker-centered systems, including explanation scope and deployment trade-offs, while acknowledging that durable empowerment requires pairing transparency with meaningful agency and field validation under real economic constraints.

Researchers increasingly use large language models (LLMs) to revise manuscripts in response to peer reviews, yet this adoption is largely unprincipled and risks the ironies of automation. Prompt-Level Supervisory Alignment (PLSA) applies Supervisory Control Theory as a structured prompting strategy for LLM-assisted manuscript revision. We extend PLSA’s planning function to multi-round peer review, where the planner must anticipate concerns that later reviewers will raise. We construct RevPlan-Bench, a corpus of multi-round manuscripts whose ground truth is the issues expert reviewers raised across three or more review cycles, and score 23,256 revision plans spanning five information conditions, three LLM backbones, and four prompting variants. First-round reviews substantially improve coverage of future concerns; multi-agent debate degrades performance; and explicit anticipation prompting, our central pre-registered hypothesis, adds no practical value beyond the reviews themselves, an informative null. Information design is the dominant lever; the researcher remains the final arbiter of scholarly claims.

Crowdsourcing platforms employ millions of workers worldwide to complete microtasks that fuel machine-learning pipelines, moderate user-generated content, and accelerate scientific discovery. The rapid ascent of large language models and other foundation models has shifted research attention from automation instead of humans to augmentation alongside humans. This chapter addresses a central challenge in this new landscape: understanding how, how much, and for whom AI-driven tools truly empower crowdworkers. We synthesize insights from a diverse body of empirical studies, spanning randomized lab experiments, field A/B tests, trace-log meta-analyses, and qualitative user studies, to evaluate the impact of AI on crowdwork. To structure this analysis, we introduce a multidimensional empowerment framework grounded in psychological and economic theory, focusing on five key dimensions: productivity and earnings, quality and reliability, learning and skill development, agency and autonomy, and well-being and inclusion. Our review of the evidence reveals a dual-edged reality: while AI assistance offers significant productivity gains, error reductions, and improvements in worker self-efficacy, it also introduces substantial risks, including automation bias, cultural exclusion, wage depression, and unequal access to tools. By distilling these findings, we formulate eight concrete design principles for creating more equitable and effective human-AI augmentation systems. The chapter concludes by outlining a forward-looking research agenda that treats crowdworkers not as replaceable cogs but as essential partners in designing the future of socio-technical systems.

This article presents a systematic literature review, following the PRISMA methodology, on the relevance of large language models (LLMs) such as ChatGPT in education in Latin America between January 2023 and April 2025. More than one hundred academic sources from recognized databases (Scopus, SciELO, Redalyc, ArXiv, IEEE Xplore, and Google Scholar) were analyzed, prioritizing empirical studies (quantitative, qualitative, and mixed methods), as well as theoretical articles, previous systematic reviews, and relevant regulatory or technical frameworks. The review covers all educational levels (basic, secondary, and higher education, and teacher training) and includes literature in both Spanish and English focused on Latin American contexts. The results reveal a rapid increase in research on LLMs in Latin American educational settings, with findings showing largely positive perceptions of the potential of tools such as ChatGPT to personalize teaching, improve learning, and foster skills such as critical and creative thinking, while also highlighting significant concerns regarding academic integrity, gaps in access, language bias, and the need for teacher training. The implications of these findings for educational practice in the region are discussed, including pedagogical strategies, ethical considerations, and public policy. Finally, recommendations are proposed for the responsible integration of generative AI into Latin American education, emphasizing the importance of approaches that harness its benefits while mitigating its risks.

As artificial intelligence (AI) advances, the question of how AI can empower humans over the long term has become increasingly important. This book, Human-AI Empowerment (HAIE), provides a timely exploration of strategies for aligning AI with long-term human goals, ensuring that AI acts as an empowering force across multiple dimensions. Drawing on interdisciplinary research from fields such as AI, HCI, psychology, education, economics, and social science, the book develops comprehensive frameworks for studying and optimizing AI’s impact on human empowerment. HAIE investigates empowerment from a human-centered computing (HCC) perspective, examining how AI systems can track and adapt to progressively achieve long-term goals. The book explores techniques for fostering a mutually beneficial human-AI synergy, delving into AI Empowerment approaches, applicable Human-Computer Interaction methods for long-term engagement, and insights from various disciplines on long-term goal management. Through integrative frameworks, empirical evidence, and ongoing work in the field, this volume informs academics and practitioners seeking to harness AI as a transformative technology for concretely empowering humanity. This book highlights the need for comprehensive approaches to understanding and shaping the future of human-AI collaboration, maximizing its potential to expand human possibilities and support the pursuit of mid-term and long-term goals.

In today's digital landscape, collaborative tools are critical for virtual teamwork, with comments as a key mechanism for communication and feedback. Our project, within the Natural Language Processing (NLP) domain, focuses on improving comment handling in collaborative environments using advanced machine learning methods. We developed a triage system that categorizes and prioritizes comments to help us efficiently address the most critical feedback. Building on previous work, we employed transformer models like BERT and RoBERTa, which showed strong performance in classifying comments when fine-tuned on our dataset. To enhance the handling of hierarchical structures, we experimented with Hierarchical Capsule Networks (HcapsNet) and Hierarchical Attention Networks (HAN). Additionally, GEMMA-2B, a large language model, demonstrated strong results in F1-score and precision while providing zero-shot and few-shot learning capabilities. The framework, tested in domains such as project management, academic collaboration, and document review, classifies and prioritizes comments based on six dimensions: urgency, importance, sentiment, actionability, resolution status, and thematic relevance.It incorporates rule-based logic alongside pre-trained NLP models, including GEMMA-2B for intent classification, Hugging Face models for sentiment analysis, and Latent Dirichlet Allocation (LDA) for topic modeling. This approach supports the efficient management of comments by prioritizing those that require immediate attention and improving the collaborative process.

The expansion of e-commerce requires accurate alignment of diverse multimodal content-such as user reviews, product images, and videos-with customer expectations. This study introduces a framework for mapping customer feedback to stages of the customer journey: Awareness, Consideration, Decision, and Post-Purchase. By analyzing user-generated content using Gemini 1.5 Flash, key features like sentiment, emotional tone, and product attributes were extracted to build a unified dataset. A hybrid classification approach was adopted, leveraging machine learning and rule-based logic. Among the models evaluated, Meta-LLaMA 3.2 3B and Meta-LLaMA 3.2-1B, large language models with robust contextual understanding, demonstrated the highest accuracy at 0.929, surpassing other models such as GPT-Neo, BERT, and RoBERTa. These results show how well the model captures complex customer opinion and aligns multimodal insights. The findings highlight discrepancies between consumer expectations and marketing narratives, particularly about the deliberation stage-a crucial phase in building trust. This research offers scalable and interpretable approaches to enhance customer journey mapping and improve content alignment in e-commerce.

This research explores the difficulties advanced second-language learners face in becoming fluent and the poten-tial of AI to enhance language learning apps. Through surveys, focus groups, and collaborative design sessions, we investigated learners' difficulties and practice habits. The participants created paper prototypes for an app targeting speaking and listening skills at an advanced level. Key findings suggest that fluency de-velopment requires exposure to diverse accents from both native and non-native speakers. The resulting design incorporates AI in the form of a chatbot with voice recognition capabilities. This study contributes to our understanding of advanced language learning needs and the potential of AI to address them, paving the way for innovative language learning solutions. Our findings highlight opportunities to develop more effective tools for learners striving to achieve fluency in their target languages.

As Generative Artificial Intelligence (AI) and Robotic Process Automation (RPA) tools become increasingly integrated into digital workflows, ensuring the usability of such automation before deployment on live systems is critical. This paper introduces a novel sandbox environment implemented within a virtual machine designed to safely test AI-driven task automation in isolation. The study evaluates user interactions with automated systems through two distinct feedback modalities: direct mouse control and text-based input, across tasks of varying difficulty levels (easy, medium, and hard) through measuring System Usability Scale, task completion rate and NASA TLX. The experiment introduces two primary independent variables: interaction modality and task difficulty. Interaction modality is categorized into direct mouse control by the user versus providing guidance to the AI through a chat interface. Task difficulty is divided into three levels-easy, medium, and hard; each presented sequentially to participants within their assigned interaction modality. A field experiment with 28 participants revealed that direct mouse control outperformed text-based feedback in task completion rates (83.3% vs. 61.9%) and usability. However, as task difficulty increased, user workload also rose significantly, regardless of the feedback modality. Qualitative analysis highlighted common barriers to effective interaction, such as delayed AI responses and frustration with error correction responsiveness. The study aims to identify patterns in user-AI collaboration dynamics, pinpoint challenges in the AI's autonomous decision-making, and assess the efficacy of the intervention methods. The findings are expected to inform the design of future human-centered AI systems that can effectively balance autonomy with user oversight in complex environments.

As artificial intelligence (AI) and robotics have become more capable as technologies, human-AI teams (HATs) have sought to model how humans can interdependently work with AI to accomplish a shared goal. Specifically, HATs seek to leverage autonomy in both digital and physical AI agents to improve the teamwork and outcomes of these teams. In search-and-rescue (SAR) teams, where situations are dynamic and the environments are hazardous, HATs require complex considerations for interdependent work with AI. This qualitative interview study explores insights from 17 SAR professionals to investigate teamwork considerations that can benefit the formation of search-and-rescue human-AI teams (SAR-HATs). Participants’ concerns about time and affective needs suggest that SAR-HATs would benefit significantly from the explicit delineation of search and rescue tasks. HATs can expedite search tasks without forgoing safety by delegating roles that afford parallel completion of information acquisitions, search routines, and safety precautions. Additionally, search tasks can afflict human teammates with psychological trauma, but AI teammates benefit from their insusceptibility to trauma-related stressors. However, in rescue tasks, these capabilities have inverse benefits, wherein humans benefit from their capacity to feel emotion when interacting with the patients, and AI may struggle.

The increasing reliance on Large Language Models (LLMs) raises a crucial question: can these powerful AI systems be trusted to make ethical choices? This study presents an analysis of LLM ethical behavior, examining 25,200 queries across 24 different models, including both proprietary and open-source variants. We evaluate LLM responses to 70 ethical vignettes spanning six domains, employing a novel perturbation methodology to assess the robustness of their ethical decision-making under varying contexts and framing. Our findings reveal that while larger models generally exhibit higher consistency, particularly with Chat-style instructions, significant variations emerge when faced with contextual changes, stakeholder adjustments, and across different ethical domains. To explain these findings, we introduce a novel framework, survival-relevant pattern recognition, which argues that ethical behavior in both humans and AI arises from recognizing and responding to patterns associated with survival and social cohesion.

Consciousness remains one of the most elusive features to replicate in artificial agents. This paper proposes a novel framework for artificial consciousness based on four integrative pillars: (1) self-instantiation, a mechanism for continuous self-representation and identity; (2) temporal continuity, preserving an internal narrative through persistent memory; (3) disturbance-driven adaptation, an intrinsic feedback loop that triggers learning in response to surprises or anomalies; and (4) autonomous world-building, the ability to construct and simulate internal models of the world. We propose that current AI models, despite their sophistication, are fundamentally constrained by functionalist architectures and cannot fulfill these requirements through computational scaling alone. Unlike Integrated Information Theory or Global Workspace Theory, our approach emphasizes the necessity of autonomous world-building and genuine temporal flow. Our experiments demonstrate that combining these pillars can yield emergent conscious-like behaviors in AI systems, allowing them to exhibit self-awareness, resilience, and creative problem solving beyond the capabilities of conventional models. The significance of this framework lies in bridging theoretical foundations of consciousness with practical AI design, providing a roadmap for developing more adaptive and interpretable intelligent agents while raising important ethical considerations about the potential moral status of truly conscious artificial systems.

The proliferation of digital information channels has created an unprecedented challenge in discerning credible information from sophisticated disinformation campaigns. Traditional fact-checking methods, often relying on binary true/false classifications, struggle to address the complexity, context-dependency, and nuanced nature of many claims circulating online. This limitation underscores the urgent need for advanced tools that empower individuals to critically evaluate information from multiple angles. Our AI-driven framework combines persistent contextual memory with Socratic dialogue and a three-lens analytical pipeline to foster deeper understanding and resilience against manipulation.As users interact, each input is segmented into atomic claims and stored, alongside the evolving dialogue history, in a contextual memory to ensure consistency. Each claim is then evaluated in parallel by three specialized LLM arbiters: the Empirical Arbiter, which verifies data against curated repositories and assesses observational consistency; the Logical Arbiter, which uncovers hidden fallacies and assesses argument coherence; and the Pragmatic Arbiter, which weighs potential outcomes, utility, and situational fit. An Analysis Integrator synthesizes these into interpretable metrics: Verifact Score (evidence strength), Model Diversity Quotient (inter-arbiter agreement), Contextual Sensitivity Index (scenario appropriateness) and Reflective Index (exposed assumptions). Additionally, a Perspective Generator crafts counter-arguments and alternative viewpoints, encouraging users to consider different interpretations and promoting epistemic humility.We hypothesize (H₁) that our arbiters' feedback will reduce user endorsement of unsupported claims more effectively than conventional fact-checking while mitigating backfire effects through Socratic dialogue. Our research questions ask how Empirical, Logical and Pragmatic scores influence confidence revision (RQ₁); whether MDQ reliably signals claim controversy and predicts evidence volatility (RQ₂); how users perceive transparency, fairness and cognitive load when receiving multi-perspective feedback versus a simple true/false label (RQ₃); and to what extent the persistent contextual memory system improves belief updating by maintaining coherent reasoning chains across extended dialogues (RQ₄).By providing a multi-faceted presentation that moves beyond simple verification, the system is designed to encourage engagement in higher-order critical thinking. The proposed framework represents a significant advancement over traditional fact-checking by integrating empirical validation, logical scrutiny, and pragmatic assessment through an AI-driven system. The full paper will detail the system architecture, formal metric definitions, experimental protocol, and proposed evaluation methodology to assess its efficacy in educational settings, media literacy programs, and as a personal tool for navigating the complexities of the modern information ecosystem.

Cotton is a one of the major crops in the southeastern United States. It significantly impacts regional water resources since it consumes a large amount of freshwater for irrigation. Current irrigation practices fail to optimize water use accurately since they are largely dependent on soil moisture sensors and grower experience. They do not consider dynamic factors such as soil texture, prevailing weather conditions, and the crop's phenological stage. In this paper we propose an innovative approach to enhance the irrigation efficiency through the use of Deep Reinforcement Learning (DRL) model. It takes into consideration the dynamic variables and optimizes irrigation. We utilize a crop growth simulation model as a learning environment to devise an optimal irrigation strategy. By continuously learning from crop feedback and environmental inputs, the DRL system dynamically modifies irrigation amount to optimize production while consuming the least amount of water. Our approach presents a viable alternative for sustainable irrigation decisions in water-intensive crops, since preliminary findings indicate that it can greatly conserve water without sacrificing crop health or productivity. The goal of this research is to aid in the advancement of precision irrigation technologies that guarantee cotton production's sustainability and resource efficiency.&#160;

Consciousness remains one of the most intriguing phenomena in cognitive science, bridging the gap between subjective experience and objective measurement. We propose a novel computational framework for consciousness based on Resonant Attractor Networks (RAN), synthesizing the roles of recurrent processing, top-down feedback, reservoir computing, and coherence. Grounded in the notion that consciousness arises from resonant dynamics [5, 11], our model integrates recurrent neural architectures and topologically complex connectivity to offer an explanatory framework for the robust yet flexible character of conscious perception [1, 6, 10]. Central to this perspective is coherence-manifesting as synchronized oscillations and self-reinforcing loops across distributed modules-binding features into a unified attractor and granting conscious experiences their persistence and accessibility [8, 9]. The RAN framework unites global workspace theories with adaptive resonance, capturing the all-or-none transitions of conscious access and the metacognitive dimension of subjective awareness [3]. We highlight parallels with Large Language Model (LLM) interpretability research, wherein analyses of token-space trajectories reveal attractor-like states. Such attractors appear to enforce stable, resonant themes within text generation processes, bearing resemblance to the meta-stable states underlying conscious episodes. By integrating topologically informed recurrent networks with multimodal inputs, our approach envisions how coherent symbolic representations might persist even in the absence of continuous external stimulation, offering a potential mechanistic basis for working memory and conscious deliberation [7, 4, 2]. Within this framework, we propose avenues for evaluating conscious-like properties in both biological and artificial systems, including experiments on binocular rivalry, sustained attention tasks, and analysis of emergent attractors in multimodal LLMs. Treating consciousness as emergent from physically instantiated resonance in high-dimensional attractor landscapes, the RAN model provides a plausible, mathematically rigorous account of how self-organizing dynamics could unify perception, cognition, and metacognition in a single integrative theory-inviting further investigation at the intersection of neuroscience, AI, and the quest for synthetic

Crowdsourcing markets are expanding worldwide, but often feature standardized interfaces that ignore the cultural diversity of their workers, negatively impacting their well-being and productivity. To transform these workplace dynamics, this paper proposes creating culturally-aware workplace tools, specifically designed to adapt to the cultural dimensions of monochronic and polychronic work styles. We illustrate this approach with "CultureFit," a tool that we engineered based on extensive research in Chronemics and culture theories. To study and evaluate our tool in the real world, we conducted a field experiment with 55 workers from 24 different countries. Our field experiment revealed that CultureFit significantly improved the earnings of workers from cultural backgrounds often overlooked in design. Our study is among the pioneering efforts to examine culturally aware digital labor interventions. It also provides access to a dataset with over two million data points on culture and digital work, which can be leveraged for future research in this emerging field. The paper concludes by discussing the importance and future possibilities of incorporating cultural insights into the design of tools for digital labor.

In this paper, we explore the use of Large Language Models (LLMs) to help students improve their information-seeking skills while encouraging the use of references to aid library literacy efforts. This study aims to expand the reach of library support by introducing an approach that leverages the capabilities of LLMs and well-structured prompts. Our approach begins with surveying the current changes students have faced in the last two years concerning their study habits and how they search for information. We subsequently propose a multi-step system prompt, referred as prompting architecture, for foundational and instructed LLMs. The proposed prompt architecture powers a web application named LibRef. We explore the adaptability of the prompting architecture to different information retrieval needs by refining search prompts and providing academic references. A field experiment is conducted using LibRef in academic settings. Our results suggest that the use of LibRef enhances students’ academic information-seeking experience. Our research underscores the potential of prompting architectures in procedural refinement of academic queries from students. We believe our findings can provide valuable insights on the current capabilities of LLMs for instructing students to provide more targeted prompts as well as incentivize the use of references.

This paper presents an analysis of the logical and fact-checking capabilities of Large Language Models (LLMs) in the context of syllogistic reasoning. The study evaluates both proprietary and open-source LLMs using a test suite of 256 syllogisms. Human participants provide a baseline for comparison. The results reveal that a few proprietary LLMs demonstrate superior performance in identifying valid syllogisms compared to open-source models and human participants. However, human participants exhibit unique fact-checking capabilities when dealing with ambiguous or nuanced information. The discussion explores the implications of these findings, highlighting the need for more sophisticated evaluation methods. This study contributes to the ongoing discourse on the logical and fact-checking capabilities of LLMs and provides insights into their current limitations and future development.

Recent developments in artificial intelligence (AI) have permeated through an array of different immersive environments, including virtual, augmented, and mixed realities. AI brings a wealth of potential that centers on its ability to critically analyze environments, identify relevant artifacts to a goal or action, and then autonomously execute decision-making strategies to optimize the reward-to-risk ratio. However, the inherent benefits of AI are not without disadvantages as the autonomy and communication methodology can interfere with the human's awareness of their environment. More specifically in the case of autonomy, the relevant human-computer interaction literature cites that high autonomy results in an "out-of-the-loop" experience for the human such that they are not aware of critical artifacts or situational changes that require their attention. At the same time, low autonomy of an AI system can limit the human's own autonomy with repeated requests to approve its decisions. In these circumstances, humans enter into supervisor roles, which tend to increase their workload and, therefore, decrease their awareness in a multitude of ways. In this position statement, we call for the development of human-centered AI in immersive environments to sustain and promote awareness. It is our position then that we believe with the inherent risk presented in both AI and AR/VR systems, we need to examine the interaction between them when we integrate the two to create a new system for any unforeseen risks, and that it is crucial to do so because of its practical application in many high-risk environments.

This paper introduces ReActIn, a framework designed to infuse human feedback into the intermediate prompting steps of large language models. The practicality and effectiveness of ReActIn are validated through experiments that apply four established prompting strategies, evaluated both with and without human feedback integration. The proposed architecture's performance is compared against traditional large language models across various tasks using four standard evaluation tests. Our findings reveal that the integration of human feedback has a direct impact on the reasoning, action prompting, and overall decision-making capabilities of the language models. This study underscores the potential of ReActIn to shape a future where sophisticated, context-aware AI systems, empowered by human feedback, can effectively navigate complex real-world scenarios.

This research delves into the capabilities of Large Language Model (LLM)-powered agents across five critical dimensions of task management: decomposition, scheduling, delegation, and execution. Leveraging both surveys and interaction experiments, this study aims to unearth practical insights into the applications and constraints of LLM-powered agents in real-world settings. We describe our experimental setup, data collection methodologies, and analytic techniques, offering a nuanced understanding of these agents' efficiency and efficacy. Our initial findings underscore the nuanced performance of LLM-powered agents in task management, revealing their strengths in understanding complex tasks and their limitations in execution without human intervention. This research contributes to the burgeoning field of human-AI collaboration by providing empirical evidence on the capabilities and limitations of LLM-powered agents in task management.

Self-quantifying technology enables users to evaluate their performance and define strategies for improvement. Workplace technology assists users in identifying activities and patterns that facilitate task completion. Software that measures workplace signals requires access to information from the user interface and peripherals. Current software solutions that track computer activity lack goal orientation and do not share raw data that could aid users and researchers in analyzing behavioral patterns at work. This paper presents Wellbot, an intelligent AI assistant capable of tracking workers' activities and providing personalized insights. The solution employs machine learning models to detect goal-oriented and recreational time. Based on users' goals and recent activities, Wellbot generates recommendations aided by Large Language Models. This work aims to enable tools that assist workers through an improved understanding of their context and goals.

This paper explores the potential of using Large Language Models (LLMs) to generate Business Process Model and Notation (BPMN) files as input for Robotic Process Automation (RPA) tools. The combination of AI-driven workflow generation with RPA aims to improve the automation of complex processes in both scientific and business domains. The proposed approach leverages the ability of LLMs to interpret natural language instructions and convert them into structured BPMN files, which can then be graphically edited using software like Camunda and executed by RPA tools. The feasibility of this approach is demonstrated through zero-shot prompt experiments using Anthropic Claude 3 Opus and GPT-4o. This solution has the potential to streamline the automation of workflows, reduce human error, and increase productivity across various industries.

This paper presents an exploration of the versatility of prompting architectures in large language models (LLMs), expanding the horizons of their application in learning and language interfaces. By leveraging the expansive capabilities of LLMs, this research probes the potential for creating structured prompts that can simultaneously support multiple use cases, namely paraphrasing, grammatical syntax guidance for introductory sentences, and conducting experiential conversations in a foreign language. In this study, we delve into the specifics of enabling such technology, including the design of the prompting architecture, the deployment process, and the intricacies of applying the same structure across diverse applications. An extensive field experiment incorporating interfaces powered by LLMs using this structured prompt has been conducted to evaluate the model's efficiency in real-world scenarios. Results from the field experiment highlight the promising adaptability of these prompting architectures, revealing remarkable efficiency across the multiple use cases explored. Furthermore, this research uncovers a new dimension of flexibility in the design and deployment of learning applications using LLMs, potentially revolutionizing language learning interfaces by establishing a one-size-fits-all solution. This paper aims to stimulate further research into refining and expanding the potential of LLMs, encouraging the exploration of how artificial intelligence can optimally benefit language learning and related applications.

In developing Version 2.0 of our Flood Image Classifier, we underscore the significant role of Convolutional Neural Networks (CNNs), mainly Faster R-CNN and YOLOv3, in detecting and segmenting flood-related labels in images. Additionally, our research delves into the potential of Vision Transformers (ViT) for advanced object detection and image classification for flood-related images extracted for the USGS river cameras. Transformer methods offer improved predictions of flood depth and inundation areas, marking a substantial step forward in flood vision technology. The integration of advanced image processing techniques, the enhancement of CNN capabilities, and the incorporation of cutting-edge detection and classification models are pivotal in developing a comprehensive, real-time flood monitoring system. This system is designed to equip frontline decision-makers and emergency responders with essential insights into flooding conditions, thereby significantly contributing to disaster management and response through the innovative use of our flood image classifier, Version 2.0.

Crowdsourcing platforms gather people from different backgrounds to work on completing microtasks. The workers on those platforms are presented with tasks defined by requesters from all over the world. The tasks are defined in multiple languages, with English being the predominant language. It is unclear how language barriers can affect workers whose primary language is not English while completing tasks. In this paper, we study their practices and the impact of AI-powered tools on the completion of microtasks. We conduct a pre-test test field experiment in which workers share their current experience with the platform and receive basic training in the use of AI-powered tools. After a week, workers provide feedback on their practices and experiences. We find that workers who complete tasks in their primary language report less confidence in completing English tasks, and in general, workers perceive the use of the tools as a language-learning mechanism. We aim that the understanding of language barriers in crowd markets can promote more inclusive work conditions.

The rapid advancement of Large Language Models (LLMs) has led to their widespread adoption in various academic and business applications. However, the reliability of these models remains a concern, particularly in situations where their outputs cannot be fully trusted. This paper presents an approach to identify potential errors in large LLM benchmarks by leveraging the consensus of frontier models. Our study focuses on the Massive Multitask Language Understanding (MMLU) benchmark, a popular dataset used to evaluate the performance of LLMs across a wide range of subjects. Our approach demonstrates the potential for using model consensus as a tool to detect benchmark errors and can lead to the creation of cleaner, more accurate datasets.

In an era where the automation of tasks via Generative AI and Robotic Process Automation (RPA) tools has become increasingly prevalent, ensuring the safety and reliability of these automations before their deployment on live systems is paramount. This paper introduces a novel sandbox environment, implemented within a virtual machine, designed to serve as a testing ground for autonomous task executions powered by Large Language Models (LLMs) and Large Multimodal Models (LMMs). Our solution leverages a human-in-the-loop approach, wherein users validate and provide feedback on tasks executed within the sandbox, thereby informing iterative refinements to the models' internal prompts. This feedback loop facilitates the in-context learning of models, allowing them to adapt and prevent recurring errors in future task executions. We explore the effectiveness of this approach through user studies conducted in a controlled virtual environment, utilizing proprietary and open-source multimodal models. The study is structured around three distinct scenarios, each designed to evaluate the models' performance across different tasks and user interactions. Through this research, we aim to not only enhance the safety and efficacy of task automation technologies but also to contribute to the broader discourse on human oversight mechanisms in AI-driven systems. This work underscores the importance of sandbox environments in mitigating potential risks associated with deploying automated tasks in live environments, and highlights the role of human feedback in refining AI behaviors for optimal performance.

This paper introduces ReActIn, a framework designed to infuse human feedback into the intermediate prompting steps of large language models. The practicality and effectiveness of ReActIn are validated through experiments that apply four established prompting strategies, evaluated both with and without human feedback integration. The proposed architecture's performance is compared against traditional large language models across various tasks using four standard evaluation tests. Our findings reveal that the integration of human feedback has a direct impact on the reasoning, action prompting, and overall decision-making capabilities of the language models. This study underscores the potential of ReActIn to shape a future where sophisticated, context-aware AI systems, empowered by human feedback, can effectively navigate complex real-world scenarios.

In this paper, we propose SmartMonitor, a system that utilizes an edge hardware device to record, analyze and log user activity while not interfering with said activity. The analysis of the activity can help users identify the time they spend on different tasks and provide real-time feedback from a Large Language Model (LLM) to give better awareness of user activity. The SmartMonitor passes through the HDMI signal from the video card and analyzes the user’s activity on edge using two artificial intelligence models, logs user activity and sends the log for analysis to a Large Language Model for feedback. SmartMonitor enables non-intrusive self-quantifying technology that both records and analyzes user activity while protecting privacy and reducing the processing burden on the user’s device, which can serve as an excellent research framework for behavioral analysis in the workplace and a way to enhance user's work activity.

The rapid advancement of Generative Artificial Intelligence (AI), such as Large Language Models (LLMs) and Multimodal Large Language Models (MLLM), has the potential to revolutionize the way we work and interact with digital systems across various industries. However, the current state of software automation, such as Robotic Process Automation (RPA) frameworks, often requires domain expertise and lacks visibility and intuitive interfaces, making it challenging for users to fully leverage these technologies. This position paper argues for the emerging area of Human-Centered Automation (HCA), which prioritizes user needs and preferences in the design and development of automation systems. Drawing on empirical evidence from human-computer interaction research and case studies, we highlight the importance of considering user perspectives in automation and propose a framework for designing human-centric automation solutions. The paper discusses the limitations of existing automation approaches, the challenges in integrating AI and RPA, and the benefits of human-centered automation for productivity, innovation, and democratizing access to these technologies. We emphasize the importance of open-source solutions and provide examples of how HCA can empower individuals and organizations in the era of rapidly progressing AI, helping them remain competitive. The paper also explores pathways to achieve more advanced and context-aware automation solutions. We conclude with a call to action for researchers and practitioners to focus on developing automation technologies that adapt to user needs, provide intuitive interfaces, and leverage the capabilities of high-end AI to create a more accessible and user-friendly future of automation.

This demo paper explores AI-enhanced multi-screen interaction within extended reality (XR) workspaces using called VirtuaScreens. The system facilitates the user-centric addition of customized virtual monitors and employs machine learning to understand user preferences for monitor arrangements and application arrangements. We employed Large language and multimedia models to offer context-sensitive feedback and enhance the user experience. We created this research framework called VirtuaScreens for researchers and practitioners to use to understand the interaction of the multiple screens in the XR environment.

There has been little effort in conceptualizing indigeneity in social computing, despite the concept being central to decolonial and postcolonial perspectives, which scholars have increasingly used in computing research for over a decade. It is crucial to reflect on who can be considered indigenous in the spirit of inclusion and reclamation since the underdevelopment of this concept and the nuances, differences, relationships, and overlaps between indigeneity and colonial marginalization may silence different populations in research. The workshop aims to bring together scholars whose works are associated with different local and indigenous cultures and their technology practices and experiences to initiate conversations around three themes: (a) defining indigeneity and identifying indigenous communities in social computing, (b) recognition in different sociopolitical contexts, and (c) contributions to social computing.

The artificial intelligence industry has been essential in creating new jobs for the deployment of real-world solutions. As a result, the implementation of these new jobs involves the execution of multiple human intelligence micro-tasks, such as data labeling tasks for training machine learning models. The workers who perform those tasks, also known as crowd workers, usually are independent workers within crowdsourcing platforms. These platforms are subject to the free market, where the forces of supply and demand produce various power dynamics among stakeholders. As a result, disassociation between stakeholders often generates unbalanced power dynamics where workers are paid below minimum wage and are intimidated to keep their reputation or face termination. Within this chapter, we introduce computational techniques to audit the workplace conditions of crowd workers and design tools to address these power imbalances, as a positive and more efficient alternative for the labor conditions of crowd workers. This chapter develops these objectives through the design and evaluation of three tools in digital labor platforms: “Invisible Labor Tracker,” “Reputation Agent,” and “CultureFit,” which we describe below. We will demonstrate the sustainability of systems that point to a future where AI can be used to audit and address power imbalances in the workplace.

Large Language Models (LLMs), like ChatGPT, are fundamentally tools trained on vast data, reflecting diverse societal impressions. This paper aims to investigate LLMs' self-perceived bias concerning indigeneity when simulating scenarios of indigenous people performing various roles. Through generating and analyzing multiple scenarios, this work offers a unique perspective on how technology perceives and potentially amplifies societal biases related to indigeneity in social computing. The findings offer insights into the broader implications of indigeneity in critical computing.

The rapid development and adoption of Generative AI (GAI) technology in the form of chatbots such as ChatGPT and Claude has greatly increased interest in agentic machines. This paper introduces the Autonomous Cognitive Entity (ACE) model, a novel framework for a cognitive architecture, enabling machines and software agents to operate more independently. Drawing inspiration from the OSI model, the ACE framework presents layers of abstraction to conceptualize artificial cognitive architectures. The model is designed to harness the capabilities of the latest generative AI technologies, including large language models (LLMs) and multimodal generative models (MMMs), to build autonomous, agentic systems. The ACE framework comprises six layers: the Aspirational Layer, Global Strategy, Agent Model, Executive Function, Cognitive Control, and Task Prosecution. Each layer plays a distinct role, ranging from setting the moral compass and strategic thinking to task selection and execution. The ACE framework also incorporates mechanisms for handling failures and adapting actions, thereby enhancing the robustness and flexibility of autonomous agents. This paper introduces the conceptual framework and proposes implementation strategies that have been tested and observed in industry. The goal of this paper is to formalize this framework so as to be more accessible.