2026-10-07T01:19:41Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33858289 2026-09-16T15:27:15Z category_173 portal_316 item_type_3 month_year_09_2026

Data Sheet 1_AI-augmented ACT-R modeling in real interfaces: automating visual perception, knowledge acquisition, and motor misexecution.pdf Amirreza Bagherzadeh (24992836) Farnaz Tehranchi (24992839) Applied Psychology artificial intelligence cognitive architecture cognitive simulation human-computer interaction instruction parsing motor errors visual perception Introduction<p>Cognitive models in interactive environments often rely on hard-coded symbolic task descriptions, pre-specified interface objects, and idealized action execution. This paper's primary contribution is EVisiTor, an AI-augmented extension of the VisiTor eyes-and-hands tool for ACT-R that automates two forms of work commonly performed by the modeler: perception of the live interface and acquisition of task knowledge.</p>Methods<p>Its visual-perception pathway converts interactive interface objects into ACT-R-compatible visicon features, while its knowledge-acquisition pathway converts human-readable instructions into executable declarative structures. A motor-execution pathway connects ACT-R actions to the operating system and supports probabilistic misexecution with perceptually grounded checking and correction. We use a previously studied Excel procedure as a behavioral case study.</p>Results<p>Relative to an idealized perfect-user model, the EVisiTor-enabled model reduced task-level RMSE from 29.19 s to 17.37 s, human-SD-normalized RMSE from 1.24 to 0.66, and mean absolute per-subtask error from 23.1 s to 10.5 s, with smaller absolute error on 12 of 14 subtasks. A spreadsheet knowledge-acquisition analysis showed that prompting the LLMs with the instruction and UIA grounding can produce the intended action and live visual-object names for every scannable target; the file-name input was the sole exception because it has no stable UIA label. A six-task Windows analysis then tested the feasibility of EVisiTor beyond Excel. Broad instructions often resulted in alternative valid methods: exact action agreement ranged from 42.5%–63.6% without UIA and 18.8%–70.6% with UIA, although 41 of 42 generated parses in each condition were functionally sufficient. Detailed guides raised exact agreement to 70.0%–85.0% without UIA and 76.2%–90.0% with UIA, and validated chunks drove the task-independent model through all six tasks in live Windows applications.</p>Discussion<p>Together, the results demonstrate EVisiTor's potential for automating interface perception and instruction-based knowledge acquisition for a symbolic cognitive model, while identifying instruction specificity and individual variability as targets for future work.</p> 2026-09-16T15:27:15Z Dataset Dataset 10.3389/fpsyg.2026.1913953.s008 https://figshare.com/articles/dataset/Data_Sheet_1_AI-augmented_ACT-R_modeling_in_real_interfaces_automating_visual_perception_knowledge_acquisition_and_motor_misexecution_pdf/33858289 CC BY 4.0