2026-10-14T11:03:49Z https://api.figshare.com/v2/oai

oai:figshare.com:article/34050262 2026-10-01T19:00:06Z category_7 category_45 category_734 category_931 portal_87 item_type_3 month_year_10_2026

GeoAgentbench: a dynamic execution benchmark for tool-augmented agents in spatial analysis Bo Yu (241082) Cheng Yang (273624) Dongyang Hou (10843800) Chengfu Liu (25158301) Jiayao Liu (20971625) Chi Wang (131921) Zhiming Zhang (176541) Haifeng Li (142063) Wentao Yang (205781) Medicine Sociology Biological Sciences not elsewhere classified Information Systems not elsewhere classified GeoAI tool-augmented agents spatial analysis benchmark <p>The integration of large language models (LLMs) into geographic information systems (GIS) marks a paradigm shift toward autonomous spatial analysis. However, evaluating LLM-based agents remains challenging due to the complex, multi-step nature of geospatial workflows. Existing benchmarks primarily rely on static text or code matching, neglecting dynamic runtime feedback and the multimodal nature of spatial outputs. To address this gap, we introduce GeoAgentBench (GABench), a dynamic and interactive evaluation benchmark tailored for tool-augmented GIS agents. GABench provides a realistic execution sandbox integrating 133 atomic GIS tools and encompasses 57 representative spatial analysis tasks across six core GIS domains. Recognizing that precise parameter configuration is a key determinant of execution success in dynamic GIS environments, we design the parameter execution accuracy (PEA) metric, which utilizes a last-attempt alignment strategy to quantify the fidelity of implicit parameter inference. Complementing this, a vision-language model (VLM)-based verification protocol is proposed to assess data and spatial accuracy and cartographic style adherence. To support diagnostic comparisons across reasoning strategies, we instantiate four representative agent baselines, including direct tool invocation, ReAct, Plan-and-Solve, and a hybrid Plan-and-React baseline that combines global workflow decomposition with stepwise reactive execution. Extensive experiments with seven representative LLMs show that GABench can reveal distinct capability boundaries in long-chain orchestration, parameter-level execution, multimodal map-product generation, and runtime error recovery. The results further indicate that the hybrid planning–reactive strategy provides a strong reference baseline under dynamic GIS execution, while also exposing persistent limitations of current LLM agents in complex spatial analysis. Overall, GABench establishes a reproducible benchmark for assessing and advancing tool-augmented agents in autonomous GeoAI. The source code and dataset are available at: <a target="_blank" href="https://github.com/GeoX-Lab/GABench">https://github.com/GeoX-Lab/GABench</a>.</p> 2026-10-01T19:00:06Z Dataset Dataset 10.6084/m9.figshare.34050262.v1 https://figshare.com/articles/dataset/GeoAgentbench_a_dynamic_execution_benchmark_for_tool-augmented_agents_in_spatial_analysis/34050262 CC BY 4.0