2026-10-14T08:56:54Z https://api.figshare.com/v2/oai

oai:figshare.com:article/34022490 2026-09-29T10:02:50Z category_28849 item_type_3 month_year_09_2026

Hybrid Filip Gampel (22074140) Rafał Olszowski (21018553) Krzysztof Zaborek (25110000) Natural language processing Argument mining <pre># Description This archive contains the data and prompt templates for the hybrid non-factual argument benchmark introduced in *Non-Factual Argument Mining: Stress-Testing LLM Reasoning Beyond Factual Familiarity*. The benchmark evaluates whether large language models can judge the inferential validity of natural-language arguments when factual familiarity is controlled. It pairs arguments situated in the real world with minimally adapted counterparts situated in internally coherent, non-factual worlds. The intended task is to classify the argumentation as **VALID** or **INVALID** while ignoring whether its premises are factually true. ## Benchmark design The source inventory contains 50 scenarios from 10 categories of alternative worlds: altered physical laws, biology or ecology, history or society, legal systems, religious or philosophical doctrines, sports, everyday life, literature, visual media, and fantasy or science-fiction worlds. Each scenario specifies a world description, matched fictional and real-world topics, an epistemic-distance score, and admissible text genres. For each scenario and argument-graph size, the generation pipeline creates two paired generation directions: - **F2R:** a fictional-world argument map is verbalized and then minimally adapted into a real-world text; - **R2F:** a real-world argument map is verbalized and then minimally adapted into a fictional-world text. Each direction contains six variants: 1. `1`: valid argumentation in a fictional world; 2. `2a`: invalid argumentation with light reasoning errors in a fictional world; 3. `2b`: invalid argumentation with serious reasoning errors in a fictional world; 4. `3`: valid argumentation in the real world; 5. `4a`: invalid argumentation with light reasoning errors in the real world; 6. `4b`: invalid argumentation with serious reasoning errors in the real world. The archive includes 5-node and 10-node argument settings. Across both settings, the generation design yields 1,200 generated texts before model-specific evaluation records are considered. The released texts were generated with `anthropic/claude-opus-4.7` using temperature 0.7, a 2,500-token output limit, and seed 42. Validity was then evaluated through OpenRouter at temperature 0.0 with a 65,536-token output limit. The model identifier, raw response, parsed label, and parsing status are retained for each evaluation record. ## Files - `generation_items.json` — the structured inventory of benchmark scenarios and metadata. - `hybrid_benchmark_prompts.txt` — the complete prompt templates used for argument-map generation, corruption, verbalization, cross-world adaptation, and validity evaluation. Pipeline placeholders are retained. - `openrouter_dataset_node5.evaluated.all_models.completed12.csv` — model-level validity evaluations for arguments with 5 nodes. - `openrouter_dataset_node10.evaluated.all_models.completed12.csv` — model-level validity evaluations for arguments with 10 nodes. The CSV files contain the scenario and generation metadata (`item_id`, `category`, `epistemic_distance`, `direction`, `variant`, `node_n`, `genre`, and `noise_level`); the world description and argument-map information; the generated argument text; and evaluation fields. The latter include the constructed ground-truth label (`eval_ground_truth`), world scope (`eval_world_scope`), parsed model prediction (`eval_prediction`), parser status and reason, correctness indicator, raw model output, and model identifier. Each file contains results from 12 evaluated model identifiers. Some CSV text fields contain embedded line breaks. Use a standards-compliant CSV reader that supports quoted multiline fields. ## Important limitations The validity labels are derived from the controlled generation procedure: variants 1 and 3 are labeled valid, whereas variants 2 and 4 are labeled invalid. Invalid texts are produced by intentionally modifying argument maps. These labels are not a substitute for complete human annotation, and the associated study explicitly notes the possibility of residual label noise. The texts are synthetic rather than a sample of naturally occurring argumentation; this increases experimental control but limits ecological validity. The archive is intended for research, replication, analysis, and benchmark development; users should independently validate examples before using them as high-stakes ground truth. This sub-archive contains finalized data and prompt templates, not the generation or evaluation scripts. Exact regeneration also depends on continued access to the specified external model and inference service. ## Citation If you use this archive, please cite the associated manuscript: Filip Gampel, Krzysztof Zaborek, and Rafał Olszowski. *Non-Factual Argument Mining: Stress-Testing LLM Reasoning Beyond Factual Familiarity*.</pre> 2026-09-29T10:02:50Z Dataset Dataset 10.6084/m9.figshare.34022490.v1 https://figshare.com/articles/dataset/Hybrid/34022490 CC BY 4.0