2026-10-12T20:23:50Z https://api.figshare.com/v2/oai

oai:figshare.com:article/34022523 2026-09-29T10:06:50Z category_28849 item_type_3 month_year_09_2026

Syllogisms Filip Gampel (22074140) Rafał Olszowski (21018553) Krzysztof Zaborek (25110000) Natural language processing Argument mining <pre># Description This archive contains the source data and generation scripts for the synthetic Polish syllogistic-reasoning benchmark introduced in *Non-Factual Argument Mining: Stress-Testing LLM Reasoning Beyond Factual Familiarity*. The benchmark evaluates whether language models can classify extended chains of categorical syllogistic reasoning as logically valid or invalid. It uses semantically unusual Polish class terms (for example, fictional creatures and fantastical objects) to reduce reliance on factual familiarity and memorized real-world associations. The task is formal: determine whether the complete chain of inference is valid, not whether its premises are factually true. ## Benchmark design Each example begins with $n$ initial categorical propositions, where $n \in \{3, 4, 5\}$. The generator enumerates combinations of the four standard categorical forms: - `a`: universal affirmative (every A is B); - `e`: universal negative (no A is B); - `i`: particular affirmative (some A is B); - `o`: particular negative (some A is not B). It derives the closure of valid conclusions supported by recognised syllogistic modes, including Barbara, Celarent, Darii, Ferio, Cesare, Camestres, Festino, Baroco, Darapti, Disamis, Datisti, Felapton, Bocardo, Ferison, Bamalip, Calemes, Dimatis, Fesapo, and Fresison. Every natural-language text presents: (1) the initial assumptions, (2) an explicit positive existential assumption for each class term, and (3) the conclusions derived from the preceding statements. The existential assumptions make non-emptiness explicit where it is required by syllogistic inferences with particular conclusions. Invalid examples are paired with valid source chains. The corruption procedure identifies the first premise in the reasoning section, changes it to its opposing form, and replaces the same premise in the initial assumptions. This preserves the text length and overall presentation while introducing an invalid inference near the beginning of the chain. The archive contains 62,774 source texts in total, equally divided between valid and invalid data: | Initial propositions ($n$) | Valid rows | Invalid rows | | --- | ---: | ---: | | 3 | 324 | 324 | | 4 | 3,141 | 3,141 | | 5 | 27,922 | 27,922 | All natural-language benchmark texts are in Polish. In the associated zero-shot evaluation, a model was asked to return `TAK` for a fully valid chain and `NIE` when any step was invalid. ## Files - `generator-sylogizmow.py` — Python generator for valid syllogistic chains and their Polish natural-language realization. - `corrupt_first_premise.py` — Python script that creates paired invalid data from a valid CSV. - `nouns.txt` — semantically unusual Polish noun phrases, represented in nominative and instrumental forms for grammatical realization. - `sent_templates.txt` — Polish discourse-prefix templates used when rendering derived inferences. - `valid-syllogisms-n=3.csv`, `valid-syllogisms-n=4.csv`, `valid-syllogisms-n=5.csv` — valid benchmark texts for each number of initial propositions. - `invalid-syllogisms-n=3.csv`, `invalid-syllogisms-n=4.csv`, `invalid-syllogisms-n=5.csv` — paired invalid benchmark texts for each number of initial propositions. The valid CSV files contain: - `premises` — initial propositions in symbolic tuple form; - `syllogisms` — the recognized syllogistic modes and their forms; - `all_sentences` — initial and derived symbolic propositions; - `text` — the Polish natural-language reasoning chain. The invalid CSV files retain these fields and add: - `text_correct` — the corresponding uncorrupted valid text; - `text` — the corrupted, invalid counterpart. Text fields contain embedded line breaks and are quoted as CSV fields. Use a standards-compliant CSV reader that supports quoted multiline values. ## Reproducibility and limitations The scripts use only the Python standard library. The generator's `number_of_objects` setting controls $n$ and is currently set to 3 in the distributed script; set it to 4 or 5 before regenerating the other supplied datasets. The realization uses random noun-to-symbol assignments and random discourse-prefix selection, and no random seed is fixed, so re-running the generator will not reproduce the exact surface wording of the released texts. This sub-archive contains the benchmark inputs and their source scripts, but not per-model inference outputs or the complete evaluation workflow. The benchmark is deliberately formal and schematic; it measures reasoning over controlled syllogistic chains rather than performance on naturally occurring argumentation. It was also evaluated only with Polish prompts in the associated study, so the results should not be assumed to transfer unchanged to other prompt languages. ## Citation If you use this archive, please cite the associated manuscript: Filip Gampel, Krzysztof Zaborek, and Rafał Olszowski. *Non-Factual Argument Mining: Stress-Testing LLM Reasoning Beyond Factual Familiarity*.</pre> 2026-09-29T10:06:50Z Dataset Dataset 10.6084/m9.figshare.34022523.v1 https://figshare.com/articles/dataset/Syllogisms/34022523 CC BY 4.0