2026-10-11T23:04:17Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33813915 2026-09-15T17:54:10Z category_7 category_12 category_734 category_106 portal_5 item_type_3 month_year_09_2026

<p>Median values by prompt type with effect sizes.</p> Jalil Ahmadpour (24949953) Medicine Cell Biology Biological Sciences not elsewhere classified Science Policy kincaid grade 16 bonferroni correction confirm 8 – 98 top structural metrics outputs exhibit graduate occupy semantic positions model pairs differ model embedding consistency next three ). large language models structural properties showing div >< p claude sonnet 4 claude opus 4 38 – 0 35 – 0 01 – 0 structural properties embedding dimensionality centroid positions 666 outputs seven models range models gemma ). 14 ). weight gemma validated methodology quality filtering paragraph count p </ medium effects median flesch making claims lexical profile level readability largest effects hoc tests dunn ’ domain relevance domain baseline calibration sampling calibration distributions 83rd percentile 05 ), <div><p>Large Language Models (LLMs) can generate text describing scientific concepts, but the characteristics of these outputs remain poorly understood. We present a multi-dimensional characterization framework analyzing 9,666 outputs from seven models (GPT-4.1, GPT-5.2, GPT-5.5, o4-mini, Claude Sonnet 4.5, Claude Opus 4.5, and the open-weight Gemma-3-27B) across five scientific domains. Rather than making claims about creativity or novelty, we measure five independent dimensions: coherence, domain relevance, lexical profile, structural properties, and semantic position using four sentence embedding models spanning 2020–2024. After quality filtering (99.2% coherence, 99.9% domain relevance pass rates), we find that outputs exhibit graduate-level readability (median Flesch-Kincaid grade 16.3) and occupy semantic positions at the 83rd percentile of calibration distributions. All 39 metrics differ significantly across models (Kruskal-Wallis, <i>p</i> < 0.05), with structural properties showing the largest effects ( = 0.35–0.54) and semantic position showing small-to-medium effects ( 0.01–0.14). Dunn’s post-hoc tests with Bonferroni correction confirm that all model pairs differ on the top structural metrics (21/21 pairs for paragraph count, 20/21 for the next three). Centroid positions are robust to calibration sampling (bootstrap cosine similarity 0.99) and exceed a shuffled-domain baseline in 96.8–98.7% of cases; cross-model embedding consistency is moderate (Spearman = 0.38–0.67) and is not explained by embedding dimensionality. Temperature effects replicate across two independent full-range models (GPT-4.1 and Gemma). This work provides calibrated measurements and validated methodology for future research without making interpretive claims about novelty.</p></div> 2026-09-15T17:53:56Z Dataset Dataset 10.1371/journal.pone.0357892.t005 https://figshare.com/articles/dataset/_p_Median_values_by_prompt_type_with_effect_sizes_p_/33813915 CC BY 4.0