2026-10-14T10:12:28Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33201684 2026-09-18T21:17:59Z category_29182 category_29179 item_type_3 month_year_09_2026

Experimental Materials and Dataset - "Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation" Talita Ribeiro (15442517) Allan Andrade (24546379) Empirical software engineering Automated software engineering Software documentation API knowledge README generation large language models LLM-as-a-judge This is the complete replication package for the ISE 2026 paper "Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation". It contains every artifact behind every number reported in the paper: the prompts, the generated README files, the raw human and automated evaluations, and the analysis scripts that turn them into the reported statistics.What the study didREADME-Gen is a structured prompting approach in which the README template is fixed a priori. The template is derived from two empirical studies of README content and from the Theory of Robust API Knowledge (ATRAK), with each section bound to one Knowledge Element. <ul><li>Model: gpt-4.1-mini-2025-04-14.</li><li>Subjects: twelve GitHub repositories across JavaScript, Python and Bash/Shell, deliberately mixing popular projects (>30k stars) and obscure ones (<300).</li><li>Input: no repository artifacts are injected into the prompt. The model writes the README from prior knowledge and whatever the provider retrieves at inference time.</li><li>Evaluation dimensions: structural completeness, ATRAK adherence, and correctness.</li><li>Evaluation protocols: two independent protocols, a human evaluator and an LLM judge (Claude Fable 5). The judge operates as a tool-using agent that installs each project in a throwaway environment and executes every documented snippet.</li><li>Baseline: README-AI, run under the same model and the same rubric.</li></ul> Contents 209 files, 8.1 MB, organised in pipeline order: <ul><li>1_inputs/: the twelve repositories with language, stars, popularity group and Core Functionality ground truth, plus the run manifest.</li><li>2_prompts/: the exact generation prompt, and the generic, tool-agnostic evaluation and comparison prompts (reusable beyond this study).</li><li>3_generated_readmes/: 48 generated files. 36 are from README-Gen (three independent runs per repository) and 12 are from README-AI.</li><li>4_evaluations/: the raw evaluation data. It includes the human evaluator's worksheet (CSV and XLSX), 74 score CSVs, and 48 LLM-judge reasoning files. The reasoning files record, per rule, the commands executed, their output and the verdict.</li><li>5_comparison/: a secondary artifact demonstrating that the comparison workflow generalises. It produces no number used in the paper.</li><li>6_analysis/: reproduce.py, which recomputes every statistic, table and figure in the paper directly from 4_evaluations/, and its outputs.</li></ul> Reproducing <ul><li>Install 6_analysis/requirements.txt (pandas, scipy, matplotlib).</li><li>Run python 6_analysis/reproduce.py.</li></ul> The script reads only 4_evaluations/. No summary spreadsheet is consulted, so a reader can verify the aggregation rather than trust it. Deleting 6_analysis/outputs/ and re-running regenerates every table, figure and statistic identically. Both the human and the automated evaluations are rebuilt from the judges' original outputs. Documentation <ol><li>README.md: maps the folder layout and the file conventions.</li><li>METHODOLOGY.md: explains how each artifact was produced and states plainly what was not controlled:</li></ol><ul><li>the judge was run at its then-current version rather than a pinned snapshot;</li><li>ground truth is the released artifact and default branch, not a pinned commit;</li><li>no execution timeout was fixed.</li></ul><ol><li>CLAIMS.md: maps every quantitative claim in the paper to the artifact that produces it.</li></ol> Not included <ul><li>README-Gen's own source code (the CLI/HTTP application) is not part of this package. 2_prompts/generation_prompt.md is the complete record of what determines the tool's output. Regenerating a README requires only that prompt, an OpenAI key and the pinned model.</li><li>The earlier, superseded round of automated scoring is also not included. 4_evaluations/ is the single source of truth.</li></ul> 2026-09-18T21:17:59Z Dataset Dataset 10.6084/m9.figshare.33201684.v2 https://figshare.com/articles/dataset/Experimental_Materials_and_Dataset_-_Literature-Grounded_Generation_of_README_Files_with_Large_Language_Models_A_Dual_Human_LLM_Evaluation/33201684 CC BY 4.0