2026-10-14T06:50:01Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33904614 2026-09-17T17:36:11Z category_15 category_45 category_734 category_106 category_122 portal_5 item_type_3 month_year_09_2026

<p>Inter-rater reliability for expert evaluations.</p> Yu-Ju Lin (389524) Neuroscience Sociology Biological Sciences not elsewhere classified Science Policy Mental Health study examined whether proposed solution strategies matched concept sheets holm &# 8211 rated ideation skills preliminary design refinement five design attributes stage design ideation specific analytic criteria perceived cognitive processing cognitive support tool bonferroni correction within rated ideation outputs subjective task experience human &# 8211 assisted workflow differed ideation outputs perceived support assisted workflow design brief specific post counterbalanced within whereas human uniform advantage task questionnaire subjects experiment related ratings problem framing outcome family increasingly used evaluator perspective creator perspective creative practice controlled evidence conditions using combined text assisted condition affective response <div><p>Generative artificial intelligence is increasingly used in creative practice, but controlled evidence on its role in early-stage design ideation remains limited. This study examined whether a combined text-and-image AI-assisted workflow differed from human-only brainstorming in expert-rated ideation outputs, preliminary design refinement, and subjective task experience. A counterbalanced within-subjects experiment was conducted with 32 undergraduate design students, each of whom completed both conditions using the same design brief. Ideation outputs were evaluated by three expert raters across six task-specific analytic criteria, participant-matched concept sheets were assessed across five design attributes, and participants completed a task-specific post-task questionnaire after each condition. After Holm–Bonferroni correction within each outcome family, AI-assisted outputs received higher ratings for problem framing, mechanism, creator perspective, and evaluator perspective, whereas human-only outputs received higher ratings for innovation and proposed solution strategies. No statistically robust condition differences were found for the five design attributes. Participants reported higher ratings for perceived cognitive processing, self-rated ideation skills, and affective response in the AI-assisted condition. These findings indicate different profiles of evaluated ideation output rather than a uniform advantage for either condition. Perceived support in the AI-assisted workflow did not translate into a uniform advantage across originality-related ratings or into superior expert-rated preliminary design refinement.</p></div> 2026-09-17T17:35:49Z Dataset Dataset 10.1371/journal.pone.0358542.t008 https://figshare.com/articles/dataset/_p_Inter-rater_reliability_for_expert_evaluations_p_/33904614 CC BY 4.0