2026-10-07T05:50:44Z https://api.figshare.com/v2/oai

oai:figshare.com:article/34001649 2026-09-26T01:18:37Z category_29398 category_29401 category_29410 item_type_3 month_year_09_2026

Shanghainese Word Frequency (v1.0): a spoken-weighted frequency list with rank uncertainty Verbavia (25112554) Computational linguistics Corpus linguistics Language documentation and description Shanghainese Shanghai Wu Wu Chinese wuu word frequency frequency list corpus linguistics spoken corpus dialect lexicon bootstrap dispersion <p dir="ltr">A word frequency list for Shanghainese (Shanghai Wu, ISO 639-3 wuu): 2,524 ranked words, each with counts in conversational speech and in written Wu, a combined frequency per million (0.75 × spoken + 0.25 × written), document range, Gries' DP, and a bootstrap 95% rank interval.</p><p dir="ltr">Limits. The spoken corpus is small: 20 speakers, about 4 hours, 52,412 tokens. Ranks are reliable only to rank 236, and there only to within about ±50%; at no depth are they within ±25%. Below rank 236 the order is indicative only. Split-half Spearman between random halves of the speakers is 0.80 (top 100), 0.77 (top 300) and 0.71 (top 500). Spoken and written counts agree weakly (ρ = 0.36 over 833 shared words), which is why speech is weighted 0.75. Transcription conventions affect some counts: 仔 and 哉 never occur in the conversations while 了 occurs 972 times. No native speaker has reviewed the list.</p><p dir="ltr">Sources (counts only; no source text is redistributed). Spoken: MagicData, ASR-CShhiDiaCSC Chinese Shanghai Dialect Conversational Speech Corpus, via the Hugging Face copy TingChen-ppmc/Shanghai_Dialect_Conversational_Speech_Corpus, revision f10f366. Written: the Wu Chinese Wikipedia dump of 2026-09-01, filtered to colloquial Shanghai-type Wu: 2,405 of 48,393 articles, 459,388 tokens. Historical check, not part of the ranking: Pott (1907) and Edkins (1868) from Project Gutenberg, 34,534 tokens; 86% / 81% / 77% of the top 100 / 300 / 500 words appear there.</p><p dir="ltr">Method. OpenCC conversion and a hand-written spelling map (the transcripts write 吾 for 我, and 伐 for both the question particle and the negator 勿); two context rules for 伐 and 呃 showed about 2–5% error in hand-checked samples of 80. Segmentation uses a 3,113-entry hand-written lexicon with a unigram model re-estimated by hard EM; particles such as 仔, 个 and 伐 are words and are never joined to a neighbour. Each word has a Wu Association romanisation, part of speech, flags and an English gloss written for this dataset. The release rebuilds byte for byte from pinned sources (seed 20260926).</p><p dir="ltr">Files. The zip holds the list (output/shanghainese_word_frequency.tsv), per-source counts, JSON files with every quoted number, the lexicon, the code and the documentation.</p><p dir="ltr">Licence. CC0 1.0: the release contains only counts, statistics and original lexicon, glosses and code. The source corpora keep their own licences (MagicData: CC BY-NC-ND 4.0; Wikipedia: CC BY-SA 4.0) and are not redistributed.</p><p dir="ltr">Compiled by the <a href="https://verbavia.com/words/shanghainese/" target="_blank" rel="noreferrer">Verbavia project</a>. With thanks to MagicData, the contributors to the Wu Chinese Wikipedia, F. L. Hawks Pott and J. Edkins.</p> 2026-09-26T01:18:37Z Dataset Dataset 10.6084/m9.figshare.34001649.v1 https://figshare.com/articles/dataset/Shanghainese_Word_Frequency_v1_0_a_spoken-weighted_frequency_list_with_rank_uncertainty/34001649 CC0