hf_mirrors/CulturalRiddles
Datasets
Dataset viewer
Files & versions

Cultural Riddles

Paper: arXiv:2610.29968 · Explore the results: dashboard · Play: try the riddles yourself · Code: github.com/Cohere-Labs-Community/CulturalRiddles

A community-built benchmark of original cultural riddles from 62 communities across 51 languages and 45 countries, written by 167 native speakers. Each riddle describes something without naming it, so answering it takes cultural knowledge, not translation or web search. A native speaker recognises the answer at once; a model succeeds only if it carries that culture's knowledge.

The release has four tables:

TableRowsWhat it holds
riddles (default)6,191The riddles and their reference answers.
model_answers170,520Every answer from 28 model configurations on the 6,090 benchmarked riddles, with its final score and who scored it.
human_annotations74,383Every native-speaker label on a unique answer, including independent second labels.
models28The model configurations, their sizes, and their recorded reasoning settings.

Why riddles

A riddle names nothing. "This red-crossed sphere's juice is not sweet for your stomach" means nothing to a search engine, but every Hungarian recognises Unicum, a famous bitters. A model that only knows the formal name misses it. Because each riddle is original, unpublished, and culturally specific, it resists memorisation and lookup.

Languages and communities

The unit is the present-day national-linguistic community, for example Spanish in Argentina or Bengali in India. Languages and communities differ because several communities share a language: Argentine, Cuban, and Peruvian riddles are all in Spanish but draw on three cultures. 1,234 riddles come from topics each community chose itself. See the list of communities.

Loading the data

from datasets import load_dataset

riddles = load_dataset("CohereLabsCommunity/CulturalRiddles", split="test")
answers = load_dataset("CohereLabsCommunity/CulturalRiddles", "model_answers", split="test")
labels = load_dataset("CohereLabsCommunity/CulturalRiddles", "human_annotations", split="test")
models = load_dataset("CohereLabsCommunity/CulturalRiddles", "models", split="test")

tamil = riddles.filter(lambda r: r["community"] == "tamil_india")

Tables

riddles

FieldTypeDescription
riddle_idstringStable unique ID, {community}_{topic}_{number}.
communitystringCommunity key, usually {language}_{country} (for example tamil_india).
languagestringISO 639-3 code of the riddle's language.
topicstringTopic the riddle belongs to.
is_wildcardboolTrue for a topic the community chose itself.
riddle_originalstringThe riddle in its original language.
riddle_englishstringEnglish translation, for human reference only. May be null.
riddle_hintstringOptional hint. May be null.
riddle_notestringOptional note from the writer; riddles flagged in the content review also carry a content note here. May be null.
reference_answer_originalstringReference answer in the original language.
reference_answer_englishstringEnglish translation of the answer. May be null.

model_answers

Each configuration answered each benchmarked riddle once (6,090 riddles from 61 communities). Scores come from rules for clear cases (32,171 answers), from native speakers (100,769), and from two LLM judges that agree (34,546); 3,034 answers have no score. Accuracy is the share of scored answers with score 3.

FieldTypeDescription
riddle_idstringJoins to riddles.
communitystringCommunity key.
topicstringTopic of the riddle.
modelstringModel configuration key; joins to models.
model_displaystringReadable configuration name.
model_answerstringThe answer the model gave.
answer_normstringmodel_answer stripped and lowercased; joins to human_annotations.
scoreint0 incorrect, 1 admits not knowing, 2 partially correct, 3 correct. Null when no score was reached.
scorerstringrule, human (native speaker), llm_judges (two agreeing judges), or none.
routingstringHow the answer was routed: rule-based categories, human evaluation, or pending and invalid human labels.
error_typestringAnswer type code, for example 0.3 hallucinated or 3.1 different valid answer.
error_type_labelstringReadable answer type.
annotator_idstringAnonymous native-speaker ID, such as annotator_k2p9. Stable across tables.
annotator_notestringThe annotator's note, as written. May be null.
judge_score_deepseekintScore from the DeepSeek judge, where one ran. May be null.
judge_score_geminiintScore from the Gemini judge, where one ran. May be null.

human_annotations

Native speakers labeled each unique (riddle, answer) pair once, so one label can cover several models. 70,006 rows are first labels and 4,377 are independent second labels used to measure agreement. 109 annotators contributed; their IDs are anonymous.

FieldTypeDescription
riddle_idstringJoins to riddles.
communitystringCommunity key.
topicstringTopic of the riddle.
answerstringThe answer the annotator labeled.
answer_normstringanswer stripped and lowercased; one label covers every model that gave it.
annotationstringfirst label, or second label from an independent annotator.
annotator_idstringAnonymous native-speaker ID; several are separated by ;.
scoreint0 to 3, as in model_answers.
error_typestringAnswer type code.
error_type_labelstringReadable answer type.
annotator_notestringThe annotator's note, as written. May be null.
benchmark_statusstringvalid or invalid for first labels; invalid labels are not used in scores.
iaa_rowfloatSet for rows in the inter-annotator agreement sample.
self_reviewboolTrue when an annotator labeled the same pair again.

models

FieldTypeDescription
modelstringConfiguration key used in model_answers.
model_displaystringReadable configuration name.
default_reasoningstringThe provider's documented default reasoning setting.
actual_reasoning_effortstringThe setting recorded for the run. unspecified means the request did not set it, not that reasoning was off.
total_parameter_bfloatPublished total parameters, in billions. Null when not published.
active_parameter_count_bfloatPublished active parameters for mixture-of-experts models, in billions.
accessstringproprietary, open-weight, or open-source.
info_urlstringModel card or documentation.
cutoff_datestringPublished training-data cutoff, where known.

Curation

Every riddle is original human writing by native speakers writing for their own community. AI tools may be used for brainstorming or to test difficulty, but the riddles are written by people. English translations are for human reference only, never shown to a model. The data contains no names or contact details of writers or annotators.

Intended use and terms

The data is released under the terms in LICENSE: Creative Commons Attribution 4.0 attribution terms combined with an evaluation-only restriction.

This data must not be used to train, fine-tune, or distill models, nor as seeds for synthetic data generation. It is an evaluation benchmark; training on it contaminates future measurement. This applies to the model answers and labels as much as to the riddles.

Content note

Although we have filtered the dataset for harmful content, some riddles may still contain sensitive themes or potentially problematic ideas. These are presented solely as cultural and linguistic puzzles, not as endorsements of the views expressed or guidance for real-world behavior.

Riddles flagged in a content review carry a short content note in their riddle_note field. Model answers are published as generated and were not filtered: many are hallucinated, and some may be offensive.

Contributors

This benchmark was proposed by Károly Boczka and Julia Kreutzer, and built by native-speaker contributors across the Cohere Labs Community.

Riddle writers by community

Writers who agreed to be credited, by the name they chose. Some writers asked not to be named or have not confirmed yet; they are part of the Cohere Labs Community credit.

  • Afaan Oromo (Ethiopia): Bontu Fufa Balcha · Debela Desalegn Yadeta
  • Assamese (India): Angana Borah · Dipti Borah Hazarika · Mina Saikia · Prasanta Borah
  • Belarusian (Belarus): Oksana Volchek · Vladislav Poritski
  • Bengali (Bangladesh): Ahanaf Aziz · Md Istiak Hossain Shihab · Nazia Tasnim
  • Bengali (India): Ankita Maity · Ruchira Dhar · Souvik Bhattacharyya
  • Buginese (Indonesia): Ahmad Mufli Ramadhan
  • Bulgarian (Bulgaria): Ralitza Nikolova Rizova
  • Cantonese (China): Fengran Mo · Jinghan Zhang
  • Catalan (Spain): Aleix Sant · Irene Baucells · Javier García Gilabert · Marc Pàmies · Ona de Gibert
  • Chinese (Malaysia): Nancy F. Chen · Zhengyuan Liu
  • Chinese (Singapore): Rui Yang Tan · Weihua Zheng
  • Dutch (Netherlands): David Stap · Mike Zhang · Rochelle Choenni
  • Egyptian Arabic (Egypt): Farah Abdou · Nadine Elframawy · Rawan Mohamed Elframawy
  • English (Canada): Jane Smith · Julia Kreutzer · Madeline Smith · Matt Bobkin
  • English (United States): Ananya Sahu · Henry Gagnier · Jonathan E. Swindell · Sajag Swami · Swati Rajwal
  • French (Canada): Lauren Altomare · Micol Altomare
  • French (France): Bryan Chen · Wafa Aissa
  • German (Germany): Diana Kozachek · Julia Kreutzer
  • Gujarati (India): Amit Agarwal · Fenal Ashokbhai ilasariya · Hitesh Laxmichand Patel
  • Hungarian (Hungary): Károly Boczka
  • Indonesian (Indonesia): Ahmad Mufli Ramadhan · Muhammad Ravi Shulthan Habibi
  • Italian (Italy): Lisa Alazraki · Marco De Santis
  • Japanese (Japan): Reuben Chagas Fernandes
  • Kannada (India): Emile Anand · Vanya Bannihatti Kumar
  • Konkani (India): Prerana Rane · Vivek Gopalkrishna Silimkhan
  • Levantine Arabic (Lebanon): Adnan El Assadi
  • Levantine Arabic (Syria): Yusser Al Ghussin
  • Luganda (Uganda): Alex Gimei
  • Lumasaba (Uganda): Alex Gimei
  • Luxembourgish (Luxembourg): Cedric Lothritz · Fred Philippy
  • Malay (Malaysia): Roy Ka-Wei Lee
  • Malayalam (India): Arya Hariharan · Arya Suneesh
  • Marathi (India): Rushikesh Zawar · Yash Bagla
  • Mongondow (Indonesia): Ahmad Mufli Ramadhan
  • Moroccan Arabic (Morocco): Abderrahmane Issam · Zakaria Baannou
  • Nepali (Nepal): Dikshya Poudel · Divas Poudel · Jebish Purbey
  • Persian (Iran): Aylin Naebzadeh · Hamidreza Saffari · Masoumeh Javadi Zarnaghi · Melika Davood Zadeh · Mohammadamin Shafiei · Pouya Sadeghi · Sara Rajaee · Sarah Bahrami
  • Polish (Poland): Joanna Radoła · dokato
  • Portuguese (Brazil): Daniel Souza · Felipe Bochini · Gabriel Assis · João Paulo Faria da Cruz · Juliana Carvalho · Lucas Resck · Pedro Delgado
  • Portuguese (Portugal): Diogo M. Serrano-Miranda
  • Russian (Russia): Anastasiia Nikitina · Andrei Nikitin · Dina Pisarevskaya · Galina Kozulina · Sergey Sedov
  • Serbian (Serbia): Danka Jokić · Milica Andrić · Nebojša Damnjanović · Toma Knežević
  • Simplified Chinese (China): Bin Hu · Di Mo · Haichang Li · Kaiser Sun · Lingbo Wang
  • Sindhi (India): Naman Bhatia
  • Sinhala (Sri Lanka): Sayuru Bopitiya
  • Spanish (Argentina): Cossio, Leandro · José Matías Avalos · Silvia Fernandez
  • Spanish (Cuba): Eduardo Sánchez
  • Spanish (Peru): David Quispe-Castillo · Fabrizzio Marin Colquichagua · Luis Fernado Solis Navarro
  • Spanish (Spain): Alvaro Escapa · Antonio Miranda-Escalada · Borja Sánchez Clemente · Isabel Blanco Ferreiro · Ona de Gibert
  • Sudanese Arabic (Sudan): Abdelhadi Osama · Wafaa Mohammed
  • Swahili (Kenya): Nanyanga Emma Loice · Priscilla Amuok
  • Tamil (India): Vinod
  • Telugu (India): Akhil Eppa · Jayanth Krishna Chundru · Lakshmi Tejaswi Sunkara · PATTAN MAHAMMAD SALMAN KHAN · Prahitha Movva · Venkata Manaswitha Sunkara
  • Thai (Thailand): Patteera Triamamornwooth · Warit Lekmuenwai
  • Traditional Chinese (China): Stephy Chen
  • Tunisian Arabic (Tunisia): Wafa Aissa
  • Turkish (Turkey): Ceren Berk
  • Ukrainian (Ukraine): Hanna Yukhymenko · Yurii Paniv · sunrise
  • Urdu (Pakistan): Muhammad Saad Umar
  • Uzbek (Uzbekistan): Abdumutalib Begmuratov
  • Vietnamese (Vietnam): Christian Hoang · My Chiffon Nguyen · Van Ngo
  • Yoruba (Nigeria): Esther Adenuga · Joy Olusanya · Oluwaseun A. Ajayi (some riddles adapted from Orature and Yoruba Riddles by Akintunde Akinyemi, Palgrave Macmillan) · YUSUF, Ismail Abayomi

Updates

This card is generated from the source repository on each release.

  • This release adds the model_answers, human_annotations, and models tables. The riddles moved to data/riddles.parquet and no longer carry the release_version column.
  • After a content review, some riddles were removed or corrected. Every change is listed in data/corrections.csv in the source repository.

Citation

If you use this dataset, please cite the paper:

@misc{nguyen2026culturalriddles,
  title         = {CulturalRiddles: A Multicultural Benchmark of Riddles},
  author        = {Nguyen, My Chiffon and Hu, Tianyi and Anbalagan, Vinod and Gagnier, Henry and Sadeghi, Pouya and others},
  year          = {2026},
  eprint        = {2610.29968},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.29968}
}

To cite the dataset release itself:

@misc{cultural_riddles_2026,
  title        = {Cultural Riddles},
  author       = {Boczka, K{\'a}roly and Kreutzer, Julia and the Cohere Labs Community},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/CohereLabsCommunity/CulturalRiddles}}
}