Paper: arXiv:2610.29968 · Explore the results: dashboard · Play: try the riddles yourself · Code: github.com/Cohere-Labs-Community/CulturalRiddles
A community-built benchmark of original cultural riddles from 62 communities across 51 languages and 45 countries, written by 167 native speakers. Each riddle describes something without naming it, so answering it takes cultural knowledge, not translation or web search. A native speaker recognises the answer at once; a model succeeds only if it carries that culture's knowledge.
The release has four tables:
| Table | Rows | What it holds |
|---|---|---|
riddles (default) | 6,191 | The riddles and their reference answers. |
model_answers | 170,520 | Every answer from 28 model configurations on the 6,090 benchmarked riddles, with its final score and who scored it. |
human_annotations | 74,383 | Every native-speaker label on a unique answer, including independent second labels. |
models | 28 | The model configurations, their sizes, and their recorded reasoning settings. |
A riddle names nothing. "This red-crossed sphere's juice is not sweet for your stomach" means nothing to a search engine, but every Hungarian recognises Unicum, a famous bitters. A model that only knows the formal name misses it. Because each riddle is original, unpublished, and culturally specific, it resists memorisation and lookup.
The unit is the present-day national-linguistic community, for example Spanish in Argentina or Bengali in India. Languages and communities differ because several communities share a language: Argentine, Cuban, and Peruvian riddles are all in Spanish but draw on three cultures. 1,234 riddles come from topics each community chose itself. See the list of communities.
from datasets import load_dataset
riddles = load_dataset("CohereLabsCommunity/CulturalRiddles", split="test")
answers = load_dataset("CohereLabsCommunity/CulturalRiddles", "model_answers", split="test")
labels = load_dataset("CohereLabsCommunity/CulturalRiddles", "human_annotations", split="test")
models = load_dataset("CohereLabsCommunity/CulturalRiddles", "models", split="test")
tamil = riddles.filter(lambda r: r["community"] == "tamil_india")riddles| Field | Type | Description |
|---|---|---|
riddle_id | string | Stable unique ID, {community}_{topic}_{number}. |
community | string | Community key, usually {language}_{country} (for example tamil_india). |
language | string | ISO 639-3 code of the riddle's language. |
topic | string | Topic the riddle belongs to. |
is_wildcard | bool | True for a topic the community chose itself. |
riddle_original | string | The riddle in its original language. |
riddle_english | string | English translation, for human reference only. May be null. |
riddle_hint | string | Optional hint. May be null. |
riddle_note | string | Optional note from the writer; riddles flagged in the content review also carry a content note here. May be null. |
reference_answer_original | string | Reference answer in the original language. |
reference_answer_english | string | English translation of the answer. May be null. |
model_answersEach configuration answered each benchmarked riddle once (6,090 riddles from 61 communities). Scores come from rules for clear cases (32,171 answers), from native speakers (100,769), and from two LLM judges that agree (34,546); 3,034 answers have no score. Accuracy is the share of scored answers with score 3.
| Field | Type | Description |
|---|---|---|
riddle_id | string | Joins to riddles. |
community | string | Community key. |
topic | string | Topic of the riddle. |
model | string | Model configuration key; joins to models. |
model_display | string | Readable configuration name. |
model_answer | string | The answer the model gave. |
answer_norm | string | model_answer stripped and lowercased; joins to human_annotations. |
score | int | 0 incorrect, 1 admits not knowing, 2 partially correct, 3 correct. Null when no score was reached. |
scorer | string | rule, human (native speaker), llm_judges (two agreeing judges), or none. |
routing | string | How the answer was routed: rule-based categories, human evaluation, or pending and invalid human labels. |
error_type | string | Answer type code, for example 0.3 hallucinated or 3.1 different valid answer. |
error_type_label | string | Readable answer type. |
annotator_id | string | Anonymous native-speaker ID, such as annotator_k2p9. Stable across tables. |
annotator_note | string | The annotator's note, as written. May be null. |
judge_score_deepseek | int | Score from the DeepSeek judge, where one ran. May be null. |
judge_score_gemini | int | Score from the Gemini judge, where one ran. May be null. |
human_annotationsNative speakers labeled each unique (riddle, answer) pair once, so one label can cover several models. 70,006 rows are first labels and 4,377 are independent second labels used to measure agreement. 109 annotators contributed; their IDs are anonymous.
| Field | Type | Description |
|---|---|---|
riddle_id | string | Joins to riddles. |
community | string | Community key. |
topic | string | Topic of the riddle. |
answer | string | The answer the annotator labeled. |
answer_norm | string | answer stripped and lowercased; one label covers every model that gave it. |
annotation | string | first label, or second label from an independent annotator. |
annotator_id | string | Anonymous native-speaker ID; several are separated by ;. |
score | int | 0 to 3, as in model_answers. |
error_type | string | Answer type code. |
error_type_label | string | Readable answer type. |
annotator_note | string | The annotator's note, as written. May be null. |
benchmark_status | string | valid or invalid for first labels; invalid labels are not used in scores. |
iaa_row | float | Set for rows in the inter-annotator agreement sample. |
self_review | bool | True when an annotator labeled the same pair again. |
models| Field | Type | Description |
|---|---|---|
model | string | Configuration key used in model_answers. |
model_display | string | Readable configuration name. |
default_reasoning | string | The provider's documented default reasoning setting. |
actual_reasoning_effort | string | The setting recorded for the run. unspecified means the request did not set it, not that reasoning was off. |
total_parameter_b | float | Published total parameters, in billions. Null when not published. |
active_parameter_count_b | float | Published active parameters for mixture-of-experts models, in billions. |
access | string | proprietary, open-weight, or open-source. |
info_url | string | Model card or documentation. |
cutoff_date | string | Published training-data cutoff, where known. |
Every riddle is original human writing by native speakers writing for their own community. AI tools may be used for brainstorming or to test difficulty, but the riddles are written by people. English translations are for human reference only, never shown to a model. The data contains no names or contact details of writers or annotators.
The data is released under the terms in LICENSE: Creative Commons Attribution 4.0 attribution terms combined with an evaluation-only restriction.
This data must not be used to train, fine-tune, or distill models, nor as seeds for synthetic data generation. It is an evaluation benchmark; training on it contaminates future measurement. This applies to the model answers and labels as much as to the riddles.
Although we have filtered the dataset for harmful content, some riddles may still contain sensitive themes or potentially problematic ideas. These are presented solely as cultural and linguistic puzzles, not as endorsements of the views expressed or guidance for real-world behavior.
Riddles flagged in a content review carry a short content note in their riddle_note field. Model answers are published as generated and were not filtered: many are hallucinated, and some may be offensive.
This benchmark was proposed by Károly Boczka and Julia Kreutzer, and built by native-speaker contributors across the Cohere Labs Community.
Writers who agreed to be credited, by the name they chose. Some writers asked not to be named or have not confirmed yet; they are part of the Cohere Labs Community credit.
This card is generated from the source repository on each release.
model_answers, human_annotations, and models tables. The riddles moved to data/riddles.parquet and no longer carry the release_version column.data/corrections.csv in the source repository.If you use this dataset, please cite the paper:
@misc{nguyen2026culturalriddles,
title = {CulturalRiddles: A Multicultural Benchmark of Riddles},
author = {Nguyen, My Chiffon and Hu, Tianyi and Anbalagan, Vinod and Gagnier, Henry and Sadeghi, Pouya and others},
year = {2026},
eprint = {2610.29968},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.29968}
}To cite the dataset release itself:
@misc{cultural_riddles_2026,
title = {Cultural Riddles},
author = {Boczka, K{\'a}roly and Kreutzer, Julia and the Cohere Labs Community},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/CohereLabsCommunity/CulturalRiddles}}
}