Sarvam MMLU-Indic (Marathi)

Translated MMLU (Massive Multitask Language Understanding) benchmark in 10 Indian languages including Marathi. Contains multiple-choice questions spanning science, humanities, social sciences, and more. Standard benchmark for evaluating how well Marathi LLMs compare to English ones.

Benchmark Marathi language models on MMLU-Indic to measure knowledge and reasoning capabilities in Marathi.

Homepage HuggingFace

Quick Start

from datasets import load_dataset
# ds = load_dataset('sarvam/mmlu-indic', 'mr')
print('Sarvam MMLU-Indic Marathi')
print('Multiple-choice knowledge benchmark in Marathi')

Modality

text

Size

~296K questions across 10 languages; Marathi subset included

License

Open

Format

JSON / Parquet

Language

Update Frequency

static

Organization

Sarvam AI

Schema

Field	Type	Description
question	string	Multiple-choice question in Marathi
choices	list[string]	Answer choices
answer	string	Correct answer
subject	string	Subject/topic area

Build With This

Create a Marathi model knowledge gap analyzer identifying which subjects have weakest performance on MMLU

Develop a curriculum-aligned Marathi QA benchmark extending MMLU with Maharashtra-specific educational content

Build a Marathi model comparison leaderboard tracking MMLU-Indic scores across different model families

AI Use Cases

Marathi LLM evaluation and benchmarkingKnowledge assessment across domainsCross-lingual model comparisonIdentifying LLM knowledge gaps in Marathi

Related Datasets

BiasShades Marathi (LLM Bias Evaluation)

text

FLORES-200 Benchmark

Text (parallel, Marathi)

Last verified: 2026-03-09