A Text Embedding Benchmark for Brazilian Portuguese
Native, not translated. 93 embedding models ranked on 22 Brazilian-Portuguese tasks, built only from text written in Portuguese (no machine-translated benchmarks), with confidence intervals, significance tests, and an analysis of which tasks actually separate models.
We gratefully acknowledge Verda (DataCrunch Oy) for the GPU compute credits that supported this work.
93 models on native Brazilian-Portuguese tasks, ranked by the 22-task mean. The top 15 are shown below; the full interactive table, the IRT ranking, and per-category views live on Hugging Face.
Open-weight models: quality versus size on a log scale. The dashed line is the Pareto frontier, the best 22-task mean reachable at each parameter budget. Hover any point for details.
| # | Model | Params | License | mean22 |
|---|---|---|---|---|
| 1 | gemini-embedding-001 CLOSED | — | Proprietary | 0.682 |
| 2 | Qwen3-Embedding-8B OPEN | 7.6B | Apache-2.0 | 0.670 |
| 3 | KaLM-Embedding-Gemma3-12B-2511 OPEN | 11.8B | Tencent-KaLM | 0.670 |
| 4 | voyage-context-4 CLOSED | — | Proprietary | 0.668 |
| 5 | Octen-Embedding-8B OPEN | 7.6B | Apache-2.0 | 0.667 |
| 6 | Qwen3-Embedding-4B OPEN | 4.0B | Apache-2.0 | 0.662 |
| 7 | voyage-context-3 CLOSED | — | Proprietary | 0.657 |
| 8 | voyage-3-large CLOSED | — | Proprietary | 0.655 |
| 9 | voyage-4-large CLOSED | — | Proprietary | 0.653 |
| 10 | SFR-Embedding-Mistral OPEN | 7.1B | CC-BY-NC-4.0 | 0.652 |
| 11 | BidirLM-1.7B-Embedding OPEN | 1.7B | Apache-2.0 | 0.651 |
| 12 | BOOM_4B_v1 OPEN | 4.0B | Apache-2.0 | 0.650 |
| 13 | embeddinggemma-300m OPEN | 308M | Gemma | 0.649 |
| 14 | codestral-embed CLOSED | — | Proprietary | 0.649 |
| 15 | Linq-Embed-Mistral OPEN | 7.1B | CC-BY-NC-4.0 | 0.647 |
The leaderboard now spans 169 models, 131 tasks, a retrieval benchmark with private data, and image, audio and video, but still no native Portuguese. Where MTEB-BR fits.
Read →The MTEB team extends the playbook to video and audio with a 23-task benchmark. What it found, and why the method matters for text embeddings too.
Read →The leaderboard’s top model is a closed API, yet the cost–quality frontier is shallow and a free open-weight model ties the leader.
Read →Translated benchmarks quietly flatten the differences between models. Here is what changes when you evaluate on Portuguese that was written in Portuguese.
Read →A multilingual model spends most of its parameters on tokens you never use. We cut EmbeddingGemma-300M to 157M for Portuguese, with zero training.
Read →MTEB-BR is described in a preprint on arXiv (cs.CL), covering the benchmark design, the statistical layer, IRT task discrimination, and a cross-leaderboard validity analysis.
@article{stekel2026mtebbr,
title = {MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese},
author = {Stekel, Tardelli Ronan Coelho},
journal = {arXiv preprint arXiv:2607.04581},
year = {2026}
}
Want your embedding model on the leaderboard? We accept submissions through either channel; pick whichever fits. Every score is reproducible from public scripts, so each new row can be audited.
Share the model ID and any prompt or pooling details. Best for a quick request.
Open a discussion ↗Prefer the code side? File an issue or a pull request on the benchmark repository.
Open an issue ↗