Trimming embedding vocabularies: half the parameters, ~99% of the score

A multilingual embedding model keeps most of its parameters in one place: the token-embedding matrix. EmbeddingGemma-300M devotes roughly $262{,}144 \times 768 \approx 201\text{M}$ of its ${\sim}308\text{M}$ weights to a vocabulary that spans a hundred languages. If you only ever serve one language, the other languages’ embeddings are dead weight — memory and bandwidth you pay for on every load and never use.

embedding-vocab-trimmer removes that dead weight. It shrinks a model’s vocabulary to a single language while leaving the encoder bit-for-bit unchanged, with no training and no GPU. On Portuguese, trimming EmbeddingGemma-300M to a 64k vocabulary keeps 99.4% of the full model’s MTEB-BR score at about half the parameters.

The result, first

Evaluated on MTEB-BR — the 22 native PT-BR tasks — with the transformer encoder and Dense heads identical at every vocab size; the only variable is the embedding matrix.

Vocabulary	Parameters	MTEB-BR mean_22	% of full
16k	~119M	0.5950	91.7%
24k	~125M	0.6263	96.5%
32k	~131M	0.6201	95.5%
48k	~144M	0.6418	98.9%
64k	~157M	0.6453	99.4%
128k	~207M	0.6491	≈100%
Full EG-300M	~308M	0.6490	100%

Quality recovers monotonically above 32k. At 64k the trim reaches 99.4% of the full model’s score at 51% of the parameters — the sweet spot. The 128k model ties the full model within measurement noise (0.6491 vs. 0.6490). Below ~24k the model loses the fine-grained distinctions that retrieval and reranking depend on, so the score drops faster than the parameter count. The score being compared against — $0.6490$ for the full EmbeddingGemma-300M — is the same 22-task mean that model earns on the MTEB-BR leaderboard, which is what makes a compression claim like “99.4% of quality” checkable rather than rhetorical.

Method

Let $\mathcal{V}$ be the full vocabulary of the multilingual model, $|\mathcal{V}| = V$, and $d$ the embedding dimension. The embedding matrix is $E \in \mathbb{R}^{V \times d}$, where row $E_v$ is the embedding of token $v$.

1. Corpus-based frequency estimation

Given a target-language corpus $\mathcal{C}$, tokenize it with the original tokenizer $\tau$ and count how often each token appears:

$$f(v) = \sum_{x \in \mathcal{C}} \sum_{t \in \tau(x)} \mathbf{1}[t = v], \qquad v \in \mathcal{V}$$

2. Vocabulary selection

Let $\mathcal{S} \subset \mathcal{V}$ be the mandatory special tokens (pad, bos, eos, unk, and high-frequency byte-fallback tokens). The trimmed vocabulary of size $K$ keeps the most frequent tokens plus the special set:

$$\mathcal{V}_K = \underset{v \,\in\, \mathcal{V} \setminus \mathcal{S}}{\text{Top-}K}\{f(v)\} \;\cup\; \mathcal{S}$$

A contiguous re-indexing bijection $\sigma: \mathcal{V}_K \to \{0, \ldots, |\mathcal{V}_K|-1\}$ preserves the original relative order of token ids.

3. BPE merge consistency

A BPE vocabulary is an ordered list of merge rules $\mathcal{M} = \{(a_i, b_i) \to c_i\}$. A merge survives only when all three tokens do:

$$\mathcal{M}_K = \{(a, b) \to c \;\in\; \mathcal{M} \;\mid\; a \in \mathcal{V}_K \;\land\; b \in \mathcal{V}_K \;\land\; c \in \mathcal{V}_K\}$$

This is the step most implementations overlook: keeping a merge whose product $c \notin \mathcal{V}_K$ makes the tokenizer emit a token id that no longer exists in the embedding matrix — silently producing garbage embeddings, or an index error at inference time.

4. Embedding submatrix extraction

The trimmed embedding matrix selects the surviving rows in the new index order:

$$E_K = E\bigl[\sigma^{-1}(0),\; \sigma^{-1}(1),\; \ldots,\; \sigma^{-1}(|\mathcal{V}_K|-1)\bigr]$$

The encoder, pooling, and Dense heads are copied unchanged, giving $\theta_K = (E_K,\, \theta_\text{enc},\, \theta_\text{pool},\, \theta_\text{dense})$. For every surviving token, $E_K[\sigma(v)]$ is bit-for-bit identical to $E[v]$. No weight is modified, fine-tuned, or distilled.

5. Parameter reduction

Only the embedding term changes:

$$P = V \cdot d + P_\text{enc}, \qquad P_K = |\mathcal{V}_K| \cdot d + P_\text{enc}, \qquad \Delta P = (V - |\mathcal{V}_K|)\, d$$

For EmbeddingGemma-300M ($V = 262{,}144$, $d = 768$, $P_\text{enc} \approx 107\text{M}$) trimmed to $K = 64{,}000$:

$$\Delta P = (262{,}144 - 64{,}000) \times 768 \approx 152\text{M parameters} \quad (-49\%)$$

Why quality holds

Because the encoder is identical across all trim sizes, quality loss comes solely from tokenization changes for out-of-vocabulary tokens. As $K$ grows, coverage of the language’s actual token distribution approaches unity and the score converges to the untrimmed baseline:

$$\lim_{K \to V} \text{MTEB}(f_{\theta_K}) = \text{MTEB}(f_\theta)$$

Empirically the convergence is fast: at $K = 64{,}000$ on Portuguese, $\text{MTEB}(f_{\theta_K}) / \text{MTEB}(f_\theta) = 99.4\%$.

Architecture

multilingual model                       language-trimmed model
+-------------------------+              +-------------------------+
| embed_tokens 262144x768 |  -- trim --> | embed_tokens  64000x768 |  ← only this shrinks
+-------------------------+              +-------------------------+
| transformer encoder     |  (unchanged) | transformer encoder     |
| pooling + Dense heads   |  (unchanged) | pooling + Dense heads   |
+-------------------------+              +-------------------------+
        ~308M params                             ~157M params

Which models are worth trimming

The key metric is the embedding fraction $\rho = (V \times d) \,/\, P_\text{total}$ — the share of parameters that live in the embedding matrix and can therefore be removed. Models with small encoders and large multilingual vocabularies (over 200k tokens) are the best candidates.

Model	$V$	$d$	Emb (M)	Total (M)	$\rho$
sentence-transformers/LaBSE	501,153	768	384.9	471	81.7%
intfloat/multilingual-e5-base	250,002	768	192.0	278	69.1%
paraphrase-multilingual-mpnet-base-v2	250,002	768	192.0	278	69.1%
google/embeddinggemma-300m	262,144	768	201.3	308	65.4%
intfloat/multilingual-e5-large	250,002	1024	256.0	560	45.7%
BAAI/bge-m3	250,002	1024	256.0	568	45.1%
Qwen/Qwen3-Embedding-0.6B	151,669	1024	155.3	596	26.1%
intfloat/e5-mistral-7b-instruct	32,000	4096	131.1	7,111	1.8%

The pattern is consistent: encoder or bi-encoder models with a multilingual tokenizer (XLM-RoBERTa = 250k; Google SentencePiece = 262k; LaBSE = 501k) concentrate parameters in the embedding matrix and yield the largest reductions. Decoder-only models (Mistral, Qwen, LLaMA) have small vocabularies relative to their encoder, so trimming barely helps.

Try it

pip install -r requirements.txt

python trim_vocab.py \
    --model google/embeddinggemma-300m \
    --corpus-config por \
    --vocab-size 64000 \
    --output ./embeddinggemma-pt-br

--corpus-config is the language code for the mining corpus (por, fra, deu, spa); pass --corpus-dataset to mine any other HuggingFace text dataset. A ready-made Portuguese build is on Hugging Face at tardellirs/embeddinggemma-pt-br.

What it is not

This is a compression method, not an enhancement. Trimming the vocabulary preserves quality; it does not improve it. Fine-tuning, layer pruning, and distillation from a larger teacher were all evaluated and each reduced MTEB-BR by 0.02–0.04 points. The base model is at its representational ceiling for the target language; trimming recovers deployment efficiency at no quality cost, but cannot exceed it. Validated on BPE tokenizers with byte-fallback (Gemma/EmbeddingGemma); the merge-filtering logic may need minor adaptation for other tokenizer families.

Vocabulary trimming is an ablation alongside MTEB-BR, not part of the benchmark ranking. The benchmark is what makes it measurable: without a native Portuguese score to compare against, “99.4% of quality” would just be a number.

Code, equations, and all seven data points: github.com/tardellirs/embedding-vocab-trimmer (Apache-2.0; trimmed models inherit the base model’s license — the example build follows the Gemma Terms of Use).