🏆 KabEncBench Leaderboard

Track A: Frozen Scale-Normalized Probe Protocol

To establish a fair, reproducible, and leakage-free evaluation of contextual encoders across diverse architectures, parameter sizes, and depths, KabEncBench enforces strict evaluation invariants:

1. 100% Frozen Backbones: The encoder trunk weights are never modified during downstream task probe training. Zero gradient updates touch the backbone.
2. Learned Layer Mixing (ScalarMix): Instead of arbitrarily selecting the last layer, representations are computed via a learnable softmax scalar mix across all $L+1$ layer representations (including the embedding table), ensuring fair layer capacity utilization.
3. Standard Linear Probes: A single linear layer maps pooled representations to label spaces without non-linear MLPs or recurrent heads.
4. Unified Hyperparameter Sweep: For every model and task, AdamW is swept across a fixed 5-learning-rate grid with 5 random seeds per learning rate for 20 epochs. The best learning rate on the development split is evaluated once on the strictly held-out test split.

Bradley–Terry Elo Tournament & Bootstrap

Rankings are determined by simulating a paired head-to-head tournament on all 100,609 item-level predictions across all 9 benchmark tasks. An item where Model A produces the correct classification while Model B errs constitutes a win for Model A. Pairwise Elo ratings are calculated via logistic Bradley–Terry maximum likelihood estimation anchored at FacebookAI/xlm-roberta-base = 1,000.0. 95% confidence intervals are computed using 2,000 paired cluster-bootstraps.

Quickstart: Using KabEncBench

from datasets import load_dataset

# Load syntax (UPOS) task
syntax = load_dataset("agbalu/KabEncBench", "syntax")

# Load morphology aspect classification
morph = load_dataset("agbalu/KabEncBench", "morph")

# Load spelling minimal pairs diagnostic
spell = load_dataset("agbalu/KabEncBench", "diag_spell")