Results for Tabular Machine Learning

Until 2025, fitting a new table meant fitting a new model: engineer features, tune XGBoost, validate, repeat. Every CSV was a fresh problem, although tables share a great deal of structure. Tabular foundation models (TFMs) change the unit of work. A transformer pre-trained on millions of synthetic tables reads the training rows as context and predicts the query rows in one forward pass, with frozen weights (in-context learning, ICL); the result approximates the Bayesian posterior predictive rather than the output of a model fitted to that table. On the September 2026 TabArena board, every 2026 flagship default beats the best tuned and ensembled gradient-boosted trees, so the 2025 question of whether deep learning can handle i.i.d. tabular data is closed. Five results did not follow from that headline.

The numbers come from the TabArena board and its BeyondArena companion, snapshot of 14 September 2026; the companion Technical Overview carries the full snapshot and the model-by-model detail.

Takeaway 1: The parameter paradox

In language models, parameter count is the first-order lever. In tabular data, the 2026 evidence points the other way: prior design and attention layout matter more than raw size. The September 2026 TabArena snapshot (Elo calibrated so that a default random forest scores 1000) puts Google's TabFM, reported by third parties at about 1.6B parameters since Google has not published a count, at 1800 Elo. LG's EXAONE Tabular, at 20.8M parameters, trails by 18 Elo points. Amazon's Mitra-v2 (September 2026) pushes further: at 77M parameters, about 5% of TabFM's reported size, it claims parity with TabFM and EXAONE on the full TabArena and on TALENT, under Amazon's own one-hour protocol that includes fine-tuning; it is not yet on the official board. Taken together, these results suggest that for structured data the quality of the synthetic prior, not the number of weights, is the engine of generalization.

Figure 1. The parameter paradox. Disclosed parameter counts against Elo on the full TabArena board. TabFM's size is a third-party figure (from the Mitra-v2 and EXAONE reports); Mitra-v2 and LimiX-2M appear as dashed markers because they are not on the board. TabPFN-3 and TabICLv2 have not disclosed parameter counts. Sources: TabArena snapshot 2026-09-14; arXiv 2608.25774, 2609.04540, 2509.03505, 2606.04485, 2506.10707.

Solving representation collapse

Smaller models reach the frontier by fixing architectural bottlenecks rather than adding layers. LimiX-2M reaches TabPFN-v2 accuracy with 2M parameters, in its own evaluation, by using RBF feature expansion to counter low-rank collapse. How features are projected into the latent space appears to return more than sheer scaling does.

"Mitra-v2 matches the 1.6B-parameter TabFM with only 5% of its size (77M parameters), delivering frontier performance at a fraction of the cost."

— Yefan Tao et al., Mitra-v2 Technical Report (arXiv 2609.04540), Amazon, 2026

Takeaway 2: Zero-shot is the default for small data

For tables under 1,000 rows, the cold-start problem is no longer open. Zero-shot TabICLv2 and TabPFN-3 out-rank tuned and ensembled CatBoost, LightGBM and XGBoost on the BeyondArena tiny and small subsets, and TabICLv2 beats tuned XGBoost, CatBoost and LightGBM on roughly 80% of TabArena datasets without a single gradient update. This works because the prior does the regularising: the model returns an approximate posterior predictive given the rows in context and has nothing to overfit to, where a boosted ensemble tuned on the same 500 rows has plenty.

Figure 2. Small data. BeyondArena Elo on the i.i.d. tiny (<1K rows) and small (1K–10K rows) subsets, with default XGBoost as the 1000 baseline. Zero-shot TabPFN-3 and TabICLv2 out-rank CatBoost, LightGBM and XGBoost even after tuning and ensembling. Source: BeyondArena subsets on the TabArena Space, snapshot 2026-09-14.

"It does not require hyperparameter tuning and still outperforms heavily tuned XGBoost, CatBoost, or LightGBM on TabArena on ~80% of datasets."

— Jingang Qu et al., TabICLv2 repository README (github.com/soda-inria/tabicl), 2026

Takeaway 3: Thinking mode, optimization without gradients

2026 added a lever between zero-shot use and fine-tuning: test-time compute, marketed as "thinking mode" and introduced with TabPFN-3-Plus. The weights stay frozen; the model spends extra optimization effort once, at fit time, to condition its context. Prior Labs reports up to 15% higher TabArena Elo over vanilla TabPFN-3 from this alone, and a win over a four-hour AutoGluon 1.5 extreme pipeline in about 80% of cases at 10x the speed. Both are the vendor's numbers; the Plus variant is not on the public board. It is also API-only, which draws a sharp line between open-weight deployment and managed inference.

Adaptation without catastrophic forgetting

Because the weights never move, there is no catastrophic forgetting and no gradient update to audit; the cost is paid once at fit time and prediction latency is unchanged. The same logic covers the other 2026 context-side levers, learned context compression (TACO), nearest-neighbour contexts and bounded contexts for streams (CURE), all of which act on what a frozen model sees rather than on the model.

"Thinking mode spends more effort on fitting stage, delivering up to 15% improvement (measured on TabArena ELO improvement) over vanilla TabPFN-3. [...] You pay the optimization cost once at fit time, so you can use it for one-time optimization that fuels recurring predictions."

— Prior Labs, TabPFN-3 release notes (docs.priorlabs.ai/changelog/tabpfn-3), May 2026

Takeaway 4: Accuracy and calibration come apart, twice

Fine-tuning most TFMs on a downstream table lowers accuracy and raises expected calibration error at the same time; the model becomes more certain and less right. TabPFNv2 is the documented exception: for that architecture full fine-tuning is the best adaptation, converges faster than LoRA, and keeps its calibration. The reason SFT fails elsewhere is that the model behaves as an implicit retriever, weighting context rows by learned similarity; the gains from fine-tuning TabPFNv2 correlate at 0.89 with an explicit retriever (ModernNCA), and gradient updates that do not preserve that similarity structure destroy it.

The second gap needs no fine-tuning at all. On 112 TALENT datasets, zero-shot TFMs post the highest AUC yet lower conditional coverage under conformal prediction than gradient-boosted trees. On 97 regression datasets, rankings shift substantially once proper scoring rules replace point metrics such as RMSE. For a regulated deployment, a TFM chosen on accuracy still needs conformal or temperature calibration before its predictive distribution is trusted.

"Our results reveal a performance-uncertainty trade-off: although TFMs achieve the highest predictive performance, measured by AUC, they exhibit lower conditional coverage under conformal prediction, measured by SSCS, compared to GBDTs."

— José Lucas De Melo Costa et al., High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models (arXiv 2605.28554), ESANN 2026

Takeaway 5: Where trees still win, and how to use a TFM next to them

TFMs are not a replacement for gradient-boosted trees outside the i.i.d. regime. BeyondArena, the TabArena team's June 2026 stress test over 142 datasets, calibrates its Elo so that a default XGBoost scores 1000 on every subset. On temporal and grouped splits the 2026 leaders trail tuned RealMLP, TabM and the GBDTs, and on the temporal split TabICLv2 falls to 835. On tables of 100K rows and above every TFM evaluated there sits below the default XGBoost: TabPFN-3 at 935, TabICLv2 at 724, against tuned models at 1,230 to 1,350. One practical pattern follows: use the TFM's column embeddings to profile a messy table (distribution shifts, hidden feature types) and keep a tuned GBDT as the production model for large or shifting data.

Figure 3. BeyondArena by regime. Elo per subset, calibrated so default XGBoost scores 1000 in every column; teal cells beat that baseline, red cells fall below it. TabPFN-2.6 is omitted because most of its scores on the shifted and large subsets are imputed. Source: BeyondArena subsets on the TabArena Space (Purucker et al. 2026, arXiv 2606.30410), snapshot 2026-09-14.

Bridging the generality gap

The TFM is the better tool for understanding a messy schema and feature roles; the GBDT remains the better tool for large-scale, non-i.i.d. execution. Splitting the work that way feeds higher-fidelity inputs to the production model and sidesteps the sequence-length limits of today's transformers. Two related facts belong here. Stacking TFMs is not a lever either: six leading models are near-redundant (mean pairwise Q-statistic 0.961) and the best stacking scheme adds 0.18% accuracy at 253x the compute. And benchmark rank does not transfer to enterprise tables: SAP finds that rankings of TabPFN, TabICL and ConTextTab on public benchmarks do not carry over to internal data in either direction, so shortlist on the board and decide on your own data.

"TFMs fail to compete with traditional models on non-IID (temporal and grouped), large-scale, high-dimensional, and high-cardinality datasets."

— Lennart Purucker et al., Beyond IID: How General Are Tabular Foundation Models, Really? (arXiv 2606.30410), 2026

What is left

The row wall fell in 2026: TabPFN-3 and TabICLv2 fit a million-row table in one context on one GPU. Winning on those tables did not follow, and neither did winning on temporal splits, grouped splits or high-cardinality categoricals. The width wall stands at 100 to 200 features for the models that scale in rows. The relational track is moving fastest and by two routes at once: KumoRFM-2 operates natively over multi-table databases, while TabPFN-Rel flattens the relational context into one table and still ranks first on RelArena-α ahead of graph-based baselines. The prior, not the parameter count, is the lever the field has agreed on; with the TabICLv2 generator public since July, it is also the one anyone can now pull.