TIB: Sample-wise Tempered Information Bottleneck for Multimodal Attribution beyond Alignment Assumption

MIx Group · University of Birmingham

NeurIPS 2026

Paper comparison of TIB, NIB, M2IB, Grad-CAM, Fast-IG, and Chefer on aligned and misaligned image–text pairs — **Attribution under semantic misalignment.** The image is unchanged, while its caption switches from describing the scene to an unrelated bicycle race. TIB shows a weaker response for the mismatched pair in this paper example. Explore the comparison below.

Abstract

Multimodal attribution often uses one modality as proxy supervision to explain another, implicitly assuming semantic alignment between paired inputs. In realistic image–text data, this assumption can fail. We analyze how standard information bottleneck objectives allocate explanatory capacity to spurious cross-modal dependencies, producing forced and over-confident explanations. We introduce TIB, a sample-wise Tempered Information Bottleneck that jointly learns a feature retention mask and a scalar predictive variance. This input-dependent uncertainty adapts fitting pressure to cross-modal reliability, while a separate compression term regulates retained information. The paper evaluates this mechanism on controlled misalignment benchmarks and real-world multimodal datasets.

Method overview

Explain what’s there.
Account for what isn’t.

When one modality explains another, the paired input becomes proxy supervision. If that supervision is unreliable, a fixed fitting pressure can produce a forced explanation.

Keep the backbone frozen.

Capture intermediate image or text features from a pretrained vision–language encoder. Learn a bottleneck for this single pair.

Image / textFrozen encoder

Learn what to retain.

A feature-wise mask h preserves useful information. A sample-wise variance σ² controls how much noise replaces suppressed features.

Mask hVariance σ²

Adapt the fitting pressure.

Tempered likelihood weighting makes the fitting term sensitive to predicted variance. A separate KL term charges for retained information.

ReliabilityAttribution

The stochastic bottleneck · Eq. 8

z = h ⊙ f + σ (1 − h) ⊙ ε

retained features + uncertainty-aware noise
ε ∼ N(0, I)

Aligned and misaligned pairs

Switch the caption and comparison method to inspect the paper visualizations on the same image.

Original image of a person with four dogs — Input imageUnchanged

TIB image attribution with an aligned caption — TIBOurs

M2IB image attribution with an aligned caption — M2IBComparison

Paper figure These are fixed visualizations from the supplied figure. Parameter changes below do not regenerate them.

View full figure

A meaningful caption can support a meaningful explanation. The challenge is what happens when that correspondence disappears.

Interactive mechanism explorer

A few parameters.
Different responsibilities.

This scalar illustration follows the implemented equations, with feature f = 1 and target dimension d = 1. Adjust a control to inspect its effect.

Mechanism explorer

Controls explain the mechanism. They do not train a vision–language model in your browser.

β Compression weight 0.10Watch the compression bar change. At a fixed mask and variance, β does not change the distribution. α Tempering exponent 0.50Watch the variance-sensitivity curve change. At σ² = 1, the current coefficient stays at 1 for every α. σ² Predicted variance 1.00A learned quantity, manually varied here. Larger values increase injected noise at fixed h. h Retention mask 0.50h = sigmoid(mask logit). Near 1 keeps the feature; near 0 replaces it with noise. r Fitting residual 0.25Watch the fitting bar change. r = 1 − cosine similarity is held independently of z in this illustration. K Monte Carlo samples 5Changes the displayed sample count, not the underlying distribution.

Eq. 6 / Tempered objective

Which pressure changes?

sg[(σ²)^α] · [½ log σ² + r / σ²] + β · KL

Loss contributions · β changes compression; r changes fitting

Variance sensitivity · α changes the slope

Every curve meets at σ² = 1.

Direct fitting coefficient1.000(σ²)^α−1

KL to N(0, 1)0.443Unweighted capacity

Weighted likelihood0.250

Compression penalty β · KL0.044

Total illustration loss0.294

At σ² = 1, the direct fitting coefficient is 1 for every α.

sg means stop-gradient: the stabilizing weight is detached. The variance still affects the likelihood, noise, and KL. These plots show the objective at fixed h and σ², not an optimization run.

Eq. 8 / Stochastic representation

What reaches the next layer?

Live

q(z | x)Monte Carlo samplesOriginal f = 1

Retained mean0.500h · f

Noise std. dev.0.500σ · (1 − h)

Sample mean—Depends on K and seed

Noise seed

Configuration reference

Explore all 19 configuration fields. Defaults come from AttributionConfig; the downloadable code also includes the selected CLIP ViT-B/32 branch profile.

Browse all 19 parameters

Predicted σ² and the optimized mask h are learned per pair. They are distinct from configuration fields such as α, β, and the initial mask logit.

BibTeX

Cite TIB

@inproceedings{huang2026tib,
  title     = {{TIB}: Sample-wise Tempered Information Bottleneck for Multimodal Attribution beyond Alignment Assumption},
  author    = {Qiming Huang and Peixi Liu and Jianbo Jiao},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

Download .bib