Image and text do not share a coordinate system

Summary

A linear classifier separates image from text with 100% in-sample accuracy in all three models. That result measures native coordinate conventions, not the absence of shared content. It does not report the median paired angle or the dispersion around each modality mean.

After whitening and a rank-32 orthogonal Procrustes rotation fitted on training pairs, a held-out image retrieves its caption above chance. The lift is 9.9× in Kimi K3, 3.2× in Inkling, and 9.2× in Qwen3-Omni. The fitted rotation removes part of the basis mismatch between modalities. The size of this effect differs by architecture.

Contents

What was measured

For each model, we use paired image and text vectors from the decoder boundary, where projected image features and text embeddings enter the same residual stream. Kimi K3 contributes 128 caption pairs. Inkling and Qwen3-Omni contribute 96 each. Every figure below draws the same vectors. Only the coordinate treatment changes.

Three quantities are reported per model. Raw separability is the accuracy of a linear classifier on the native coordinates. Cone half-angle is the median angle between a sample and its own modality mean direction. Paired rotation is the angle between an image vector and the text vector of the same pair.

Native coordinates separate trivially

A single linear boundary separates the modalities in all three models. The classifier reaches 100% in-sample accuracy because each modality has a different offset and basis. It identifies the modality from a vector without using the paired content.

Figure 1

Kimi K3n = 128

100% separableencoder + projector

Inklingn = 96

100% separableencoder-free

Qwen3-Omnin = 96

100% separableencoder + projector

● image● textlarge markers are modality means

Figure 1. First two components of one joint PCA over normalized decoder-boundary vectors, per model. Each model keeps its own coordinate system. PC1 in Kimi K3 is not PC1 in Inkling, so positions are not comparable across panels.

Dispersion differs by architecture

Each encoder-projector model has a narrow cone for one modality and a wide cone for the other, in opposite assignments. Kimi K3 has a 1.6° median text cone. Qwen3-Omni has a 2.0° median image cone. The encoder-free model has wide cones for both modalities.

Figure 2

Kimi K3n = 128

image 24.5°text 1.6°

Inklingn = 96

image 18.2°text 21.4°

Qwen3-Omnin = 96

image 2.0°text 19.6°

● image● textpale wedge is the 90th percentile

Figure 2. Median angle between a sample and its own modality mean direction. The two encoder-projector models each have one narrow and one wide modality, in opposite assignments.

Figure 3

Kimi K3n = 128

169.0° medianp10-p90 163.0-174.0°

Inklingn = 96

148.0° medianp10-p90 126.0-170.0°

Qwen3-Omnin = 96

161.0° medianp10-p90 152.0-169.0°

image raytext ray at median paired angle

Figure 3. Angle between each image vector and the text vector of the same pair. Kimi K3 is within 11.0° of antipodal with a p10-p90 range of 11.0°. The corresponding range for Inkling is 44.0°. The required paired rotation differs by model.

Held-out retrieval after rank-32 alignment

We whiten each modality on training pairs and fit a rank-32 orthogonal Procrustes map from the image basis to the text basis. A held-out image then retrieves its caption by nearest neighbor. The lift over chance is 9.9× for Kimi K3, 3.2× for Inkling, and 9.2× for Qwen3-Omni.

Table 1

ModelR@1over chancelinear CKA

Kimi K331%9.9×0.41

Inkling13%3.2×0.27

Qwen3-Omni38%9.2×0.19

Table 1. Held-out caption retrieval after training-only whitening and rotation, next to raw linear CKA. Candidate pools have different sizes, so lift over chance is the comparable quantity.

Method

imagetextgrey rows in X and Z are held out

X_{\mathrm{img}}

training + held-out

W_{\mathrm{img}}

training statistics

Z_{\mathrm{img}}

rank 32

Whiten and truncate image vectors in their own training basis.

X_{\mathrm{txt}}

training + held-out

W_{\mathrm{txt}}

training statistics

Z_{\mathrm{txt}}

rank 32

Whiten and truncate text vectors independently.

Z_{\mathrm{img}}^{\top}

training rows only

Z_{\mathrm{txt}}

training rows only

C

32 × 32

Cross-covariance from the 96 training pairs only.

C

U

\Sigma

V^{\top}

R = UV^{\top}

Discard Σ and retain only rotation R. The map has no scaling or translation.

Z_{\mathrm{img}}

held out

R

Z_{\mathrm{txt}}

held out

Apply R to held-out image vectors, then rank text vectors by cosine similarity over 32 candidates.

Method. The transport map consists of whitening, rank-32 truncation, and one orthogonal rotation. We fit it on 96 pairs and evaluate it on 32 held-out pairs. The orthogonal constraint prevents R from changing scale or location. R only rotates the image basis into the text basis.

Figure 4

Native

unmodified decoder-boundary vectors

Angular

L2-normalized, cosine geometry only

Mean removed

joint image-text mean subtracted

Rank-32 alignedfit on training pairs, evaluated on held-out pairs. Drag to rotate.

● image● textthe text-series color marks pairs above the p75 residual in the current projection

Figure 4. The sequence shows how each operation changes overlap for the same Kimi K3 pairs. Panels one through three describe the full sample. In the final panel, we fit whitening and rotation on 75% of pairs. Connectors join matching held-out pairs. We measure residuals in the displayed projection, so the ranking and highlighted set change with the viewing angle.

Limits of the projection

After whitening, one unit equals one modality-specific training standard deviation. The transformation sets equal cluster extents. These extents do not measure the native representations. Radius, area, and distance from the origin no longer represent native activation magnitude.

Figure 4 displays three dimensions of a 32-dimensional aligned space. Apparent overlap in this projection does not establish correspondence. The primary evidence is held-out retrieval computed in the full representation.

Cite

@misc{modalitytransport2026,
  author = {Korolev, Konstantin},
  title  = {Image and text do not share a coordinate system},
  year   = {2026},
  url    = {https://advprop.ai/notes/modality-transport/}
}

J-lens over visual features in Kimi and Inkling

Image and textdo not share acoordinate system

What was measured

Native coordinates separate trivially

Dispersion differs by architecture

Held-out retrieval after rank-32 alignment

Limits of the projection

Complementary pair