From Conditional Denoising
to Learned Information

The direction proposed here is task-dependent preservation of the conditional mean under learned attention. The aim is not to recover every input coordinate. It is to characterize which distinctions must survive for a specified denoising or editing target, and to identify when depth improves their usable representation.

01 / ReconstructionWhat the slide proves, once the model is fixed

Let $X^{(1)},\ldots,X^{(N)}\in\mathbb R^{P\times d}$ be indexed clean images or token arrays. Draw $J$ with masses $\pi_n>0$, $\sum_n\pi_n=1$, independently of $\varepsilon\sim\mathcal N(0,I_{Pd})$. At a fixed $0<t\leq1$, define

Fix the target token $i$. Positional labels and trained parameters are treated as known; if a position is not uniquely identified by the actual representation, its possible indices must instead be included in the conditioning model. Repeated training examples retain their probability mass.

First suppose $S\in\mathbb R^{H\times P}$ is externally fixed and independent of $(J,\varepsilon)$. Set $A=SZ$, let $b_i$ be the $i$th coordinate vector in $\mathbb R^P$, and define

After consistent vectorization, the component law and posterior are

Here $\varphi_\Sigma$ denotes a centered Gaussian density on its appropriate support. Because $Z_i$ is retained, the target residual is fixed once $J$ is known:

This is the slide’s expression with explicit prior masses and a declared sampling law. The factor $SS^\top$ is indispensable: the heads share corrupted tokens. Positive definiteness holds exactly when $S_{-i}$ has full row rank, where $S_{-i}$ removes column $i$. Otherwise one evaluates a Gaussian density in coordinates on the common support, rather than setting the likelihood to zero. These are consequences of linear Gaussian conditioning.^[1]

Equations involving $1/t$ do not define the endpoint $t=0$. At $t=1$, the noise equals the observed corrupted input, so a representation retaining $Z_i$ estimates $\varepsilon_i$ exactly. Time-integrated claims require a specified time law and integrability; the propositions below are stated at fixed $t$.

02 / Exact developmentCondition on the logits, then on the values

2.1 A model with learned projections

Keep the finite-mixture corruption model (1). For head $h$, use fixed learned matrices $Q_h,K_h\in\mathbb R^{k_h\times d}$ and $V_h\in\mathbb R^{v_h\times d}$. These parameters are frozen during the probabilistic calculation. Claims about training or generalization require a separate argument. We use a standard unmasked, bias-free dot-product head:^[2]

There is no layer normalization, dropout, preceding nonlinear block, or quantization in this model. Known affine positional terms can be incorporated by subtracting their known logit offsets. Such extensions should not be silently substituted for (5).

Condition on $C:=Z_i=c$. Define the query-dependent measurement matrix

The complete row is equivalent to its $P-1$ log-ratios for each head. Conditional on the query, those ratios are linear observations of the other tokens. This is the useful algebraic simplification; the attention output itself remains nonlinear before conditioning.

2.2 From log-ratios back to attention probabilities

For independent simplex coordinates $s_{hj}$, $j\ne i$, the determinant of the log-ratio transformation for a single head is $(\prod_js_{hj})^{-1}$. If the joint row law has full dimension as assumed above,

The Jacobian is common to every $n$, so it cancels in the posterior. It does not make the exogenous-row likelihood correct: the row-dependent evidence in the middle factor of (10), and the conditional residual covariance $R$, remain essential.

If $M$ is rank-deficient, replace the redundant log-ratios by coordinates spanning its image, and condition using $M^+$ and the induced Gaussian support. One must not apply a full-dimensional simplex density to dependent heads. Masks also alter the observed support and may remove the self-token anchor used in (6).

2.3 Recovery, loss, and stability

If $\operatorname{rank}M=d$, each $Z_j$ is determined by $(c,\ell)$; $R=0$ and the values add no information. If the rank is smaller, the rows leave an affine fiber with directions in $\ker M$. The values constrain additional combinations on that fiber. This is a statement about retained rows, not a claim that a conventional attention block exports them.

Moreover, $\Delta\ell_{hj}\approx\Delta s_{hj}/s_{hj}-\Delta s_{hi}/s_{hi}$. Small nonzero singular values or nearly saturated probabilities can make exact identifiability practically useless. This bound holds with the query fixed; query uncertainty also perturbs $M(c)$ and must be propagated. A serious empirical comparison must specify weight precision and clipping.

2.4 An exact sufficiency test on the remaining fiber

The rank of $M$ alone is not the relevant stopping criterion: the values may resolve its null directions, and some unresolved directions may carry no distinction between hypotheses. This can be tested without requiring full token recovery.

Let $z_{-i}\in\mathbb R^{(P-1)d}$ stack the other tokens in a fixed order. For a retained observation $o=(c,\ell,a)$, define

In this display, the bottom block is the horizontal concatenation of the $D_j$ matrices. The fiber of the retained observation is exactly an affine solution set $F(o)z_{-i}=b(o)$: the first block fixes the log-ratios, and therefore the row; the second fixes the values. For projected values retained together with the row, replace each bottom block by $W_OD_j$.

The two-component target distinction is exact. If $N=2$ and $\mu_{1,i}\ne\mu_{2,i}$, preserving $\mathbb E[\varepsilon_i\mid Z]$ is equivalent to preserving the component posterior, because (4) is then an injective affine function of one posterior probability. Thus (12b) is necessary and sufficient for zero information-loss term in (16). If $\mu_{1,i}=\mu_{2,i}$, the local noise mean is already determined by $c$, even when the component posterior is lost. With more components, equality of conditional means can likewise hold without equality of posteriors.

The practical consequence is a computable diagnostic for the finite-mixture reference experiment: report the unresolved mean contrasts, alongside the conditioning margins and measured risk. Once the row itself is discarded, the observation is a union over latent-row fibers, and this row-conditioned criterion cannot be applied as if the row were still available.

03 / The observation boundaryWhat if the network discards the row?

Suppose the actual retained output is $(C,Y)$ with $Y=W_OA$, rather than $(C,S,A)$. In (9), replace $\nu_n$ by $W_O\nu_n$ and $\Gamma$ by $W_O\Gamma W_O^\top$. The exact conditional law is obtained by integrating over the unobserved log-ratios:

This is a mixture of conditional Gaussian measures, allowing singular components. Both their means and covariances depend on the logits. It is not generally one Gaussian. In a scalar two-token construction, output-only attention already admits multiple inverse branches; retaining only the branch used to generate a sample would leak unavailable information.

For an arbitrary deterministic representation $T=G_\theta(Z)$, the universal object is the pushforward component measure $\mathsf Q_{n,\theta}=(G_\theta)_\#\mathcal N(\mu_n,t^2I)$. With $\mathsf Q_\theta=\sum_n\pi_n\mathsf Q_{n,\theta}$, the posterior is the Radon–Nikodym derivative

If $G:\mathbb R^D\to\mathbb R^m$ is locally Lipschitz, $m\leq D$, and has rank $m$ almost everywhere under the input law, the coarea representation at almost every output is

Use the full-rank-almost-everywhere hypotheses, not merely a maximum-rank condition; positive input mass on critical sets can create singular output mass.^[3] When $Z_i$ is not retained, (4) must also be replaced: the numerator for the conditional noise mean integrates $(z_i-\mu_{n,i})/t$ over the same fibers. Component responsibilities alone then do not determine the target.

04 / What should be optimized?An exact decomposition of four sources of risk

Let the square-integrable target be $U=\varepsilon_i$ for denoising, or a separately specified task variable. Let $T=G_\theta(Z)$ be the actual representation. Define $m_Z=\mathbb E[U\mid Z]$, $m_T=\mathbb E[U\mid T]$, and $R_Z^*=\mathbb E\|U-m_Z\|^2$. Suppose the predictor $g(T)$ lies in a fixed linear output subspace $\mathcal S$, with orthogonal projector $\Pi$.

4.1 A sharper width criterion

For fixed $T$, let $C_T=\mathbb E[m_Tm_T^\top]$. Optimizing over all fixed output subspaces of dimension at most $r$ and all measurable predictors in them gives

Indeed, the minimizing predictor in a fixed subspace is $\Pi m_T$, and the minimizing subspace spans the leading eigenvectors of $C_T$. With a free affine offset, use $\operatorname{Cov}(m_T)$ instead. An input-dependent skip to the output can invalidate the fixed-subspace premise.

The relevant spectrum is that of the predictable target, not merely the target covariance. A target can have large variance entirely due to irreducible uncertainty. This distinction strengthens the interpretation of a width bound without claiming that it predicts final image quality.

4.2 Quantifying the cost of an incorrect likelihood

Returning to $U=\varepsilon_i$, for an observation retaining $Z_i$, compare the exact responsibilities $r$ with an approximation $\widetilde r$. Let $\widetilde m=(Z_i-\sum_n\widetilde r_n\mu_{n,i})/t$, and let $D_i=\max_{n,m}\|X_i^{(n)}-X_i^{(m)}\|$. Then

The first identity is orthogonality to functions of the same observation; the second follows by coupling the two finite distributions and bounding the distance between paired clean tokens. A posterior discrepancy is consequential only through target differences. Wrong posteriors can coincide in conditional mean when the candidate target values agree.

05 / Depth without ambiguityWhat additional layers can and cannot add

For a deterministic full-state network $H_{\ell+1}=F_\ell(H_\ell)$,

Thus later full states cannot improve the unrestricted Bayes risk by creating information about the original target. They can preserve that information while making the conditional mean easier for a restricted readout to realize. Comparing two separately trained architectures is also different from following the states of one fixed network.

For a local token, attention can access other tokens at each layer. Its successive representations need not form nested information sets. Define instead the cumulative local history $\mathcal F_\ell=\sigma(T_i^0,\ldots,T_i^\ell)$ and $m_\ell=\mathbb E[U\mid\mathcal F_\ell]$. Then

This martingale-increment identity measures target-relevant information gained by the cumulative observer. It is not a monotonicity theorem for the final token alone, because the network may discard earlier distinctions. Any depth study should declare whether it probes the full state, a single current token, or the cumulative history.

A useful sufficient preservation condition is available for a residual block $F(h)=h+f(h)$. If $f$ is globally $\kappa$-Lipschitz with $\kappa<1$, then

The reverse triangle inequality proves injectivity and a stable inverse on the image. This is a strong sufficient condition, not an assumption satisfied by ordinary transformer blocks by default. Normalization, compression, and subsequent projections still require separate analysis.

06 / Literature positionWhat is already advanced, and what remains

The constructive implication is to study where the terms in (16) move. Alignment could improve finite-budget realization without proving information sufficiency. A wider head can reduce a fixed-subspace obstruction when one exists. Additional layers can improve computation or local access while also discarding information. These explanations can coexist, and the proposed measurements should distinguish them.

Accordingly, “learnable attention” by itself is too broad a contribution claim. The more defensible objective is an observation-matched account of what learned attention retains, how stable that retention is, and which part of the achievable task risk it changes.

07 / Proposed directionA research programme with a falsifiable central question

7.1 Make the target and observation explicit

For noise estimation use $U=\varepsilon_i$. For a future steering application, define an edit request $q$ and a measurable target $U_q$ that specifies the requested change and protected quantities; condition on $q$ throughout. No generation metric alone identifies this task. Start with one declared family, not a generic promise of controllable generation.

Given any square-integrable target, the ideal representation objective has the equivalent forms

This supplies the conceptual objective, not a tractable gradient oracle. A trained probe gives an upper bound on the optimal representation risk; poor probe performance does not prove information loss. In the finite-mixture reference model, exact or controlled-integration posteriors can establish the missing calibration.

7.2 Three stages, each with an exit condition

1. Establish the exact first-block reference. Implement (10), the row-marginalized kernel (13), and the frozen-row approximation on the same hypotheses, corruptions, query, and learned matrices. Include duplicate heads, rank loss, nearly saturated softmax, and finite-precision row observations. Verify posteriors and noise estimates against change-of-variable or low-dimensional integration checks. Exit condition: the proposed estimator must represent the declared observation; no latent branch or hidden row may be revealed only to one method.
2. Separate representation quality from realization. Train a small model on independently sampled corruptions, then freeze it for exact evaluation. Independent corruptions test the declared finite distribution; generalization to unseen clean images requires a separate held-out image distribution. Compare learned rows, fixed rows, and ordinary row-discarding outputs. Match observation access, output rank, parameter budget, and training budget where a causal comparison is claimed. Report the individual terms of (16) where computable, otherwise clearly labeled bounds. Exit condition: if an advantage disappears under matched observations or after calibrating the readout, do not claim better information preservation.
3. Introduce depth only after the first-block discrepancy is understood. Probe full-state, current-token, and cumulative-token observations separately. Track (20), readout error, and conditioning as depth changes. Layer-normalized and deep-state likelihoods require new analysis or controlled approximation. Exit condition: retain a depth claim only if it explains a measured difference beyond extra parameters, access, or probe capacity.

7.3 A realistic theoretical target

Corollary 1 already settles exact posterior sufficiency for the row-and-value observation, and exact local denoising sufficiency for two components with distinct target means. The next theorem should address the actual row-discarding observation or quantitative loss under bounded row perturbations, rather than re-prove that solved special case. Full input recoverability is a useful special case, but target sufficiency is the more economical condition. A stability theorem should include a nonzero-singular-value margin and a probability margin for the observed rows, or explicitly describe their failure sets.

Learning can then be addressed under a restricted parameterization. Equation (10) alone does not show that gradient descent discovers good $M(c)$, value maps, or output projections. Rank changes and changing supports also make naive differentiation of singular-density formulas unsafe. If a smooth observation-noise model is introduced for analysis, declare it as a changed experiment and study its noise dependence; do not describe numerical jitter as an exact repair.

7.4 The claim this work could earn

A successful result would have the form: for this target family, corruption law, retained observation, precision regime, and learning procedure, a specified representation reduces a measured component of excess risk relative to matched alternatives, with a proved or empirically bounded failure regime. That is narrower than solving transformer denoising in general, but it is a claim that an expert can scrutinize and that a subsequent architecture decision can use.

The present deliverable earns a more limited statement: a conditional Gaussian construction and an exact sufficiency criterion for the declared learned first-block observation, a risk decomposition linking that observation to architecture, and a sequence of tests that could invalidate the proposed direction. It does not yet establish originality, trainability, or a steering advantage.

Appendix / A necessary qualificationTraining-loss lower bounds do not imply endpoint-error lower bounds

The archived RAE preprint’s Theorem 2 claims a positive sampling-endpoint discrepancy from a covariance-tail loss bound. Its proof, physical p.21, equates the derivative of a norm with the norm of a velocity difference and substitutes an expected squared regression bound for a pointwise drift bound. Neither step is generally valid.^[6] The following counterexample directly addresses the printed statement.

Let clean $X$ and noise $E$ be independent $\mathcal N(0,I_n)$, set $Z_t=(1-t)X+tE$, and use the velocity target $W=E-X$. With $a(t)=(1-t)^2+t^2$, Gaussian conditioning gives

The exact field is globally 1-Lipschitz in $z$ because $|(2t-1)/a(t)|\leq1$. Both this flow and the admissible zero predictor $g=0$ return the same endpoint $z(0)=z(1)$ for every common starting point. Yet $\operatorname{Cov}(W)=2I_n$, so the printed right-hand side with bottleneck width $r<n$ and Lipschitz constant 1 is $2(1-e^{-1})(n-r)>0$.

This is a local mathematical qualification of the archived version, not an author-issued erratum or a claim about its empirical results. It leaves the fixed-output-space regression obstruction intact. It prevents an unjustified move from instantaneous estimation error to inevitable terminal image error: trajectory differences can cancel.

Positive endpoint or distributional guarantees require their own hypotheses and analysis. The present programme therefore evaluates denoising risk and generated outcomes separately.

Verification / Reading scopeWhat has been checked

The companion numerical checks compare the conditional factorization to an independently assembled augmented Gaussian law, compare a fully invertible two-token case to a finite-difference change-of-variable Jacobian, recover the existing frozen-row counterexample, and verify the orthogonal risk decomposition on a finite probability space. Additional fiber checks cover both posterior sufficiency and a controlled failure, the two-component target distinction, and the conditional log-odds variance identity. Singular duplicate-head cases are checked in reduced coordinates. These are bounded consistency checks, not proofs of the general propositions or a model-training study.

This response builds on local library Chapters 48, 49, and 75. Selected primary-source sections were read, not every paper in full. The additional targeted search did not establish novelty of Proposition 1 or of the proposed programme. In particular, its ingredients are standard Gaussian conditioning and log-ratio coordinates; the potentially useful contribution lies in their application to an explicitly matched observation and a verified learning comparison.

The HTML is self-contained, including mathematical fonts. It can be read offline or printed. Equations are rendered from retained TeX source, and the computational check script accompanies the draft.

References / Exact versionsPrimary sources and boundaries of use

1. Kevin P. Murphy. Multivariate Gaussians. Author notes, 28 September 2007.Linear transformations and conditional Gaussian formulas, physical pp.1 and 3. The singular-support and attention-specific arguments here are derived separately.
2. Ashish Vaswani et al. Attention Is All You Need. NeurIPS 2017.Scaled dot-product and multi-head definitions, §3.2. The restricted first-block model in this response excludes normalization and other architecture details.
3. Luigi Negro. Sample Distribution Theory Using Coarea Formula. arXiv:2110.01441v2, 3 December 2021.Selected full-rank results, especially Theorem 4.4 on physical p.9. The maximum-rank-only formulation in this preprint’s Theorem 4.7 is not used. The later journal text was not inspected.
4. Hongkang Li, Hancheng Min, and Rene Vidal. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs. arXiv:2604.10074v1, 11 April 2026.Selected physical pp.2–7, especially Theorem 1, p.4, and the limitations on p.7. Appendix proof not independently audited; reported training not reproduced.
5. Sihyun Yu et al. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025; arXiv:2410.06940v4, 18 June 2025.Physical pp.5–9; the quoted FID values are Table 2, p.7, SiT-L/2, ImageNet-256, 400k steps, DINOv2-L with NT-Xent, without classifier-free guidance. These are source-reported measurements.
6. Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion Transformers with Representation Autoencoders. arXiv:2510.11690v1, 13 October 2025.Selected pp.4–6 and 20–21. Theorem 1 is qualified through an independent projection argument; Theorem 2 is not adopted, for the reason shown in the appendix.
7. Shengbang Tong et al. Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders. arXiv:2601.16208v1, 22 January 2026.Selected pp.1 and 3–7; wide-head and noise-schedule qualifications primarily p.5. Author-reported results; no independent training reproduction.