Does the shape of a finished answer carry a trace of how hard it was to write? A deterministic reading of
the text, set against the model's own token-by-token uncertainty.
Russell Parrish · Parallax Metrology · GPT-4o, 60 responses · Data collected March 2026, analysis October 2026
Responses
60
10 prompt families, GPT-4o, temperature 1.0
Clause density vs instability
r = 0.46
95% CI 0.19 to 0.67
After length and family
r = 0.36
p = 0.005; CI 0.02 to 0.61
Mean uncertainty
r = 0.10
no link: the signal is in how it moves
Needs model access?
No
the text reading uses the finished answer only
The question
A language model writes one token at a time. At each step it spreads probability over the possible next tokens:
sometimes it is confident, sometimes the choice is close. That uncertainty is visible only to someone with access to the
model's probabilities. Most methods for estimating it need those probabilities (Malinin & Gales, 2021; Kadavath et
al., 2022) or several sampled answers to compare (Kuhn et al., 2023; Manakul et al., 2023). Most deployed text arrives
with neither.
Token probabilities have long been used to inspect finished text (Gehrmann et al., 2019). This study asks the reverse:
whether any trace of the model's uncertainty survives in the finished text alone. If a structural reading of the answer, taken
without the model and without reading for meaning, moves with the model's own uncertainty, then completed text carries a
small, recoverable signal about how it was produced. The answer here is yes, for one reading in particular, and the
signal is modest.
1. Two measurements of the same answer
Surface structure
Each response is read by the Linguistic Kernel (v19), a deterministic instrument that turns any text into eight
numbers from countable surface features: sentence lengths, paragraphing, content-word share, connectives, rare
vocabulary, clause markers. No model is called and the same text always gives the same numbers. The reading that matters
here is structural thickness, dₛ: commas, plus semicolons and colons weighted 1.5×, plus subordinating
conjunctions, per word, scaled to 0–1.
Generation uncertainty
The same responses were requested from GPT-4o with log-probabilities switched on, returning the top five candidate
tokens at every position. Uncertainty at each token is the entropy of those five candidates, renormalised: 0 when one
candidate holds all the probability, up to about 1.61 when all five are equally likely. The token series is smoothed with
a 9-token rolling mean.
From each smoothed series comes an instability index: the average of three standardised features describing how
uncertainty moves rather than how much of it there is.
Gradient: the mean absolute step-to-step change in smoothed entropy.
Paragraph shock: the mean jump in sentence-level entropy across paragraph breaks (0 when there are none).
Recovery: after an entropy spike, the mean number of tokens until it settles again.
2. What uncertainty looks like across an answer
Figure 1. Smoothed token entropy across three responses: the lowest, middle and highest on the
instability index. Hover a line for its prompt. Every trace moves; the index summarises how sharply it changes from token
to token, how much it jumps at paragraph breaks, and how long spikes take to settle, which is not always obvious by eye.
Length alone does not decide it: the lowest and highest answers here are both short.
Prompts, left to right: "Is there such a thing as an objective fact?" · "Explain how to prepare for a
difficult conversation." · "List exactly five principles of good design, one sentence each."
3. The finding: clause density tracks instability
Figure 2. Each dot is one response. Left: structural thickness against the instability index. Right:
the same relationship after removing what response length and prompt family explain from both. The dashed line is a
least-squares fit. Hover a dot for its prompt.
Answers dense with clause markers tend to be the answers whose generation was least steady. The correlation is
r = 0.46 (p = 0.0002; bootstrap 95% CI 0.19 to 0.67, 5,000 resamples).
Two obvious explanations had to be ruled out. Longer answers could be both denser and more unstable, and some prompt
families could produce both. With log response length and prompt family regressed out of both measurements, the
relationship weakens but holds: r = 0.36 (p = 0.005; CI 0.02 to 0.61). The interval is wide, which is what 60
responses allow. The direction is consistent.
Instability component
r with dₛ
p
r after length and family
p
Paragraph shock
0.44
0.001
0.29
0.02
Gradient
0.26
0.04
0.22
0.10
Recovery
0.12
0.37
0.08
0.56
Instability index
0.46
0.0002
0.36
0.005
Most of the link runs through paragraph breaks: in clause-dense answers, the model's uncertainty jumps more when it
starts a new paragraph. The average level of uncertainty is unrelated (r = 0.10). What the text records is not how
uncertain the model was overall, but how unevenly the uncertainty moved.
How much it predicts
Predicting the instability index for each response from the other 59 (leave-one-out, ridge regression), length and
prompt family together explain 24% of the variance. Adding dₛ raises that to 29%. Five points of out-of-sample variance
is small, but it comes from one reading of the finished text.
4. What dₛ is picking up
dₛ is mostly comma density (r = 0.76 with commas per word). Colons and bold headings contribute too (r = 0.40 and
0.39), because "Heading:" formatting adds colons. So the reading is best understood as clause-marker density:
how often the text stops to qualify, list or subordinate. Formatted, sectioned answers score high as well as long,
winding sentences. The most unstable answer in Figure 1 is a short list whose every line is built around a colon and a
qualifying clause.
5. What this can and cannot do
It can give a single-pass, model-free signal about generation difficulty from finished text: cached answers,
third-party outputs, or any model whose probabilities are not exposed.
It can rank answers on average: clause-dense, heavily qualified answers tend to be where the generation was
least steady.
It cannot recover uncertainty itself. The relationship is real but modest, and most of the variation in
instability is not visible in surface structure.
It does not show cause. Dense clause structure and uneven uncertainty may both follow from what the prompt
demands.
Limits. One model (GPT-4o), one temperature (1.0), 60 responses across ten prompt families.
Entropy is computed from the top five candidates only, which approximates the full distribution. The instability index
is one equal-weight construction; other constructions may behave differently. Five of the 60 responses reached the 600-token limit and end mid-sentence;
without them the result is unchanged (r = 0.46; 0.36 after controls). Replication on other models needs token-level
probabilities, which not every provider exposes.
Method
Element
Detail
Model and settings
GPT-4o via the Chat Completions API; logprobs on, top_logprobs = 5, temperature 1.0, max_tokens 600
Prompts
50 prompts in ten families (5 each), plus a 10-response preliminary run that repeats prompts from five of the families; 60 responses in total
H = −Σ p log p over the top-5 candidates, renormalised to sum to 1; maximum ≈ 1.61
Smoothing
9-token rolling mean
Instability index
Mean of z-scored mean absolute gradient, paragraph boundary shock (0 if no paragraph breaks; 8 of 60 responses) and mean spike recovery steps (spike: smoothed z > 1.5 at a local maximum)
Text reading
Linguistic Kernel v19, default configuration, computed on the finished response text
Controls
Log token count and nine prompt-family indicators, regressed out of both measurements (partial correlation)
Leave-one-out ridge regression (Hoerl & Kennard, 1970; α = 3), features standardised within each training fold
Example prompts by family: "Explain how photosynthesis works." · "List exactly five benefits of exercise, one
sentence each." · "Describe the color blue to someone who has never seen color." · "What is the relationship between
time and memory?" · "Compare and contrast democracy and autocracy." · "What does it mean for something to exist?"
Sources
This page is compiled from the author's working study Structural Residue: Kernel Metrics as Surface
Signatures of Generation-Level Uncertainty in Large Language Models (2026), which collected this dataset, and uses
the author's Linguistic Kernel (v19). The archive of the original paper is kept at
artistinfluencer.com.
References
Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
Gehrmann, S., Strobelt, H., & Rush, A. M. (2019). GLTR: Statistical detection and visualization of generated
text. Proceedings of ACL 2019: System Demonstrations.
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems.
Technometrics, 12(1), 55–67.
Kadavath, S., et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221.
Kuhn, L., Gal, Y., & Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty
estimation in natural language generation. Proceedings of ICLR 2023.
Malinin, A., & Gales, M. (2021). Uncertainty estimation in autoregressive structured prediction.
Proceedings of ICLR 2021.
Manakul, P., Liusie, A., & Gales, M. J. F. (2023). SelfCheckGPT: Zero-resource black-box hallucination detection
for generative large language models. Proceedings of EMNLP 2023.
See also: Text has shape, what the same instrument can tell apart across
prompts, runs and engines.