1 Introduction

Shallow cumulus fields in the trades organize into recognisable mesoscale patterns [1, 2]. These have been labelled by crowds [3], summarised by hand-designed metrics [4–6], and placed in a continuous space whose four leading dimensions are characteristic length, void size, directional alignment and horizontal variance of cloud-top height [7]. Learned representations recover organization regimes with distinct radiative properties and environmental conditions and allow transitions between regimes to be followed [8–10], and self-supervised features have been compared with other networks for classifying the patterns [11]. We are not aware of a test of whether such an embedding adds out-of-sample prediction of how a cloud field evolves beyond established metrics, the large-scale environment, viewing geometry and measured cloud properties.

Cloud cover in the trades has a pronounced daily cycle, mediated partly by the changing frequency of patterns and strongly related to near-surface wind [12]. The MODIS instruments on Terra (about 10:30 local time) and Aqua (about 13:30) observe the same field three hours apart. We ask which description of the morning scene better predicts the change in cloud fraction between the two overpasses, 13 organization metrics or a general-purpose image embedding, and then which known quantities account for the difference.

Contributions: (i) an out-of-sample comparison of organization metrics, computed two ways, and two general-purpose embeddings on a physical outcome, in an original sample, additional years and new regions; (ii) controls for viewing geometry, nonlinear cloud cover, cloud optical and cloud-top properties and high cloud, separately and together; (iii) a test, against a noise baseline, of how much of the embedding’s extra prediction named measures reproduce, and which ones; (iv) a working practice in which every confirmatory analysis was fixed and hashed before it was run, with amendments recorded (Appendix D).

2 Data

Scenes. MODIS Terra corrected-reflectance true-colour images (NASA GIBS) of fixed $21^\circ \times 14^\circ$ boxes at $2100 \times 1400$ pixels (about 1.1 km per pixel), one box per region and day. Regions 1–3 are the three trade-wind regions of [3]; regions 4–6 (north-east Pacific, South Atlantic and South Indian Ocean trades) were chosen, and their seasons declared, before any of their data were fetched (Figure 1, Table 1). Sample A uses the image files shown to the labellers of [3]; samples B and C use the same GIBS layer requested directly, with the same box and size.

Outcome. The change in MODIS cloud fraction from Terra to Aqua (GIBS cloud-fraction layers, decoded with their published colormaps), over the pixels valid in both passes; box-days with less than half of the box valid in both are excluded. The outcome comes from the MODIS cloud product, not from the true-colour image the predictors are computed on. Comparator. Image statistics (grey-level percentiles, cloud brightness, image cloud cover, swath gaps), morning MODIS cloud fraction, the Aqua swath gap, day-of-year harmonics, region-season indicators, ERA5 [13, 14] 10 m and 100 m wind, direction, shear and wind variability at the Terra overpass, and 15 MERRA-2 [15] variables (Appendix E). Organization metrics. Thirteen metrics from the cloudmetrics package [16]: Iorg, SCAI, COP, open-sky fraction, object count, mean and maximum object length, mean eccentricity, orientation, fractal dimension, spectral slope, spectral anisotropy and median spectral length scale. They are computed two ways. Image mask (CONV): on a brightness mask of the true-colour scene (more than 30 grey levels above a local 10th-percentile background), on gap-free 256-pixel windows, median over windows; this version was fixed for earlier analyses. MODIS field (CONVM): on the MODIS Terra cloud-fraction field over the whole scene, as in [7], with cloud fraction $\geq 0.5$ as the mask and the spectral metrics on the field itself; its valid fraction and mean cloud fraction are included. Viewing geometry (GEOM). MODIS cloud fraction depends on view zenith [17], and Terra and Aqua see each box from different angles. Both satellites repeat their ground track every 16 days, so for a fixed box the geometry of both passes is set by the day’s phase in that cycle; region-by-phase indicators reproduce Terra’s box-mean view zenith with $R^2$ 0.995. The block is these indicators plus Terra’s box-mean view zenith, sun zenith, relative azimuth, glint angle and view-zenith spread from MOD08_D3. Aqua’s angles could not be retrieved (Appendix D) and enter through the phase indicators only. Cloud optical and cloud-top properties (CP). Morning Terra cloud optical thickness, cloud water path and daytime cloud-top height [18], by two processing routes of the same MODIS retrievals: (i) GIBS image layers decoded with their colormaps (quantized, all phases), and (ii) the MOD08_D3 collection 6.1 Level-3 daily product at $1^\circ$ (liquid phase for optical thickness and water path). Per box-day: mean, median, 90th percentile and retrieved fraction (12 variables each). The two agree closely for cloud-top height (Spearman 0.99 for the box mean) and moderately for optical thickness (0.70–0.79) and water path (0.63–0.70). High cloud. Share of retrieved cloud-top-height pixels above 3 km and above 5 km (decoded layer), and the Terra and Aqua ice retrieval fractions (MOD08_D3, MYD08_D3).

3 Methods

Embeddings. DINOv2 ViT-S/14 [19] (self-supervised on a curated set of web images) and CLIP ViT-B/16 [20] (contrastive training on web image-text pairs); no satellite-specific training or fine-tuning was done here. Each scene is cut into a $6 \times 4$ grid of 350-pixel tiles; tiles with at least 10% no-data pixels are dropped; each tile is resized to 224 pixels and its class-token embedding taken; the scene embedding is the median over valid tiles. Each embedding is reduced to 29 principal components fitted, without the outcome, on the Terra scenes of sample A; for sample A, refitting the components within each training fold gives the same gain (0.068 against 0.069). Scenes with fewer than four valid tiles are excluded. Gains. Ridge regression (penalty chosen by internal cross-validation over 30 values from $10^{-3}$ to $10^{4}$; median imputation and standardisation fitted within training folds), leave one calendar year out, fitted and evaluated separately in each sample. The gain of a block $X$ beyond a set $Z$ is $R^2(Z \cup X) - R^2(Z)$ on the pooled out-of-fold predictions. Uncertainty: year-block bootstrap with refitting (500 draws). A gain is called bounded when its point estimate and its 2.5% bootstrap quantile are both above zero; this is a predeclared bootstrap criterion used for the named readings, not a correction for multiple comparisons. Protocols and chronology. Each confirmatory analysis was written as a protocol naming its readings (for example, survives all controls or not beyond all controls, by a stated rule), run once on random-normal outcomes, then hashed (SHA-256) with its scripts and inputs before the real run (Appendix D); protocols were frozen locally, not registered publicly. The outcome, comparator, image-mask metrics and model were fixed while testing a structural image instrument (Appendix B), over about thirty earlier analyses that also used samples B and C. The embeddings, both cloud-property sets, the MODIS-field metrics, the geometry block and the high-cloud measures were introduced afterwards, so the comparison could not have been tuned in the embedding’s favour. Samples B and C were declared before their data were fetched, but they are not untouched for this paper. The primary reading is the gain beyond all controls; the other readings are named in Appendix D.

Design choices.

• Outcome. A same-day change between two overpasses of the same field removes slow seasonal and regional differences and is measured by a product independent of the image used for prediction. Because the two passes are different instruments with different viewing geometry, geometry is controlled explicitly.

• Comparator. It contains what is known without the image’s structure (cloud amount, wind, thermodynamics, season), so that a gain measures information in the arrangement and appearance of the scene.

• Ridge. The blocks are many and correlated; ridge is stable and its gains are easy to interpret. A boosted-tree comparison is reported (Appendix C); the claims here are for linear prediction.

• Leave one year out and year-block bootstrap. Neighbouring days share weather; holding out whole years prevents leakage between them, and resampling years makes the intervals reflect year-to-year variation.

• 29 components. Fixed in advance to match the structural instrument’s 29 coordinates. The gain rises to about 64 components and then levels off or falls (Appendix C).

• Nonlinear cover. Cubic splines of morning MODIS cloud fraction, image cover and their product, with knots at fixed percentiles of each variable (outcome-free).

• Noise baselines for naming. When named measures are used to reproduce an embedding’s contribution, the baseline is not zero, because any fitted embedding direction shares variance with them; a first dry run of the analysis in Section 4.4 showed 13–18% reproduction on pure noise, and readings were judged against noise outcomes thereafter.

• Tile median. Robust to swath gaps and edge tiles; the tile mean gives the same result (Appendix C).

4 Results

4.1 Embeddings against organization metrics

The comparator alone reaches out-of-fold $R^2$ of 0.37, 0.34 and 0.48 in samples A, B and C; the image-mask metrics raise this to 0.41, 0.40 and 0.53, and DINOv2 to 0.48, 0.48 and 0.57 (Table 1). Over the comparator, DINOv2 gains 0.102, 0.119 and 0.072, more than the image-mask metrics (0.046, 0.066, 0.052) and the MODIS-field metrics (0.058, 0.062, 0.065) in every sample (Table 2, Figure 2a). Beyond the image-mask metrics it adds 0.069, 0.073 and 0.045, each bounded: 12, 12 and 10 per cent less squared error. The MODIS-field metrics add 0.036, 0.030 and 0.036 beyond the image-mask ones; beyond the MODIS-field metrics DINOv2 adds 0.084, 0.089 and 0.049 (reading beyond modis field metrics), and beyond both 0.061, 0.062 and 0.039. The gain beyond the metrics is positive in every held-out year (11, 8 and 11 years) and in each of the six regions (0.038 to 0.105; Appendix C). In an earlier frozen run, a greyscale version of the embedding gained as much as the colour one (post hoc comparison).

CLIP adds 0.063, 0.065 and 0.050 beyond the image-mask metrics, bounded in samples A and C; in B its interval has a long lower tail (reading: second model replicates). Each embedding adds beyond the other (DINOv2 beyond CLIP 0.021, 0.024, 0.012, all bounded; CLIP beyond DINOv2 0.015, 0.016, 0.017, bounded in A and C).

4.2 Controls

Table 3 and Figure 3 add each control to the comparator and image-mask metrics.

Viewing geometry. Geometry by itself is a strong predictor of the outcome (0.110, 0.086, 0.044), as expected for a difference between two instruments’ cloud fractions. DINOv2 still adds 0.058, 0.067 and 0.041 beyond it (reading survives view geometry). Within box-days where Terra views the box closer to nadir (box-mean view zenith at or below the sample median) it adds 0.060, 0.049 and 0.040 (holds near nadir).

Nonlinear cover. With cover splines, DINOv2 adds 0.063, 0.071 and 0.043 (survives nonlinear cover).

Cloud properties. The cloud optical and cloud-top properties alone gain about as much as the embedding in samples A and B (decoded 0.068, 0.077; Level-3 0.072, 0.082) and less in C (0.019 and 0.047), so the two overlap in predictive value. Beyond the decoded properties DINOv2 adds 0.027, 0.029 and 0.033 (partly absorbed), and beyond the Level-3 properties 0.029, 0.031 and 0.026 (embedding beyond level-3 microphysics), all bounded.

All controls. Together, the MODIS-field metrics, geometry, cover splines, Level-3 cloud properties and high-cloud measures add 0.233, 0.215 and 0.148 beyond the comparator and image-mask metrics. Beyond all of them DINOv2 adds 0.020, 0.023 and 0.019, each bounded (primary reading: survives all controls), which is 5.5, 5.9 and 5.8 per cent less squared error. CLIP beyond all controls adds 0.010, 0.015 and 0.020, bounded only in C.

4.3 Shallow scenes and cloud-top height

In box-days with at most 10% of cloud tops above 3 km and at most 5% ice retrievals (about 40% of each sample), DINOv2 adds 0.032, 0.029 and 0.024 beyond the comparator and metrics, positive in all three samples but bounded only in C (reading not shown in shallow scenes; the subset holds about 40% of the rows and its intervals are up to twice as wide, and in A the lower bound is $-0.0008$); in the other box-days it adds 0.057, 0.072 and 0.053. Correlating the embedding’s extra out-of-fold prediction with each cloud-property, meteorology and label variable (descriptive), the strongest association in all three samples is with mean morning cloud-top height (Spearman +0.20, +0.18, +0.17), then the 90th percentile of cloud-top height and, negatively, optical thickness (Figure 4); no meteorological variable is among the 15 strongest. Part of the embedding’s advantage therefore comes from scenes containing deeper or higher cloud; for shallow cumulus alone the evidence neither establishes the gain nor rules it out.

4.4 What the embedding’s extra prediction is

To ask what the embedding adds in named terms, we took each scene’s out-of-fold extra prediction (with DINOv2 minus without, over comparator and image-mask metrics) and predicted it, out of fold, from named measures only: the structural reader’s 29 coordinates (Appendix B), the MODIS-field metrics, the cloud optical and cloud-top properties, the high-cloud measures, viewing geometry and cover splines. Any direction in the embedding, even one fitted to noise, is partly reproduced by measures that share variance with it, so each result is compared with the same procedure applied to 20 noise outcomes per sample.

All named measures together reproduce 0.29, 0.28 and 0.21 of the extra prediction ($R^2$), above the noise baseline’s 95th percentile in samples A and B (0.21, 0.22) but not in C (0.35); reading named reproduce part. The embedding itself reproduces only 0.53, 0.47 and 0.56 of its own out-of-fold extra prediction, so the named measures reach about half of what is reproducible. The reproduction comes from the cloud properties (0.20, 0.20, 0.08) and high-cloud measures (0.08, 0.08, 0.06). The structural reader reproduces 0.018, 0.044 and 0.067, below what it reproduces of noise-fitted directions (0.040, 0.047, 0.128), and the MODIS-field metrics behave alike; neither names the outcome-relevant part of the embedding, and neither names it better than the other (no clear difference). On the outcome, the named measures absorb 80, 80 and 50 per cent of the embedding’s gain; in a Shapley split over the six blocks, cloud properties and high cloud carry 0.038, 0.039 and 0.012 of it and the structural reader 0.004, 0.007 and 0.002. The named variables that track the extra prediction most, with the same sign in all three samples, are a lower liquid retrieval fraction, more ice retrievals, more cloud tops above 5 km and 3 km and higher cloud tops. What the embedding adds is largely the presence of higher, icier and less retrievable cloud in the morning scene, a matter of cloud type and vertical structure rather than horizontal arrangement.

4.5 Albedo change

On the CERES SSF1deg-Hour top-of-atmosphere albedo change between the overpasses (rows and comparator of an earlier stage, which include morning CERES albedo and cloud amounts), DINOv2 gains 0.071 and 0.067 in samples A and B. Viewing geometry adds nothing to albedo change ($-$0.001, $-$0.005), and DINOv2 adds 0.072 and 0.069 beyond it (reading albedo survives geometry). This second outcome, a radiative quantity from a different instrument, shows the same embedding advantage.

4.6 Pattern labels

The crowd labels of [3] (area shares of Sugar, Gravel, Fish and Flower) add nothing bounded to afternoon cloud change, either beyond the metrics (0.003) or beyond the metrics and the embedding ($-$0.001). The embedding also predicts how far the labellers agreed on a scene (Appendix F).

5 Discussion

Under linear prediction, a self-supervised image model trained on general web images carries more information about the three-hour change in cloud fraction over the trade-wind oceans than 13 established organization metrics, whether these are computed on an image mask or on the MODIS cloud field. The advantage holds in additional years and new regions, in every held-out year and region, with viewing geometry controlled, and for a second embedding trained with a different objective. Cloud optical and cloud-top properties overlap with much of it; with every control we could construct, 0.019–0.023 of $R^2$ (about 6% of the remaining squared error) remains.

The controls are generous to the metrics. The MODIS-field metrics and both cloud-property sets come from the same instrument whose afternoon-minus-morning difference is the outcome, while the embedding sees only a JPEG rendering of the morning scene. Cloud-top height and optical thickness are not only nuisance variables: how deep and how thick the clouds are is part of how a field is organized, so controlling for them can remove organization information together with any confound. The gain beyond all controls is therefore a conservative estimate of what the embedding adds, as is the 29-component reduction (Appendix C).

Two qualifications define the scope. First, the advantage is concentrated in scenes with some high or ice cloud; where the field is shallow cumulus alone it is about half as large, positive in all three samples but bounded only in the new regions, and the association with cloud-top height points the same way. Second, a boosted-tree model with the same inputs finds little beyond the cloud properties in two of three samples (Appendix C), so part of the embedding’s advantage under ridge may be nonlinear structure that the embedding makes linearly accessible, rather than information absent from the other variables.

For organization research the practical implication is a baseline. A metric proposed to capture an aspect of organization can be checked against a general-purpose embedding at small cost (about 0.2 s per scene on a laptop GPU), and an effect attributed to organization can be checked for whether an embedding, with geometry and cloud properties controlled, explains it better. Section 4.4 names about half of what the embedding adds: higher, icier and less retrievable cloud, which existing retrievals already describe; the horizontal-arrangement measures, including a structural reader, do not name it. Horizontal variance of cloud-top height, one of the four dimensions of [7], remains a candidate for part of the rest. Three further tests could strengthen or bound the result: a same-instrument outcome with fixed geometry (geostationary cloud fraction at the two overpass times), satellite-pretrained or fine-tuned embeddings, and a named description of what the embedding responds to in the scenes where its extra prediction is largest.

Limitations.

• Scenes are true-colour JPEG renderings, not calibrated radiances; sample A uses an earlier rendering of the same layer than B and C.

• The outcome is a difference between two instruments’ cloud-fraction products; geometry is controlled through Terra’s angles and the 16-day phase, not Aqua’s measured angles. A same-instrument outcome with fixed geometry (geostationary cloud fraction at the two times) was not tested.

• Fixed boxes mix local evolution with advection (wind is in the comparator).

• The Level-3 cloud properties are at $1^\circ$ and liquid-phase for optical thickness and water path; the decoded ones are quantized. Neither is a pixel-level Level-2 control.

• Two general-purpose embeddings were tested; satellite-pretrained models were not.

• The comparison design was refined on sample A, and samples B and C were used in earlier analyses (Methods).

• The associations in Section 4.3 are correlations of a prediction difference, not a decomposition.

Data and code availability

Imagery, cloud-fraction and cloud-property layers: NASA GIBS; MOD08_D3 and MYD08_D3 via NASA Harmony (LAADS DAAC). Wind: ERA5 through the Open-Meteo archive. Meteorology: MERRA-2. CERES: SSF1deg-Hour Edition 4A. Labels: [3] (not redistributed). Scripts, frozen protocols with their hashes and amendments, and the full working record (a laboratory book with every analysis, including withdrawn readings) are available from the author on request; the laboratory book is published alongside this paper.

Acknowledgements

We acknowledge the use of imagery and data from NASA’s Global Imagery Browse Services (GIBS), part of NASA’s Earth Science Data and Information System (ESDIS), and from the LAADS DAAC, the NASA Langley Atmospheric Science Data Center, the Copernicus Climate Change Service (ERA5) and the NASA Global Modeling and Assimilation Office (MERRA-2).

Appendices

Appendix A. Refereeing a disagreement with a third instrument

In an earlier analysis we compared the image cloud mask (I) and the MODIS cloud mask (M) with an independent lidar (CALIPSO, L) along its track, asking whether the scene-level disagreement between I and M that persists across the two overpasses is the error of one instrument. After controlling the track cloud amounts, the persistent disagreement correlated with the departures of both I and M from the lidar (Terra: rank partial correlations +0.065 and +0.069; reading both), which we read as a definitional difference shared by both. On a track pixel, $|I - M| \le |I - L| + |M - L|$, so scenes where I and M disagree more must show more departure from L in at least one of them, and the original analysis could not separate attribution of error from the amount of disagreement. Holding the track’s own I–M disagreement fixed removed the association (+0.008 and $-$0.022; reading neither; $n = 4,947$ box-days), and the earlier reading was withdrawn.

Appendix B. A structural instrument as a third arm

The comparison was first designed to test a deterministic structural reader of images: 29 named coordinates computed without training on gap-free 256-pixel greyscale windows (tone distribution, texture contrast and homogeneity at two scales, void and island geometry, orientation and drainage-like structure), median over windows. On afternoon cloud change it gains 0.048, 0.064 and 0.040 over the comparator, less than DINOv2, and adds 0.004, $-$0.001 and 0.005 beyond metrics and DINOv2 (not bounded). Its only bounded contribution beyond the embedding was on disagreement between cloud products in the new regions; against all 384 embedding dimensions read on the same windows it shrank to +0.004 and +0.001, so it largely reflects the 29-component reduction. With equal block sizes the structural reader accounts for 35–47% of the embedding’s variance, with leading out-of-fold canonical correlations of 0.91–0.94. Its coordinates are named and inspectable, but the embedding’s leading components are not where the outcome signal lies (5 components give 0.022–0.033 of gain; Appendix C), and the reader does not name the embedding’s outcome-relevant extra prediction either (Section 4.4). In this domain the named measures that explain the embedding are cloud retrievals, not the structural reader.

Appendix C. Robustness and secondary analyses

Computed on the rows of Table 2; not frozen except the boosted-tree rows (frozen, without a reading). Gain of DINOv2 beyond comparator and image-mask metrics.

Per-year and per-region values are the gain within each subset of the pooled out-of-fold predictions. Boosted trees: 300 iterations, learning rate 0.05, 15 leaves, at least 40 rows per leaf, predictions averaged over three seeds; the gain exceeds the noise floor only in sample C.

Appendix D. Protocols

Readings named in advance: second model (second model replicates); cloud properties decoded (partly absorbed) and Level-3 (embedding beyond level-3 microphysics); viewing geometry (survives view geometry); near nadir (holds near nadir); MODIS-field metrics (beyond modis field metrics); shallow scenes (not shown in shallow scenes); cover splines (survives nonlinear cover); all controls, primary (survives all controls); albedo with geometry (albedo survives geometry); naming the extra prediction (named reproduce part, no clear difference). The headline gain beyond the image-mask metrics was computed in the frozen head-to-head run as a reported gain; that protocol’s named readings concerned the structural instrument.

In the frozen run of the last protocol, every arm containing the cover splines produced bootstrap intervals of order $10^{1}$ to $10^{7}$ in two samples: knots fixed on $[0,1]$ left near-constant basis columns (morning cover never falls below about 0.2), which exploded under standardisation. A declared amendment moved the knots to outcome-free percentiles and dropped near-constant columns, checked on noise outcomes before the run; only the two readings containing splines were re-run. Aqua’s MYD08_D3 angles could not be retrieved: the subsetting service’s variable names for them do not match the file contents, and it returned files without them. Entry numbers refer to the laboratory book released with the code.

Appendix E. Comparator, geometry and high-cloud variables

Image: cloud cover from the image mask, grey-level mean, standard deviation and 10th, 25th, 50th, 75th and 90th percentiles, cloud brightness, swath-gap fraction and position. MODIS: morning cloud fraction over the common area, common-area fraction, Aqua swath-gap fraction and position. Calendar: day-of-year sine and cosine, region-season indicators. ERA5: 10 m and 100 m wind components and speeds, shear, 10 m speed variability, 10 m direction sine and cosine. MERRA-2: SST, EIS, LTS, EIS variability, 700 hPa relative and specific humidity, 700 and 500 hPa vertical velocity, total column water vapour, 2 m air minus sea-surface temperature, 2 m relative and specific humidity, SST advection, boundary-layer top, 850 hPa temperature. Geometry: region $\times$ 16-day phase indicators (48), Terra box-mean view zenith, sun zenith, cosine and sine of relative azimuth, glint angle, view-zenith standard deviation. High cloud: share of cloud tops above 3 km and 5 km, Terra and Aqua ice retrieval fraction.

Appendix F. Labeller agreement

On scenes with at least three labellers, the same embedding predicts mean pairwise box overlap between labellers (gain 0.039 on the comparator of an earlier stage). The endpoint’s repeatability between the Terra and Aqua images of the same day is only 0.23–0.28, which limits what any predictor can achieve, so this gain is a sizeable share of the predictable part.

References