Site edition 2026-11-03. Local file paths in the record refer to the working repository; data access credentials are not part of this edition.

Cloud organization lab book

1. What this pass set out to settle

Whether the Sugar/Flower/Fish/Gravel cloud-organization data can carry a declared study: are the labels and images obtainable, is the human-disagreement endpoint a stable property of a scene, can the frozen measurement release read these tiles, and how do its readings respond to processing that should not change the clouds.

2. Data obtained (Gate 2)

• Labels: public GitHub repo raspstephan/sugar-flower-fish-or-gravel at HEAD 938affc3. No licence file: research use assumed until the authors confirm. The label file contains labeler user names and hashed IPs: kept local, not for publication.
• Reproduces the paper: 9,744 scenes, 30,310 classifications, 67 labelers, 3,004 distinct days, 2007 to 2017, 3 regions, 5 region-seasons of about 1,950 scenes each.
• Images: the exact 2100 x 1400 JPEGs the labelers saw are still served by the Zooniverse CDN (12 of 12 fetched). Fresh NASA Worldview re-renders are similar but not identical (correlation 0.948 to 0.979; mean absolute grey difference 7.1 to 11.5; swath-gap pixels 8.5% to 14.3%): reprocessing since 2018, so the labelers' copies are the source of record for any claim about the labels.
• Radiation endpoint (CERES SSF Aqua FM3 Edition 4A, collection C2724240405-LARC_CLOUD): listed in NASA's public catalog, 24 hourly files of about 63 MB for the sample date 2007-03-07. Download needs a free NASA Earthdata login, which has not been created. PENDING.

Top: tiles as the labelers saw them. Bottom: 2026 Worldview re-render. Black wedges are gaps between satellite swaths; grey haze is sun glint.

3. Independent units (Gate 3)

Terra and Aqua tiles of the same date and region show the same cloud field about 3 hours apart, so the unit is the day-region: 4,987 units in all, 4,546 with at least 3 labelers. Adjacent days are also autocorrelated; a study should block by week or month within region-season. Labeler identity is a nuisance: each scene's disagreement depends on which 3 or 4 of the 67 people saw it.

4. The disagreement endpoint (Gate 4)

Two candidate scene-level measures, computed from labels only: pairwise box IoU per pattern (the Rasp et al. convention) and pixel split D (mean of 4p(1-p) over marked pixels). Scenes with at least 3 labelers: median IoU 0.124 (IQR 0.049 to 0.242); median D 0.222 (IQR 0.200 to 0.262).

Reliability check: if disagreement belongs to the scene, the Terra and Aqua views of one field should agree on it.

5. Measurement identity (Gate 1)

• Frozen release stk-measurement-repair/1.0.0 rebuilt in vendor/kernel_1.0.0/ from the byte-identical snapshots in the research-context package (the release folder itself has moved). All three hashes match the recorded release values. Used through measure() only; no blocked legacy coordinate requested.
• Runtime: Python 3.14.3, NumPy 2.4.2, SciPy 1.17.0, scikit-image 0.26.0, OpenCV 4.13.0, Darwin arm64. This matches the release's recorded test environment.
• Declared representation for this pass: labelers' JPEG, PIL greyscale, one 368 x 368 px window fully inside the gap-free area (mask eroded 8 px), nearest the tile centre. 368 px is about 400 km: the largest square that fits every sample tile. That is small next to Fish structures (up to about 1,000 km); see Gate 5.

6. Processing response (Gate 5)

Each cell is the median over 12 tiles of |reading under the change minus base reading|, in units of the between-field standard deviation (6 independent fields, so the scale itself is rough). Green under 0.25, amber under 0.5, red at or above 0.5. with_gaps is a contamination check, not a control. terra_vs_aqua is the real cloud field 3 hours later.

• framing-sensitive (8): delta_y, x_p, theta, rc_drc, rc_dr, blob_size_cv, drain_confidence, gap_theta
• processing-sensitive (3): d_s, ds_high, nils
• scale-bound (8): g4_slope, blob_count, n_islands, island_size_cv, island_spread, g1_centroid_path_length, g3_unsigned_turn_variance, g3_turn_mean_crossing_density
• stable in this envelope (31): delta_x, r_v, mu, rho_r, sdi, g2_void, g2_nvoids, g2_AR, g2_cut, g4_oent, sci, tex_contrast_fine, tex_entropy_fine, tex_homogeneity_fine, tex_energy_fine, tex_contrast_coarse, tex_entropy_coarse, tex_homogeneity_coarse, tex_energy_coarse, blob_density, blob_size_mean, resistance_mean, ds_low, ds_mid, edge_density, ler, gap_rv, tonal_shadow, tonal_mid, tonal_high, g1_centroid_turn_mean

7. Where this leaves the candidate

• Gate 1: PARTIAL. Identity, hashes and runtime are settled; the scene representation is not (window size and aggregation).
• Gate 2: SUPPORTED for labels and imagery (licence unknown); PENDING for CERES (needs Earthdata login).
• Gate 3: units counted (4,546 day-regions with at least 3 labelers); precision for a declared endpoint still to estimate.
• Gate 4: the disagreement endpoint (C1) has low but non-zero repeatability; box IoU is usable, pixel split is not. The radiation endpoint (C2) is now the stronger primary candidate.
• Gate 5: PARTIAL. Robust to recompression, reprocessing and contrast for most coordinates; gaps, scale and framing need the representation fixed first.

8. Next steps (as planned at entry 1; see entry 2)

1. Define a gap-aware multi-window scene representation; rerun the Gate 5 controls on it (no outcomes).
2. Create an Earthdata login; pull CERES SSF footprints for the sample dates; check overpass matching and coverage.
3. Rebuild the comparator: cloud fraction plus the Janssens et al. organization metrics on the same windows.
4. Then freeze subset, endpoint, splits (day-region, blocked by month) and smallest worthwhile gain, before any outcome is read.

Entry 2: afternoon cloud change, declared test

9. Change of primary endpoint

CERES albedo needs an Earthdata login, which was not available at this stage. The endpoint was moved to one that the images themselves support and that the literature treats as physically meaningful: the change in cloud cover between the Terra pass (about 10:30 local) and the Aqua pass (about 13:30) over the same region and day. Vial, Vogel and Schulz (2021) report that the four patterns have distinct daily cycles (Fish peaking near noon, Sugar later in the day) and that pattern frequency drives the daily cloudiness cycle. Albedo is close to additive over pixels (Eastman et al. 2024), so cloud cover is the first-order radiative quantity available without CERES.

10. Scene representation and the swath gaps

• Gap mask: connected regions of grey < 3 larger than 20,000 px (the black swath wedges); dark ocean specks are not gaps. Valid area = not gap, eroded 8 px.
• Cloud mask: local background as the 10th percentile over 151 px (gaps filled by normalised Gaussian fill first, so the wedge edge does not create a false cloud band); cloud where grey exceeds background by more than 30.
• Windows of 256 px on a fixed grid, kept only when fully inside valid pixels, at most 16 per scene, evenly spaced. Kernel and expert metrics are read per window and summarised by the median. Scenes with fewer than 4 windows are excluded.
• dCF uses only pixels valid in both images; pairs sharing less than 50% valid area are excluded. Gap size and position for both passes are in the comparator.
• Kernel subset rule (images only, no outcomes): keep a coordinate when its median change under JPEG q40, the Worldview re-render and a grid shift is below 0.5 of its across-scene SD. Kept 29 of 50: k_r_v, k_mu, k_rho_r, k_theta, k_g2_void, k_g2_nvoids, k_g2_AR, k_g2_cut, k_g4_oent, k_g4_slope, k_tex_entropy_fine, k_tex_homogeneity_fine, k_tex_energy_fine, k_tex_contrast_coarse, k_tex_entropy_coarse, k_tex_homogeneity_coarse, k_tex_energy_coarse, k_blob_count, k_blob_density, k_blob_size_mean, k_island_spread, k_drain_confidence, k_edge_density, k_nils, k_gap_theta, k_tonal_shadow, k_tonal_mid, k_tonal_high, k_g1_centroid_path_length.

11. Extraction

9,744 scenes read (all labeler images); 55 unreadable (45 with no usable window, 10 truncated JPEGs) and removed by the 4-window rule. P1 pairs: 4,757 day-regions with both passes, 4,663 after the common-area rule. P2 scenes: 6,784.

12. Results (declared)

Leave-one-year-out (11 years), ridge with internal alpha selection; dR2 = R2(B + kernel) − R2(B); 95% interval from 2,000 bootstrap resamples of years; null from within-year permutation of the kernel block (200 for P1, 50 for the secondaries). A = conventional statistics, B = A + expert organization metrics. Decision rule: dR2 ≥ 0.02 with lower bound > 0 is SCOPED_CONTRIBUTION_DEMONSTRATED.

In P1 the real dR2 exceeded every permuted value. A + kernel (no expert metrics) reached R2 0.429, above B (0.369).

13. Checks on the P1 result (descriptive, not pre-declared)

• Single influential pair. One pair has dCF = +0.325 (Region1 DJF, 2017-02-22, a real afternoon increase; cloud cover 0.27 to 0.60). Without it: dR2 0.054, 95% CI [0.025, 0.095].
• Years. Gain is positive in all 11 held-out years; largest in 2007 (0.195) and 2017 (0.217). Without both years: dR2 0.032.
• Model form. Gradient boosting on B alone reaches R2 0.462, higher than ridge B + kernel (0.441). Over that stronger baseline the kernel adds 0.025 [0.002, 0.049]. Part of the ridge gain may therefore be non-linear structure in the conventional features that the kernel expresses linearly; the gain that survives a flexible baseline is about 0.025, with a lower bound just above zero.
• Rank agreement. Spearman between dCF and prediction: 0.623 (B), 0.652 (B + kernel).

Gain by region and season:

Gain by swath-gap position (larger gap of the two passes; every pair has a gap of at least 1%):

The gain is similar for central and edge gaps (0.066 and 0.076), so it does not appear to come from the gap's position.

Which coordinates carry it (drop one coordinate, loss in leave-one-year-out R2; standardized coefficient from a full fit):

No single coordinate carries more than 0.007; the gain is spread. The two largest are orientation readings (k_theta, k_gap_theta). Whether they read wind-aligned cloud streets (physical) or view-geometry streaking near the swath edge is open.

14. Status

• P1 (afternoon cloud change): SCOPED_CONTRIBUTION_DEMONSTRATED under the declared rule (ridge). Over a gradient-boosting baseline the added value is smaller (about 0.025, lower bound 0.002). Holds without the influential pair and without the two strongest years.
• P1 secondary (dGrey): DETECTABLE_BELOW_WORTHWHILE.
• P2 (labeler agreement): NOT_SUPPORTED_IN_TESTED_SCOPE with the declared comparator; DETECTABLE_BELOW_WORTHWHILE with pattern shares added. Overall R2 for labeler agreement stays below 0.08; the endpoint itself is weakly repeatable (Terra/Aqua 0.23 to 0.28), which bounds what any reader can show.
• Scope: one dataset (Rasp et al. 2020 SFFG, 2007-2017 winter-dominated, three Atlantic/Pacific regions), JPEG Worldview imagery, one window size, one cloud-mask definition. dCF comes from the same image pipeline as the predictors; it is not an independent radiative measurement.

15. Openings (not started)

1. Independent outcome: CERES SSF albedo or MODIS cloud-product cloud fraction for the same pairs (needs an Earthdata login).
2. Independent bearing on the orientation coordinates: ERA5 low-level wind direction per day-region as an added comparator. If the orientation gain persists over wind, it is not a wind proxy; if it vanishes, the kernel is reading wind from the clouds.
3. Untouched-data replication with the exact frozen specification: other years or regions from Worldview (2018 onward, or the same regions outside the labelled seasons), images only, no labels needed for P1.
4. Non-linear comparator declared up front (gradient boosting as primary), since it is the stronger baseline.

Entry 3: is the gain wind?

16. Question

The largest P1 contributors were k_theta and k_gap_theta, which measure how concentrated image-gradient directions are (anisotropy strength, not a heading). Cloud streets and their alignment depend on low-level wind, so the kernel might be reading wind from the clouds. The addendum adds ERA5 wind at Terra time (u, v and speed at 10 m and 100 m, speed shear, spread of speed across the region, direction as sine and cosine; 3 h mean over a 3×3 grid per region) to the comparator and repeats the declared P1 test unchanged.

17. Result (declared)

Secondary (gradient boosting): B + wind 0.474, + kernel 0.493; dR2 0.019, 95% CI [-0.003, 0.040]. Over the flexible baseline with wind, the gain is below 0.02 and its interval includes zero.

By region and season (ridge):

18. Orientation coordinates and wind (descriptive)

k_theta rises with wind speed (ρ 0.25), as cloud streets would predict; the expert anisotropy metric does not (ρ −0.07). With wind in the comparator, the drop-one value of k_theta falls from 0.007 to 0.0020 and of k_gap_theta from 0.005 to 0.0025; k_g4_oent keeps 0.0073. So the orientation readings carry some wind information, but the kernel's gain as a whole is mostly not wind.

19. Status after the wind addendum

• P1 ridge result holds over wind: SCOPED_CONTRIBUTION_DEMONSTRATED (dR2 0.061).
• Over the strongest baseline tested (gradient boosting with wind) the added value is 0.019 with an interval crossing zero: in this operating window the kernel's contribution is linear-model-scale, and a flexible model of conventional features plus wind recovers most of it.
• Remaining openings: an independent outcome (CERES or MODIS cloud fraction); untouched-years replication with the frozen specification; other meteorology (stability, humidity, SST) as comparator.

Entry 4: untouched years

20. Data

7,260 images downloaded (all requested). 52 could not be read by the scene code (no 256 px window fully inside valid pixels), the same kind of loss as amendment A2. 3630 Terra/Aqua pairs; after the declared P1 filters (common area ≥ 50%, Aqua ≥ 4 windows) 3558 pairs over 8 years.

21. Result (declared primary and secondary)

Transfer (models fit on 2007-2017 only, applied unchanged to the untouched years; year bootstrap):

By region and season (primary, ridge):

22. Reading

• The P1 contribution replicates in years the instrument and the test had not seen, in both blocks separately, and in every region-season.
• Over the flexible baseline the untouched-year gain is 0.026 [0.010, 0.045] (2007-2017: 0.025). In 2018-2021 alone it is 0.004 with an interval including zero; in 2003-2006 it is 0.065.
• With wind in the comparator the ridge gain is 0.017 [0.011, 0.024] (47% retained; 85% in 2007-2017).
• Transfer is the strictest form: coefficients learned on 2007-2017 add 0.051 (ridge) and 0.047 (boosting) on the untouched years, both with intervals above zero.

Entry 5: an independent outcome (CERES)

23. Question

P1's outcome (dCF) comes from the same images the kernel reads. CERES is a separate instrument on each satellite: a broadband radiometer measuring reflected sunlight. The addendum asks whether the kernel's Terra-time reading predicts the CERES-measured afternoon change in top-of-atmosphere albedo (dAlb, Aqua minus Terra over 1° cells seen by both passes), beyond the entry-2 comparator plus CERES's own morning state (all-sky and clear-sky albedo, cloud amount, low cloud, solar geometry, common-cell share). Model, folds, bootstrap, permutation null and decision rule are unchanged.

24. Alignment and validity

The day offset between image date and CERES day was chosen per region and satellite on 2007 levels only (Spearman, image cloud fraction vs CERES total cloud), then frozen:

Same-day everywhere except Region 2 Terra (−1 day), the box next to the date line, where the morning pass falls on the previous GMT day. Agreement between image-derived and CERES quantities on the analysis set (4577 pairs):

25. Result (declared)

By region and season (primary, ridge):

26. Reading

• On an outcome measured by a different instrument, the kernel adds a detectable amount that is positive in every region-season, below the 0.02 worthwhile threshold. The comparator here is stronger than in P1 (R2 0.550 vs 0.369), because CERES's own morning state is in it.
• Over the flexible baseline the albedo gain is 0.004 [-0.008, 0.014]: not distinguishable from zero.
• Low-cloud change is where the ridge gain clears the threshold (0.022 [0.012, 0.033]); total CERES cloud change does not (0.006). The images resolve shallow cumulus at 250 m-1 km; CERES aggregates to 1°, so some of the P1 signal sits at scales CERES averages out.
• Adding ERA5 wind and CERES inversion strength (EIS) leaves 0.013 [0.004, 0.022].
• Untouched years (frozen alignment): albedo-change gain 0.011 [-0.000, 0.022], decision NOT_SUPPORTED_IN_TESTED_SCOPE.

27. Status

• Image-derived afternoon change (P1): SCOPED in 2007-2017 (0.072), replicated SCOPED in untouched years (entry 4), mostly not wind.
• Independent radiometric outcome (CERES albedo): DETECTABLE_BELOW_WORTHWHILE with a linear model in 2007-2017; in the untouched years a similar point estimate (0.011) whose interval reaches zero, so NOT_SUPPORTED there; not separable from zero over a flexible baseline in either period. CERES low-cloud change falls from 0.022 to 0.006 in the untouched years. This marks the operating window: the kernel's added information is largest at image resolution and for cloud cover, smaller once outcomes are averaged to 1° and expressed as broadband albedo.

Entry 6: algorithm or grid?

28. Question

The kernel's gain is 0.072 on the image-derived cloud change (P1) and about 0.006 on CERES cloud change (entry 5). Two things differ between those outcomes: the cloud algorithm (P1 thresholds visible brightness in the true-colour image; the MODIS and CERES cloud masks use infrared and visible tests) and the sampling (all common pixels vs 1° cells with footprints from both passes). The addendum crosses the two: each algorithm on each sampling, same pairs, same machinery. On the CERES-cell outcomes the comparator also carries the CERES morning state from entry 5.

29. Agreement between outcomes

Restricting the image cloud change to CERES cells barely changes it (ρ 0.99). Image and MODIS cloud change agree at ρ 0.65; MODIS on CERES cells agrees with CERES cloud change at ρ 0.75.

30. Result (declared)

MODIS cloud change by region and season (ridge):

31. Reading

• Sampling does not carry the drop. The image cloud change keeps its gain on CERES cells (2007-2017: 0.055, against 0.064 on the full area for the same pairs).
• The algorithm does. With the MODIS cloud mask the gain falls to 0.018 on the full area and to about zero on CERES cells (-0.002), matching CERES cloud change (0.002).
• Under the readings declared in advance this is the case "dCFm well below P1" for 2007-2017 (about a quarter of the P1 gain): a large part of the P1 gain is tied to the visible-threshold cloud mask that the outcome shares with the images the kernel reads. In the untouched years the MODIS gain is 0.024 against 0.037 for P1, about two-thirds, and clears the worthwhile threshold.
• Over a flexible baseline the MODIS gain is 0.008 [-0.001, 0.017] (2007-2017) and 0.014 [-0.001, 0.031] (untouched); both intervals reach zero.
• The MODIS gain on the full area disappears on CERES cells once CERES morning state is in the comparator; the two changes (sampling and comparator) move together in S2, so this design does not separate them for the MODIS outcome.

32. Status

• On cloud change measured by an algorithm independent of the image threshold (MODIS, IR + visible), the kernel adds 0.018 (2007-2017, detectable) and 0.024 (untouched years, scoped), linear-model scale.
• The larger P1 figure (0.072) includes a component specific to the visible-threshold cloud mask. Candidate content of that component, untested: thin or broken cloud near the threshold, sun-glint and aerosol brightness, which a texture reading of the same image can see.
• CERES's small result (entry 5) is explained by the cloud algorithm, not by the 1° grid.

Entry 7: what is in the threshold component?

33. Question

Entry 6 left a component of the P1 gain tied to the visible-threshold cloud mask. Here the outcome is that component directly, Tthr = image cloud change minus MODIS cloud change on the same area (2 km grid), split into image-only cloud (dFP) and MODIS-only cloud (dFN), Tthr = dFP − dFN. Four candidate contents, each with a marker that does not use the image threshold: sun glint (geometry only), thin cloud (MODIS optical thickness < 3.6), broken-cloud edges (MODIS partly-cloudy retrievals), haze (MODIS aerosol optical depth). Each marker block enters the comparator as Terra value, Aqua value and their difference.

34. Descriptives and marker correlations

Tthr +0.0084 (sd 0.0368) in 2007-2017, +0.0092 (sd 0.0348) in the untouched years: on average the image threshold registers slightly more afternoon cloud increase than MODIS, mostly through image-only cloud (dFP +0.0070 (sd 0.0104)).

The threshold component tracks change in partly-cloudy retrievals (+), change in thin-cloud fraction (−) and glint fraction (−).

35. Result (declared)

36. Reading

• Sun glint does not account for the threshold component (retains 91% / 77%), and does not change the headline: with the glint block in the comparator the P1 gain is 0.073 against 0.072 (2007-2017) and 0.036 against 0.037 (untouched). Glint geometry is known before the afternoon pass, so this is a fair comparator.
• Haze does not account for it (98% / 116%).
• Thin cloud: not in 2007-2017 (82%); accounts in the untouched years (48%).
• Broken-cloud edges: partly in 2007-2017 (75%); not in the untouched years (96%).
• All four blocks together: 81% retained (2007-2017, not) and 63% (untouched, partly). Most of the kernel's gain on the threshold component is not carried by these four markers as measured here.
• T2: the gain sits in MODIS-only cloud in 2007-2017 (dFN 0.026, scoped; dFP 0.014, not supported) and in image-only cloud in the untouched years (dFP 0.032, scoped; dFN 0.013, detectable). The side differs between periods.
• T4 with all blocks is attribution, not a fair comparator: the Aqua markers are afternoon MODIS cloud properties, and the base R2 rises from 0.37 to 0.73. Over that base the kernel keeps 0.025 (35%) in 2007-2017 and 0.007 (19%) in the untouched years.

37. Status

• The headline P1 gain is not a sun-glint effect (fair comparator, both periods).
• The threshold-specific component is itself predicted by the kernel at about 0.02 (linear scale, both periods). Glint and haze do not account for that; thin and broken cloud account for parts of it, differently in each period; together the four markers leave 63-81% of it.
• Most of the P1 gain overlaps with what afternoon MODIS cloud properties record about the change (T4 attribution).
• Untested: other candidate contents of the threshold component (cloud-top brightness and texture beyond optical thickness, cloud shadow, the 2 km resampling of the threshold mask).

Entry 8: timing correction, and the large-scale state

38. Timing correction (Region2 Terra pass)

While designing this addendum: the ERA5 wind features (entries 3 and 4) placed the Region2 Terra pass at 23:12 UTC on the image date, about 24 h after the actual morning pass and after the Aqua pass. The actual pass is 23:12 UTC on the previous day, which the CERES alignment had found empirically (Region2_Terra offset −1). Regions 1 and 3 were correct. Fixed in scripts/passtime.py; superseded outputs kept in out/superseded_wind_timing/; PROTOCOL_AMENDMENTS.md.

The 2007-2017 wind result is unchanged in substance (85% of the gain retained). In the untouched years the gain over wind falls below the worthwhile threshold: 47% of the 0.037 retained, not 64%. Primary results without wind (P1, R1, CERES outcomes) are unaffected.

39. Question

On MODIS cloud change the kernel adds 0.018 (2007-2017) and 0.024 (untouched years). Does that survive the standard large-scale controls on low cloud at the time of the morning image: stability (EIS, LTS), free-tropospheric and near-surface humidity, SST and air-sea temperature difference, subsidence, cold-air advection, boundary-layer depth? MERRA-2 cloud variables and afternoon meteorology excluded.

40. Meteorology and MODIS cloud change

Each meteorological variable alone is weakly related to the change (|ρ| ≤ 0.17).

41. Result (declared)

42. Reading

• On independently detected cloud, the kernel's gain largely survives the large-scale state at the morning pass: 80% and 84% retained, about 0.014-0.020 on the linear scale. Over a flexible baseline with the same variables it is -0.004 and 0.004; both intervals reach zero.
• Adding the meteorology after wind slightly raises the kernel's gain on MODIS cloud change in both periods (0.013 → 0.014; 0.018 → 0.020): the retained fractions move by a few points with the comparator and are read with that spread.
• On the image-derived headline P1 the picture differs by period: 2007-2017 keeps 0.054 (75%), still scoped; the untouched years keep 0.011 (31%), detectable. In the untouched years most of the P1 gain is shared with wind and the large-scale state.

43. Status

• On cloud change measured by an algorithm independent of the image threshold, the kernel adds about 0.014-0.020 (linear scale) over a comparator that includes the cloud statistics, ERA5 wind and the standard large-scale controls on low cloud at the time of the morning image. Detectable in 2007-2017, scoped in the untouched years; above every within-year permutation in both.
• Over a flexible baseline that gain is not resolved from zero.
• Untouched-years figures over wind were corrected downward (section 38).

Entry 9: three new regions

44. Question and design

Does the kernel's gain on MODIS cloud change over cloud statistics, wind and the large-scale state (entry 8: 0.014 in 2007-2017, 0.020 in the untouched years, both in the three SFFG regions) appear in regions never used for anything? Three trade-cumulus boxes of the same size: NE Pacific east of Hawaii (Region4, DJF + MAM), South Atlantic SE trades (Region5, JJA + SON), South Indian Ocean SE trades (Region6, JJA + SON); every day 2007-2017. Nothing re-selected: the 29 coordinates, comparators, exclusions, outcome definitions and machinery are those of entries 2-8. A sensor swap (VIIRS) was not possible: GIBS has no non-MODIS true-colour layer from a morning pass.

Sample: 6031 pairs, 5978 with the MODIS outcome (Region4 1981, Region5 1995, Region6 2002). MODIS cloud change mean -0.024 (sd 0.047); Spearman between image and MODIS cloud change 0.77. 0 missing wind or meteorology rows.

45. Result (declared)

46. By region-season (descriptive)

47. Transfer (secondary)

Models fitted on the original regions 2007-2017 (MODIS outcome, n = 4659, comparator with wind and meteorology, no region dummies), applied unchanged to the new regions; interval from a year bootstrap.

48. Reading

• Over cloud statistics alone the kernel's gain on MODIS cloud change in the new regions is 0.019, the same size as in the SFFG regions (0.018 / 0.024). Wind and meteorology take more of it here: 0.007 retained (37%), against 80% and 84% in the SFFG regions.
• In the new regions the gain is resolved from zero under both models: ridge 0.007, boosting 0.008. In the SFFG regions the boosting interval reached zero.
• The image-derived headline P1 is scoped in the new regions over cloud statistics (0.027) and detectable over wind and meteorology (0.017).
• Five of six region-seasons are positive on the primary; Region5 SON is near zero.
• Carried across regions without refitting, the comparator alone explains little of the new regions' variance under ridge (0.098); with the kernel 0.191. Under boosting the kernel adds 0.014. Part of the ridge figure reflects the linear comparator's poor transfer, not only the kernel.

49. Status

• The kernel's contribution to independently detected afternoon cloud change, beyond cloud statistics, wind and the large-scale state at the morning pass, is present outside the regions it was developed in: 0.007 [0.003, 0.011] (linear) and 0.008 (boosting), below the 0.02 worthwhile threshold.
• Range across the three tests on this comparator: 0.007 (new regions), 0.014 (SFFG 2007-2017), 0.020 (SFFG untouched years).

Entry 10: external review, checked against the code

50. Points and checks

51. Effect size

N1: ridge gain 0.007 over comparator R² 0.528 is 1.5% less squared prediction error, about 0.7% less RMSE.

52. Status after review

• Within this pipeline, the frozen kernel subset shows small out-of-sample gains on MODIS cloud change over the implemented comparator in all three samples under ridge, and under boosting in the new regions. Its added value beyond a fully specified conventional morning-state model is not established.
• Entry 6 status line ("explained by the cloud algorithm, not by the 1° grid") is qualified: for the MODIS outcome the design does not separate algorithm from sampling (as entry 6 section text states).
• Write-up revised accordingly (WRITEUP.html, section 7 lists the limits).

Entry 11: bounded audit, and a seek test on disagreement

53. AU1: does the MODIS cloud-change gain survive stricter procedures?

Outcome dCFm; comparator C = B + P1x + Mc + wind + MERRA-2. Conditional permutation keeps the part of each kernel coordinate that C explains (median R² of kernel on C 0.77 to 0.84) and shuffles the remainder within year. Refit bootstrap resamples years and refits. AU3 adds spline terms of morning MODIS and image cover and their product. Declared readings use AU1, AU2 and AU3; the year-by-year (cluster t) and year-blocked tuning intervals are reported only.

Attribution (ridge; generic = k_tex, k_tonal, k_blob, k_edge_density, 14; distinctive = the other 15):

54. Post hoc: why does the refit bootstrap fail?

Hypothesis written before running s37: when a year is drawn twice, generalized cross-validation sees each row's duplicate and picks too small a penalty, producing the long left tail (to -0.106 in the SFFG sample). s37 repeats the first 200 AU2 resamples with the same seeds and records the chosen penalties, then repeats them with year-blocked tuning, which cannot see duplicates.

The hypothesis is not supported: year-blocked tuning gives intervals of the same width, and penalties are not at the grid floor. The width comes from resampling 8 to 11 years (draws with 5 or 6 distinct years carry the tails). The gain has the same sign in 9 of 11, 8 of 8 and 10 of 11 years; its size is not pinned under year resampling. The AU2 narrowing stands as declared.

55. SK1: does the kernel see ambiguity?

Instrument disagreement per pass, D_inst = mean |image cloud mask - MODIS cloud fraction| on the common area. Ambiguity score = boosting fitted once on labelled scenes (n 6784), kernel only, target 1 - IoU, then frozen. Comparator Cs includes image and MODIS cover and their interactions.

Declared reading: DIRECT_ONLY (SK1b passes in r1, n1; SK1a direction interval includes 0 in both untouched samples).

SK1c, labelled scenes (n 6554): Spearman of human disagreement with D_inst at the same pass 0.072 [0.0405, 0.1004] raw, 0.013 [-0.0128, 0.0374] after each comparator.

Design property found in the results: the comparator explains 93 to 97% of D_inst. Most per-pass disagreement is the difference in mean cover between the two masks, and both covers are in the comparator. SK1a and SK1b therefore address the 3 to 7% remainder (spatially cancelling disagreement), not the level.

56. Use readout

Share of the 10% of passes flagged by each predictor that fall in the true top 10% of D_inst (chance 0.10), out of fold:

57. SK1d (exploratory, labelled scenes already seen in P2)

Boosting gain on human disagreement over B + n_class (R² 0.033): all 29 coordinates 0.0316; generic 14 0.0174; distinctive 15 0.0355.

New-region passes the score ranks most (top) and least (bottom) ambiguous:

Top row: scenes with several cloud forms in one frame (sheets, streets, thin veils, glint). Bottom row: single fields of discrete cumulus clusters.

58. Status

• MODIS cloud-change gain (ridge): survives the conditional permutation and the nonlinear morning state in all three samples; same sign in 27 of 30 years; refit year-bootstrap intervals include 0 (declared NARROWED, AU2). Boosting in the new regions: same pattern.
• Instrument disagreement: the kernel adds held-out information beyond cover in both untouched samples under ridge (0.0033, 0.0027; conditional p 0.005); boosting does not. The human-trained ambiguity score adds nothing beyond cover.
• Human and instrument disagreement coincide through cover only. Human disagreement is carried more by the distinctive coordinates than by the generic ones (exploratory).

Entry 12: is disagreement a mixture? (SK2)

59. Design

Hypothesis from the SK1 contact sheet: observers disagree where a scene holds several forms of organization at once. New representation from per-window readings (s38, 14,297 scenes, 4 unreadable): the across-window IQR of the 29 kernel coordinates and k_mix, the mean pairwise distance between a scene's windows in standardised kernel space. Matching conventional spread (17 window metrics) and m_mix sit in the comparator. New outcome D_cancel = 2 min(FP, FN): the part of image-vs-MODIS disagreement that runs in both directions and cancels in the mean (SK1 found cover explains 93 to 97% of the total, so the cancelling part is where structure can act).

Dry run with noise outcomes: declared reading NOT_FOUND_IN_TESTED_SCOPE; ridge point gains within ±0.001, boosting point gains up to ±0.024 on noise, so boosting point estimates are read only through their bootstrap bounds.

60. Results

SK2b, human disagreement (labelled Terra scenes, n 3410, comparator R² 0.043): mixture gain ridge -0.0159 [-0.1158, 0.0118], boosting 0.0001 [-0.0284, 0.0280]; conditional p 0.990; k_mix vs residual -0.011 [-0.0444, 0.0265].

Declared reading: NOT_FOUND_IN_TESTED_SCOPE.

Mixture: ridge point gains small and positive in all three samples, beyond the conditional null (p 0.005 each), with a positive direction (k_mix vs residual interval excludes 0 in the new regions and the SFFG sample); refit year-bootstrap intervals include 0, so the pass rule is not met. Human disagreement shows no relation to mixture.

SK2c (declared, reported): the window-median kernel over cover terms gains on D_cancel in both untouched samples with refit bootstrap intervals above 0 (ridge). The gain is 4 to 6 times the SK1b gain on total disagreement (0.0033, 0.0027).

61. Use readout (declared)

Share of flagged passes in the true top 10% of D_cancel (chance 0.10), boosting out of fold:

62. Post hoc (s40, after the frozen run): what carries the median-kernel gain on cancelling disagreement?

Not declared. Ridge, leave one year out, refit year bootstrap 200 draws. Generic = k_tex, k_tonal, k_blob, k_edge_density (14); distinctive = the other 15.

Coordinates with the largest single-coordinate gain over Cs (summed over the untouched samples); gain and, in brackets, Spearman with the Cs residual:

Sign of each coordinate's relation to the Cs residual agrees between untouched years and new regions for 23 of 29 coordinates (Spearman of the two profiles 0.76).

Use: share of the 10% of passes flagged that fall in the true top 10% of D_cancel (ridge, out of fold), and mean Cs residual from lowest to highest decile of kernel lift (prediction with kernel minus prediction without):

Glint (s41, post hoc): the comparator holds the glint-angle area fractions, but the image shows the glint itself. Gain within passes with no area at glint angle under 25 degrees: untouched years 0.0112 (n 1,031), new regions 0.0196 (n 2,309); with some: 0.0115, 0.0191. The gain does not depend on glint.

New-region passes where the kernel raises (top) and lowers (bottom) the expected cancelling disagreement most, relative to cover alone:

63. Status

• Mixture of forms (across-window spread) is not found to predict cancelling instrument disagreement or human disagreement at the declared standard; operating window: 256-pixel windows, IQR spread, these two outcomes. The ridge signal is beyond the conditional null but not stable under year resampling.
• The gain is carried mainly by the generic coordinates (coarse texture contrast and homogeneity, edge density, blob size): more cancelling disagreement than cover predicts where coarse contrast is low and edges many; distinctive coordinates add 0.0029 [0.0009, 0.0039] beyond generic in the new regions and 0.0011 [-0.0032, 0.0030] in untouched years. Per-coordinate signs agree between the untouched samples for 23 of 29 coordinates.
• The kernel's window medians predict where the image mask and MODIS disagree in both directions at once, beyond both covers, in untouched years and new regions (ridge 0.014 and 0.017, intervals above 0). This is the clearest instrument-disagreement result so far.

Entry 13: two production products (VM)

64. Design

SK2c (entry 12) used our image threshold mask as one of two observers. VM replaces it with two production products crossing at about 13:30 local time: MODIS Aqua (MYD06) and VIIRS Suomi NPP (CLDPROP), each as a GIBS cloud-top-height snapshot (cloudy = retrieved cloud top). Per-pixel overpass times come from CMR granule footprints; only 11 km blocks seen by both within 10 minutes count. Outcome D_cancel_VM = 2 min(MODIS-only, VIIRS-only). New-region Aqua images for 2018-2021 were fetched and read for this test (s42; 2,193 images, 38 unreadable). Passes: 929 (A) and 1065 (B), 10 years each.

Outcome-only check before freezing: disagreement in both directions rises with the overpass gap and is flat within 10 minutes (cloud drift), supporting the cut:

65. Results

VM3 (transfer of the frozen SK2c kernel lift): Spearman with the comparator residual 0.014 [-0.0549, 0.0856] (A), -0.041 [-0.1029, 0.0251] (B).

Declared reading: NOT_FOUND_IN_TESTED_SCOPE.

Use readout (top-10% precision, ridge out of fold):

66. Post hoc: the ridge comparator in sample B

The ridge comparator R² in sample B is negative. s47: in the 2014 fold, generalized cross-validation picks a penalty of 0.005 and one Region 4 pass (2014-05-02, mostly swath gap) is predicted at 1.18 against 0. Clipping columns to the 0.5-99.5th percentile (s48 variant 1) and year-blocked penalty choice (s48 variant 2) do not repair it; removing that pass leaves the comparator R² at -0.17. With a fixed penalty of 10 (best comparator-only R² in sample B, chosen without the kernel):

VM1_A fixed alpha 10: R2_C 0.458 dR2 -0.0094 years with positive gain 4 of 10
VM2_A fixed alpha 10: R2_C 0.460 dR2 -0.0115 years with positive gain 2 of 10
VM1_B fixed alpha 10: R2_C 0.286 dR2 -0.0148 years with positive gain 2 of 10
VM2_B fixed alpha 10: R2_C 0.272 dR2 -0.0008 years with positive gain 6 of 10

Under the fixed penalty and under boosting the kernel's gain is negative in both samples. The reading does not depend on the ridge pathology.

67. What the two products disagree about

Mean covers and one-way parts on near-coincident blocks:

Near-coincident disagreement between the two products is almost entirely one way: MODIS marks cloud where VIIRS does not (2 km against 1 km pixels; different retrieval thresholds). VIIRS-only cloud is about 1% of blocks, so the in-place part has little variance. Passes the kernel flags most and least (sample B), each with the MODIS-only (red) / VIIRS-only (blue) map; blue bands are the MODIS swath gap, outside the analysis:

68. Status

• VM: the kernel does not predict where MODIS Aqua and VIIRS SNPP disagree in place, at 11 km blocks within 10 minutes, in either sample, under either model; the SK2c lift does not transfer. Operating window: the kernel sees where a simple threshold mask parts from MODIS; between these two production products, in-place disagreement is near its floor.
• The disagreement between these products is a one-way amount difference (MODIS cover exceeds VIIRS by about 0.2 to 0.25 on near-coincident blocks). Whether the kernel predicts the size of that offset beyond cover is a different question, not tested here.

Entry 14: the quiet coordinates (exploratory sweep)

69. Why

Every test so far used the 29 coordinates kept by the s09 rule (images only, before any outcome), as one block, pooled over all scenes. The other 21 coordinates the kernel reads were dropped for processing sensitivity and never tested; several are silent in most scenes. A coordinate that moves only in some regimes is diluted by pooling. The sweep looks at each of the 50 coordinates (window median) and its across-window spread, one at a time, within regimes.

70. Method

Residual r0 = outcome minus ridge prediction (penalty 10, leave one year out) from the test's comparator; r1 = the same with the 29-coordinate block added (what a coordinate says beyond the 29). Cells: all, image-cover terciles, seasons, regions, glint-area low and high terciles. A (feature, cell) is nominated when Spearman with the residual has the same sign and |rho| ≥ 0.15 in both odd and even years (n ≥ 40 each). Chance: the residual shuffled within year, 40 draws. Outcomes: SK2c (image-vs-MODIS cancelling disagreement), VMc (MODIS-VIIRS in-place disagreement), VMo (MODIS-VIIRS one-way offset fM - fV, comparator without fM).

SK2c: far above chance, also beyond the 29-block. VMc: above chance beyond the comparator, at chance beyond the 29-block. VMo: within the chance range.

71. Replication of the SK2c nominations beyond the 29-block

Same residual and cell rules in samples the sweep did not use (Terra passes). Sign agreement with discovery: untouched years 92 of 92, new regions 69 of 69 (chance 0.5 each; nominations are correlated with one another, so these are not independent tests). Replicated magnitudes are smaller than in discovery.

Dropped coordinates among the replicated features, with their s09 processing ratios (median change / scene SD; keep rule < 0.5):

They were dropped for grid-shift sensitivity (reading changes when the window grid moves), not for compression; the grid is fixed per scene, so this adds noise rather than bias.

72. Reading

• Beyond cover and beyond the 29 coordinates, cancelling image-vs-MODIS disagreement is consistently higher where scenes have many small blobs, many edges and fine contrast, and lower where several coordinates vary more across a scene's windows (spread of k_mu, edge density, k_gap_rv, k_rc_dr, k_sdi, k_delta_y, k_x_p). The relation is strongest in high cover, low glint and DJF, and replicates in sign in untouched years and new regions.
• Several coordinates dropped by the s09 rule (k_rc_dr, k_rc_drc, k_gap_rv, k_sdi, k_x_p, k_delta_y, k_g1_centroid_turn_mean, k_g3_unsigned_turn_variance) carry part of this.
• For MODIS-VIIRS outcomes the sweep finds nothing beyond chance once the 29-block is in.
• Exploratory: nominations were chosen on discovery data. Aqua passes have never been used for cancelling disagreement and are available for a frozen confirmation.

Entry 15: AQ, the quiet coordinates on Aqua (frozen)

73. Question

Do the entry-14 nominations (6 coordinates dropped by s09 and 10 across-window spreads) carry over to a second instrument and capture time? Aqua images are taken by the second MODIS about 3 hours after Terra and are compared with the Aqua MODIS cloud product. Cancelling disagreement (2 min(FP, FN)) had never been computed for Aqua. Days and regions are shared with Terra, so this is a partly independent bearing.

74. Declared reading

QUIET_COORDINATES_CONFIRMED. AQ2 (sign agreement) passes in all three samples; AQ1 (gain) passes in r1 and n1, not in orig. All AQ1 passes come from ridge; boosting gains are near zero or negative in every sample.

75. AQ1 and AQ4: gain over Cs + 29

AQ1 = 16-column block; AQ4 = the 6 dropped-coordinate medians and their 6 spreads (reported). The ridge rule stayed on GCV in all samples (comparator R² positive).

76. AQ2: sign agreement over the 69 pairs

The pairs overlap (shared features and nested cells), so the shuffle distribution is wide (99th percentile 0.93-0.97); 69 of 69 is above it in every sample. Noise dry run: 0.54.

77. AQ3 (regime) and use readout

The declared regime (high cover or low glint) holds a larger gain only in orig; in r1 and n1 the gain is as large or larger outside it.

78. Post hoc (not part of the reading): fixed-penalty ridge

The orig GCV ridge gain (+0.052) with a bootstrap bound below zero matches the leverage pattern of entry 13. With penalty fixed at 10 (s54, 200 year draws):

The +0.052 was the GCV choice; the stable orig gain is about +0.004 and not bounded above zero. r1 and n1 are unchanged.

79. Reading

• Direction carries to the afternoon instrument: every nominated (feature, regime) relation keeps its sign on Aqua in all three samples.
• Size is small: about +0.006 to +0.008 R² beyond cover, image statistics and the 29 coordinates, linear only (boosting finds nothing), in the two samples outside the discovery years and regions. In the discovery years and regions it is not bounded above zero.
• The regime concentration seen on Terra (high cover, low glint, DJF) does not carry over; on Aqua the relation is spread over the sample.
• Use: top-10% flagging gains 0 to 2.7 points of precision.

Entry 16: can the quiet signal be amplified? (exploratory)

80. Ceiling: how much leftover disagreement belongs to the scene

Residual = cancelling disagreement minus its prediction from Cs + 29 (ridge penalty 10, leave one year out), per satellite. Terra and Aqua view the same box on the same day about 3 hours apart. The correlation of their residuals estimates the share of leftover variance that persists with the scene rather than being measurement noise.

In the samples outside discovery, about 45% of leftover variance persists across the 3-hour gap (about 0.06-0.09 of total variance). The quiet index accounts for under 1% of that persistent part. The Aqua index correlates more with the Terra residual than with its own.

81. Fixed-sign index (no fitted weights)

The 16 AQ columns rank-normalised within sample and summed with their discovery signs (one column).

The fixed index gains less than the fitted block (roughly half). Its rank correlation with the residual (0.11-0.17) is two to three times its linear correlation (0.02-0.06).

82. Pooling curve

Correlation of mean index with mean residual over random groups of k passes within region x season x year (20 partitions).

The correlation rises with pooling only in new-regions Aqua (0.06 to 0.12). Elsewhere it is flat or falls, and in the discovery years and regions it turns negative at k ≥ 16. Averaging Terra and Aqua on the same box raises the single-pass correlation from about 0.05 to about 0.07.

83. Reading

• The ceiling is not the limit: a sizeable persistent part of the leftover disagreement exists, and the quiet coordinates read only a small share of it.
• Neither a fixed-sign index nor pooling passes amplifies the signal; it is a per-scene relation, not a slope that accumulates over many passes.
• The gap between rank and linear correlation places the relation in the ordering of scenes (and likely the tails) rather than in a proportional relation.

Entry 17: what the persistent leftover is (exploratory)

84. Groups

Per sample, paired boxes whose leftover (residual on Cs + 29) is in the top quartile on both Terra and Aqua ("high"), in the bottom quartile on both ("low"), and a random 150 of the rest ("mid"). High: orig 521, r1 425, n1 664. Regions and seasons are represented about evenly.

85. Profile: high vs rest (standardized difference, Terra pass)

3 samples; of 105 variables, 52 have the same sign in all three. The 25 largest among those:

Apart from the outcome itself, the high boxes have more image-cloud-MODIS-clear disagreement (FP, about +0.8 SD) and, structurally, smaller and more uniform blobs, less anisotropic spectra, smoother coarse texture and lower clustering (iorg). These variables are in the comparator linearly; the differences are what remains.

86. Maps: where the disagreement sits

Disagreement does not stay in the same place between the passes (correlation about 0.05) and is not tied to fixed places in the region (about 0.1, no higher in the high group). Persistence belongs to the kind of scene, not to a location. Most disagreement sits at cloud edges in every group.

87. Sunglint beyond the controlled area

In the high boxes the red appears in large solid bands on washed-out bright sea that move between the Terra and Aqua passes. Share of FP (per box common area) by glint angle, and the area within 25 degrees:

In orig and r1 the extra FP of the high boxes sits in the glint core (< 25 degrees) at a similar glint area: 2-3 times the FP of low and mid boxes there. In the new regions the extra FP is spread across angles. Glint-core FP falls with 10 m wind (Spearman -0.36, -0.31, -0.23), and leftover is largest in the calmest wind quintile. Calm sea gives bright, concentrated glint that the brightness threshold reads as cloud; the comparator controls glint area, not glint brightness.

88. Calm-glint terms in the comparator

Added: wind, 1/(wind+1) and their products with the glint-area fractions (ERA5 at the Terra time).

89. Reading

• Part of the persistent leftover is calm-sea glint read as cloud by the image threshold. With calm-glint terms, persistence drops from 0.19 to 0.10 (orig), 0.45 to 0.38 (r1) and 0.45 to 0.41 (new regions).
• Calm-glint terms also take 25-55% of the AQ quiet-block gain; the remainder stays positive in every sample (+0.002 to +0.006).
• Most of the persistent part remains after calm glint. It follows the kind of scene, not a place; the high boxes lean to fields of small, uniform, isotropic and weakly clustered cloud.

Entry 18: chasing the scene-type leftover (exploratory)

90. Ladder: does any available description absorb the persistent part?

MODIS markers (thin cloud, COT retrieved, partly cloudy, AOD, AOD retrieved) exist for orig and r1 only. Met is at the Terra time for both passes. No rung takes persistence below 0.28 in r1 or 0.34 in the new regions. Boosting lowers it by about 0.09 in r1 and less elsewhere, with lower R² than ridge in r1 and the new regions.

91. Time structure

Correlation of the leftover (Cs + 29 + calm glint, ridge) with the same region at later days, with other regions on the same date, and share of variance in year means and region x year x month means.

No shared signal across regions on the same date and almost none in year means: not a processing or calibration drift. The leftover carries over from day to day in the same region and fades within about a week, the time scale of a weather regime.

92. Reading

• About 40% of leftover cancelling disagreement (outside discovery) is shared by passes 3 hours apart and carries over 1-3 days in the same region, fading by about 7 days.
• It is not explained by reanalysis met (stability, humidity, subsidence, SST, boundary layer), MODIS cloud and aerosol markers, the full kernel reading with window spreads, conventional organization metrics, calm glint, or a nonlinear model of all of these.
• It is not tied to a place, a date across regions, or a year.
• The disagreement therefore follows a regime-scale property of the scene that none of the descriptions available here measures.

Entry 19: haze, dust and high thin cloud (exploratory)

93. Question

Entry 18 left a leftover shared by Terra and Aqua and carried over a few days. Candidate: aerosol layers (dust, sea salt, sulphate, smoke) or high thin cloud that brighten the image without MODIS counting the pixel as cloud, which would raise image-cloud-MODIS-clear disagreement (FP).

94. Ladder

R² changes by at most 0.006. Persistence moves by at most 0.01 (r1 0.383 to 0.374, new regions 0.500 to 0.495, ridge).

95. Correlations (Spearman): with the L1 leftover / with FP

FP falls as aerosol and cloud amount rise (log AOD −0.46 to −0.60; high cloud −0.25 to −0.48). These relations are already carried by the comparator: with the leftover every field is within ±0.13, and high cloud is negative.

96. Reading

• Reanalysis aerosol by species and cloud layers at the pass time do not explain the persistent leftover.
• The sign runs opposite to the haze hypothesis: FP is larger on cleaner, less cloudy days.
• Scope: MERRA-2 is 0.5° x 0.625° and the test uses box means, so sub-box layers and thin cirrus that the model does not carry are not seen here.

Entry 20: day sequences, and predicted sunglint brightness (exploratory)

97. Runs

Run = 3 or more consecutive days in one region with the box-day leftover at or above the sample's 85th percentile. Observed high runs: r1 45, new regions 74. With values shuffled within region and year (200 shuffles): mean 10.4 and 17.1, 99th percentile 18 and 25.

The strongest run per region (by length x mean) was drawn day by day with one day either side, both passes; one low run per sample as contrast.

98. What the sequences show, measured

Per run: mean wind (ERA5, Terra time), image-cloud-MODIS-clear disagreement (FP), share of FP at glint angle below 40°, and area below 40°; run days against the flanking days.

• In all six high runs the FP share in the glint zone is higher on run days than on flanking days (by 0.02 to 0.40), with the glint-zone area within 0.08.
• In five of six the red patches sit inside the grey sunglint wash. Region2 (r1, Jan 2004) is the exception: diffuse FP over a field of small broken cloud.
• Wind is lower on run days in the high runs, and also in the two low runs, where the glint share does not rise.

99. Predicted glint brightness (Cox-Munk)

From geometry and wind only: slope variance 0.003 + 0.00512 W, facet tilt = half the glint angle, Fresnel 0.02. Box features on the MODIS common area: mean, mean on the MODIS-clear part, 90th percentile, area above 0.05 and above 0.15, and their logs. Added to Cs + 29 + calm glint.

100. Quiet block (AQ, 16 columns) on top of it

101. Reading

• High-leftover runs occur about four times as often as chance.
• Most visible runs are calm-sea sunglint the image mask calls cloud and MODIS does not.
• Physically predicted glint brightness takes 0.04-0.05 of the Terra-Aqua persistence under ridge (about 11%) and about a third of the 1-day carry-over; boosting persistence does not move.
• The quiet-block gain changes by 0 to −24% and stays positive in all four sample-satellite cases.
• Scope: wind is one box-mean value at the Terra time; glint geometry is approximate (track tilt ignored), so local calm patches under the glint are not resolved.

Entry 21: the outcome with the sunglint zone masked out (exploratory)

102. Results

103. Reading

• Masking the glint zone does not lower the Terra-Aqua persistence of the leftover; it rises (r1 0.428 to 0.490, new regions 0.424 to 0.519 under the 40° mask; boosting likewise), although less area is measured.
• Mean FP outside the glint zone is the same as or slightly higher than over the whole box: on average the image-cloud-MODIS-clear disagreement is not concentrated in glint.
• High runs fall (r1 32 to 26, new regions 68 to 40) and stay about three times the shuffled count. The 1-day carry-over falls in the new regions (0.140 to 0.082) and slightly in r1.
• The quiet-block gain stays positive in all four sample-satellite cases under both masks, reduced by 16-62% under the 40° mask.
• This revises the reading of entry 20. The strongest visible runs are dominated by calm-sea glint, and glint contributes part of the day-to-day carry-over, but the persistent shared leftover is mostly outside the glint zone and remains unexplained by every description tested (met, MODIS markers, full kernel, aerosol and cloud layers, glint physics, glint masking).

Entry 22: the shared leftover by cloud type and local cloud context (exploratory)

104. Design

• Image context (all samples). Per pass, on the 2-px block grid, each block is classed by its ~20 km neighbourhood (9 × 9 blocks): sparse, broken or overcast by image cloud cover; small-cumulus field where at least half the neighbourhood cloud lies in connected objects of 25 px or fewer; thin where at least half the cloud pixels are faint (30-45 grey levels above background; the cloud threshold is 30).
• Human cloud type (original sample only, the one with labels). A block is pattern P where at least half of that scene's labelers marked P; Terra and Aqua images were labelled separately.
• Masks as in entry 21: the MODIS common area is cut to blocks inside (keep only) or outside (remove) a context, the same area for both passes (a block counts if it is in the context on either pass); box-days with less than 20% kept are dropped, and each variant is compared with the unmasked outcome on the same box-days.
• Composition predictors (image only, no MODIS input, so they cannot carry the outcome): per-pass area shares of each context and of clear neighbourhoods, cloud cover in small and in large objects, faint share, edge density; for the original sample also the labelers' mean area per pattern.
• FP share = fraction of the pass's whole-box FP falling inside the kept area (descriptive).

105. Masks

106. Composition rungs and labelled patterns

Original sample, L1 leftover by the box-day's dominant labelled pattern (mean of both passes):

107. Reading

• The cancelling disagreement pairs image-cloud-MODIS-clear area with the reverse elsewhere in the box, so keep-only masks that separate the two lower it partly by construction. The remove masks are the cleaner reading.
• r1 and new regions: removing overcast neighbourhoods raises persistence (0.377 to 0.500; 0.411 to 0.536), as does removing thin cloud (0.428; 0.497); removing sparse, broken or small-cumulus areas leaves it about unchanged. Keeping only sparse or only small-cumulus areas keeps most of it; keeping only broken, overcast or thin areas lowers it to 0.12-0.18. Overcast and thin cloud carry mostly pass-specific disagreement; the shared component is spread through the open-cloud parts of the scene rather than confined to one context.
• Cloud-context composition raises R² markedly (r1 Terra 0.840 to 0.898, Aqua 0.861 to 0.913; new regions 0.881 to 0.907, 0.884 to 0.918) but lowers persistence only by about a tenth (0.377 to 0.343; 0.411 to 0.374). Image-measured cloud type explains much of each pass's leftover and little of the part the two passes share.
• Original sample: persistence is low there (0.102, as in entry 17). Labelled pattern areas lower it to 0.090. What there is sits with open shallow cumulus: boxes dominated by Gravel 0.345 against 0.008-0.083 for the other patterns, and keeping only Sugar-marked areas gives 0.418 against 0.172 on the same box-days. This agrees in direction with the r1 and new-region reading (open cloud, not overcast or thin).
• Remaining candidates for the shared component: processes inside open shallow-cumulus fields below the 2-km block scale (for example sub-pixel cumulus treated differently by the two cloud masks), and properties of the regime not described by cloud geometry.

Entry 23: sub-pixel cumulus at 250 m (exploratory)

108. Design

• 250-m cloud uses the scene rule (grey minus the 1.1-km scene background above 30). Per 2-px block (8 × 8 subpixels): 250-m cover f250, and cover in elements of 16 subpixels or fewer (≤ 1 km², below one MODIS cloud-mask pixel).
• The MODIS side of the outcome is the GIBS Cloud_Fraction_Day layer (MOD06/MYD06 5-km Cloud_Fraction from the MOD35 mask), sampled on the 2-px block grid.
• Image-only predictors per pass: f250, sub-km cover, small-element share of cloud, 1.1-km patch cover, 1.1-km minus 250-m cover ("over"), spread of f250 across blocks. Control: 1.1-km patch cover alone. Each pass's leftover is regressed on its own predictors (ridge, leave one year out, within the subsample) and Terra-Aqua persistence recomputed.
• Outcome decomposition on FP blocks (image cloud > MODIS fraction): sum(ci − cf) = sum(ci − f250), the 1.1-km image counting more than 250 m resolves, + sum(f250 − cf), MODIS reporting less than 250 m resolves. FN blocks likewise. Descriptive only (uses MODIS).

109. What 250 m shows

110. Persistence with sub-pixel predictors

111. Reading

• Sub-kilometre cumulus is a small part of the cloud in these scenes (1.5-4% of 250-m cloud area), and where MODIS reports a fraction of 0.1 or less, 250 m finds about 3% cover.
• Sub-pixel predictors explain none of the leftover (R² about 0 out of sample) and leave Terra-Aqua persistence unchanged in both samples and for both leftovers (for example r1 0.283 to 0.278, new regions 0.353 to 0.354). The sub-pixel cumulus explanation, in this form, is not supported.
• FP blocks are mostly cloudy at 250 m (0.67-0.76 cover); 58-66% of FP is the MODIS fraction falling below the 250-m cover, and 34-42% the 1.1-km image counting more than 250 m. FN is almost entirely (96-97%) MODIS reporting cloud where neither image resolves it. On the highest-leftover box-days the MODIS-side FP component is about 40% larger than on the lowest (r1 0.028 against 0.019; new regions 0.026 against 0.018), while the image-side component is about the same.
• This pattern (MODIS lower inside clouds, higher in the gaps beside them) is what aggregating a cloud field to the 5-km MOD06 cell would produce. A resolution-mismatch description of the cancelling disagreement, computed from the image alone, has not yet been tested.

Entry 24: resolution matching between the image mask and the MODIS fraction (exploratory)

112. Design

• Per pass, the full-resolution image cloud mask is smoothed with an s × s moving mean over valid pixels (s = 3, 5, 9, 15, 25 pixels; about 3, 6, 10, 17, 28 km) and averaged to the 2-px block grid (ci_s; s = 1 is the unsmoothed ci of the outcome).
• Effective scale (descriptive): correlation of the MODIS fraction with ci_s over the MODIS common area.
• Matched outcome: the cancelling disagreement 2·min(FP, FN) of ci_s against the MODIS fraction, with image cover replaced by the smoothed cover in the design. Its leftover after L1 + context and the Terra-Aqua persistence of that leftover are recomputed.
• Image-only blur predictors: mean |ci − ci_s| per scale, per pass (no MODIS input), added to L1 and to L1 + context on the unsmoothed outcome.

113. Effective scale

114. Outcome and persistence

115. Reading

• The MODIS fraction agrees best with the image mask smoothed over about 17 km (s = 15; r1 correlation 0.70 to 0.80, new regions 0.67 to 0.77), somewhat broader than one 5-km MOD06 cell; the gridded product behaves as a smoother field than its nominal cell size.
• Matching the image to that scale before comparison lowers the cancelling disagreement by about half, with the minimum at s = 9 (r1 Terra 0.067 to 0.035, Aqua 0.081 to 0.045; new regions 0.057 to 0.030, 0.068 to 0.036). About half of the disagreement in each pass is resolution mismatch.
• The shared Terra-Aqua part does not follow it down. Persistence of the matched-outcome leftover is 0.316-0.339 (r1) and 0.348-0.371 (new regions) for s = 3 to 15, against 0.343 and 0.374 unmatched; only the coarsest smoothing (s = 25) reaches 0.318 and 0.306, where the mean disagreement is rising again.
• Image-only blur predictors add about 0.001-0.002 to R² and lower persistence by under 0.01 (0.343 to 0.334; 0.374 to 0.369).
• The resolution mismatch is mostly pass-specific. The shared component survives the best-matching smoothing tested; that match is incomplete (correlation 0.80 at best), but within it the shared part behaves as a disagreement about cloud amount at the MODIS scale rather than about where cloud edges fall inside a cell. Whether the effective scale comes from the MOD06 product or from GIBS resampling is not separated.

Entry 25: cloud near the detection threshold (exploratory), and corrections from the document audit

116. Design

• Image contrast c = grey minus the scene background (10th percentile over 151 px); the scene rule calls cloud at c > 30.
• Moved threshold. The cancelling disagreement is recomputed with the image threshold at 20, 25, 35, 40 and 50, image cover terms in the design replaced accordingly; leftover after L1 + context and Terra-Aqua persistence recomputed. If the shared component were cloud near the image threshold, moving the threshold would change it.
• Image-only predictors (no MODIS input): share of valid pixels in contrast bands 15-30, 30-45, 45-60, ≥ 60 and mean contrast of cloud pixels, added on the stored outcome.
• Descriptive (uses MODIS): contrast in FP blocks (image cloud > MODIS fraction + 0.25) and FN blocks (MODIS fraction > image cloud + 0.25); in r1, MODIS COT in those blocks. Compared between the top and bottom 15% of the shared leftover (mean of the two passes' leftovers).

117. Moved threshold and band predictors

118. Brightness of the disagreement areas

119. Reading

• Moving the image threshold from 20 to 50 changes image cover by about 0.15 and the cancelling disagreement by a factor of about three, but the persistence of its leftover stays between 0.32 and 0.40 in both samples and is highest at the strictest threshold (r1 0.345 at 20, 0.401 at 50; new regions 0.368 and 0.393). The shared component does not depend on where the image threshold sits.
• Contrast-band predictors raise R² (r1 Terra 0.898 to 0.923; new regions 0.907 to 0.921) and lower persistence by 0.02 in r1 and 0.004 in the new regions. In r1 they lower the 1-day carry-over from 0.131 to 0.080; in the new regions it does not fall (0.103 to 0.107).
• Near-threshold pixels are about as common on the highest-leftover days as on the lowest, in FP and FN blocks alike. FN blocks have image contrast about half the threshold (mean 15); in r1, MODIS retrieves an optical thickness in 12% of FN blocks and calls 85% of those thin (COT < 3.6), equally on high and low days. FP blocks have retrieved COT in 17% of cases.
• The clearest image-side correlate is the mean contrast of image-cloud pixels (Spearman +0.19 r1, +0.14 new regions; 97 against 92 and 100 against 95 grey levels, high against low days). Contrast is measured against the scene background, so this cannot tell brighter cloud from a darker background estimate.
• Cloud near either mask's detection threshold, in this form, is not supported as the source of the shared component.

120. Corrections from the document audit (2026-11-01)

• Entry 16 described the Terra-Aqua correlation of leftovers as variance that "persists with the scene rather than being measurement noise". It measures what the two passes share. Both passes apply the same image-mask rule and the same MODIS algorithm to the same kind of scene, so a systematic scene-dependent difference between the two measurements is also shared (calm glint, entry 17, is one). Later entries should be read with "shared" not implying "physical".
• The wind figure "87% retained" (README and ASSESSMENT) predates the entry-8 timing fix; after the fix it is 85% (2007-2017) and 47% (untouched years). Corrected in README, ASSESSMENT and memory.
• The last reading of entry 24 was revised the same day: the shared component survives the best-matching smoothing tested, which is incomplete (correlation with MODIS 0.80 at best); the effective scale was not separated between the MOD06 product and GIBS resampling.
• WRITEUP.html is generated by scripts/build_writeup.py; an earlier hand edit to the HTML was lost on rebuild and has been restored into the script, with sections now covering entries 1-25.
• Untested candidates raised by the audit: (a) the image mask's background estimate (10th percentile over about 166 km) sits on clear sea in open scenes and is raised in cloudier ones, a regime-dependent bias shared by both passes; the mean-contrast correlate above is consistent with it but does not separate it; (b) the quiet-coordinate relations (fine structure with more cancelling disagreement) have the signature of the pass-specific resolution component of entry 24, which would explain why they read under 1% of the shared leftover.

Entry 26: the image mask's background estimate (exploratory)

121. Design

• Candidate (a) of entry 25: the scene rule calls cloud where grey minus a background exceeds 30; the background is the 10th percentile of grey over 151 px (about 166 km), computed on a 1/4 grid. In open scenes it sits on clear sea; where most of a window is cloud it is raised, and the mask under-calls cloud. Both passes share the rule, so a regime-dependent bias of this kind would be shared.
• Alternative backgrounds (threshold 30 throughout): 5th percentile over 151 px; 10th percentile over 75 px; 10th percentile over 301 px (computed on a 1/8 grid for run time; on smoke-test days it matched a 1/4-grid computation closely); a single scene-wide clear-sea level (2nd percentile of valid grey), which cannot rise under cloud. Matched outcome for each, image cover terms replaced, leftover after L1 + context and Terra-Aqua persistence recomputed.
• Image-only background predictors on the stored outcome: mean and SD of the background above the clear-sea level, that elevation under image cloud, share of pixels with elevation above 15, and the clear-sea level itself.
• Descriptive: the same measures on the top and bottom 15% of the shared leftover (mean of the two passes' leftovers).

122. Alternative backgrounds and background predictors

123. Background elevation and the shared leftover

124. Reading

• The scene background sits on average 25 (r1) and 36 (new regions) grey levels above the clear-sea level, comparable to the threshold of 30. With the clear-sea level as background, image cover rises from about 0.36 to 0.59 (r1) and 0.51 to 0.68 (new regions), the cancelling disagreement nearly doubles, and R² falls (r1 Terra 0.898 to 0.663). The background estimate is a large part of how the image mask differs from MODIS.
• No background tested removes the shared component. Persistence of the leftover ranges from 0.34 to 0.46 across the five backgrounds in both samples (r1 0.343 stored, 0.341 to 0.456; new regions 0.374 stored, 0.342 to 0.456). The 75-px window gives the highest value in both samples (0.456); the 5th percentile gives the lowest in the new regions (0.342).
• Background predictors raise R² by about 0.01 and change persistence by −0.013 (r1) and +0.019 (new regions).
• Background elevation is slightly lower, not higher, on the highest-leftover days (r1 26.7 against 28.3; new regions 39.7 against 41.3; Spearman −0.05 to −0.13 for every measure). The mean-contrast correlate of entry 25 therefore reflects brighter cloud rather than a darker background estimate.
• A regime-dependent bias of the image background estimate, in this form, is not supported as the source of the shared component. With the image threshold (entry 25), smoothing scale (entry 24) and background estimate (this entry) each varied, the image-mask settings tested leave persistence between about 0.32 and 0.46.

Entry 27: VIIRS in place of MODIS as the Aqua-time reference (exploratory)

125. Design

• Question: is the shared Terra-Aqua part specific to the MODIS reference? The Terra outcome keeps its reference (image against the MODIS Terra cloud fraction). The Aqua image is compared against five references on the same area: the MODIS Aqua cloud fraction (control), the MODIS Aqua cloud-top-height mask (cloudy where a height is retrieved; MYD06), the VIIRS SNPP cloud-top-height mask (CLDPROP, MODIS-VIIRS continuity algorithm, a different sensor at 750 m), and both height masks smoothed over 3 × 3 grid pixels (about 7 km).
• Area: MODIS common area, restricted to blocks where both MODIS Aqua and VIIRS SNPP granules cover the block and their start times differ by 40 min or less (20 min in a second set), as in the VM preparation (entry 13).
• Measures: Terra-Aqua correlation of the leftovers (persistence) and their covariance. An Aqua reference that adds its own pass-specific disagreement (time offset, viewing geometry, resolution) raises the Aqua leftover's spread and lowers the correlation without changing the covariance; the covariance ratio to the control measures how much of the shared part each reference carries.

126. Persistence and shared covariance by Aqua reference

127. MODIS-VIIRS cover difference and the shared leftover

128. Reading

• Restricting to the VIIRS-covered area changes persistence of the control little (r1 0.378 stored to 0.362; new regions 0.410 to 0.353, within 40 min).
• With VIIRS as the Aqua reference, persistence falls to 0.08-0.21, but the VIIRS leftover has about 2.5 times the spread of the control's and correlates 0.35-0.42 with it: VIIRS adds a large pass-specific disagreement. The shared covariance with VIIRS is 1.20 (0.83-1.60) and 1.07 (0.72-1.37) times the control within 40 min, and 1.24 (0.89-1.58) in the new regions within 20 min; in r1 within 20 min, the smallest set, it is 0.64 (−0.08 to 1.24). The 3 × 3 smoothed VIIRS mask gives 0.46-1.02. In three of four sets the shared part carried by the VIIRS reference is at least that carried by MODIS.
• The MODIS height mask covers more than the image (0.66-0.77 against Aqua image cover 0.34-0.48 on these areas), so its cancelling disagreement is small (mean 0.013-0.028); it carries 0.40-0.82 of the control's shared covariance.
• MODIS reports more cloud than VIIRS on every area (height masks 0.22-0.26 apart; cloud fraction 0.08 above the VIIRS mask), and slightly more so on high-leftover days (Spearman +0.16 for the height masks, +0.08 to +0.12 for the cloud fraction).
• The shared component is not specific to the MODIS sensor or its products: it is carried when the Aqua-time reference is VIIRS SNPP. Not separated by this test: the cloud-mask algorithm family (the VIIRS continuity mask descends from the MODIS mask), the Terra side (still MODIS), and the image rule, which is common to both passes in every variant.

Entry 28: the quiet-coordinate gain on resolution-matched outcomes (exploratory)

129. Design

• Question from the document audit (entry 25, candidate b): the quiet-coordinate relations (fine structure with more cancelling disagreement) resemble the pass-specific resolution component of entry 24. If the block reads that component, its gain shrinks on the resolution-matched outcome (image mask smoothed s × s px before comparison with MODIS) and appears on the removed part (stored minus matched outcome).
• Same comparator and block as the frozen AQ1 test; for matched outcomes the image cover terms are recomputed from the smoothed mask. Terra passes are included (the discovery pass of SK2c).

130. Gain of the quiet-coordinate block by outcome

131. Reading

• On the matched outcomes the gain is as large as or larger than on the stored outcome in all four sample-pass sets (Aqua: r1 +0.0059 to +0.0076 at s = 9, new regions +0.0075 to +0.0119; Terra: r1 +0.0046 to +0.0081 at s = 15, new regions +0.0061 to +0.0110). Measured against the variance left by the comparator, it is about the same (r1 Aqua 0.041 stored, 0.031 at s = 9; new regions 0.064 and 0.057). Sign agreement of the replicated pairs stays at 0.97-1.00.
• On the removed part (the resolution component) the gain is +0.0003 to +0.0019, with intervals including or touching zero in every set; sign agreement falls to 0.75-0.97.
• The quiet coordinates read the part of the cancelling disagreement that survives matching the image to the MODIS scale, not the resolution component. Candidate (b) of entry 25 is not supported.

Entry 29: a geostationary reference outside the MODIS cloud-mask family (exploratory)

132. Design

• Reference: NOAA GOES-R ABI Clear Sky Mask (binary cloud mask, 2 km at nadir, full disk every 10 min, enterprise algorithm, not derived from MOD35), with quality-flagged pixels removed, sampled on the box grid by nearest neighbour, and smoothed 3 × 3 as a second form.
• Timing: each image pixel is assigned its MODIS L2 granule (CMR footprints: MOD06_L2 for Terra, MYD06_L2 for Aqua; 10-px edge erosion), and each granule the GOES scan whose estimated time over the box is nearest. A geostationary reference exists at both overpass times, so the MODIS reference can be replaced on both passes or on one.
• Combinations: both MODIS (control, same area), both GOES, and the two mixed pairs. Measures as in entry 27: persistence, 1-day carry-over of the Terra leftover, and the shared covariance relative to the control.

133. Persistence and shared covariance by reference pair

134. Reading

• The image disagrees with GOES about 2.5 times as much as with MODIS (mean D 0.18-0.20 against 0.06-0.08); mean cover is image 0.35-0.36, GOES 0.45-0.46, MODIS 0.48-0.50, so the larger disagreement is mostly in where cloud is placed, in both directions (geolocation, parallax and pixel size differ between the two views).
• With GOES on both passes the leftover carries over from day to day far more than the control's (1-day carry-over 0.39 against 0.08 for both regions; Region1 0.32 against 0.17, Region3 0.47 against 0.11), and the shared covariance is 6.1 (3.9-9.7) times the control's, from Region1 (6.0) rather than Region3 (1.0, −0.3 to 2.4). A fixed-geometry reference sees each place from the same angle every day, so its own errors are shared by the two passes and persist across days; the both-GOES pair measures that component as well and does not isolate the one under study.
• The mixed pairs share no GOES error between passes. Terra MODIS with Aqua GOES carries 0.93 (0.39-1.34) of the control's shared covariance for both regions (Region3 1.13, 0.80-1.55; Region1 0.16, −0.99 to 0.73). Terra GOES with Aqua MODIS carries 0.24 (−0.69 to 1.07; Region1 0.65, Region3 0.48, both intervals wide). The Terra-time GOES leftover correlates only 0.21-0.32 with the control's, against 0.48-0.61 at Aqua time.
• Inconclusive at this sample size: one mixed pair is consistent with the shared component being carried by an independent algorithm, the other is not informative, and the two regions disagree within wide intervals. The test is limited by the archive start (about 390 box-days per region, three years) and, for the both-GOES pair, by the geostationary reference's own persistent error.

Entry 30: state of the line at consolidation

135. Threads and status

136. Candidates tested for the persistent shared leftover

Terra-Aqua persistence of the leftover cancelling disagreement, untouched years (r1) / new regions (n1), before and after each candidate; for reference swaps, the shared covariance relative to the MODIS control. All exploratory.

Common to every variant tested: the image mask rule (applied to both passes) and the MODIS Terra reference. Shared between passes is not the same as physical: a systematic scene-dependent measurement difference would also be shared.

137. Extensions not run

• GOES-R reference on new scenes from 2022 onward (outside the frozen samples), to resolve entry 29.
• A conventional organization-metric set and an established image embedding under the same frozen protocol, with run time and hardware, to measure proximity.
• Cloud-street alignment relative to the wind or shear direction (the orientation readings measure strength of alignment, not heading).
• A non-MODIS sensor for the morning image, if a morning true-colour source becomes available.

Entry 31: CALIPSO lidar as referee for the persistent shared leftover (frozen reading)

138. Design

• CALIPSO trails Aqua by about 73 s, so its lidar gives an independent active measurement of cloud along a line through each box at Aqua time. Each 333 m profile is cloudy if any detected layer is classified cloud with CAD score ≥ 20. Profiles are binned to the 2.2 km map grid; a pixel needs at least 3 profiles and must lie in the MODIS common area outside the Aqua glint zone (glint angle ≥ 40°).
• Along the track: L = lidar cloud fraction, I = image mask (Aqua), M = MODIS Aqua cloud fraction; E_I = mean |I − L|, E_M = mean |M − L|.
• The Terra leftover (from s56) is about 3 h before the lidar, so its association with Aqua-time track error reflects only what the two passes share. Primary statistics: Spearman of the Terra leftover with E_M and E_I, as partial correlations with the ranks of track-mean L, I and M held fixed, and their difference. Frozen readings: MODIS_SIDE, IMAGE_SIDE, BOTH, NEITHER.

139. Primary and secondary readings

Primary frozen reading: BOTH. The partial correlations are 0.069 (MODIS error) and 0.065 (image error), both intervals above zero, difference 0.005 with an interval spanning zero. Raw correlations are near zero; the association appears once the track-mean cloud amounts are held fixed. The reading is the same with the low-cloud-only definition and with CAD ≥ 70.

140. By region (each region read on its own)

141. Who the lidar sides with inside the disagreement

Track pixels where image and MODIS disagree by at least 0.5: FP = image cloud, MODIS clear; FN = MODIS cloud, image clear. Pixel-weighted lidar cloud fraction in each, for all box-days and for the s56 high and low leftover groups.

142. Reading

• The shared part of the leftover goes with both passive measurements departing from the lidar, by similar amounts (frozen reading BOTH). The associations are small (partial ρ about 0.07); a line sample through a box is a noisy per-day referee.
• Per region the reading is BOTH in Region5 and Region6 (partial ρ 0.10-0.13) and NEITHER in Region4 (both near zero). The shared component is resolved by the lidar in two of three regions and not in the third.
• The Aqua leftover, which also contains the pass-specific part, reads IMAGE_SIDE (NEITHER at CAD ≥ 70): at Aqua time the part not shared with Terra goes more with the image mask's error than with MODIS's.
• Along the track the lidar reports less cloud than either passive measurement (L 0.43, image 0.50, MODIS 0.61); MODIS's mean error against the lidar is larger than the image's (0.24 against 0.18). FN pixels outnumber FP about 8 to 1 on the track. Where only the image sees cloud, the lidar sees cloud in 63% of profiles; where only MODIS does, in 32%. On high-leftover days the lidar confirms more of the MODIS-only cloud than on low days (0.35 against 0.25), while the FP side is unchanged (0.63 against 0.64).
• Declared caveats (protocol): the lidar samples a line, not the pixel area; it detects thin cloud and haze edges that neither passive measurement is designed to see; glint exclusion removes some near-nadir track sections.

Entry 32: does the kernel flag scenes where both passive measurements depart from the lidar? (frozen reading)

143. Design

• Outcome: the departure of the passive cloud measurements from the CALIPSO lidar along the track (entry 31), E_P = (E_I + E_M) / 2.
• Comparator, all from passive data: the sample base (tonal statistics, season and region terms, conventional organization metrics), scene cover and glint terms (as SK2c/AQ), and the passive track terms (track-mean image and MODIS cover, track cancelling disagreement, FP and FN shares, pixel count). No lidar quantity.
• Added: the 29 window-median kernel coordinates read on the Aqua image. Reading FLAG_SUPPORTED if the gain on E_P is positive and its 1.25% bootstrap quantile is above zero under ridge or boosting.

144. Results

Frozen reading: FLAG_SUPPORTED. Ridge adds +0.0206 R² to a comparator at 0.509, bounded above zero at 1.25% and beyond the conditional null; boosting adds +0.0121, not bounded. The gain holds for each passive measurement separately (image +0.0170, MODIS +0.0222).

145. Secondary analyses

• Lidar cover added to the comparator. With track-mean L in the comparator, R² rises to 0.946 and the kernel's gain falls to +0.0005 (ridge) and -0.0016 (boosting). Given the passive covers, the departure is almost fully set by how much cloud the lidar sees; the kernel's contribution is information about that lidar cover which the passive covers do not carry.
• AQ1 block added to the kernel. No further gain (+0.0200 for kernel plus block against +0.0206 for the kernel alone, ridge).
• Use readout. Flagging the top 10% of predicted departure catches the top 10% of actual departure at precision 0.612 with the comparator alone and 0.616 with the kernel added (boosting; base rate 0.100). The passive comparator already ranks the hardest scenes; the kernel's gain is spread over the rest of the range.

By region the gain is bounded above zero in Region4, near zero in Region6 and negative in Region5 (the 29-coordinate ridge does not transfer across years within that region). Region4 is the region where the shared leftover was not resolved by the lidar in entry 31.

Entry 33: head-to-head against conventional metrics and an established image embedding (frozen reading, with a colour follow-up)

146. Design

• Tasks: the line's frozen outcomes, each with its own comparator, rows and folds: afternoon cloud change P1 (three samples), Terra and Aqua cancelling disagreement (two samples each), and the lidar departure of entry 32.
• Blocks: C0 = the comparator without the conventional organization metrics; CONV = the 13 conventional metrics (Iorg, SCAI, COP, open sky, object count and lengths, eccentricity, orientation, fractal dimension, spectral slope, anisotropy, length scale), already in every frozen comparator; K = the 29 kernel coordinates; E = DINOv2 ViT-S/14 (Meta, lvd142m weights; a general-purpose self-supervised natural-image model), CLS embedding of a 6 × 4 grid of 350-px tiles, tile median, PCA to 29 components fitted on the SFFG Terra scenes.
• Readings: KERNEL_BEYOND_EMBEDDING if K adds beyond C0 + CONV + E (ridge, bounded above zero) in at least 5 of 8 task-samples; EMBEDDING_BEYOND_KERNEL likewise for E beyond C0 + CONV + K. Before the freeze, the dry run showed boosting gains up to +0.02 on pure-noise outcomes and none for ridge, so readings use ridge and boosting is reported against its measured noise floor.

147. Results (ridge R² gains; bold = bounded above zero)