Real-World Setup
We deploy three pairs of cameras over a two-month period, enabling evaluation through comparisons with co-located measurements from other instruments.
NeurIPS 2025 (Spotlight)
TL;DR: Cloud4D estimates physically-realistic 3D cloud fields using ground-based cameras at a 25 m spatial resolution and 5 s temporal resolution.
Why? To obtain better measurements for the development of high-resolution weather and climate models.
We deploy three pairs of cameras over a two-month period, enabling evaluation through comparisons with co-located measurements from other instruments.
We render our predictions from a top-down view and compare against satellite imagery (MODIS and Sentinel-2).
Our predictions match satellite results, while obtaining measurements every five seconds rather than every one day (MODIS) or every five days (Sentinel-2).
(a) Liquid Water Content
(b) Liquid Water Path
We compare against radar retrieved (a) liquid water content and (b) liquid water path (height-summed liquid water content). Our predictions match retrieved radar values, while retrieving full 3D properties rather than only along a single vertical ray.
(a) Cloud Layer Model: Cloud4D leverages the spatial structure of clouds and uses a homography to map images to cloud layers. Taking advantage of the vertically thin nature of cloud layers, Cloud4D uses the homography and formulates the cloud layer estimation as an easier 2D-to-2D task.
(b) 3D Refinement: A sparse 3D transformer then refines the predictions using a full learned 3D prior attending over the whole volume.
(c) Wind Retrieval (inference): Lastly, at inference time, we estimate the liquid water content at multiple time steps and extract horizontal wind profiles by tracking our predictions over time.
Cloud4D is trained using physics-based large-eddy simulations (LES), resulting in physically-meaningful clouds which we render using Blender.
@inproceedings{
lin2025cloudd,
title={Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution},
author={Jacob Lin and Edward Gryspeerdt and Ronald Clark},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=g2AAvmBwkS}
}