Testing Neural Compression at Scale: Embed2Scale at the ESiWACE3 Hackathon

Addressing the exponential growth of weather and climate model data requires urgent solutions that safely reduce file sizes while preserving critical data properties. To tackle this challenge, the ESiWACE3 Data Compression Hackathon brought together climate researchers, modeling teams, and data compression developers in a hybrid event featuring experts from DKRZ, NVIDIA Earth-2, the University of Helsinki, Embed2Scale partner Forschungszentrum Jülich, and Asterisk Labs. The event aimed to establish practical data compression solutions for high-value datasets across leading data modeling and compression teams.

As part of the program, Embed2Scale partner Conrad M. Albrecht (University of Oxford) delivered a presentation titled “NeuCo-Bench: Lossy Neural Compression at the Test”. The talk explored key pillars of neural compression and evaluation for Earth Observation data:

  • High-Resolution EO Data: Utilizing high-resolution Earth Observation datasets sampled globally across all seasons.
  • Neural Compressors: Highlighting leading neural compression solutions capable of achieving compression rates up to 7,000x, as demonstrated in the 2025 CVPR EarthVision data challenge.
  • Probing Embeddings: Evaluating how effectively neural compressors preserve critical information using NeuCo-Bench, an extensible, model-agnostic benchmarking framework that tests fixed-size representations across downstream tasks (such as land-cover proportion estimation, cloud detection, and biomass estimation) using linear probes and a scoring system balancing accuracy and stability.
Testing Neural Compression at Scale: Embed2Scale at the ESiWACE3 Hackathon
Testing Neural Compression at Scale: Embed2Scale at the ESiWACE3 Hackathon

The event also featured a presentation published through the European Geosciences Union (EGU) preprint repository, ClimateBenchPress (v1.0): A Benchmark for Lossy Compression of Climate Data. Together, both contributions reflect a growing cross-community effort across Earth Observation, climate science, and high-performance computing to establish transparent, standardized evaluation for compressed datasets and neural representations.