<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v3.0 20080202//EN" "https://jats.nlm.nih.gov/nlm-dtd/publishing/3.0/journalpublishing3.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" specific-use="SMUR" dtd-version="3.0" xml:lang="en">
<front>
<journal-meta>
<journal-id journal-id-type="publisher">EGUsphere</journal-id>
<journal-title-group>
<journal-title>EGUsphere</journal-title>
<abbrev-journal-title abbrev-type="publisher">EGUsphere</abbrev-journal-title>
<abbrev-journal-title abbrev-type="nlm-ta">EGUsphere</abbrev-journal-title>
</journal-title-group>
<issn pub-type="epub"></issn>
<publisher><publisher-name>Copernicus Publications</publisher-name>
<publisher-loc>Göttingen, Germany</publisher-loc>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.5194/egusphere-2024-3183</article-id>
<title-group>
<article-title>Compressing high-resolution data through latent representation encoding for downscaling large-scale AI weather forecast model</article-title>
</title-group>
<contrib-group><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Liu</surname>
<given-names>Qian</given-names>
</name>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Gong</surname>
<given-names>Bing</given-names>
</name>
<xref ref-type="aff" rid="aff3">
<sup>3</sup>
</xref>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Zhuang</surname>
<given-names>Xiaoran</given-names>
</name>
<xref ref-type="aff" rid="aff4">
<sup>4</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Zhong</surname>
<given-names>Xiaohui</given-names>
<ext-link>https://orcid.org/0000-0001-6706-2335</ext-link>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Kang</surname>
<given-names>Zhiming</given-names>
</name>
<xref ref-type="aff" rid="aff5">
<sup>5</sup>
</xref>
</contrib>
<contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Li</surname>
<given-names>Hao</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
</contrib>
</contrib-group><aff id="aff1">
<label>1</label>
<addr-line>Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, 200433, China</addr-line>
</aff>
<aff id="aff2">
<label>2</label>
<addr-line>Shanghai Academy of Artificial Intelligence for Science, Shanghai, 200232, China</addr-line>
</aff>
<aff id="aff3">
<label>3</label>
<addr-line>Department of Electrical and Computer Engineering Shanghai Normal University, Shanghai, China</addr-line>
</aff>
<aff id="aff4">
<label>4</label>
<addr-line>Jiangsu Meteorological Observatory, Nanjing, Jiangsu, 21008, China</addr-line>
</aff>
<aff id="aff5">
<label>5</label>
<addr-line>Nanjing Meteorological Observatory, Nanjing, Jiangsu, 21009, China</addr-line>
</aff>
<pub-date pub-type="epub">
<day>22</day>
<month>10</month>
<year>2024</year>
</pub-date>
<volume>2024</volume>
<fpage>1</fpage>
<lpage>19</lpage>
<permissions>
<copyright-statement>Copyright: &#x000a9; 2024 Qian Liu et al.</copyright-statement>
<copyright-year>2024</copyright-year>
<license license-type="open-access">
<license-p>This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this licence, visit <ext-link ext-link-type="uri"  xlink:href="https://creativecommons.org/licenses/by/4.0/">https://creativecommons.org/licenses/by/4.0/</ext-link></license-p>
</license>
</permissions>
<self-uri xlink:href="https://egusphere.copernicus.org/preprints/2024/egusphere-2024-3183/">This article is available from https://egusphere.copernicus.org/preprints/2024/egusphere-2024-3183/</self-uri>
<self-uri xlink:href="https://egusphere.copernicus.org/preprints/2024/egusphere-2024-3183/egusphere-2024-3183.pdf">The full text article is available as a PDF file from https://egusphere.copernicus.org/preprints/2024/egusphere-2024-3183/egusphere-2024-3183.pdf</self-uri>
<abstract>
<p>The rapid advancement of artificial intelligence (AI) in weather research has been driven by the ability to learn from large, high-dimensional datasets. However, this progress also poses significant challenges, particularly regarding the substantial costs associated with processing extensive data and the limitations of computational resources. Inspired by the Neural Image Compression (NIC) task in computer vision, this study seeks to compress weather data to address these challenges and enhance the efficiency of downstream applications. Specifically, we propose a variational autoencoder (VAE) framework tailored for compressing high-resolution datasets, specifically the High Resolution China Meteorological Administration Land Data Assimilation System (HRCLDAS) with a spatial resolution of 1 km. Our framework successfully reduced the storage size of 3 years of HRCLDAS data from 8.61 TB to just 204 GB, while preserving essential information. In addition, we demonstrated the utility of the compressed data through a downscaling task, where the model trained on the compressed dataset achieved accuracy comparable to that of the model trained on the original data. These results highlight the effectiveness and potential of the compressed data for future weather research.</p>
</abstract>
<counts><page-count count="19"/></counts>
</article-meta>
</front>
<body/>
<back>
</back>
</article>