Behind the colorful pictures of the ocean, monthly reports about the climate, and graphics about billion-dollar disasters lives a lot of data. NCEI’s role as the primary environmental archive for NOAA covers an expanding territory of information.
Why Archiving Matters
NOAA invests billions of dollars into Earth Observing Systems, and archiving data ensures the greatest value to the Nation from those data. Providing retrospective environmental information lends context for what is happening today to inform decision making, to improve community and economic resiliency, and to understand changes in our Earth environment. The archive serves as a resource for many purposes—to improve warnings for tsunamis, help coastal fishing and tourism, support farmers who produce crops and livestock, and prepare for severe weather and changes in climate, to name only a few.
The value of our archive is apparent from our data users. Each week, our customer service representatives and staff document the contacts we have with the public and those who use our data. We want to know who needs or uses NOAA environmental data and how the information is being applied to solve problems and make decisions. Based on our documentation and analysis, every sector of the U.S. economy listed in the North American Industry Classification System (NAICS) uses NCEI data.
Every data point in our national archive is a building block for a better future. And, the archive continues to grow. As of late 2023, NCEI was responsible for over 60 petabytes of data with exponential growth expected from new observing platforms and technology. This is in addition to a physical archive in climate-controlled storage facilities, which contains over 38,490 boxes of documents, more than 135,000 reels of microfilm, and about 800,000 pieces of microfiche.
Information about the past has demonstrated the potential to have tremendous positive impacts on our quality of life. Environmental data offer an irreplaceable picture of our atmosphere, land, ocean, seafloor, and space that informs decisions today and those of the next generations.
What is Archiving?
Archiving, in essence, is a more specific term that basically means saving. NCEI saves today’s information for use by current and future generations. Archiving more precisely involves storing, describing, processing, and providing the data with the ultimate goal of preservation, broad availability, and reuse.
- We store data in all types of ways—digitally on tape robotics systems, the cloud, and on the hard drives of computer servers; in physical form, such as paper records in a secure, climate-controlled facility; and on reel-to-reel film, video, and digital image formats.
- We describe data by telling the story of the data including who, what, where, why, and when it was collected, what we call metadata. This is documentation that describes the kind of artifact or digital record that we steward. It could include a date, a place, a timeframe, a format, a measurement type (℃ or inch, for instance), a geographical location, instrumentation that was used, how it was processed, or the name of the observer.
- We process data in some instances from its raw form so that others can use it more easily, both for today and in the future.
- We preserve data by ensuring it is safely stored in a format that can be used for generations, adding information that describes the data and how to use it.
- We provide data by making it broadly available and independently understandable with information that describes the data and how to use the information.
Simply storing data does not ensure it will be usable again. That might be easier to grasp with a story about your grandparents. Those old photographs in the attic may be stored and accessible, but without any details about who and what are in the pictures, they’re just paper. Data are far more valuable when they come with information. Our ongoing archival stewardship adds value to environmental data over time and responds to changes in technology and science. So rather than letting your grandparents’ photos sit in a dusty box, our processes would involve digitizing the image, adding a caption (who, what, where, when), and then storing it in the cloud for others to use.
Data Come in Many Forms, From Many Sources
A major challenge of stewarding environmental information for future generations is its diversity in format and source. The kind of data and their formats differ greatly in our archive. Many factors determine those characteristics, including the observing system and the observation being collected. Video footage of an Okeanos Explorer expedition is collected and stored in a specific format that’s not the same as a monthly precipitation value from Alaska or visualizations of space weather from the satellite DSCOVR.
The same is true for data collected by a glider than that of a weather satellite. A glider can gather samples of micronutrients, such as plankton, while a satellite can record the concentration of ozone in the atmosphere. These would be documented differently in our archive. In short, data comparisons and integration are not always apples to apples. That’s where data management standards and best practices become important and why we set those standards universally. Those standards allow us to describe all types of data in a consistent manner.
Not only does NOAA manage a wide array of observing systems, including nine satellites orbiting Earth, but NCEI receives information and datasets through NOAA-funded projects, federal offices, cooperative institutes, and partners from around the globe. To manage this diversity of data, NCEI works closely with our partners across NOAA and the international community to understand, appraise, and prepare data for archiving. All data go through a scientific appraisal and approval process that refers to our Archive Collecting Policy, which defines the data we archive.
How does this information get delivered to NCEI? It depends on what kind of data is submitted. Historically, we have primarily received data through hand-delivered documents or media. More and more, however, data are “ingested” through digital portals, such as Send2NCEI (S2N), or by direct digital delivery.
Potential data contributors should email NCEI to discuss a submission, including whether it is within NCEI’s scope and to determine the best method to submit. Also, Advanced Tracking and Resource Tool for Archive Collections (ATRAC) is recommended to submit requests to send NCEI data. Our staff members have the knowledge to navigate metadata creation, guide data format choices, and understand best practices and other nuances of the data submission process.
