U.S. flag

An official website of the United States government

Archive

NCEI provides archive services for much of the data collected by NOAA scientists, observing systems, and research initiatives. We manage a comprehensive collection of environmental information from a broad range of time periods, observing systems, scientific disciplines, and geographic locations. Our world class stewardship services preserve data for future use on behalf of our science-user communities and the general public.

Data Sources

NCEI receives data from many different sources, including individual researchers, other government organizations, and research institutions from all over the world. The data submitted to the NCEI archive are subject to one of the three following quality control procedures:

NOAA Scientists

Data from NOAA scientists and contractors are subject to the NOAA Procedural Directives that provide guidelines for Data Management Planning (archiving responsibilities), Data Documentation (metadata requirements), Data Access (discoverability), and Data Citation (credit and citations).

NOAA Funded Projects

Data providers who received NOAA funding through a grant, cooperative agreement, or contract should review project-specific guidelines for data archive responsibilities and related peer-reviewed publications. Those guidelines can be found in the Data and Publication Sharing Directive for NOAA Grants, Cooperative Agreements, and Contracts.

Non-NOAA Funded Data

Data collected without NOAA funding or support must go through a scientific appraisal process to be considered for the archive, and is subject to the NESDIS non-NOAA data policy upon approval.

Submitting Your Data

NCEI follows a robust, well defined archive process to preserve data for future use. We accept data, metadata, and products that add value to the core NOAA data collection, and contribute to the long-term preservation of the global environmental data record.

Submission Methods

Contact Us

Before contributing data, first time submitters should email ncei.archive@noaa.gov to consult with an archive specialist. Our staff can help you choose the correct submission method and provide a detailed explanation of the data submission process.

Contact Us

ATRAC

Use the Advanced Tracking and Resource Tool for Archive Collections (ATRAC) to submit repeating or multiple delivery data, or data that exceeds 20 GB.

ATRAC Guidelines

Launch ATRAC

S2N

Use Send2NCEI to submit non-repeating or single delivery data less than 20 GB.

Launch S2N

Data Submission Format Guidance

NCEI prefers open file formats maintained by a standards organization rather than proprietary or product-specific options. Self-describing formats like netCDF and HDF include valuable metadata that support data compatibility and long-term information preservation. PDF or PDF for archiving (PDF/A) are preferred for archived documents. Above all, choose a format that accommodates characteristics of the data being encoded. Additionally, consider the format’s long-term (decades into the future) compatibility with data access software and infrastructure. For example, tabular data in comma-separated UTF-8 or ASCII files are much more likely to be accessible to future operating systems than proprietary spreadsheet formats (e.g., Lotus 1-2-3).

NCEI recommends using the National Archives and Records Administration (NARA) transfer formats for digital data. Refer to the Library of Congress Sustainability of Digital Formats page when considering formats outside of the NARA list. All submissions are subject to review, and may need to undergo changes to format, filename, packaging, and other criteria to meet system or preservation requirements before being approved.

Frequently Used Formats
Generic Data Types Preferred Format(s) Comment(s)
Tabular Data
  • Comma Separated Values (.csv)
  • Tab Separated Values (.tsv)
  • NetCDF with CF conventions (.nc)
  • OpenOffice Calc (.ods)
  • Fire History Exchange Format (FHX2)
  • Tuscon Decadal Format
  • Paleo Submission Template
 ASCII or UTF/UTF8/UNICODE encoding; See additional NetCDF guidance for many kinds of observation data (i.e, profiles, time series, trajectories, etc.)
Geospatial Information Systems (GIS)
  • GeoTIFF
  • Keyhole Markup Language (.kml)
  • Laser (LAS) File Format
  • OpenGIS GML
  
Text
  • Portable Document Format/Archival (.pdf)
  • OpenOffice Document (.odt)
  • UTF-8 and ASCII (.txt)
 Text documents must meet NOAA accessibility guidelines for compliance with Section 508
Metadata ISO 191xxx (series of international Geographic Metadata) (.xml) The preferred representation of metadata in these standard formats is eXtensible Markup Language (xml)
Moving Images
  • MPEG-2
  • MOV
 Please contact ncei.info@noaa.gov before submitting video data. Be sure to assess potential copyright and privacy issues before archiving.
Still Images TIFF  
Others
  • GRIdded Binary or General Regularly-distributed Information in Binary (GRIB)
  • Binary Universal Form for the Representation of meteorological data (BUFR)
  • Water Column Sonar (*.raw, *.wcd, etc)
  • Bathymetric sonar (*.all, *.gsf, etc).
  • RDF/SKOS
  

File Naming Conventions

The names of files submitted to NCEI must:

  • Be comprehensible and readable
  • Exclude special characters (such as punctuation and symbols)
  • Use appropriate file format extensions

Additional file naming and packaging rules may apply for data based on the submission pathway, or data that must adhere to specific program guidelines. NCEI personnel are available to review and provide feedback on files as needed.

Supporting Documentation

An archive requires sufficient information to read, understand and characterize data holdings in accordance with documentation standards. This documentation contextualizes data, allowing current and future users to:

  • Read the encoded format including descriptions of the data variables/content
  • Understand data lineage pertaining to how it was created
  • Characterize data quality to assess its usability
  • Identify and distinguish data
  • Trust the data integrity

Common supporting documents include user guides, format specifications, algorithm theoretical basis documents, and read-me files. Other forms of documentation may be appropriate for archiving to support data stewardship. NCEI may request additional documentation in order to meet these requirements.

The submitter is responsible for ensuring that documents comply with Federal Section 508 accessibility requirements. The NOAA Central Library Section 508 compliance page has many resources, such as how to create 508 compliant documents and tools for checking accessibility.

Collection Metadata

NCEI requires standardized metadata for every collection or dataset to support discoverability, understandability and interoperability. This documentation includes a standard description, instructions for accessing and citing the data, and other pertinent details. Collection and metadata creation is supported through the various data submission methods.

Data Licensing

From the NESDIS/NCEI Open Data Policy:

GENERAL POLICY AND GUIDANCE

  1. Environmental data and information produced by NOAA or any Federal agency are available fully and openly to data users unless explicitly exempt from release by Federal law, regulation or policy. These data are in the public domain in the United States. To serve our international partners and customers, NCEI will work to ensure a Creative Commons CC0 license is applied.
  2. Environmental data and information collected or created under NOAA grants or cooperative agreements must be made visible, accessible, and independently understandable to general users in a timely manner (typically no later than two (2) years after the data are collected or created) (See 5d.). Except where limited by law, regulation, policy or security requirements, NCEI will strongly encourage providers to apply a Creative Commons CC0 (preferred) or CC BY 4.0 license in grant or agreement language, or upon submission.
  3. NCEI may archive data produced by non-Federal entities for which the creator either surrenders or maintains intellectual rights. NCEI will seek to minimize access restrictions for environmental data and information acquired from non-Federal sources, and will encourage providers to apply for a Creative Commons CC0 (preferred) or CC BY 4.0 license. If it is determined that accepting limited-access data is not in the best interest of the public, NCEI may decline to accept data. (See 5e.)

NCEI Archive Services

Data Discovery and Access

NCEI has a number of search and visualization tools that allow users to find data using custom queries that pull from the entire archive, as well as individual products and datasets. Following the archive process early in your data gathering and collection efforts can optimize its discoverability with these systems.

Data Identification and Attribution Using DOIs

NCEI may mint Digital Object Identifiers (DOIs) for data held in its archive in accordance with the NOAA Data Citation Procedural Directive. Once minted, a DOI provides a unique and persistent identifier that allows users to accurately cite and locate data obtained from NCEI. Data without an assigned DOI can be cited, but may not have a persistent URI or have citation metadata published in the DataCite discovery service. See FAQs below for more information.

Data Preservation

Data stewarded by NCEI must include metadata about provenance, authenticity, the technical environment necessary to use the data object, what preservation actions have been taken, and what intellectual property rights apply to the data object. To provide long term preservation for digital and analog data holdings, NCEI follows guidance from many best practice communities to ensure that data are accurate and authentic for the long term.

As mandated by Federal regulations, data maintained by NCEI are scheduled in accordance with the appropriate NOAA Records Schedules. These schedules identify many records series which indicate the requirements for how long those records (data and metadata) must be retained, and other conditions for retention and preservation. Each record series must be approved by the National Archives and Records Administration (NARA).

To support and ensure data preservation, NCEI routinely performs regular media migration, data integrity checks (e.g., comparing cryptographic hash values prior to and following file movement between storage media), daily and weekly media content backups, and other practices to ensure access to data objects. NCEI follows guidance from NARA for backup processes, storage media, etc.

FAQs

Why should I archive my data?

The goal of data preservation is to ensure that your data are independently understandable for future use and re-use. To assist you, NCEI follows a well defined archive process based on the NOAA Procedural Directives that contextualizes data and documentation to ensure that they remain valuable and functional for as long as possible. Because all data acquired, observed, processed or recovered by NOAA is subject to these guidelines, it’s important to incorporate them early in a project’s lifecycle to ensure data compliance and long term viability.

How does NCEI decide if my data will be archived?

NCEI generally follows the guidance outlined in the NOAA Procedure for Scientific Records Appraisal and Archive Approval (2008). NCEI also follows a policy for receiving and using non-NOAA data issued by our parent organization, the NOAA National Environmental Satellite, Data, and Information Service (NESDIS). This policy establishes criteria to authenticate incoming non-NOAA data, ensure consistency and sustainability, and maintain security. Much of the appraisal process is based on the information you provide to NCEI before submitting data. NCEI considers many factors, including but not limited to:

  • What type of data is being presented to NCEI to archive? For general categories of the types of data acquired by NCEI, please refer to the NCEI Archive Collecting Policy.
  • Is the original observation data and metadata to be archived, a "data product," developed from other data sources, or both? If a "data product," is the original observation data archived elsewhere and if so, where?
  • Is this a one-time submission of a relatively small data product (less than about 20GB total), or would the product require a frequently-updated or repeating submission of new/revised data? This helps NCEI advise about the most appropriate data submission process for the data.
  • How large (in MB/GB/TB/PB) is the data or data product to be archived? 
  • Are consistent file naming conventions being used? Please describe the file naming convention.
  • What format is used to represent the distribution analysis product? NCEI prefers data in formats that do not require specific proprietary software for re-use.
  • Is there descriptive metadata for these data and/or product(s)? NCEI uses the ISO 19115 family of metadata standards, with NASA Global Change Master Directory (GCMD) keywords, to describe NCEI archival data holdings.
  • Are additional services to discover, access, visualize, or distribute the analysis product needed, or would the standard suite of NCEI discovery and access tools (e.g., https, ftp, THREDDS, Live Access Server) be sufficient?

How should I submit my existing relational database (RDBMS) to NCEI for archiving?

Relational database management systems (RDBMS) present many unique challenges for long-term preservation and access. Please contact ncei.info@noaa.gov prior to submitting data in an RDBMS to determine what is the best solution for your RDBMS data.

What are the acceptable file formats for data archived at NCEI?

There are many "acceptable” file formats. There is no 'one size fits all' format, so it is best to use a standards-based, community-recognized, non-software specific file format for encoding the data. The NCEI list of preferred formats, which are based on guidance by the National Archives and Records Administration and the Library of Congress.

An 'ideal' format is:

  • Not software specific (i.e., is not limited to use by a single, specific program or application)
  • Based on well-documented national or international standards based format specifications
  • Suitable for the type of data that are encoded in the format

Can I get a Digital Object Identifier (DOI) for data at NCEI?

Please notify NCEI while preparing the data for archiving that you would like to obtain a DOI. Data must be archived with NCEI in order to be eligible for a NOAA Data DOI assignment. NCEI follows the guidance in the NOAA Data Citation Procedural Directive and will work to assign a DOI, as necessary.

Does NCEI accept "DNA data"?

Not at this time due to the archive storage resource constraints. NCEI is reviewing options to better support these data. NCEI may accept summarizations of genetic information, but contact ncei.info@noaa.gov prior to submitting genetics data.

Can NCEI embargo access to data I submit until my paper is published?

Possibly. NCEI follows the guidance in the NOAA Data Access Procedural Directive for providing access to data provided by federal staff. NCEI follows the guidance in the NOAA Data and Publication Sharing Directive for NOAA Grants, Cooperative Agreements, and Contracts for providing access to data provided by non-federal staff.

Does NCEI archive software that I developed for use with my data?

NCEI does not currently archive software, scripts or other model/product generating technologies that were not written by NCEI staff.

What is “Science Ready Data”?

NCEI refers to 'science ready data' as data that are processed from engineering units to units used in scientific analyses (e.g., voltages measuring the electrical conductivity of seawater would be engineering units, but salinity calculated from the voltages would be the 'science ready data'). Satellite data are often referred to as 'Level 0', 'Level 1', etc., so 'Level 0' are the initial values measured by the satellite based instrument and sent to Earth, where specific algorithms are applied to transform those 'engineering units' into 'science ready data'.

What data licenses will NCEI accept?

NOAA has adopted the well-known and internationally recognized licenses from Creative Commons. NCEI encourages external data submitters to assign Creative Commons 1.0 Universal Public Domain Dedication (CC0-1.0) onto their data, which ensures maximum usage of the data throughout the industry, for anyone's use and for any purpose.

Submitters who wish to maintain copyright on their data can use the Creative Commons Attribution 4.0 International License (CC BY 4.0). Under this license, the data may be used by anyone, for any purpose, so long as credit is given to the provider when using the data in its original form.

NCEI may accept data submitted under other licenses. Data submitters with questions are encouraged to contact ncei.info@noaa.gov to consult with an archive specialist.

For contributed data that have not been licensed by the data submitter, will NCEI apply for a license?

No. Data submitters should license the data before submitting to NCEI.

Has NCEI retroactively licensed their data holdings?

No. As with new acquisitions, all existing holdings must be licensed by the data submitter or originator. Because works of the US government are in the public domain in the United States, NCEI can assign a CC0 license to that portion of its holdings. Beyond that, NCEI may reach out to other data submitters to recommend licenses for their submissions. This work has not begun.