spacer link to MAST page spacer logo image spacer
 
link to STScI page


STScI's Internet connections will be UNAVAILABLE Saturday, 28 October 2017 from 7:00 AM - 1:00 PM. STScI web servers, SSH and similar services will be unreachable from the Internet.

MAST HLSP Spectral Data Delivery Standards

Spring 2013

This page describes FITS spectra data standards for HLSP deliveries. Also please read the general HLSP guidelines page. Under some circumstances we will accept spectra in ASCII format; the requirements for this format are the same as for FITS with respect to header keywords (the metadata description of the data). FITS keywords fall into three categories: required, recommended and optional. more | less

We have tried to generalize the FITS keywords requirements across a number of spectra types, therefore some of these keywords may not be applicable to your data, especially for composite spectra. We suggest you consult a MAST staff member prior to delivery, to be sure your data abides by the FITS and MAST delivery standards.

MAST has put together the following information because it is essential for data deliveries. If the archive cannot extract the information they need from dataset headers, MAST will attempt to calculate it based on original exposures when possible. If MAST is not able to extract or calculate what it needs based on the delivered dataset headers, then some of the data will not be searchable and will not be displayed across multi-mission search results.

[General Information]   [FITS File Formats]   [FITS & WCS Standards]   [ASCII Standards]  

[Composite vs Single Spectra]   [Observed vs Rest-Frame]   [Keyword Nomenclature]   [Units]  

[Required Keywords]   [Recommended Keywords]  [Optional Keywords]  [Examples

General Header Information

In general, spectra should be delivered as final product, extracted, FITS files. Ancillary spectra data products (earlier products) are recommended as deliveries with final products. The following 5 types of spectra are accepted:
  • 1-D spectra (traditional wavelength vs flux), in FITS format [ASCII format accepted]
  • 2-D spectra (long slit)
  • Grism spectra (i.e. HST-Grism)
  • IFU (Integral Field Unit) 3-D spectra (i.e. JWST-NIRSPEC)
  • Echelle spectra (stored in Multi-Extension FITS)

MAST requires that FITS headers contain the following information (see Required keywords below): more | less

  • Relevant ID information (program ID, proposal ID, etc) if appropriate
  • Time coverage, in order of preference (more than one may be present):
    • Flux-weighted midpoint of integration.
    • Mid-point of integration.
    • Start time of integration + total length of integration.
  • List of exposures (origin of the spectra, only needed for MAST Missions)
  • Pointing location (RA, DEC)
  • World Coordinate System (WCS, for wavelengths)

FITS File Formats

The preferred spectral delivery format is FITS, but ASCII is also accepted. For tabular, extracted, 1-D spectra in either FITS or ASCII, we suggest three data columns: wavelength, flux and flux_error. Examples are available below. Spectra can be specified in at least 4 different ways within a FITS file: more | less

The following list enumerates what MAST will accept in order of preference; the links provided show the mandatory keys for that extension based on the FITS Standard v3.0 (Pence et al. 2010) paper:

  1. FITS Binary Table Extension - rows and columns of data in binary representation
  2. FITS ASCII Table Extension - rows and columns of data in ASCII character format
  3. FITS 1-D "Image"/Spectra - a 1-dimensional array of pixels, like in a primary array
  4. FITS N-D Image/Spectra - a N-dimensional array of pixels, like in a primary array

The FITS paper also provides a summary of the mandatory keywords for the above file types.

Expand for non-FITS (ASCII) details | Retract/Close details

Another option is to use non-FITS, ASCII tables of 1-D spectra, which are also accepted by MAST. This format equates to the FITS Binary or ASCII Table Extension formats, but in straight ASCII files. More information is provided below.

FITS & WCS Spectra Standards

The above requirements translate into FITS header keywords below. These keywords and files themselves must abide by the FITS standard, and therefore cannot have errors when being checked by tools like 'fverify' or 'fitsverify'. more | less

For 1-D or N-D FITS spectra/images, please read the spectral WCS resources for WCS header details.

For multi-extension FITS files (i.e. FITS Tabular data, Echelle), please note that many of the keywords cannot be in the 0th extension or HDU, they need to be within the corresponding extension(s) which contain the data.

ASCII Spectra Standards

ASCII spectra deliveries are accepted as long as the data are well-defined and documented. Header keywords must be provided at the top of the ASCII file or as an associated data pair (one file contains the header information and the other contains the data themselves). Please abide by the FITS keyword naming scheme as described below; all required keywords must be present.

Expand for ASCII formatting and header details | Retract/Close details

The data format (columns) must be documented within the file itself and/or the supporting README file. For 1-D tabular spectra, we suggest three data columns: wavelength, flux and flux_error. All columns and rows must contain data values; blanks are not allowed. For blank values, please use value 'NaN' or 'NULL' in the ASCII file, but not both within the same file. Within a data line, fields are separated by one or more whitespace characters (space or tab). A field is either a sequence of non-whitespace characters or a sequence of characters between two matching quote characters (single (') or double (") quotes) - spaces are therefore allowed in quoted fields.

Specification of keywords and description of the column names in the ASCII file should be done as follows:

  • Use the '#' symbol to precede all keyword text lines (non-data text lines)
  • The next 8 characters are used for FITS keyword name; buffer the 8-chars name with spaces (see "NAXIS" keyword in example below)
  • The 9th character is an equal sign '=' followed by a at least a single space
  • The keyword value can be a number or string; strings must be in single quotes
  • Keyword comments are written after a "/" symbol
  • The FITS 80-character line length limit applies here: no more than 79 characters in total can be used on any given line of text; the 80th char is the end of line/return char
  • The FITS "COMMENT = " keyword should be used to describe any additional information
  • The "COMMENT = " keyword value is always a string within single quotes

The following is an ASCII header example of keyword, value pairs, including the 4 data column descriptions for wavelength, flux, error, and data quality. Note that these map directly onto FITS header keywords so that MAST can build FITS files from ASCII tabular spectra deliveries.

                                                           
#TELESCOP= 'HST'                / telescope used to acquire data                 
#INSTRUME= 'STIS'               / instrument used to acquire data                              
#RA_TARG =            82.586460 / right ascension of target (deg) (J2000)        
#DEC_TARG=            -7.434805 / declination of target (deg) (J2000)               
#DATE-OBS= '2003-04-28'         / UT date of start of first exposure             
#TIME-OBS= '09:20:38'           / UT start time of first exposure                
#
#XTENSION= 'BINTABLE'           /Written by IDL:  Mon May 25 12:26:32 2009       
#BITPIX  =                    8 /                                                
#NAXIS   =                    2 /Binary table                                    
#NAXIS1  =              1152000 /Number of bytes per row                         
#NAXIS2  =                    1 /Number of rows                                  
#PCOUNT  =                    0 /Random parameter count                          
#GCOUNT  =                    1 /Group count                                     
#TFIELDS =                    4 /Number of columns                               
#EXTNAME = 'E230H-2013_310X005N_52757' /Extension name                           
#EXTNO   =                    1 /Extension number                                
#TFORM1  = '64000D  '           /Real*8 (double precision)                       
#TTYPE1  = 'WAVE    '           /Column 1: Wavelength                            
#TUNIT1  = 'Angstroms'          /Units of column 1                               
#TFORM2  = '64000E  '           /Real*4 (floating point)                         
#TTYPE2  = 'FLUX    '           /Column 2: Flux Density                          
#TUNIT2  = 'erg/s/cm^2/A'       /Units of column 2                               
#TFORM3  = '64000E  '           /Real*4 (floating point)                         
#TTYPE3  = 'ERROR   '           /Column 3: Photometric Error                     
#TUNIT3  = 'erg/s/cm^2/A'       /Units of column 3                               
#TFORM4  = '64000I  '           /Integer*2 (short integer)                       
#TTYPE4  = 'DQ      '           /Column 4: Data Quality                          
#TUNIT4  = 'unitless'           /Units of column 4  
#
#COMMENT = 'Delivered to MAST from the StarCat HLSP project'
#END                                                                             

In most cases we will translate ASCII 1-D spectra into FITS files when we ingest the datasets into MAST. Additional examples of ASCII spectra files are available below.

Composite vs Single Spectra

Composite data deliveries of MAST Missions are required to contain a list of the original datasets from which the combined dataset was made. more | less

Composite data are defined as spectra which were constructed from other spectra (e.g., spectra from multiple instruments stitched together). The most common examples of composite data are highly processed science products made from other spectra, or 1-D spectra extracted from 2-D or 3-D datasets.

The list of original datasets from which the dataset was made can be a FITS table as a separate HDU in the composite dataset, or an ASCII list of exposure names (e.g. IPPPSSOOT convention for HST, etc.) per composite dataset.

An example composite map from the CLASH HLSP project can be viewed here where the 2nd column is the ipppssoot name for HST; for other missions, the data ID is required, see example for the FUSE Magellanic Cloud HLSP project.

Observed vs Rest-Frame

Spectra should be delivered in the observed-frame (rather than the rest-frame). The archive will never correct spectra to be in the rest-frame. more | less

The data provider can include values in the header which are needed to do that correction (barycentric correction, observed Doppler shifts). In addition, spectra corrected to the rest-frame may be submitted along with the uncorrected spectra, but should never be provided as the sole data product.

Header Keyword Nomenclature

For a single spectrum, the keyword will contain a value; if the spectrum is a composite, that keyword can contain the value of "MULTI", followed by a list of N similar keywords which encompass all the values needed to explain the composite data using a series of related keywords. more | less

For example, a single spectrum can have FITS keyword 'INSTRUME' = "COS". For a composite spectrum made by stitching COS and STIS spectra, the keys would be as follows:

  • INSTRUME = "MULTI"
  • INSTRU01 = "COS"
  • INSTRU02 = "STIS"

To designate multiple keywords, we denote this as [nn].

Keyword Value Units

Some keywords have standard units and do not need to be explicitly specified in the headers. In cases where you need to specify units for header keywords, this can be done in one of two ways: more | less

  1. KEYWORD + KEYWORD_UNIT pair
  2. You can specify the keyword units using a second, similar keyword where the keyword name contains the string "_UNIT" or "UNIT". Please remember to stay within the FITS 8-character keyword name limit. E.g.:

    RADVEL = 1688 / The observed Radial Velocity of the object in velocity units.
    RVUNIT = 'km/s' / The units of RADVEL, one of {m/s, km/s}.

  3. KEYWORD = value / [unit] in brackets within the FITS comment
  4. Comments can contain the keyword unit within brackets; this should be the first text following the standard fits comment delimiter " / ", the single slash. Please remember to stay within the FITS 80-character line length limit, which includes the comments. E.g.:

    RA_TARG = 2.0952854139 / [deg] right ascension of target



KeywordDescription
  
             // DATA DESCRIPTION KEYWORDS
TELESCOP observatory (e.g. HST, IUE, ISO, VLT) [Expand TELESCOP values List | Retract/Close List]

INSTRUME single instrument alone, or instrument/detector names, or instrument/detector-subdetector names (e.g. "STIS" or "IUE" or "FOS" or "STIS/FUV-MAMA") or "MULTI" for composite [View Current List]
INSTRU[nn] if INSTRUME="MULTI", then an element from allowed values of INSTRUME [View Current List]
TARGNAMEtarget name (according to raw data or catalog); "MULTI" for multi-object/composite case
RA_TARG The right ascension of the object for object spectra, or the center of the image for 2D grism spectra (deg)
DEC_TARG The declination of the object for object spectra, or the center of the image for 2D grism spectra (deg)
  
             // DATE AND TIME KEYWORDS
DATE-OBS date and time of observation start in the ISO standard 8601 format: YYYY-MM-DDThh:mm:ss.sss; start of first observation if composite
(DATE-OBS) Alternative, accepted date format (yyyy-mm-dd); must be paired with TIME-OBS
(TIME-OBS) Alternative, accepted time format (hh:mm:ss); must be paired with DATE-OBS (yyyy-mm-dd)
EXPTIME effective exposure time (seconds) for single exposure, or "1" for composite (which are typically in units of counts/second, see EXP* keywords below).
EXPSTART start time of observation, or first exposure if composite [MJD]
EXPEND end time of observation, or last exposure if composite [MJD]
  
             // For Tabular Spectra: FITS BINARY/ ASCII TABLE EXTENSION KEYWORDS
SIMPLE T / FITS standard
XTENSION Type of extension: BINTABLE or ASCII table
EXTNAME Extension name (single word)
(EXTNO) Extension number - recommended keyword
BITPIX number of bits that represent a data value
NAXIS 2 / denoting that the included data array is two-dimensional: rows and columns.
NAXIS1 the number of 8-bit bytes in each row of the table
NAXIS2 number of rows in the table
PCOUNT Random parameter count
GCOUNT Group count
TFIELDS Number of columns
TBCOL[n] [FITS ASCII table ONLY] Integer specifying the column in which field n starts (starting at 1).
TFORM[n] [FITS BINARY table ONLY] Format of column (valid data types)
TTYPE[n] Name of column (e.g. WAVE, FLUX, ERROR, DQ)
TUNIT[n] Column units (e.g. Angstroms, erg/s/cm^2/A, unitless)
  
             // For 1,2,3-D Spectra: WORLD COORDINATE SYSTEM (WCS) KEYWORDS
SIMPLE T / FITS standard
RADESYS astrometric reference system (e.g. FK5, ICRS)
BITPIX number of bits that represent a data value
NAXIS number of data axes in this file
NAXIS[n] number of points on axis [n], the size of each dimension
DISPAXIS axis that represents the dispersion dimension; one of {1,2,3}.
CTYPE[n] description of axis (array)
CUNIT[n] units of axis (array)
CRPIX[n] pixel at axis origin
CRVAL[n] value at axis origin
CD[n]_[n] spectrum coordinate matrix; mandatory term CD1_1 for FITS 1-D spectra, i.e. Linear dispersion (Angstrom/pixel); else, required all 4 terms 1_1, 1_2, 2_1 and 2_2 for FITS 2-D and higher dimensions
  

For multi-extension FITS files (i.e. FITS Tabular data, Echelle), please note that many of the keywords cannot be in the 0th extension or HDU, they need to be within the corresponding extension(s) which contain the data. Please check for additional required keywords for Echelle data.


RECOMMENDED FITS HEADER KEYWORDS


KeywordDescription
  
APERTURE for HST, the name of the aperture of the instrument (STIS apertures)
PROPOSID Proposal ID (when available)
HLSPLEAD Full name of the lead of HLSP project (Firstname M. Lastname)
PR_INV_L Last name of principal investigator
PR_INV_F First name of principal investigator
PR_INV_M Middle name or initial of principal investigator
  
EPOCH Epoch of the observation (considered as deprecated; use EQUINOX when possible)
BC The barycentric correction one needs to ADD to correct for the Earth's rotation, Earth-Moon revolution, and orbit around the Sun. (km/s) [more]
  
WAVEMIN minimum wavelength n'th order
WAVEMAX maximum wavelength n'th order
WAVEUNIT Unit for the MINWL, MAXWL and wavelength coordinates in the file {nm, ang, micron, mm, wavenum}
AIRORVAC are the wavelengths in air or vacuum? One of {air, vac}.
SPECRES spectral resolution (delta lambda) at a given wavelength provided by WAVERES.
WAVERES The reference wavelength of SPECRES in units of WAVEUNIT, the resolving power "R" (often misquoted as the "resolution") would then be given as WAVERES / SPECRES.
  
WAVECENT effective wavelength, or central wavelength
FLUXMAX maximum flux
FLUXMIN minimum flux


OPTIONAL FITS HEADER KEYWORDS


KeywordDescription
  
HLSPNAME High Level Science Product (HLSP) project name, long form
HLSPACRN High Level Science Product (HLSP) project acronym, short form
CITATION citation reference of this project, for publications (else in HLSP README file)
DETECTOR detector of the instrument (same order as INTRUME) or "MULTI" for composite exposure
DETECT[nn] list of detectors of the instrument of composite exposure
GRATING grating of the instrument
FILTER filter for single exposure or "MULTI" for composite exposure [View Current List]
FILTER[nn] if FILTER="MULTI", filters list when composite exposure [View Current List]
POSANGL position angle
  
EXPDEFN string describing how EXPTIME was calculated, i.e. 'MEAN', when composite exposure
EXPMIN if EXPDEFN='MIN', numerical value = minimum composite exposure
EXPMAX if EXPDEFN='MAX', numerical value = maximum composite exposure
EXPMODE if EXPDEFN='MODE',numerical value = mode of composite exposure
EXPMED if EXPDEFN='MED', numerical value = median of composite exposure
EXPMEAN if EXPDEFN='MEAN', numerical value = mean of composite exposure
EXPSUM if EXPDEFN='SUM',numerical value = sum of composite exposure
  
RADVEL The observed Radial Velocity (RV) of the object in velocity units. Include CCF/Bisector info where possible.
RVUNIT The units of RADVEL, one of {m/s, km/s}.
REDSHIFT The observed redshift of the object, expressed as a "z" value.
RVERR 1-sigma uncertainty for RADVEL in units of RVUNIT
VHELIO Heliocentric RV in the same velocity units as in RVUNIT.
VLSR RV of the object in the Local Standard of Rest in units of RVUNIT.
VGSR RV of the object in the Galactic Standard of Rest in units of RVUNIT.

Note: Supplemental Data (cross-correlation function, line bisectors, IRAF-style wavelength solutions and masks)

more | less

  • The cross-correlation function (CCF) should be provided if a radial velocity (RADVEL) is measured with such a technique. Other similar products should be provided if using broadening functions, etc. (This is to be able to assess (a)symmetry of the function's shape, search for signals from additional sources, etc.)
  • If high spectral resolution, line bisectors would be another good product to provide if an RV measurement is provided.
  • IRAF-style wavelength solution parameters (e.g., best-fit coefficients in functional fit to identified reference lines) can be supplied, along with the linelist and/or calibration spectra used to determine the wavelength solution.
  • If masks or corrections used to remove contaminating lines are not in the FITS files already, we encourage users to include these files.

HLSP Spectra Header Examples

Expand for Examples | Retract/Close Examples

MAST has been ingesting and distributing HLSP data products for over 10 years. During this period, the requirements for HLSP data deliveries have expanded in order to help unify all datasets housed at MAST for ease of multi-mission searching. The example HLSP headers may not abide by all the requirements listed above because they were delivered prior to some requirements being written. We encourage the data delivery teams to provide data sample so that all header and data issues can be worked out prior to the actual delivery for ingestion into the archive.

The following projects contain many examples of spectra file headers, which can be used as a guideline for your datasets:

Back to general HLSP guidelines

Back to top