STScI's Internet connections will be UNAVAILABLE Saturday, 28 October 2017 from 7:00 AM - 1:00 PM. STScI web servers, SSH and similar services will be unreachable from the Internet.
MAST HLSP Catalog Data Delivery Standards
This page describes FITS catalog data standards for HLSP deliveries. Also please read the general HLSP guidelines page. Under some circumstances we will accept catalogs in ASCII format; the requirements for this format are the same as for FITS with respect to header keywords (the metadata description of the data). FITS keywords fall into three categories: required, recommended and optional. more | less
MAST has put together the following information because it is essential for data deliveries. If the archive cannot extract the information they need from dataset headers, MAST will attempt to calculate it based on original exposures when possible. If MAST is not able to extract or calculate what it needs based on the delivered dataset headers, then some of the data will not be searchable and will not be displayed across multi-mission search results.
FITS File Formats
The preferred catalog delivery format is FITS, but ASCII is also accepted. For tabular, photometric catalogs in either FITS or ASCII format, we highly recommend ra, dec, flux and flux_error information (columns) per source. Examples are available below. Catalogs can be specified in at least 3 different ways within a FITS file: more | less
The following list enumerates what MAST will accept in order of preference; the links provided show the mandatory keys for that extension based on the FITS Standard v3.0 (Pence et al. 2010) paper:
The FITS paper also provides a summary of the mandatory keywords for the above file types.
Another option is to use non-FITS, ASCII tables of catalogs (e.g. SExtractor format), which are also accepted by MAST. This format equates to the FITS Binary or ASCII Table Extension formats, but in straight ASCII files. More information is provided below.
Lastly, we will also accept CSV format, where the first line is always the list of column names, or otherwise listed in documentation.
The above requirements translate into FITS header keywords below. These keywords and files themselves must abide by the FITS standard, and therefore cannot have errors when being checked by tools like 'fverify' or 'fitsverify'. more | less
For multi-extension FITS files (i.e. FITS Tabular data), please note that many of the keywords cannot be in the 0th extension or HDU, they need to be within the corresponding extension(s) which contain the data.
The data format (columns) must be documented within the file itself and/or the supporting README file. For tabular, photometric catalogs in either FITS or ASCII format, we highly recommend ra, dec, flux and flux_error information (columns) per source. All columns and rows must contain data values; blanks are not allowed. For blank values, please use value 'NaN' or 'NULL' in the ASCII file, but not both within the same file. Within a data line, fields are separated by one or more whitespace characters (space or tab). A field is either a sequence of non-whitespace characters or a sequence of characters between two matching quote characters (single (') or double (") quotes) - spaces are therefore allowed in quoted fields.
SExtractor output tables are an accepted form of delivered catalogs; keywords must be in a supplemental README file or at the top of the catalog, using the designation outlined below. See SExtractor output example file with header and data to confirm allowed format. For non-SExtracor files, please follow the ASCII format outlined here.
Specification of keywords and description of the column names in the ASCII file should be done as follows:
The following is an ASCII header example of keyword, value pairs, including the 4 data column descriptions for wavelength, flux, error, and data quality. Note that these map directly onto FITS header keywords so that MAST can build FITS files from ASCII tabular spectra deliveries.
#TELESCOP= 'HST' / telescope used to acquire data #INSTRUME= 'STIS' / instrument used to acquire data #RA_TARG = 82.586460 / right ascension of target (deg) (J2000) #DEC_TARG= -7.434805 / declination of target (deg) (J2000) #DATE-OBS= '2003-04-28' / UT date of start of first exposure #TIME-OBS= '09:20:38' / UT start time of first exposure # #XTENSION= 'BINTABLE' /Written by IDL: Mon May 25 12:26:32 2009 #BITPIX = 8 / #NAXIS = 2 /Binary table #NAXIS1 = 1152000 /Number of bytes per row #NAXIS2 = 1 /Number of rows #PCOUNT = 0 /Random parameter count #GCOUNT = 1 /Group count #TFIELDS = 4 /Number of columns #EXTNAME = 'E230H-2013_310X005N_52757' /Extension name #EXTNO = 1 /Extension number #TFORM1 = '64000D ' /Real*8 (double precision) #TTYPE1 = 'WAVE ' /Column 1: Wavelength #TUNIT1 = 'Angstroms' /Units of column 1 #TFORM2 = '64000E ' /Real*4 (floating point) #TTYPE2 = 'FLUX ' /Column 2: Flux Density #TUNIT2 = 'erg/s/cm^2/A' /Units of column 2 #TFORM3 = '64000E ' /Real*4 (floating point) #TTYPE3 = 'ERROR ' /Column 3: Photometric Error #TUNIT3 = 'erg/s/cm^2/A' /Units of column 3 #TFORM4 = '64000I ' /Integer*2 (short integer) #TTYPE4 = 'DQ ' /Column 4: Data Quality #TUNIT4 = 'unitless' /Units of column 4 # #COMMENT = 'Delivered to MAST from the StarCat HLSP project' #END
Composite Dataset Traceability
Composite data are defined as products which were constructed from other datasets.
The list of original datasets from which the dataset was made can be in the README file or provided within the dataset. Please discuss these options with MAST prior to delivery.
An example composite map from the PHAT HLSP project can be viewed within the PHAT catalog header where the "IM1*" through "IM5*" header keywords denote the observational details regarding the original 5 datasets that were used to create the catalog.
Header Keyword Nomenclature
Catalogs are generally composites of a variety of datasets. In order to capture some keywords which is a composite, that keyword can contain the value of "MULTI", followed by a list of N similar keywords which encompass all the values needed to explain the composite data using a series of related keywords. more | less
For example, a catalog can be built from a single observation with FITS keyword 'INSTRUME' = "COS". For a composite catalog made extracting fluxes from several images, say STIS and COS data, the keys would be as follows:
To designate multiple keywords, we denote this as [nn].
Keyword Value Units
Some keywords have standard units and do not need to be explicitly specified in the headers. In cases where you need to specify units for header keywords, this can be done in one of two ways: more | less
You can specify the keyword units using a second, similar keyword where the keyword name
contains the string "_UNIT" or "UNIT". Please remember to stay within the FITS 8-character
keyword name limit. E.g.:
Comments can contain the keyword unit within brackets; this should be the first text following
the standard fits comment delimiter "/", the single slash. Please remember to stay within the
FITS 80-character line length limit, which includes the comments. E.g.:
|// DATA DESCRIPTION KEYWORDS|
|TELESCOP|| observatory (e.g. HST, IUE, ISO, VLT)
[Expand TELESCOP values List |
|INSTRUME|| single instrument alone, or instrument/detector names,
or instrument/detector-subdetector names
(e.g. "WFPC2" or "ACS/WFC" or "ISOCAM" or "STIS/FUV-MAMA") or "MULTI" for composite
[View Current List]||INSTRU[nn]|| if INSTRUME="MULTI", then an element from allowed
values of INSTRUME [View Current List]||FILTER|| filter for single exposure or "MULTI" for composite
exposure [View Current List] ||FILTER[nn]|| if FILTER="MULTI", filters list when composite
exposure [View Current List]||
// For Tabular FITS Data:
ASCII TABLE EXTENSION KEYWORDS||SIMPLE|| T / FITS standard||XTENSION|| Type of extension: FITS BINTABLE or FITS ASCII table ||EXTNAME|| Extension name (single word)||(EXTNO)|| Extension number - recommended keyword||BITPIX|| number of bits that represent a data value||NAXIS|| 2 / denoting that the included data array is two-dimensional: rows and columns.||NAXIS1|| the number of 8-bit bytes in each row of the table||NAXIS2|| number of rows in the table||PCOUNT|| Random parameter count ||GCOUNT|| Group count ||TFIELDS|| Number of columns ||TBCOL[n]|| [FITS ASCII table ONLY] Integer specifying the column in which field n starts (starting at 1). ||TFORM[n]|| [FITS BINARY table ONLY] Format of column
(valid data types) ||TTYPE[n]|| Name of column (e.g. WAVE, FLUX, ERROR, DQ) ||TUNIT[n]|| Column units (e.g. Angstroms, erg/s/cm^2/A, unitless) |
|TARGNAME||target name (according to raw data or catalog)|
|RA_TARG||right ascension of the target [deg] (J2000)|
|DEC_TARG||declination of the target [deg] (J2000)|
|EPOCH||Epoch of the observation (considered as deprecated; use EQUINOX when possible)|
|EQUINOX||Equinox of celestial coord. system|
|PROPOSID||Proposal ID (when available)|
|HLSPLEAD||Full name of the lead of HLSP project (Firstname M. Lastname)|
|PR_INV_L||Last name of principal investigator|
|PR_INV_F||First name of principal investigator|
|PR_INV_M||Middle name or initial of principal investigator|
|HLSPNAME||High Level Science Product (HLSP) project name, long form|
|HLSPACRN||High Level Science Product (HLSP) project acronym, short form|
|CITATION||citation reference of this project, for publications (else in HLSP README file)|
|DETECTOR||detector of the instrument (same order as INTRUME) or "MULTI" for composite exposure|
|DETECT[nn]||list of detectors of the instrument of composite exposure|
|APERTURE||for HST, the name of the aperture of the instrument (i.e. STIS apertures)|
|EXPDEFN||string describing how EXPTIME was calculated, i.e. 'MEAN', when composite exposure|
|EXPMIN||if EXPDEFN='MIN', numerical value = minimum composite exposure|
|EXPMAX||if EXPDEFN='MAX', numerical value = maximum composite exposure|
|EXPMODE||if EXPDEFN='MODE',numerical value = mode of composite exposure|
|EXPMED||if EXPDEFN='MED', numerical value = median of composite exposure|
|EXPMEAN||if EXPDEFN='MEAN', numerical value = mean of composite exposure|
|EXPSUM||if EXPDEFN='SUM',numerical value = sum of composite exposure|
MAST has been ingesting and distributing HLSP data products for over 10 years. During this period, the requirements for HLSP data deliveries have expanded in order to help unify all datasets housed at MAST for ease of multi-mission searching. The example HLSP headers may not abide by all the requirements listed above because they were delivered prior to some requirements being written. We encourage the data delivery teams to provide data sample so that all header and data issues can be worked out prior to the actual delivery for ingestion into the archive.
The following projects contain many examples of catalog file headers, which can be used as a guideline for your datasets: