spacer link to MAST page spacer logo image spacer

[Top] [Prev] [Next] [Bottom]

Chapter 2 - Mikulski Archive for Space Telescopes (MAST)

The Multimission Archive at STScI supports a variety of astronomical data archives, with a primary focus on scientifically related data sets in the optical, ultraviolet, and near-infrared parts of the spectrum. See http://archive.stsci.edu/missions.html for a full list of the mission, survey, and catalog data distributed by MAST. MAST is a component of NASA's distributed Space Science Data Services (SSDS).

The staff of the Archive Sciences Branch (ASB) and the Multi-mission Archive at STScI (MAST) provides:

  • world-wide technical and scientific leadership in archive system design
  • secure storage and reliable retrieval services for data from HST and all MAST-supported missions
  • user-friendly and scientifically useful search and cross-correlation tools
  • development and support for inter-archive communication and data transfer standards
  • accurate and useful mission archive documentation
  • helpful user support services with a 1 business day response time

MAST archives a variety of spectral and image data with a range of data characteristics. MAST provides a large suite of searches, including customized searches for each mission. It also provides several cross-mission search tools.

MAST also archives sets of community contributed High-Level Science Products (HLSPs). Users may search for HLSPs by target or coordinates by using the HLSP search page. MAST actively solicits submission of High-Level Science Products related to our missions and we provide guidelines for contributing them to MAST.

The MAST Users Group provides essential user perspectives on archive operations and development, including suggesting priorities for short and long term operational and scientific enhancements to the archive.

Although there is no cost involved in retrieving data from MAST, researchers are requested to include an acknowledgement (as shown in the MAST Data Use Policy and in Chapter 1) in publications that make use of MAST.

All the information in this chapter, and more, is available on the archive website. It should always be consulted for the most current information. This web site is the best place to start learning about MAST and what it can do to enable your research. What follows is an outline of material available on the web.

2.1 MAST Interfaces

The MAST web interface provides access to all of our missions and is likely to be the most convenient option for most users. The Web tool Starview provides access to HST and FUSE observations. Some specialized tasks can only be performed or may be more conveniently accomplished with StarView.

2.1.1 Finding Data

MAST provides several ways to find data of interest. These are listed on a searches web page (http://archive.stsci.edu/searches.html) for your convenience. Each mission has a specialized search form.

MAST provides several cross-mission searches. Use the VizieR/MAST Cross Correlation search or NED/MAST Cross Correlation search to cross-correlate a catalog with MAST missions. A full list of cross-mission and speciality searches is provided on the searches web page.

2.1.2 Obtaining Data

2.1.2.1 Data Archive and Distribution Service (DADS)

HST and FUSE data are distributed from the original HST archiving system called DADS. Users request data after selecting observations of interest from the search results on the web or via StarView. The data will be retrieved and (in the case of the newer HST instruments) reprocessed with the most current calibrations and staged to the users disk or to an ftp area on this server. HST and FUSE data are retrieved in batch mode. The current mean retrieval time can be found on the archive status page (http://archive.stsci.edu/help/archive_status.html). HST data may also be distributed on DVD. A user requesting proprietary data from HST and FUSE observing programs must be an authenticated user to obtain these data. Non-proprietary data may be retrieved without STScI SSO credentials, via anonymous ftp, although users may always use their STScI SSO accounts.

2.1.2.2 HSTonline

MAST stores final calibrated data for HST/STIS (datasets taken prior to 2005) and HST Legacy Instruments GHRS, FOS and FOC on spinning disk. The data for the GHRS and FOC instruments were processed through the OTFR pipelines at the Space Telescope European Coordinating Facility (ST-ECF) using the best available software and calibration files at this time. The reprocessing effort is documented in Instrument Science Reports ( FOC-99, FOC-100, GHRS-92 and GHRS-093 ). The links are to pdf files containing these Instrument Science Reports. The enhancements to the STIS calibration are discussed in the Update on STIS Closeout Calibration memo, posted on the STIS website.

This alternate interface to the HST legacy data is provided to allow direct access to the data on disk. All these data may also be retrieved via DADS. Both interfaces provide the same data for FOS, FOC, GHRS and the pre-SM4 STIS data.

The search interface for the Legacy Instruments is very similar to the general HST search interface. The main difference is that data marked for retrieval will be downloaded directly to your computer in a tar file. The download does not utilize the DADS batch retrieval method and should therefore be faster.

The data are also available via anonymous ftp. FTP to archive.stsci.edu logging in as anonymous. CD to pub/hstonline. The data are stored in a set of subdirectories following the format /pub/hstonline/XXXX/datasetname where XXXX is the first four letters of the dataset name, effectively grouping the data by proposal. For dataset X0ND0101T the path would be pub/hstonline/x0nd/xond0101t.

2.1.2.3 Other MAST data and HLSPs

All MAST data, other than HST and FUSE, and all the HLSPs are stored on spinning disk. Users may select desired data from the search results pages for most MAST missions and download the data directly to disk via their browsers. For HLSPs, see the summary of HLSP downloads options for more information (http://archive.stsci.edu/hlsp/hlsp_download.html).

If you don't quickly find the information you need, try the site search (http://archive.stsci.edu/google_search.html). You may also email us at archive@stsci.edu.

2.3 Mission Web Search Interfaces

There is a web based interface for every MAST mission, including the Digitized Sky Survey (DSS) and Guide Star Catalog (GSC). The GALEX interface is similar, but relies on different underlying software and may differ from other missions in some respects. The heading for each form item is a link to a help page with more information on that field and how to best use it in a search.

The mission search pages are listed on the Searches page under mission specific searches. There is also a link to each mission search on the "mission navigation bar" and under the Search & Retrieval left menu pullout for the appropriate missions.

2.3.1 Search Forms

For most missions, there is a "standard form" and a "file upload" form, which allows users to upload a list of targets. The search forms consist of three sections.

  • The top or name/coordinate specification section
    On the "standard form", the top section consists of a place to enter a target name or coordinates. When you enter the target name you can choose to have the coordinates looked up via SIMBAD or NED using the Resolver options field. The search is then done using the coordinates returned from the selected resolver. You may choose to search on a character string by choosing the "Don't Resolve" option in the Resolver options field. To maximize the chance of finding the targets, surround the choice with wild cards e.g. *Jup* to find observations of Jupiter. You may also choose the search radius (the default is to 3 arcmin).

    For the "file upload form", the top section helps you describe the format of a file containing a list of targets or coordinates you wish to search on. Click on the column labels for additional help. The local file name is the name of the file on your system. The file must be an ASCII text file with one entry per line with the fields (e.g. RA, Dec, data_id and/or target_id) separated by one of the allowed delimiters. Coordinates may be in several formats including sexigesimal or decimal degrees. Use the form to designate in which column the data_id, target name or RA and Dec are to be found. If appropriate designate the delimeter (tab, |, comma (,), semi-colon(;)). Choose the desired resolver or the "Don't Resolve" option.

  • The mission specific section
    The middle section of the search page contains mission specific fields, that can be used to perform searchs on dataset name, exposure time, start time, dataset name and/or observers name. Users may add any two additional search fields using the "User-specified fields" at the bottom of the middle section of the web-based form. For more information about the available searchable fields click on the link "Field Descriptions". This brings up a page describing all the available columns (the database column name, the label, the data type, the valid values in that field) and additional notes where applicable.

  • The output options section
    The third section gives you choices for how to format the output of the search results. On the left side a list of columns is displayed. These default columns were chosen as the most commonly needed columns. You may remove any of the columns by highlighting that column and clicking on the remove button to the right. You may add columns by choosing columns from the select box below and clicking the add button to the left of the select box. You may change the order by clicking on field in the output columns box and click on the up and down buttons to the right. Clicking on the reset button will restore the default output columns settings. You may choose to sort the output by up to three columns. You may choose from four output formats: HTML, comma separated value text, Excel spread sheet, and VOTable format. The HTML format is the format that will give access to previews and other useful links as well as retrieval options. Click on the headings for additional help.

  • Other options
    In the lower right hand section of the search form are the "Show Query" and "Make Rows Distinct" checkboxes and the "Maximum Records" and "Records per Page" menus. To see the SQL query produced for your search, check the Show Query box. The SQL will be displayed at the bottom of the search result page.

    Check the Make Rows Distinct box if you would like to see a distinct list of objects with certain criteria. For example, all the objects within an IUE or FUSE object class. For this example, limit all but the "Target Name" (or RA and Dec); this is done with the "Remove" button in the Output Columns window. This action permits a selection only on unique targets names (or coordinates) that have been observed by the satellite for the object class given.

    The number of rows returned and the number of records per page are selectable using the menus.

2.2.2 Search Results Output

The user may specify one of HTML, CSV, Excel or VOTable as appropriate for the output format of the search results. The default is HTML.

2.2.2.1 HTML output

The main item of interest on the output page is the listing of rows found as a result of the search. As a default only the first 200 rows are displayed with 50 records per page. One of the output options is to increase the number of rows displayed. Be aware that when a large number of rows are displayed, the page may render very slowly. The Mark button column for proprietary HST and FUSE data is displayed with a yellow background to easily flag those data as proprietary.

Results can be sorted by clicking on the column title at the top of the display. To get a description of the column, click on the column title at the bottom of the display.

To see a preview (if available), click on the Data ID. A separate preview page will be displayed. A link, "Display all previews as thumbnails," will display all the previews as thumbnail images.

Click on the number in the Ref colummn to get a list of the publications based on these data.

Above the output display is a link to allow numeric columns to be displayed graphically using VOPlot. If a resolver (SIMBAD or NED) was used, the resolver output is displayed. If you have chosen to display the SQL used to search the database the query will be displayed. A series of additional comments explaining various links may be included in this section. A variety of links are available as part of the search output (assuming the appropriate columns have been chosen for display). The dataset name is usually a link to a preview page which includes a preview version of the image or spectra as well as additional information and links. If the "reference" column is displayed the number indicates the number of refereed journal articles in which the data has been used. For HST, the references are categorized by proposal number, so not all individual observation in that program may have been used. For HST, FUSE, IUE and EUVE, the proposal id is a link to a listing about that proposal. It includes the abstract, any associated journal articles and a listing of all the observations acquired for that program.

Some observations have been used to create "High-Level Science Products" which are hosted at MAST. The number in the column labeled High-Level Science Products indicates the number of products associated with that observation. Click on that number to see a list of the products. Also listed may be associated previews and readme files.

For spectral observations, you may coplot multiple spectra by choosing the observations to be plotted and then clicking on the "Plot marked spectra" button. You may choose a variety of scaling factors to customize the plot and replot the data. The customized plotting program may also be accessed from the preview.

To request data, mark the datasets of interest and click on the button to retrieve or download data. For HST and FUSE data, a retrieval options form will give you options for data delivery and for the type of files to be requested. For IUE, you may choose to have final data product (MXLO and MXHI) files downloaded or the "more options" file which will allow you to choose among the NEWSIPS and the original IUE data products. Data for HST and FUSE will be delivered as individual files according to the chosen data delivery options. Other MAST data will be delivered as a tar or zip file (as you choose) directly to your computer. More detailed information about retrieving data from MAST is in the section Retrieving Data.

2.2.2.2 Comma Separated Values (CSV) Output

The CSV output is a simple ASCII file containing column headings followed by comma-separated entries. In file upload mode, a blank line is inserted between database queries. (Note: this option will execute more quickly than the others since it involves the least amount of processing).

2.2.2.3 Excel output

The search results are stored as an Excel spreadsheet table (note: assumes the user's computer/browser provides support for Excel-format files).

2.2.2.4 VOTable output

Search results are returned in an XML file format developed by the VO project. For searches returning results from more than one mission and/or target, multiple "resource" tags are created. A list of the output fileds for a particular mission/catalog may be obtained by searching on that mission/catalog with a radius of 0 (zero). See the VOTable documentation at the IVOA website (http://www.ivoa.net/Documents/latest/VOT.html) for more information.

2.3 HST Pointings Search

The MAST "Pointings" interface allows a user to search for ACS, WFPC2, NICMOS, STIS and FOC exposures in a powerful way. The interface can be found at http://archive.stsci.edu/pointings/search.php. A description of how a "Pointing" is defined can be found on the pointings search help page. You can run the pointings search interface as a web service.

For each of these instruments we have assembled a searchable data table that allows users to look for sky regions (or pointings) which have been observed more than N times, observed with 2 or more filters, or have been observed more than twice with a time separation of more than (or less than) N days.

Included in each table are the total number of exposures for a pointing and the number of unique bands. The table also lists the date/time of first and last exposure for every pointing.

An example of the kind of query this table will allow is: How many high galactic latitude observations exist [for RA between 9 - 18 hours in the northern hemisphere (dec > -20) observable from the northern hemisphere in the early spring] where there has been at least two observations, and where at least one of those observerations was in the I-band, and at least one of those observations was in any other filter?

Galactic Latitude Above & below plane +/- > 20.0 degrees
Band I >0
Number of Unique Bands >1
Total Number of Exposures >1
RA 09..18
Dec >-20.0

2.4 MAST Web Services

MAST provides a number of web services. The currently available services and their allowable parameters are listed on the MAST web site at http://archive.stsci.edu/vo/mast_services.html.

2.4.1 HTTP GET Requests

A GET request allows the search parameters to be included in the URL. As such, they can be called from within programs to automate data searches. The results can be returned in a variety of formats including HTML, VOTable XML format, excel spreadsheet, and comma-separated values which can simplify ingesting results into user-written programs. In addition, submitting GET requests can bypass restrictions currently placed on the web search forms (e.g., restrictions on the max_records value).

Note: when creating URLs, special characters must be encoded. URLs may only use alphanumerics [0-9a-zA-Z], the characters "$-_.+!*'()," and special reserved characters. The characters "<" and ">" must be specified as "%3C" and "%3E" respectively. For example, to specify a parameter such as RA > 120, the GET request parameter would be RA=%3E120. Blanks can be specified using the "+" character.

2.4.1.1 Script Examples for GET Requests

Scripts for retrieving MAST search results can be written in many programming languages. Basically you just need to download and extract information from a web page. A few simple examples are shown below.

2.4.1.1.1 Sample PHP Script

Search HST archive and print results:

#!/archive/data1/bin/php -f
<:?php
#
# standalone PHP program
# (edit top line to point to local PHP executable)
#
# Submits GET request to search HST archive for given RA, Dec, and radius,
# limit results to 10 records,
# returns data set name, RA, Dec, and Target name
# as a comma-separated list
# prints out column headings and data
#

# create GET request

$request = "http://archive.stsci.edu/hst/search.php?"; //quotes url
$request .= "RA=53.084&DEC=-27.873&radius=1.0&max_records=10&";
$request .= "outputformat=CSV&action=Search";
$request
.="&selectedColumnsCsv=sci_data_set_name,sci_ra,sci_dec,sci_targname";

print "\nrequest = $request \n\n";

# download results from MAST as an array called $data
# (ignore errors)

$data = @file($request);

# print column headings (skip data types in next row)

print "$data[0]\n\n";

# now print search results

$cnt = count($data);
for ($i=2; $i<$cnt; $i++) print "$data[$i]\n";

?>

2.4.1.1.2 Sample Python Scripts

Search Pointings archive for ACS images more than 75 degrees from galactic plane with < 5 exposures in U band AND more than 1 in I band, output as comma separated list save results in file out_py.txt

#!/usr/local/bin/python

# change above line to point to local
# python executable

import urllib, urlparse, string, time

# create URL with desired search parameters

url = "http://archive.stsci.edu/pointings/search.php?"
url = url + "primary=ACS&outputformat=CSV"
url = url + "&pnt_ucountp=%3C5&pnt_icountp=%3E1&bao=and"
url = url + "&galactic=Above&galsearch=75"
url = url + "&action=Search+Exposures"

print url

# retrieve URL and write results to filename

filename = "out_py.txt"

urllib.urlretrieve(url,filename)

### Done!

2.4.1.1.3 Sample Shell Script

Search STIS spectra and output results as a VOTable XML file called out_csh.xml.

wget -O out_csh.xml -v -a logfile 'http://archive.stsci.edu/ssap/search.php?ID=stis2%26POS=327.795,28.864%26SIZE=0.03'

2.4.1.2 Mission Searches

In general, mission searches are specified in the form: http://archive.stsci.edu/[data set]/search.php?action=Search¶ms where [data set] is the MAST-defined acronym for a particular set of data, and params is the list of search parameters as defined below. Note that "action=Search" is a required parameter for mission search requests. For a pointings search however, action may equal either "Search+Pointings" or "Search+Exposures" The current list of data sets that can be searched in this manner include the following:

  • HST - Hubble Space Telescope
  • IUE - International Ultraviolet Explorer
  • HUT - Hopkins Ultraviolet Telescope
  • EUVE - Extreme Ultraviolet Explorer
  • FUSE - Far-UV Spectroscopic Explorer
  • UIT - Ultraviolet Imageing Telescope
  • WUPPE - Wisconsin UV Photo-Polarimeter Explorer
  • BEFS - Berkeley Extreme and Far-UV Spectrometer
  • TUES - Tübingen Echelle Spectrograph
  • IMAPS - Interstellar Medium Absorption Profile Spectrograph
  • HLSP - High Level Science Product
  • Pointings - HST Image Data grouped by position
  • Copernicus - Copernicus Satellite

The search parameters include both general and mission-specific items. Table 2.2 gives the links to lists of these parameters.

Table 2.2: Links to Search Parameters
Mission Link
General Search Parameters http://archive.stsci.edu/vo/general_params.html
HST-specific search fields http://archive.stsci.edu/hst/help/quickcol.html
Pointings-specific search fields http://archive.stsci.edu/pointings/help/quickcol.html
IUE-specific search fields http://archive.stsci.edu/iue/help/quickcol.html
HUT-specific search fields http://archive.stsci.edu/hut/help/quickcol.html
EUVE-specific search fields http://archive.stsci.edu/euve/help/quickcol.html
FUSE-specific search fields http://archive.stsci.edu/fuse/help/quickcol.html
UIT-specific search fields http://archive.stsci.edu/uit/help/quickcol.html
WUPPE-specifc search fields http://archive.stsci.edu/wuppe/help/quickcol.html
BEFS-specifc search fields http://archive.stsci.edu/befs/help/quickcol.html
TUES-specific search fields http://archive.stsci.edu/tues/help/quickcol.html
IMAPS-specific search fields http://archive.stsci.edu/imaps/help/quickcol.html
HLSP-specific search fields http://archive.stsci.edu/hlsp/help/quickcol.html
Copernicus-specific search fields http://archive.stsci.edu/copernicus/help/quickcol.html

In general, any parameter listed on the HTML search form can be specified in a GET request. If you are parsing the results, it would be easier and probably faster, to request the search results in CSV or VOTable format using the "outputformat" parameter.

Mission search examples:

  • Search the HST archive for observations within the specified radius (in arcminutes), centered on the given RA and Dec values (in decimal degrees), allow a maximum of 1000 records, and output the default columns as comma-separated values: (Try it)
    http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&radius=100. &max_records=1000&outputformat=CSV&action=Search

  • Search the HST archive for pep_id 9293, allow a maximum of 1000 records, and output results as comma-separated values: (Try it)
    http://archive.stsci.edu/hst/search.php?sci_pep_id=9293 &max_records=1000&outputformat=CSV&action=Search

  • Search HST as above, but return results in VOTable format (Try it)
    http://archive.stsci.edu/hst/search.php?sci_pep_id=9293 &max_records=1000&outputformat=VOTable&action=Search

  • Search HUT database for observations of M81 and return only columns for target name and data id: (Try it)
    http://archive.stsci.edu/hut/search.php?action=Search&target=m+81 &selectedColumnsCsv=hut_target_name,hut_data_id

  • Search the ACS Pointings table for the target "M 81", output format = "CSV", and more than 11 exposures in the U band: (Try it)
    http://archive.stsci.edu/pointings/search.php?pnt_ucount=%3C11&target=m+81 &primary=ACS&outputformat=CSV&action=Search+Pointings

Notes:

  1. Be sure to include "action=Search" (with capital S) in the mission searches, but not for SCS or SIAP requests.

  2. Parameter order is not important.

  3. Use a comma separated list for selected columns with the selectedColumnsCsv parameter. Don't use selectedColumnsList as seen from the search form.

  4. Parameters such as resolver=don'tresolve should probably be encoded to be resolve=don%27tresolve. It seems to work without it though. (See PHP command called urlencode).

  5. There is currently a 300 second execution time limit on archive.stsci.edu which can cause large search requests to abort. Contact MAST at archive@stsci.edu if you encounter this problem.

2.4.1.3 Simple Cone Search

MAST missions can also be searched using the NVO Simple Cone Search protocol (SCS). The base url is the same as for MAST mission searches (i.e., http://archive.stsci.edu/[data_set]/search.php) without the "action=Search" parameter. The search radius parameter "SR" must be included in the URL for the cgi script to recognize a simple cone search is being requested. Besides the search radius in decimal degrees, RA and DEC parameters are also required to perform a cone search. The results are returned in VOTable format. The SCS protocol can be used with any MAST mission and the general MAST parameters can be included as well. Examples:

  • This example will return about 50 HST observations (in VOTable format) at RA,Dec = 53.084,-27.873 and a search radius = 0.01 degrees http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&SR=0.01

  • Same as above except use a larger search radius (0.02) but only allow a maximum of 100 records to be returned http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&SR=0.02&max_records=100

  • To find all high level science products at the same coordinates: http://archive.stsci.edu/hlsp/search.php?RA=53.084&DEC=-27.873&SR=0.01

2.4.1.4 Simple Image Access Protocol (SIAP)

MAST missions can also be searched using the NVO Simple Image Access protocol (SIAP). The base URL is http://archive.stsci.edu/siap/search.php followed by the SIAP parameters. Like the Simple Cone search, the base url is similiar to that for mission searches however only missions with image data are searchable. By default, up to 300 entries will be returned but specifying max_records=nnn can override the default. Adding the parameter "representative=y" will return only "representative" images as defined for the MAST scrapbook tool. Currently the list of instruments, and High Level Science Products (HLSPs) that can be queried using the SIAP include:

  • WFPC - HST Wide Field Planetary Camera
  • WFPC2 - HST Wide Field Planetary Camera 2
  • NICMOS - HST Near Infrared Camera and Multi Object Spectrometer
  • STIS - Space Telescope Imaging Spectrograph
  • FOC - HST Faint Object Camera
  • ACS - HST Advance Camera Survey
  • UIT - Ultraviolet Imageing Telescope
  • UDF - HLSP Ultra Deep Field
  • HDF - HLSP Hubble Deep Field
  • HDF_SOUTH - HLSP Hubble Deep Field South
  • GOODS - HLSP The Great Observatories Origins Deep Survey
  • TNO - HLSP trans-Neptunian objects
  • HELIX - HLSP HST Helix Observations
  • MAOZ_ATLAS - HLSP HST Atlas of Ultraviolet Images of Nearby Galaxies

SIAP Examples:

  • Search for MAST images at RA+151.0, Dec=+69.0 and a search area of 0.5 degrees: http://archive.stsci.edu/siap/search.php?POS=151.0,69.0&SIZE=0.5

  • Dislay supported SIAP parameters: http://archive.stsci.edu/siap/search.php?format=metadata

  • Find WFPC2 observations at RA,Dec = 53.084,-27.873, format = GIF, and a rectangular search area 0,05 degrees wide: http://archive.stsci.edu/siap/search.php?POS=53.08,-27.873 &SIZE=0.05&FORMAT=gif&ID=wfpc2

  • Find "representative" WFPC2 observations as above: http://archive.stsci.edu/siap/search.php?representative=y&POS=53.08,-27.873 &SIZE=0.05&FORMAT=gif&ID=wfpc2

2.4.1.5 Simple Spectral Access Protocol (SSAP)

The Simple Spectral Access Protocol (SSAP) will be supported when the new standard has become better defined. See also the Simple Spectral Data Model. Plans are being made to support SSAP for all MAST missions containing spectral data.

2.4.2 SOAP Services

SOAP (simple object access protocol) is an XML-based messaging protocol. MAST has one SOAP-based web service which is accessed via the ADS (Astrophysics Data System). The web service allows users to enter a data set name and find where that data is archived. The service can be accessed from the web form at: http://ads.harvard.edu/ws/DataVerifier, or, using the latest ADEC naming conventions: http://vo.ads.harvard.edu/dv/DataVerifier.cgi .

2.4.3 RSS News Feed

Another MAST web service is our RSS news service. This is a list of our most recent news items which can be requested in HTML or RSS XML-format:

  • http://archive.stsci.edu/mast_news.php?out=html&desc=t&last=10 shows last 10 entries (by date) in html with descriptions

  • http://archive.stsci.edu/mast_news.php?desc=t&last=10 shows same in rss xml format

  • http://archive.stsci.edu/mast_news.php?out=html&last=10 summary in html of last 10 (no descriptions)

  • http://archive.stsci.edu/mast_news.php?out=html&id=25 show entry 25 in html

2.5 MAST Tools

MAST provides and develops tools that archival users may use to find and view data in the archives. The current tools are several types of searches, two display tools for spectra and a scrapbook. Check the Tools menu on the main archive web page for a up to date listing.

2.5.1 NED/MAST Cross Correlation Search

The NED/MAST Cross Correlation Search permits users to search the NED catalog by object name, position, object category etc. and optionally cross-correlate results with MAST mission databases.

NED search results are retreived as a web service, and the results are either displayed directly, or the retrieved coordinates are used to display MAST mission observations that are within a specified search radius of the retrieved catalog entries.

2.5.2 VizieR/MAST Cross Correlation Search

The VizieR/MAST Cross Correlation Search permits users to search for catalogs held by VizieR and then either search them internally or to use them as input for a MAST cross-mission search. There is a help page for these searches, but below is an overview of the process.

First search for a VizieR catalog. When searching for a catalog name, you will often be more successful if you search using a wild card (e.g. *Abell*). The results of the search list the catalogs found at VizieR with two options to the left of each found catalog (S and CC).

  • Click on the "S" to Search the catalog itself. This brings up a form with the same basic format as the mission searches. The top section permits entry of target name or coordinate criteria and the bottom section specifies output formats. The middle section gives you the option of qualifying the search using any of the fields present in the catalog. The field descriptions link brings up a page describing the column of the catalog as provided by VizieR.

  • Click on the "CC" to cross-correlate the catalog with any of the MAST missions (except GALEX at this writing). This form has 4 sections. The top section is the usual Target Name/coordinate specification. The second section permits you to add qualifications concerning the catalog. Again use the Field Descriptions link to see the column/field descriptions from VizieR. The third section permits you to cross-correlate with data from specific instruments. The fourth section is the normal output selection criteria. The first 10 records found per catalog entry will be displayed. The total number of records found will be indicated.

2.5.3 Specview

Specview is a 1-D spectral visualization tool written in Java for analysis of astronomical spectrograms. It is capable of reading all the HST spectral data formats, as well as data from several other missions (such as IUE, FUSE, ISO, FORS, HUT, EUVE and SDSS), and data from generic FITS and ASCII tables.

Once ingested, data can be plotted and graphically examined with a large selection of custom settings. Specview supports instrument-specific data quality handling, flexible spectral units conversions, custom plotting attributes, plot annotations, hardcopy to a PostScript file or printer, etc.

A spectral feature quick measurement tool enables the user, with a few mouse actions, to perform and record, in VOTable or FITS format, measurements on selected spectral features.

Specview can be used to overplot or combine data from the same astronomical source taken with different instruments and/or spectral bands. It has a spectral model fitting capability that enables the user to work with multi-component models (including user-defined models) in a number of ways, and fit models to data.

Specview can overplot spectral line identifications taken from a variety of line lists, including user-supplied lists.

To download Specview or for more information see the Specview page (http://www.stsci.edu/resources/software_hardware/specview).

MAST staff have written a tutorial (in Powerpoint format) that shows how to use the spectrum combination/filtering functions. See http://www.stsci.edu/resources/software_hardware/specview/tutorial.

2.5.4 MAST Coplotter

The MAST Spectral Data Coplot utility allows users to coplot multiple spectra from MAST's holdings of spectral data. For HST instruments, the plotted data are obtained from the preview files produced by CADC. For the other MAST missions, the data is extracted from the ASCII preview files. In both cases, differences may exist between these data and those found in the distributed FITS files. The current list of data sets to choose from includes: STIS, IUE, EUVE, HUT, BEFS, TUES, FOS, GHRS, and WUPPE. Other mission data sets will be ignored. See the Spectral Data Coplotter Help page for more information

2.5.5 MAST Scrapbook

The MAST Spectral/Image Scrapbook is designed to allow users to take a quick look at representative data in the MAST archive for a particular astronomical object of interest. It is particularly useful if the user is not already familiar with the datasets involved. Objects such as planets, comets and asteroids are NOT included in the Scrapbook.

The tool utilizes catalogs of "representative" spectra and images compiled by the MAST staff. Each table lists unique datasets representing either an object or unique pointing observed by MAST missions or instruments. Datasets are listed for an assortment of bandpasses, depending upon on the instrument and availability per target or pointing. A description of the automated process used to select the representative spectra and images is available. Note that these are "representative" spectra or images, not necessarily the "best".

The scrapbook help page describes the scrapbook interface and includes descriptions of the input parameters and output.

2.6 Hints for searching

The best search to use depends on what you are trying to locate. Several hints are presented in the following sections.

2.6.1 Searching for Planets, Comets and Asteroids

Using coordinates to search the archive for moving targets such as planets and comets is not usually the most efficient way of locating such observations. The most straightforward way to search for observations of such a target is to search on the target name using the "Don't Resolve" option in the Resolver form element.

For some missions, such as HST, the target name is designated by the original observer. For example, an observation of Jupiter might have the target name "CP1-JUP-NORTH-LIMB-A". To find all the items of interest you may want to use a wild card search using the minimum number of characters. To do this enter *JUP* in the target element of the search form. You must remember to choose the "Don't Resolve" option in the resolver form element. You can narrow the search using such items as the time of observation, instrument, filter etc. This type of search may take a little longer than a search by coordinates.

For a few missions, targets have been assigned an object class or category. You can utilize this to find all solar system objects. For shorter duration missions, such as HUT or UIT this might be the most efficient way of searching for planets or comets. For longer term missions such as IUE and FUSE, this type of search will return many rows unless you are able to narrow the search using other criteria such as time or exposure length.

Another way to locate observations of solar system objects is to search the abstracts of missions with a Guest Investigator program (HST, FUSE, IUE, EUVE). Entering the target of interest into these search forms will find programs that planned to observe the target. A list of the programs contains the abstract as well as links to listings of the observations associated with that program.

Using the VizieR/MAST Cross-correlation tool is another way of finding classes of targets within the MAST archive. See the section on VizieR/MAST Cross Correlation Search for more information.

2.6.2 Searching for HST data by Object Class

There are two object category fields in the archive for HST data: the "broad category" and the "target description" fields. The target description field is one of the fields explicitly listed on the HST search form. This is the longer description and probably the more useful search field. You may also search on the broad category utilizing the User-specified field options on the search form.

The format is NOT uniform and so you will need to utilize wild cards to search on a keyword or phrase. The keywords might be separated by semi-colons, spaces, or dashes so you will need to place a wild card between each word or phrase in the search criteria. For example: *GALAXY*SPIRAL*. This search will find most of the observations of interest. However, word order is important so you might need to try *SPIRAL*GALAXY* as well.

Searches on object category fields are likely to return many rows. You may wish to further qualify your search by instrument, filters etc.

Every morning a list of all the target descriptions currently in use is generated (http://archive.stsci.edu/hst/daily/target_descriptions.html). This list is linked back to the HST Science Search page, so you can read through this list (or search it with your browser's find capability) and find datasets matching the description.

2.6.3 Search HST catalog by Region in the Sky

The MAST "pointings" interface allows a user to search for ACS, WFPC2, STIS, FOC and NICMOS images in a powerful way. For each of these instruments we have assembled a searchable data table that allows users to look for sky regions (or pointings) which have been observed N times or more, observed with 2 or more filters, or have been observed more than twice with a time separation of more than (or less than) N days. An exposure is defined to be the file created after the shutter closes. The total number of exposures for each of the defined bands within a defined pointing were tallied. The definition of a pointing is different for each instrument. See the Search Help page for a more complete description of the methodology used to define a pointing. This page also includes descriptions of the search fields and the input and output columns.

2.6.4 Searching with Ranges of Coordinates

You may search on ranges of right ascension or declination using the ".." operator. For example, you can enter 21h 51m .. 21h 52m for the right ascension, and 28 51 .. 29 51 for the declination. Comparators can also be used, i.e. ">", ">=", "<", "<=". For example, > 85 as a declination value will return all observations with declination larger than 85 degrees. (Note when ranges of coordinates are specified the search radius will be ignored. Also, searches on ranges can be quite time consuming.)

2.6.5 Searching with Ranges in Time

Searching on ranges of time is important if you wish to search on any time field. If you enter the date July 19, 2002, then midnight (00:00:00) is assumed. If you enter July 2002, then the date July 1, 2002 at midnight is assumed. If you don't know the exact time of an observation then you will need to search using a range.

To search for observations before a given date, use <, and for observations after a given date, use >. For example,


  • > Jul 15 2002
  • < Jul 15 2002
  • You can use the .. operator to search on a range of dates. To find all observations in July 2002 enter:
    Jul 1 2002 .. Aug 1 2002

    This operator is inclusive on the first date and exclusive on the second.

    Finally, you can search on a list of dates or date ranges. For example,

    Jul 1 2002 .. Jul 3 2002, Dec 1 2002 .. Dec 6 2002

    will search for observations done within either one of these date ranges.

    2.6.6 Searching Across Multiple Missions

    The MAST Cross Correlation Search permits searches across all the HST instruments and other MAST missions. This interface is similar to the mission search forms. For more detailed information about search options see the help page.

    The top section is the same as the mission search forms and includes both the standard and file upload options. The second section permits selection of the instruments of interest. You may customize the search radius for any of the instruments. The third section gives some options such as the output format, but the options are not as extensive as the mission search pages.

    The output from the search is grouped by instrument. Up to 10 rows are displayed for each instrument, but the number of available rows is displayed. The output utilizes the default output fields for each instrument/mission. The HTML version includes links to the previews, the proposal search, the references and High-Level Science Products. At the bottom of the output display is a summary listing each instrument and the number of observations found per instrument. The summary is not listed in the CSV, Excel and VOTable formats.

    2.6.7 Comparing observations from two different missions

    Perhaps you are interested in seeing what targets (perhaps of a specific type of object) have been observed by two different missions. If you have a list of specific targets then the easiest thing is to just use the Cross-correlation search described just above. But perhaps you are just interested in finding out what BL Lacertae objects have been observed by both FUSE and STIS without a pre-determined list of objects. First determine the unique target names or coordinates observed by one of the instruments for that type of target. You might choose to obtain this initial list from the FUSE search, as it is easier to search by object class. Follow these steps:

    • Choose BL Lacertae Object from the list of categories on the FUSE page.
    • Remove all the output columns except the ra and dec from the default output columns form element.
    • Choose the "Make Rows Distinct" option
    • Select the CSV (Comma Separated Values) Output Format Option
    • Search
    • Save the output as a file

    After editing the file to remove any header information, you may use this file as input for the HST search.

    2.6.8 Abstract Searches

    The abstract search form can be used to find HST, FUSE, IUE, and EUVE proposal abstracts or titles containing specific words or phrases. You may also search on proposer name and proposal id. Fairly complex searches may be performed using the mission specific search forms. See Table 2.3 for links to these forms. For more details see the search form help pages.

    Table 2.3: Links to Abstract Search Forms
    Missions Link
    HST

    http://archive.stsci.edu/hst/abstract.html
    FUSE http://archive.stsci.edu/fuse/abstract.html
    IUE http://archive.stsci.edu/iue/abstract.html
    EUVE http://archive.stsci.edu/euve/abstract.html

    The results list the proposal id, the title, the PI and institution and the abstract. Search criteria are highlighted in red in the search result display. The proposal ID is a link to the proposal search. For HST, the PI name is a link to a search that lists other HST proposals by authors with that last name.

    You can find the abstract search on the Searches page under the mission specific searches. There is also a link under the Search & Retrieval left menu pullout for the appropriate missions.

    2.6.9 High-Level Science Product Searches

    There are two kinds of High-Level Science Product (HLSP) searches. One is a search by Project (http://archive.stsci.edu/hlsp/index.html) and the other is a search by target (http://archive.stsci.edu/hlsp/search.php).

    2.6.9.1 Search by HLSP Project

    This is a simple form with three search criteria: Product Type, Wavelength, and Object Type. The attributes were assigned by the MAST staff scientists. To search select one or more of each criteria. Hold down the shift key to select more than one option per category. After you click on the Search button, a list of pertinent sets of HLSP is displayed. The form includes a listing of all the HLSP sets grouped by Product type. The search results use the same display format. The title is a link to a web site giving more information about that set of data. Most sets of HLSP have a "Search" button link to the left of the title. Click on that search button to get a listing of all the HLSP products from that project. A few sets of data have not yet been completed or are not actually hosted at MAST. These do not have a search button.

    2.6.9.2 Search HLSP by target

    This search looks like a mission search form with the usual three page sections. As usual, the top section allows you to enter target names or coordinates. The second section permits you to specify specific projects and limit the search by instrument, product type and format. The third section permits you to specifiy the output content and formats.

    The output lists the project, filename and other associated information. The project id is a link to a webpage giving more information about that project. The filename is a link that permits you to download the data directly to your workstation or desktop. Most HLSP are also stored in the anonymous ftp area on archive.stsci.edu.

    Links to the HLSP searches may be found in the left navigation menu for the main MAST page and for each mission with associated HLSP. The searches are also on the searches web page linked from the top navigation bar which appears on most MAST web pages.

    2.6.10 Performing Large Searches

    You may need to perform searches and make requests for large amounts of data. The archive web based search form currently limits the number of rows returned from a query to a maximum of 1500 rows. If your search that will return a larger number of rows, you may wish to use one of the MAST web services. See http://archive.stsci.edu/vo/mast_services.html. The web page documents how to use web services to make a large search.

    If you intend to retrieve large amounts of HST data, consult the large searchs and request page (http://archive.stsci.edu/hst/bigsearch_request.html). The page documents some operation limitations that need consideration.

    2.7 Previewing Data

    Previews are available from the results pages of most of the mission searches. Preview pages for most MAST missions include the following information:

    • Title including the dataset name
    • Publication reference id, in the format for the dataset used by some publications For more information on this topic see the page on Dataset Identifiers.
    • gif representative image
    • Subset of exposure information
    • Observing program information for Guest Observer missions including a link to a proposal page.
    • List of links to: information about the previews; the FITS header (except HST); the customized plot program (for spectral data); and links specific to the mission.
    • List of journal articles utilizing the data. (For HST, papers are associated with programs, not to individual observations). Articles are listed in order of the number of citations for that paper at the ADS. Citation counts are updated monthly.

    2.7.1 HST Previews

    HST previews for science observations are created by CADC. HST previews are stored as FITS files and the gifs are created on-the-fly. Users may retrieve the preview data in a variety of formats including the original FITS file. Just beneath the gif image are two links. The link "FITS format" will download the original FITS file. The link "More preview format options" will bring up a page so that you may choose the format you desire.

    2.7.2 FUSE Previews

    FUSE previews are provided by the FUSE project. In addition to a "quicklook" preview, there are links to four gif images showing summed plots for each channel (i.e., A and B) for each detector (LiF1, LiF2, SiC1, SiC2). Following the summed detector plots is a table of links to gif images showing data from the individual exposures. There is a web page with more information about FUSE browse files or previews.

    2.7.3 IUE Previews

    IUE Previews show the extracted spectrum and the SI browse file (an image). The most commonly used previews are those constructed from the IUE Final Archive data "mx" or extracted data files. These previews are displayed by clicking on the data id in the search results page. The preview contains a link to more information about the MX previews. The IUE browse files were conceived and originally implements by Dr. Derck Massa, Dr. Nancy Oliversen, and Ms. Patricia Lawton, then members of the GSFC Astrophyics Data Facility (ADF) staff under direction of Dr. Michael Van Steenberg. Some modifications were made as part of the transition to MAST maintenance.

    Near the bottom of the preview page there are a set of links. One link is to the SI browse file, the image file from which the spectra were extracted. More information on the SI browse files is available from a link on the SI preview page.

    The IUE was operated from two different tracking stations: GSFC for NASA and Villafranca Spain for ESA. The original observing notes for the data acquired from the GSFC station have been scanned and are available from a link on the IUE preview pages. To keep down costs, the exact mapping between the scanned page to the image sequence number was not made at the time of scanning. A good guess can be made, and the link to the program that guesses which scanned page is of the desired observing script is available from the preview page. Look at the thumbnails from the pages nearby the best guess and click on the appropriate thumbnail to see the entire observing script.

    2.7.4 Thumbnails

    An option to display thumbnails of all the search results previews is available on the XMM-OM, FUSE, EUVE, IUE, HUT, UIT, WUPPE, HPOL, TUES, and BEFS search results pages. Click the "Display all previews as thumbnails" link at the top of the page. This allows users to quickly compare the images or spectra of their search results.

    2.7.5 High-Level Science Product Previews

    Several sets of High-Level Science Products have previews associated with them. See the pages of the individual HLSP Projects (http://archive.stsci.edu/hlsp/index.html) for preview availability.

    2.8 Retrieving Data From the Archive

    By its nature, the archive contains very different datasets, not all of which can be retrieved in the same manner. In addition, users have different needs and what works for one may not work for another. The following sections describe various retrieval options.

    2.8.1 Retrieving HST and FUSE Data From DADS

    HST and FUSE data, excluding previews, are stored within the Data Archive and Distribution System (DADS). The DADS system has several functions beyond the basic retrieval of data. DADS contains an On The Fly Reprocessing (OTFR) feature. Data from the currently active HST instruments are reprocessed at each request to ensure that the latest calibrations and header corrections are applied. DADS distinguishes and handles proprietary data for both HST and FUSE. You may choose to have the data staged in an archive staging area or transferred directly to your computer. Requests for HST data may also be written to DVD or CD-R.

    2.8.1.1 Registration/STScI SSO Credentials

    Authentication (i.e., registration) is only required for retrieval of proprietary data, or retrieval of any data to physical media (i.e., DVD). User may use their STScI SSO credentials when requesting public data. See the STScI's Single Sign On (SSO) Portal, for details on recovering a forgotten password, updating your STScI SSO account information (e.g., contact information, change passwords, etc.) and other useful information.

    When requesting data anonymously, enter your email address. The email address is used to notify you when your request has completed or if there is a problem with your retrieval.

    2.8.1.2 Retrieving Proprietary Data

    If you are an HST or FUSE Guest Observer/Investigator, you may already have STScI SSO credentials. That is, your old archive account may have been converted to an STScI SSO account. An e-mail was sent to all users whose archive accounts were converted to SSO. If you did not receive such an e-mail and/or have not yet been authenticated at STScI, go to the STScI's Single Sign On (SSO) Portal and create an STScI SSO account. If you encounter and problems creating the SSO account, contact support@stsci.edu.

    You can easily tell if data are proprietary as the backgound of the "Mark" button on the search results page is bright yellow.

    2.8.1.3 Retrieval Options

    In general, data are retrieved by performing a search using the web interface or Starview, and then marking the data to be retrieved. A page with a retrieval options page is displayed. You may be asked for your SSO credentials (SSO account and password). As stated above, you may enter anonymous as the username with your email address as the password if you are not requesting proprietary data.

    You will be asked to specify one of the following data delivery options:

    • FTP the data to the destination you specify on the form
    • SFTP the data to the destination you specify on the form
    • Stage - Put the data onto the Archive staging disk where you will be able to pull them to your computer using FTP. The data will remain on this staging disk for about 4 days. See section Retrieving Data from the Staging Area for more information on accessing the staging area.
    • DVD - Write the data to DVD and mail it to the address associated with the STScI SSO account name. If you request data on hard media anonymously, you will be prompted for an address.

    If you choose to have the data transfered to a specified location, you must indicate the hostname, the directory, the username and password DADS should use to log in to deliver the data. YOU are responsible for making sure that there is sufficient disk space available for the data. If the transfer fails, the data will be written to the staging disk and you will be notified via email.

    You will be asked to designate the type of files requested. This is different for HST and FUSE data. Help on the file types is available from the request options page.

    2.8.1.4 Retrieving Data from the Staging Area

    Retrieve data from the staging disk on archive.stsci.edu by logging in with your STScI SSO credentials.. If you retrieved data anonymously, you will use anonymous as your archive username and your email address as the password.

    If you issue a dir command you will see the the pub directory where other MAST data and the MAST High-Level Science Products are stored and a directory called stage. The dads retrieval data is stored in that directory in a subdirectory with your username. Data requested with the anonymous username, is stored in a subdirectory called anonymous. You will only be able to see and access these two directories. With in "your" subdirectory will be a subdirectories containing the data named by the request id.

    A typical ftp session might look like this, where account name is your STScI SSO account:

    ftp archive.stsci.edu
    accountname
    password
    cd stage/accountname
    cd accountname54321
    prompt
    binary
    mget *
    quit
    

    After you have successfully downloaded the data from your directory, you may delete the data. Although the data are removed automatically about 4 days after the request completes, during peak usage times the disk does tend to fill up, so it is helpful if you delete data that you have already retrieved. A typical session to delete files might look like this:

    ftp archive.stsci.edu
    accountname
    password
    cd stage/accountname
    cd accountname54321
    prompt
    mdele *
    cd ../
    dele accountname54321
    quit
    

    2.8.1.5 Retrieving Large Amounts of HST Data from DADS

    As the archive has expanded, some users need to make requests for large amounts of HST data. Please consult the Large Searches and Requests page at http://archive.stsci.edu/hst/bigsearch_request.html#requests. It explains the best methods and some guidelines for ensuring a smooth and complete finish to your large requests.

    2.8.2 Downloading Other MAST Data

    Other MAST data, e.g. GALEX, IUE, ASTRO, ORFEUS and VLA-FIRST, are stored on spinning disk. These data may be downloaded from the search interface or via anonymous ftp. Do see the GALEX chapter for information specific to GALEX.

    2.8.2.1 Downloading data from the search interface.

    After performing a search for observations of interest, you can to download the data to your computer utilizing the download data option on the search result page. Data download is available from the html formatted search results page but not from the other output formats.

    Choose the observations of interest by clicking in the "Mark" box. If you would like to retrieve all the data on the results page, use the "Mark all" button at the top and bottom of the page. The requested data will be packaged and downloaded in a single tar or zip file. You may choose the type of file. Click on the "Download selected datasets" button. A dialog box will open, giving you options for downloading the data. For most missions you will receive all the files for the selected datasets. The VLA-FIRST data are currently downloaded one file at a time by clicking on the Data ID.

    The options for IUE data are slightly different because two versions of the IUE data are available. The versions relate to the type of processing: the Final Archive data products (NEWSIPS) or the products from the original IUESIPS processing pipeline. There are two buttons on the search results page to download data. The "Download NEWSIPS MX files as a .tar file" option will download the files of extracted spectra for the marked data. The "More retrieval options" button will bring up a form that permits you to select which files and format you wish to download. At the top are two menu boxes to select NEWSIPS or IUESIPS data products. The default is the MX data from NEWSIPS. To download IUESIPS data you will need to select not only the files but the data type using the Download menu. There are three types: NEWSIPS (which are FITS files), IUESIPS in the "GO" format (files are in the original VICAR label format), and IUESIPS_RDAF which reads the IUESIPS GO formats and places them in the format easily utilized by the IUE RDAF software. This option is for the convenience of long-time IUE users. The IUE RDAF software can utilize all the formats. Clicking on the "Submit Request" button will bring up the dialog box with options for downloading the data.

    EUVE data may be downloaded from the search results page. The links to the data are in the column labeled Data Files. Clicking on the IMG link will bring up a download dialog box to download the image file; right clicking on the SPEC link will download the file. Similarly, VLA-FIRST data are downloaded from the search results page by clicking on the Data ID file.

    2.8.2.2 Downloading data via anonymous FTP

    MAST data other than HST and FUSE may be downloaded via anonymous ftp. FTP to archive.stsci.edu as anonymous using your email address as the password. Change directory to pub and then cd to the mission with the desired data. The HUT, UIT and WUPPE data are under a directory called astro and the BEFS, TUES and IMAPS data are under a directory called orfeus. All data are stored in a subdirectory called data with further subdirectories grouped by dataset name. For some missions the dataset name includes the targets, for others the dataset names are basically sequence numbers and you will need to know the name of the dataset you would like.

    For example, here's how to download HUT data for U Gem.

    ftp archive.stsci.edu
    anonymous
    user@institution.edu
    cd pub/astro/hut/data
    dir
    cd u-gem_152 
    prompt 
    binary 
    mget *
    

    IUE data are organized in directories by camera. Within each camera directory are directories where the data are grouped by image sequence number. For example in pub/iue/data/lwp/01000 are all the files (both NEWSIPS and IUESIPS) for image sequence numbers 01000 - 01999.

    2.8.2.3 Downloading MAST data via the browser

    The pub area has been set up so that you may display the contents of an ftp directory in a browser. For example, to see the entire set of files available for the BEFS dataset befs1001 you would use the url: ftp://archive.stsci.edu/pub/orfeus/befs/data/befs1001/. By using ftp rather than http, you should get a dialog box when you click on the file you wish to download. If you use http, then you will need to right click on the file name to see the download options.

    You could also use the "wget" tool. The general path for the data in the anonymous ftp area is: http://archive.stsci.edu/pub/mission/data/datasetdir

    2.8.2.4 Downloading ASCII versions of the spectral data

    Most of MAST's spectral data are available as ASCII text files. The files are available from the preview page or via your browser. In general, the url follows this template: http://archive.stsci.edu/browse/previews/mission/txt. As an example, EUVE ASCII files are at: http://archive.stsci.edu/browse/previews/euve/txt/. Note that for ASTRO and ORFEUS missions the url would include the appropriate shuttle mission name before the mission as in http://archive.stsci.edu/browse/previews/orfeus/tues/txt/. If you use http in the url name, then you will need to right click on the file name to see the download options. By using ftp in the url rather than http, you should get a dialog box when you click on the file you wish to download.

    Note that for some missions such as EUVE and FUSE, the data are not exactly the same as the contents of the fits files. The FUSE project strongly recommends that the FUSE ASCII data be used ONLY as a preview and not be used for scientific analysis.

    2.8.3 High-Level Science Products

    MAST archives a growing number of High-Level Science Products. Most are located in the anonymous ftp area and may be retrieved via browser, anonymous ftp and from the HLSP search results page as described in the page describing HLSP download options.

    MAST has archived several sets of High-Level Science Products on spinning disk in an anonymous ftp area on archive.stsci.edu. Users may also download the data from several browser interfaces. Currently, a few of the High-Level Science Products are not stored in the anonymous ftp site, but these may be moved into this area over time.

    MAST does not archive all of the HLSPs listed on the HLSP page, but includes links to the off-site websites. Obviously, these instructions are not pertinent to those "off-site" HLSP.

    2.8.3.1 Downloading Via Anonymous FTP

    If you would like to download an entire set of HLSP, the most efficient method might be to download the data via anonymous ftp. In general follow this procedure:

    ftp archive.stsci.edu
    (logon as anonymous)
    cd pub/hlsp
    

    In general, the HLSP are stored in subdirectories in the hlsp directories. A few sets of HLSP - the Hubble Deep Field, the Hubble Deep Field South, Helix Nebula, the Magellanic Cloud Planetary Nebula, and the Medium Deep Survey are in subdirectories under the pub directory.

    2.8.3.2 Downloading via Browser

    The pub area has been set up so that you may display the contents of an ftp directory in a browser. For instance, to see the entire set of products for the GOODS Version 1.0 you would use the url:
    ftp://archive.stsci.edu/pub/hlsp/goods/v1/
    By using ftp rather than http, you should get a dialog box when you click on the file you wish to download. If you use http, then you will need to right click on the file name to see the download options.

    2.8.3.3 Downloads from the High-Level Science Products Search Form

    Users who use the HLSP search interface will have a list of products meeting their search criteria displayed in the browser window. Clicking on the filename in the display will bring up a dialog box to download the file to your computer. A similar interface is available from the Web searches for HST, FUSE, IUE and EUVE. A column labeled High-Level Science Products is displayed in the search results page. Click on the number, which represents the number of HLSP associated with the observation, to see a list of the associated HLSP. Clicking on the filename will bring up a dialog box to permit a file download.

    2.9 Data Archiving Guidelines

    The information below is for projects intending to archive mission data sets within MAST. See the web page at http://archdev.stsci.edu/mast_guidelines.html for the most up to date requirements. Recommendations are included regarding:

    • data set formats for archival data (i.e., using FITS),
    • the creation of database tables to allow online catalog searches,
    • the preservation of project-specific software and documentation.

    The usefulness of mission data sets to future users will depend on how well this information is preserved. MAST staff members can work with projects to assist them in developing data products that will be useful to the general astronomical community for the long term. We can review FITS headers, for example, for compliance with the standard and to confirm the information most frequently used for searching is present.

    1. Data Set Formats

      The astronomical community has adopted the Flexible Image Transport System (i.e. FITS) format as the default standard for the exchange of data between institutions. The FITS file format is platform independent, supported by many institutions, and endorsed by both NASA and the IAU. For these reasons FITS is the recommended file format for archiving data at STScI. A description of the FITS data format recommendations can be found in the MAST Data Format Guidelines document. A online version of the FITS Standard Document and the FITS User's Guide is also available.

      It is recognized however that some archival data may be stored in other formats, particularly for those projects which preceded the recent developments in FITS. One example is the earlier-processed IUE data which is stored in a VICAR-based IUE "GO" format. In some other cases, projects have distributed data as ASCII text files or created auxiliary data sets as ASCII text files or postscript format. In these cases, no attempt will be made to reformat the data sets before being archived within MAST.

    2. Catalogs

      MAST, like other data retrieval systems, uses online database tables to search for requested data sets. In many cases, projects store the same information in both the catalog or database table and the FITS keywords. To simplify adding new data sets into MAST, it is helpful if either

      1. the project provides a target list or catalog of observations, or
      2. the FITS headers are constructed such that MAST staff members can create a catalog from the FITS keywords.

      Obviously, project-delivered catalogs would greatly simplify the archiving of mission data sets. In the absence of catalogs or target lists, we would appreciate guidance from the project staff concerning which of the FITS header keywords would be most useful for searching. We do not want to do a wholesale ingest of all header keywords

      Catalogs should contain those fields which would most help users locate the desired observation(s). Coordinates, observation date, exposure time, and target name are fairly essential (depending on the method of observation), while parameters needed for analyzing or interpreting archived data would be highly desirable. A target classification entry has been very useful for users interested in particular types of objects.

      Although the MAST uses the Sybase Database Management system, tables can be exchanged between most database systems by copying them to ASCII table files (be sure to include a sufficient number of significant figures for representing floating point values.) A WEB page containing an observation list may be an adequate replacement for a database table. In either case, a description of the individual fields within the table or list should be provided as well. The description should also define the source of the entries. For example, it should state whether the coordinates were supplied by the observer, or obtained from an existing catalog.

    3. Documentation

      Project-supplied documentation in the following categories should be made available for archive users:

      1. Project Description - General descriptions of the mission and instrumentation,
      2. Data Processing - how the data was reduced and calibrated,
      3. Data Description - Documentation on data characteristics (e.g., instrumental resolution, field of view, wavelength coverage, etc.), anomalies, (e.g., cosmic ray hits, bad pixels, scratches, etc.), measurement uncertainties, and database field descriptions,
      4. Data Format - A general description of the contents of the archived mission data sets including, for example, documentation on the FITS keyword entries. (Note generally the FITS keyword comment field alone is insufficient to properly define many keywords. Without additional documentation, many of these keywords will be of little or no use to future users.)

      Since MAST documentation will be accessed primarily from the WEB, documentation is most useful if it exists online. Most text processing formats such as LaTeX or Microsoft WORD (or standard ASCII files) can be fairly easily converted to HTML by staff members. Large documents such as user manuals or data analysis guides should be made available to users in several formats such as HTML for online access, and POSTSCRIPT and/or PDF for downloading.

    4. Software

      Some projects have written software to analyze and interpret raw and/or processed data. These programs should be archived for future users. MAST will make project-supplied software available to requesters, although support for the software itself can not be provided. MAST currently maintains for example, the IUEDAC IDL software libraries, the UIT BDR software written in C and Fortran-77, and the EUVE EUV1.8 IRAF software.

      A list of available Fits software packages is available from HEASARC. The list contains links to sites supporting general FITS readers and writers written in a variety of programming languages.

    2.10 Guidelines for Contributing High-Level Science Products to MAST

    MAST welcomes the contribution of HLSPs from the astronomical community. The purpose of this document is to establish the guidelines and policies for the contribution of HLSP to MAST. Note that contributions of HSLP are now a requirement for the HST Treasury and Legacy programs. See the Guidelines for Contributing HLSP web site for the most current requirements. If you wish to contribute HLSP or have any questions concerning these guidelines please send mail to archive@stsci.edu.

    The guidelines for archiving non-reduced data from entire missions is a separate document (see previous section or the Data Archiving Guidelines ).

    MAST encourages and welcomes HLSP including any of the following examples:

    • Products from any of the MAST-supported missions (e.g. HST, FUSE, GALEX, IUE, EUVE etc.).
    • Ground-based observations closely related to any of the MAST missions.
    • Output produced from theoretical models, provided that they are closely related to the MAST data and are not generally available through another permanent archive.
    • Object catalogs and lists of identified spectral lines

    Literature, including paper preprints and reprints and analysis products several steps removed from the HLSP are not considered HLSP and are inappropriate for this site.

    HLSP should contain a short "README" file explaining how a given HLSP was created, including details on the data reduction and pipeline processing version. It may in addition contain the following types of files and products:

    • Plots relevant to the HLSP and quicklook/preview images in common display formats such as PostScript, GIF, JPEG, PNG and PDF.
    • Thoroughly tested and well-documented specialized analysis software relevant to the HLSP, Java applets, and other image display software.

    We may need to work with you to modify the format of some figures, so please hang onto your plot-generating scripts!

    It is important that contributors designate a contact person who will be available to answer technical questions from users regarding details of the actual observations, the processing techniques, or scientific quality issues that MAST may not be in a position to answer. From experience, we expect that such queries will be infrequent.

    2.10.1 File Formats, Keywords, and Naming Conventions

    We recommend the following standard formats:

    • FITS for images and spectra; FITS ASCII tables are also useful for catalogs having a large number of columns
    • plain text ASCII (catalogs, some software, README files)
    • PDF (plots, and large README files)
    • Any common image format (e.g. GIF, JPEG, PNG) for preview, color and annotated images (MPEG is also allowed)

    File formats specific to particular software such as Microsoft, Tex, LaTex, and IDL save-set formats are discouraged.

    We also ask that these requirements be followed:

    • file names all be lower case.
    • filenames include a version number if there will be more than one version of the HLSP submitted.
    • filenames start with h_ so that they can be easily distinguished as an HLSP
    • filenames need to be unique within MAST. MAST will ensure this, but submittors may need to add some project identifier in the filename to ensure this uniqueness.
    • the total length of the filename should not exceed 39 characters.
    • FITS files should contain the keyword FILENAME with an accurate self-referencing entry.
    • FITS files should contain RA and Dec keywords as well as the RADECSYS and EPOCH keyword. For images, the coordinates should designate the center of the field. For pointed spectra, the coordinates should be of the object.
    • FITS headers should include WCS keywords.
    • for data products created from MAST data, provide a list of the individual datasets used in creating the file. This "list" may already be coded as part of the processing information in the FITS header and that is adequate. An alternative is to provide a separate file that contains the information. These lists will be used to "map" HLSP to the individual datasets used to create them.

    Since MAST is eventually going to make your HLSP available to the IVOA (International Virtual Observatory Alliance), you might consider enhanceing the information content in the header of your datafiles. You can scan through a list of optional header keywords for images and some examples from current HLSP headers. The respective information on spectra is also available.

    All FITS format files must be verified before inclusion in the MAST archive. We suggest FITS verification be done early in the production of the HLSP, to avoid delays in archiving the product(s). MAST recommends that using the FITSVERIFY validator available from HEASARC. You can also verify your FITS files online at http://fits.gsfc.nasa.gov/fits_verify.html.

    Instructions for reading and writing FITS files can be found within the standard data reduction packages such as IRAF and STSDAS, and within the CFITSIO package distributed by HEASARC at http://heasarc.gsfc.nasa.gov/docs/software/fitsio/fitsio.html.

    2.10.2 Delivery of Files to STScI

    Data may be delivered electronically or on hard media such as DVD, whichever is most convenient for you.

    Some additional information may be useful at the time of initial contact:

    • Name of a Contact Investigator for the program, along with contact information including email address and telephone number
    • Program of origin, e.g., HST/Treasury for HST Treasury program data, or HST/GO for HST Guest Observer data; Contribution from an ADP, LTSA or AISRP programs.
    • The program number, if there is any, e.g., HST proposal ID or ADP number
    • The primary target of the observations, e.g. "Hubble Deep Field" or "NGC1068"
    • A list of the missions from which the HLSP are derived, e.g., HST, IUE, and GALEX
    • References of up to three papers closely related to the contributed HLSP, excluding authors, in standard format, e.g., 2001, ApJ, 551, 23

    2.10.3 Web Support

    MAST will point to any websites the project or contributors set up. If you wish to ontribute a set of web pages, but have no place to host them, MAST may be able to host them for you. If no web site is available, MAST will maintain a page that will include basic information on the HLSP set. Even if you are planning your own web site, we will want to archive the data at MAST where we can provide a long-term home for it. We also would like you to consider having your web site point to the data here rather than to a local copy. That ensures that the same data gets delivered from both sites and allows us to easily and consistently track use of the products from the Treasury programs. We can also more easily integrate these valuable products with our other search interfaces. We are starting to make the HLSP available to the Virtual Observatory, and that process is also easier if we archive the data here.

    A list of acronyms and abbreviations used in the MAST web pages.

    The MAST GALEX team provides separate tutorials for the GALEX web site. While the GALEX site and interface have many things in common with the rest of MAST, there are some differences due to a difference in servers and software.

    This tutorial currently does not contain information about the Guide Star Catalog (GSC) interfaces and data. Please see the GSC web pages for additional information.

    2.11 MAST Documentation

    The MAST web pages are compliant with "HTML 4.01 transitional" and "CSS levels 1 and 2". Cascading style sheets (CSS) and JavaScript are used for controlling the appearance of the navigation links so be sure they are enabled in your browser. We are also in the process of verifying compliance with Priority 1 accessibility and section 508 compliance.

    The majority of MAST pages are created dynamically using a PHP-based template. The template combines a MAST header, mission-specific banners, gutters, and footers with staff/mission-generated content files. The pulldown menus use JavaScript. If JavaScript is disabled, one can still click the link and find the next level of links on the displayed page. Note JavaScript is also used in some web forms for such things as marking files for download and selecting output columns.

    MAST HTML forms either call PERL CGI scripts, or use PHP. Several programs now use XML for file formatting (e.g., the MAST mission search forms, and whatsnew and faq pages).

    2.11.1 Organization

    The MAST web site is basically organized by mission, with each mission having it's own home page. For active missions that maintain their own web site (e.g., FUSE, TUES, & GSC), links are made directly to the project-supported pages where appropriate. Also, for some missions (e.g., EUVE), all or part of the data archive actually resides at another data center but data can be retrieved transparently using MAST-style search forms.

    The MAST home page itself (http://archive.stsci.edu/index.html) contains general information and links to cross-mission tools, software, documentation, and "high-level" prepared datasets. Besides the various cross-mission search tools, seach forms are also available for each individual mission.

    All MAST pages follow a common page layout as much as possible.

    2.11.2 Use the HELP for searches and tools

    All of the searches and web tools have help options with tips for how to use them. The help button is generally placed in the top right of the screen for the searches and tools (although for some older tools the help button may be elsewhere). In each search page, the labels are all links to information about the item and when needed, information about entering qualifications for that field. In the search results pages, the column headings are links to information about the data in that column.

    Most MAST missions have a listing of the fields in the mission observation catalogs/database that are searchable via the "User defined fields" on the search pages or via the MAST web services that permit searching. Click on the "Field Descriptions" link on the search forms or on the links of interest in the description of MAST web services.

    2.11.3 Mission Documentation

    Almost all documentation about a mission and its data was received from the mission project so the level and quantity of documentation varies from mission to mission. In general you can find documentation about the mission itself (e.g. Instrumentation and Operations) by clicking on the "About mission" tab in the second navigation bar. Documentation about the mission data can be found in the "About mission Data" pullouts in the left menu.

    2.11.4 News

    MAST maintains an archive of news items. The 5 most recent news items are posted in the "News" section of the main MAST page and on the top "mission" pages. The news on the MAST page contains news for the entire MAST archive. The news on the mission pages contains news items relevant to that mission. The mission news may pertain to changes in the web site that affect all missions. To see older news items click on the "NEWS" link.

    For more information about the news item, click on the "headline". The news item will usually contain links to any additional, relevant pages.

    MAST NEWS is also available as an RSS News Feed.

    MAST has an electronic archive newsletter that is sent to a mailing list of self-subscribed recipients. MAST news is also reported in the quarterly STScI Newsletter.



    [Top] [Prev] [Next] [Bottom]