Retrieving Data from MAST

Retrieving HST and FUSE data from DADS
Downloading Other MAST data
Downloading High-Level Science Products
Return to the Tutorial Index

Retrieving HST and FUSE Data From DADS

HST and FUSE data are stored within the Data Archive and Distribution System (DADS). The DADS system has several functions beyond the basic retrieval of data. DADS contains an On The Fly Reprocessing (OTFR) feature. Data from the currently active HST instruments are reprocessed at each request to ensure that the latest calibrations and header corrections are applied. DADS distinguishes and handles proprietary data for both HST and FUSE. You may choose to have the data staged in an archive staging area or transferred directly to your computer. Requests for HST data may also be written to DVD or CD-R.

Registration
In earlier versions of DADS, you were required to register to retrieve all HST and FUSE data, but now registration is only required for proprietary data, although most users still register and use their archive accounts to request data. It is easy to register and to maintain your account. If you have forgotten your password, you may ask for it to be emailed to the email account designated at the time of registration. You may maintain your own account, keeping your address and phone numbers up-to-date with no need to interact with archive personnel. If you are requesting only public data and you are not having the data written to DVD or CD-R, you may request data anonymously. When asked for your account, enter anonymous and enter your email address as your password. The email address is used to notify you when your request has completed or if there is a problem with your retrieval.
Retrieving Proprietary Data
If you are an HST or FUSE Guest Observer/Investigator, you must contact the archive helpdesk, to let them know that you wish to request your proprietary data and supply the program ID. Archive personnel will confirm that the user is the PI of the program. If you are a PI that wishes to share data with a Co-I or collaborator, you must contact the archive helpdesk and give the archive account name you wish to authorize to access the data. Co-I's cannot request data until they are authorized in this way. You can easily tell if data are proprietary as the background of the "Mark" button is bright yellow.
Retrieval Options
In general, data are retrieved by performing a search using the web interface or Starview, and then marking the ata to be retrieved. A page with a retrieval options page is displayed. You will be asked for your archive user name and password. As stated above, you may enter anonymous as the username with your email address as the password if you are not requesting proprietary data.
You will be asked to specify one of the following data delivery options:
- Stage - Put the data onto the Archive staging disk where you will be able to pull them to your computer using FTP. The data will remain on this staging disk for about 2 weeks. See section Retrieving Data from the Staging Area for more information on accessing the staging area.
- DVD/CD - Write the data to DVD or CD-R and mail it to the address associated with the account name. You may not request data on hard media anonymously.
If you choose to have the data transfered to a specified location, you must indicate the hostname, the directory, the username and password DADS should use to log in to deliver the data. YOU are responsible for making sure that there is sufficient disk space available for the data. If the transfer fails, you will be notified via email.
You will be asked to designate the type of files requested. This is different for HST and FUSE data. Help on the file types is available from the request options page.
Retrieving Data from the Staging Area
Retrieve data from the staging disk on archive.stsci.edu by logging in with your archive username and password. If you retrieved data anonymously, you will use anonymous as your archive username and your email address as the password.
If you issue a dir command you will see the the pub directory where other MAST data and the MAST High-Level Science Products are stored and a directory called stage. The dads retrieval data is stored in that directory in a subdirectory with your username. Data requested with the anonymous username, is stored in a subdirectory called anonymous. You will only be able to see and access these two directories. With in "your" subdirectory will be a subdirectories containing the data named by the request id.
A typical ftp session might look like this:
```
ftp archive.stsci.edu
accountname
password
cd stage/accountname
cd accountname54321
prompt
binary
mget *
quit
```
After you have successfully downloaded the data from your directory, you may delete the data. Although the data are removed automatically about 4 days after the request completes, during peak usage times the disk does tend to fill up, so it is helpful if you delete data that you have already retrieved. A typical session to delete files might look like this:
```
ftp archive.stsci.edu
accountname
password
cd stage/accountname
cd accountname54321
prompt
mdele *
cd ../
dele accountname54321
quit
```
Retrieving Large Amounts of HST Data from DADS

As the archive has expanded, some users need to make requests for large amounts of HST data. Please read some information that explains the best methods and some guidelines for ensuring a smooth and complete finish to your large requests

Return to the Tutorial Index

Downloading Other MAST Data

Other MAST data, e.g. IUE, ASTRO, ORFEUS and VLA-FIRST, are stored on spinning disk. These data may be downloaded from the search interface or via anonymous ftp.

Downloading data from the search interface.
After performing a search for observations of interest, you can to download the data to your computer utilizing the download data option on the search result page. Data download is available from the html formatted search results page but not from the other output formats.
Choose the observations of interest by clicking in the "Mark" box. If you would like to retrieve all the data on the results page, use the "Mark all" button at the top and bottom of the page. The requested data will be packaged and downloaded in a single tar or zip file. You may choose the type of file. Click on the "Download selected datasets" button. A dialog box will open, giving you options for downloading the data. For most missions you will receive all the files for the selected datasets. The VLA-FIRST data are currently is downloaded one file at a time by clicking on the Data ID.
The options for IUE data are slightly different because two versions of the IUE data are available. The versions relate to the type of processing: the Final Archive data products (NEWSIPS) or the products from the original IUESIPS processing pipeline. There are two buttons on the search results page to download data. The "Download NEWSIPS MX files as a .tar file" option will download the files of extracted spectra for the marked data. The "More retrieval options" button will bring up a form that permits you to select which files and format you wish to download. At the top are two menu boxes to select NEWSIPS or IUESIPS data products. The default is the MX data from NEWSIPS. To download IUESIPS data you will need to select not only the files but the data type using the Download menu. There are three types: NEWSIPS (which are FITS files), IUESIPS in the "GO" format (files are in the original VICAR label format), and IUESIPS_RDAF which reads the IUESIPS GO formats and places them in the format easily utilized by the IUE RDAF software. This option is for the convenience of long-time IUE users. The IUE RDAF software can utilize all the formats. Clicking on the "Submit Request" button will bring up the dialog box with options for downloading the data.
EUVE data is archived at HEASARC on spinning disk. You may download the data from the search results page. The links to the data are in the column labeled Data Files. Clicking on the IMG link will bring up a download dialog box to download the image file; clicking on the EVT link will download the Event file. Similarly, VLA-FIRST data are downloaded from the search results page by clicking on the Data ID file.
Downloading data via anonymous FTP

MAST data other than HST and FUSE may be downloaded via anonymous ftp. FTP to archive.stsci.edu as anonymous using your email address as the password. Change directory to pub and then cd to the mission with the desired data. The HUT, UIT and WUPPE data are under a directory called astro and the BEFS, TUES and IMAPS data are under a directory called orfeus. All data are stored in a subdirectory called data with further subdirectories grouped by dataset name. For some missions the dataset name includes the targets, for others the dataset names are basically sequence numbers and you will need to know the name of the dataset you would like.
Example showing download of HUT data of U Gem.
```
ftp archive.stsci.edu
anonymous
user@institution.edu
cd pub/astro/hut/data
dir
cd u-gem_152 
prompt 
binary 
mget *
```
IUE data are organized in directories by camera. Within each camera directory are directories where the data are grouped by image sequence number. For example in pub/iue/data/lwp/01000 are all the files (both NEWSIPS and IUESIPS) for image sequence numbers 01000 - 01999.
Downloading MAST data via the browser

The pub area has been set up so that you may display the contents of an ftp directory in a browser. For example, to see the entire set of files available for the BEFS dataset befs1001 you would use the url: ftp://archive.stsci.edu/pub/orfeus/befs/data/befs1001/. By using ftp rather than http, you should get a dialog box when you click on the file you wish to download. If you use http, then you will need to right click on the file name to see the download options. "
HEASARC has information about the FTP area where EUVE data are stored.
You could also use the "wget" tool. The general path for the data in the anonymous ftp area is: http://archive.stsci.edu/pub/mission/data/datasetdir
Downloading ASCII versions of the spectral data

Most of MAST's spectral data are available as an ASCII text files. The files are available from the preview page or via your browser. In general, the url follows this template: http://archive.stsci.edu/browse/previews/mission/txt. As an example, EUVE ASCII files are at: http://archive.stsci.edu/browse/previews/euve/txt/. Note that for ASTRO and ORFEUS missions the url would include the appropriate shuttle mission name before the mission as in http://archive.stsci.edu/browse/previews/orfeus/tues/txt/. If you use http in the url name, then you will need to right click on the file name to see the download options. By using ftp in the url rather than http, you should get a dialog box when you click on the file you wish to download.
Note that for some missions such as EUVE and FUSE, the data are not exactly the same as the contents of the fits files. The FUSE project strongly recommends that the FUSE ASCII data be used ONLY as a preview and not be used for scientific analysis.

Return to the Tutorial Index

High-Level Science Products

MAST archives a growing number of High-Level Science Products. Most are located in the anonymous ftp area and may be retrieved via browser, anonymous ftp and from the HLSP search results page as described in the page describing HLSP download options.

Return to the Tutorial Index