In the beginning was the Hubble Data Archive (HDA), which contained data from HST. Users searched the catalog, sometimes called the DADS catalog, using Starview, a software package developed at STScI. Data were retrieved through the Data Archiving and Distribution System (DADS) and written to tapes, which were mailed to the user. As networks developed and became robust, the demand for web-based search and retrieval grew. A web interface was first made available in 1994. A few years later, the Multimission Archive at STScI (MAST), funded under a separate contract, was created to host data from optical, UV and IR space-based missions. At about the same time, on the fly calibration (OTFC), later replaced by on the fly recalibration (OTFR), was developed for some HST data. At STScI a user could find data from many missions, some available for immediate download, some delayed by OTFR processing. Starview could be used to search for data from some missions, while the web interface, sometimes called MAST web, could be used to search all missions. Some complicated searches of the HST catalog, which Starview supports, are not supported by the web. The alphabet soup of acronyms continued to grow as more missions and tools were added to the archive. The distinction between HDA and MAST lessened except in funding.

In this document the holdings at STScI, whether archived and maintained under the Hubble contract or the MAST contract, are treated as one archive, called the archive, as that's what it looks like to an outside user. No distinction is made between datasets stored on spinning disk and those stored in archive appliances. Where it is important, the distinctions between HDA and MAST are delineated. For example, there are some differences when retrieving HST or FUSE data, and some acknowledgement distinctions are made, as required by the funding sources.

While this document was in revision, work on the Hubble Legacy Archive (HLA) began. As the HLA is under development, it will not be discussed in this version of the Archive Manual.

For HST and FUSE, the archive contains the calibration reference files, such as flat fields.

For HST, the archive contains engineering files (aka observation logs or jitter files) that may be useful for diagnosing some questions about observations, and the spacecraft ephemeris.

Retrieved data from all missions are primarily in FITS format.

The archive has searchable catalogs of the data, consisting of information about the observations and targets. For HST data, the catalog is populated from the header keywords of the data files and is quite extensive.

HST data become available to the astronomical community upon the expiration of a proprietary period. For HST, most general observer (GO) and guaranteed time observer (GTO) observations have proprietary periods of a year, but some observations have shorter or longer proprietary periods. Nearly all calibration observations are made public immediately upon receipt. The archive catalog contains information on these proprietary observations.

Proprietary datasets may be retrieved only by GOs and GTOs with the appropriate authorization. See the section on Registration/STScI Single Sign On (SSO) for account information.

All publications based on HST archival data should carry the following footnote:

"Based on observations made with the NASA/ESA Hubble Space Telescope, obtained from the data archive at the Space Telescope Science Institute. STScI is operated by the Association of Universities for Research in Astronomy, Inc. under NASA contract NAS 5-26555."

In addition, if the archival research was supported by a grant from STScI, the publication should also carry the following acknowledgment at the end of the text:

"Support for this work was provided by NASA through grant number ________ from the Space Telescope Science Institute, which is operated by AURA, Inc., under NASA contract NAS 5-26555."

Please send one preprint or reprint of each refereed publication based on HST archival research to the following address:

Librarian
Space Telescope Science Institute
3700 San Martin Drive
Baltimore, MD 21218 USA

Publications based on all other data from the archive should carry the following acknowledgment:

"Some/all of the data presented in this paper were obtained from the Multimission Archive at the Space Telescope Science Institute (MAST). STScI is operated by the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS5-26555. Support for MAST for non-HST data is provided by the NASA Office of Space Science via grant NAG5-7584 and by other grants and contract."

There are many ways of searching the archive for data of interest. These include, but are not limited to, StarView, MAST (web based), SpecView, Aladin and many Virtual Observatory (VO) services. There are also special interfaces or tools, such as the CASJobs implementation for searching GALEX data. Some of these interfaces, applications and tools are described here, others are described in the MAST chapter. Still others are not discussed at all. See the MAST chapter and the on-line help for details.

The Virtual Observatory (VO) project is creating standards to facilitate the discovery and joining of data among astronomical archives. These projects include the US-Virtual Observatory (NVO) which is a member of the International Virtual Observatory Alliance (IVOA). MAST is working to make the archive data compliant with the current VO defined standards and protocols.

For most missions in the archive, retrieval can be as simple as clicking on the dataset name. However for HST and FUSE data the procedure is different.

As noted above, public data may be retrieved anonymously. Only STScI SSO credentialed (registered) and authorized users (see section 1.5, above) can retrieve proprietary data. Once you have STScI SSO credentials, you can retrieve your proprietary data by using MAST to select your datasets and choose the delivery mode.

Whatever means is used to search the archive, when retrieval is requested for HST or FUSE data the Retrieval Options page is displayed. You are required to select a delivery option for the data. Current options are

• FTP/SFTP, which will send the data directly to your home computer,

• STAGE, which will place the data on the archive computer staging disk. You must log into this computer and FTP your data to your home computer.

• DVD/external hard disk, which will be mailed to you after the data are retrieved.

For every request you will be asked for your STScI SSO credentials. For anonymous retrieval, the default, enter your e-mail address. For an SFTP/FTP retrieval, you will be required to give your home username and password so that the retrieved data can be written to your disk.

When retrieving a large number of datasets, it is better to submit several smaller requests than one large one. To avoid delay, please keep the number of datasets in each request under 100 - 200.

After a request is submitted, the archive system processes it. An e-mail notification is sent immediately when the system has accepted the request and again when the system has completed processing the request, indicating whether or not the transfer was successful. The messages will go to the e-mail address specified on the Retrieval Options page (for anonymous retrieval) or, if an STScI SSO account was used, to the e-mail address associated with that STScI SSO account.

How long a retrieval takes depends on a variety of factors, including: the type of data in the request, the size of the request, the number of requests in the system at the time, and the destination of the request (the internet connections between STScI and some sites, especially those overseas, is sometimes a significant source of delay). If everything is running smoothly, one should expect a median turn around time of an hour. If it takes more than one day, and you do not think any of the factors listed above are playing a significant role, please contact us at archive@stsci.edu.

If the data are retrieved to the staging disk (STAGE option), the data will be written to a subdirectory. Each data retrieval request will be in its own subdirectory, identified by the request ID number, which will be included in the notification message you will receive. To find your data, from your home account type:

% ftp archive.stsci.edu
% login: STScI SSO credentials
% password: ******
ftp> binary
ftp> cd /stage/username/request_ID_number
ftp> ls

Note the use of binary in the above example. Not all ftp clients automatically set the data mode. Any attempt to ftp fits files in ASCII mode will result in corrupted data files, with no errors from ftp.

For anonymous retrievals, the current default, enter your e-mail address. The subdirectory will be /stage/anonymous/request_ID_number.

After locating your data on the archive host, transfer them across the Internet using FTP as described in a later section. Because the disk space available to each user within the data directory is limited, the files created for you are temporary and are deleted automatically after a few days. Please transfer your data promptly and delete your data from the staging area after the transfer has completed. When disk space is tight, we would appreciate notification that you have completed copying over your retrieved files so that we may delete them. (Send email to archive@stsci.edu.)

All non-HST/FUSE data retrieved through the MAST web search interface is directly downloaded to the user's system as a tar or zip file. The data are also all available via anonymous ftp (archive.stsci.edu, cd pub/mission/data). Users will need to know the desired dataset names when using this option. More information on MAST retrievals is provided in the MAST Chapter of this manual.

The Operations and Engineering Division (OED) at STScI is committed to providing outstanding and timely support to archive researchers. We provide assistance and advice on methods and strategies for finding information in the archives and provide a hotseat staff for researchers who have specific problems or questions about using the archive. Archive researchers who need extensive advice on search strategies or help analyzing their astronomical data can visit STScI.

The Operations and Engineering Division (OED) at STScI is responsible for the management, scientific and technical oversight, and operation of the STScI archives. OED staff also support astronomers who wish to use public data from the archives for their own research. To provide assistance for archive researchers, the OED staff includes archive specialists (with bachelor or masters level degrees in physics or astronomy) and archive scientists (Ph.D astronomers). The support provided by the OED includes:

• Answering specific questions about data in the archives and methods for retrieving those data.
• Maintaining a Web site for archive services.
• Providing advice on strategies for searching the archives.
• Responding to problems identified by users.
• Authorizing GOs and GTOs who wish to retrieve their own proprietary data.
• Writing DVDs and, for large retrievals, external hard disks for users.
• Providing support for users who visit STScI.

1.8.4 Questions and Comments

Occasionally, a retrieval fails because of a network timeout, disk space inadequacy, or other reasons. The staff at archive@stsci.edu are available for any questions about the request.