spacer link to MAST page spacer logo image spacer
 
link to STScI page


Please take the MAST Survey.

Batch retrievals of HST data will be unavailable Thursday, October 30 for a period between 9 and 11 AM EDT . Access to HSTOnline data will available.

Large Searches and Requests

As the archive has expanded, some users need to perform searches and make requests for large amounts of HST data. We present here some information to explain the best methods and some additional guidelines for ensuring a smooth and complete finish to your large requests.

You should be aware that while DADS can handle fairly large requests, there are some limitations. These limits are discussed in the request section below. The limits may influence how you wish to search and request data. You may wish to make many small searches and requests rather than one or two extremely large searches.

Searches

  • Using web interface

    The HST web based search interface provides a means to search the HST archive. Currently, the page limits the number of rows to be returned. The default is 100 rows, but the limit can be increased to a maximum of 1500 rows. This limit is imposed due to the length of time needed to render the search results in HTML. Even 1500 rows will take some time to render in HTML. However, if you wish to make a number of searches and requests, the web interface should serve you well. You may wish to perform the same basic search, but limit the results by including a filter and/or observation time limit.

  • Using web service (HTTP GET request)

    You may use HTTP GET requests to search as well as the web form-based interface. The GET request allows the search parameters to be included in a URL in your browser. As such, they can also be called from within programs to automate data searches. The results can be returned in a variety of formats including VOTable XML format, excel spreadsheet, and comma-separated values which can simplify ingesting results into user-written programs. In addition, submitting GET requests can bypass restrictions currently placed on the web search forms (e.g., restrictions on the max_records value).

    In general, mission searches are specified in the form:
    http://archive.stsci.edu/[data set]/search.php?action=Search¶ms

    In general, any parameter listed on the HTML search form can be specified in a GET request. If you are parsing the results, it would be easier and probably faster, to request the search results in CSV or VOTable format using the output format parameter.

    Below is an example of a search for ACS observations taken with the F814W filter within the specified radius (in arcminutes), centered on a given RA and Dec. In this example, the RA and Dec are in decimal degrees but other standard formats may also be used. Note that for some instruments (including ACS) the option to use more than one filter exists. So if you are looking for a specific filter, surround the filter name with wild cards. The example uses max_records parameter to specify that up to 5000 records be displayed. In this example the default output columns will be displayed. The output format specified is CSV. If you expect more than 1500 rows to be displayed, we do NOT recommend specifying the HTML format. The time to render such a long web page is prohibitive. The other output choices are VOTable and EXCEL.

    http://archive.stsci.edu/hst/search.php?RA=36.821482&DEC=-40.933062&radius=30
    &sci_spec_1234=*F814W*&sci_instrume=ACS&max_records=5000&outputformat=CSV&action=Search

    You may specify any parameter that can be used in the web form in the URL. In the example above the instrument and filter fields are shown as an example. The search parameters that you may use are documented in a web page. This page is also available from the web search form by clicking on the Field Descriptions form heading. To specify a parameter, use the column name found in the first column of the the table. The page also lists valid values and ranges that may be useful as you formulate your search.

    You may choose to specify specific columns as output using the selectedColumnsCsv format. Use the same parameters that are used to specify the output columns. Separate your columns with commas. In the example below, the dataset name, ra and dec are displayed as output.

    http://archive.stsci.edu/hst/search.php?RA=36.821482&DEC=-40.933062&radius=30
    &sci_spec_1234=*F814W*&sci_instrume=ACS&max_records=5000
    &selectedColumnsCsv=sci_data_set_name,sci_ra,sci_dec&outputformat=CSV&action=Search

    If you are using a script to run GET requests, please make sure that you have only a few searches running at a time. Many searches tend to overrun the capacity of our current webserver.

    There are a number of MAST webservices that may be useful. They are described more fully on the mast_services web page.

  • Using the "Pointings" search The MAST Pointings search is an alternative way to search for HST image data. We have assembled a searchable data table that allows users to look for sky regions (or pointings) which have been observed N times or more, observed with 2 or more filters, or have been observed more than twice with a time separation of more than (or less than) N days. An exposure is defined to be the file created after the shutter closes. The total number of exposures for each of the defined bands within a defined pointing were tallied.

    n example of the type of query these tables will allow is: How many high galactic latitude observations taken with WFPC2 exist [for RA between 9 - 18 hours in the northern hemisphere (dec > -20) observable from the northern hemisphere in the early spring] where there have been at least two observations, and where at least one of those observerations was in the I-band, and at least one of those observations was in any other filter?

    Choose the following options:

    Galactic Latitude Above & below plane +/- > 20.0 degrees
    Exposures in Band I >0
    Number of Unique Bands >1
    Total Number of Exposures >1
    RA 09..18
    Dec >-20.0

    You will get a page listing the information found for the pointings or sky regions that meet the criteria you specified. At the top of the page is a summary of all the pointings. The numbers on the clickable buttons indicate that the number of observations included in the pointing for that bandpass. If you click on the buttons, you will get a list of the observations included in that bandpass/pointing. You can choose to retrieve data from the page listing the observations, but please obeserve the guidelines for size of a submitted observation.

    The pointings search can also be run as a Web service (GET request).

  • Using Starview Starview is a search option that can support queries that need to sample parts of the cataloged metadata other than that found on the web pages such as calibration files or instrument specific settings. Starview is a Java application that can be downloaded from http://starview.stsci.edu/html/. Click on the Help menu in the upper right corner and select Start Help Browser for more help. If you wish to use the aperture overlay functionality download the Astronomer's Proposal Tool (APT) and use the StarView bundled with it. Some online help can be found at:
    http://starview.stsci.edu/html/HowTos and http://starview.stsci.edu/javahelp/FAQ/StarViewFAQ.htm

    StarView has a problem returning large (e.g. thousands) of records. There is a problem database connection from Java that occasionally hangs the query. StarView can also run out of allocated memory for very large returns as well. Because of these limitations Starview is probably best used in a support role to examine information held in tables that are not accessible from the web interfaces.

    StarView also has an interface to the pointings table.

Requests