SIDARCHIVE Search Help

Use the SIDARCHIVE search form to search the JWST I&T data archive by test Name, target name, instrument, etc. By submitting a search query, you can then mark data for retrieval from the search results page.

Local File Name

The name of a local file containing a table or list of parameters such as coordinates, targets names, or data IDs to be uploaded to the server and used to query the database. The file must be an ASCII text file with either one entry per line (e.g., a target name, a Data ID, or a set of coordinates), or a table separated with one of the allowed delimiters, with targets, Data IDs, or RA and Dec values in the designated columns. Note in either case, only one entry per line is extracted. If, for example, a comma-separated list of target names is downloaded in one line, only the value in the designated column will be used.

The search script will perform a database search for each extracted Data ID, target, or set of coordinates, contained in the uploaded list. Target names will be resolved to coordinates. If a resolver error occurs with target names, the search will abort and the compiled results displayed. Coordinates may be given in several formats including sexigesimal and decimal degrees. If the output is requested in CSV or Excel spreadsheet format, a blank line will be used to distinguish the results of one target search from another. VOTable format incorporates separate RESOURCE tags for each database query.

Use the other form entries to specify field delimiters, RA, DEC, or Target column numbers (when the file contains a table of values), and file contents (target names or coordinates). The browse button allows users to seach local directories to locate files.

Warning: Since uploading long lists can take a while to run, uploaded files are limited to 10,000 entries.

File Contents

Specifies contents of the local file to be uploaded (e.g., coordinates, target names or Data ID's). By default, most MAST search forms assume coordinates.

Note that each target, or set of coordinates specified in the uploaded file, is treated as a separate query. The results are displayed as each query is run. If however Data ID's are entered (up to 10,000 are allowed), it is processed as a single query (e.g., select * where data_id in (did1,did2,...).

To avoid displaying column names after each target name or coordinate query, consider using the comma-separated values output format. Also keep in mind that not all target names are resolvable. For example, SIMBAD currently does not reolve most 2MASS IDs.

RA/Target/Data ID Column Number

The column number containing either the Data ID, Right Ascension (decimal degrees or sexagesimal), or target name (depending on how file_contents is set). The default is to assume the first column is to be used.

Dec Column Number

For forms allowing coordinates as input, the column number containing the Declnation (decimal degrees or sexagesimal). The default is to assume the second column contains the DEC value. If the file contents is set to Target Name or any Data ID, this parameter is ignored.

Column Delimiter

The character used to delimit table entries in the uploaded file. Allowed values include tabs( ), commas(,), vertical bar (|), or semi-colons(;). The default is to assume tabs are the delimiters. If the file contains only a single column, use tabs as the delimiter.

Retrieval Type

Package type; either "file" or "package". File refers to individual FITS files while package refers to a tar file of various data sets. The database table is comprised primarily of values extracted from either FITS keywords (file type) or a readme file (package type). Therefore some entries may only be defined for one or the other retrieval type.

Test Name

Title of test as specified in the test plan. Note an "*" can be used as a wildcard character in any string field. For example, the test name can be given a value nir* to find NIRCAM tests. Case is ignored.

Test Number

This is defined as the number of test as specified in test plan, however it is stored as a string and frequently has the same value as the test name.

Target Name

Target name as dertermined by FITSWriter although the tester is responsible for updating its value (not defined for packages). Default is usually "unknown".

Instrument

Identifier for instrument used to acquire test data.

Data Mode

Reference number used to specify ground data processing requirements for this data. Note values are stored as a string!

Dates

When specifying a date, you need to include a date and an optional time. If time is omitted then then midnight (00:00:00) is assumed. This means that a specification like "July 1 1994" will look for observations done on July 1 1994 00:00:00, not for observations done for the entire day July 1 1994. To search for observations for an entire day use the range option described below.

Note also that when entering a date with the month in numerical format, the American ordering is used; i.e., the first number is the month.

The date can have any of the following formats (the month name can be spelled out or abbreviated to three letters; case is not significant):

      Jul 15 1994
      Jul 1994 15
      15 Jul 1994
      1994 Jul 15
      1994 15 Jul
      7/15/1994  <--- Note order of month/day of month/year for numeric formats
      7-15-1994
      7.15.1994

If the day is omitted, the first day of the month is assumed. This means that a specification like "July 1994" will look for observations done on July 1 1994 00:00:00, not for observations done during July 1994. Note also that when entering a date with the month in numerical format, the American ordering is used; i.e., the first number is the month.

If a time is omitted, then midnight (00:00:00) is assumed. Otherwise, you can specify a time in any of these formats:

      14:30
      14:30:20
      14:30:20:999
      14:30:20.9
      4am
      4 PM
      04:30:20 AM

To search for observations before a given date, use <, and for observations after a given date, use >. For example,

      > Jul 15 1994
      < Jul 15 1994

You can use the .. operator to search on a range of dates:

     Jul 1 1994 .. Aug 1 1995

This operator is inclusive on the first date and exclusive on the second.

Finally, you can search on a list of dates or date ranges. For example,

     Jul 1 1994 .. Jul 3 1994,
     Dec 1 1995 .. Dec 6 1995

will search for observations done within either one of these date ranges.

Exposure Time

The total exposure times in seconds. Currently exposure time is the only numerical field in the SIDARCHIVE table. As such, you can use numerical operators in your query such as ">" , "<", or perform range searches using "..". For example, to find exposure times between 1 and 10 seconds, enter: "1 .. 10". To find exposures > 100 sec, enter "> 100".

User Specified Field n

You may use these form elements to search on any column(s) in the mission table. First, select the field you wish to search from the pulldown menu under the "User-specified field n" heading. Then, type in the qualification in the corresponding "Field Descriptions" box. Clicking on the "Field Descriptions" link, will display information on the allowed fields including the allowed range of values.

As an example, a Kepler user might select "E(B-V)" from the pulldown menu and enter "< 0.5" in the "Field Descriptions" box.

NOTE only fields which are not already included on the search form should be selected. Specifying search criteria for a field that is listed in BOTH the form and in the User Specified field, may cause either the query to fail or return unexpected results.

Output Columns

This form element allows you to choose the columns to be displayed and their order, for the search results. A set of columns that are commonly requested has been chosen as a default.

You remove output columns by highlighting the column to be removed and then clicking on the remove button to the right of the output columns list.

You may determine the order of columns by highlighting a column and then clicking the up or down buttons to the right of the list of chosen output columns.

You may also add more columns to the list, either singly or all at once. To add individual columns, select the desired column from the pull down menu beneath the list of chosen output columns, then click the add button. The column will be added to the bottom of the output column list. To add all the available columns, click the "add all" button.

Note the output column form element has its own reset button to restore the list of output columns to the values initially displayed when the page was drawn. The reset button at the top of the form is used to reset the other form elements. Clicking the "clear form" button will restore the original defaults to all isections of the form.

One warning, the amount of memory required for a search is proportional to the number of columns requested. For large requests, users may want to reduce the number of output columns to the minimum required.

Suppress Null Result Message

By default, when using "file upload mode", the message "No Records Found Matching Query" (or for non-html output, "no rows found") is displayed for each entry with no search results. Clicking this button will prevent these messages from appearing which may be useful for reducing output from large search results.

Verb

Verb is an integer parameter used by the VO community for specifying the amount of output returned for a given search request. It is only available when retrieving data as a web service but works with every MAST service. Setting verb=3 in a search request is equivalent to specifying "add all" from a search form; it will return all the available columns in the output not just the standard default fields. Currently setting verb to any other value has no effect.

Sort output by:

Choose how you want the output rows sorted. You can select up to three fields to sort on. The rows will be sorted in the order of the first sort field; if two rows have the same sort field, they will be sorted in order of the second sort field, and so on. Default sort fields may be listed, but any field from the pulldown list can be used. Specifying multiple sort fields may increase executon time. If you prefer no sorting, you can specify "null" for all 3 fields. This may speed up the query, but results will be displayed in the order in which they were originally stored in the database table.

For each field, you can select that the rows be sorted in reverse order on that field by selecting the reverse checkbox. For example, you can sort the rows with the most recent observations first by selecting Observation Date for the first sort field and selecting the reverse checkbox next to it.

One word of caution: the selected sort field can change the search results when the query finds more rows than are displayed ( i.e, when the number of found rows exceeds the value of "maximum records"). For example, for a search on a particular coordinate that finds 5,000 entries, if the search is sorted on exposure time and 1,001 rows are to be displayed (the default), then the 1,001 shortest exposures from the 5,000 found entries will be displayed which may not include the entries closest to the desired position. (It is a good idea to always sort on "ang_sep" for target or coordinate searches).

Display Coordinates

Specifies the format for displaying the primary equatorial (i.e., RA and Dec) coordinates. The options include:

Sexagesimal - The default format with Right Ascension specified as hh mm ss.sss and Declination as +/-dd mm ss.ss ( e.g., RA = 12 46 11.091, Dec = -00 30 12.08). Note an extra digit was added as of June, 2012.
Degrees - Decimal degrees for both RA and Dec with 7 significant figures to the right of the decimal point (e.g., RA=191.5461912, Dec=-0.5033333). Note 2 extra digits were added as of June, 2012. or
Hours - Decimal hours for RA and decimal degrees for Dec in same format as for decimal degrees (e.g., RA=12.7697512, Dec=-0.5033333). Note decimal hours = decimal degrees/15.0. (Note 2 extra digits were added as of June, 2012.)

Any other coordinate fields contained in the searched mission catalog will be displayed in their original format.

Maximum Records

This value specifies the maximum number of rows returned in a single query. For the standard mission search forms, the current default is 1,001, but values from 1 to 25,001 are allowed. For the file upload forms in which multiple targets, data ID's, or coordinates can be specified, the default is set to 20 rows per file entry, with allowed values ranging from 1 to 5,000.

Note when displayed as HTML, the latest search scripts will display 100 records per page. Links to the additional pages are shown on the results page. This paging feature however does require javascript to be turned on.

Users should be cautioned about retrieving a large number of records (i.e., > 10,000 - 15,000) in HTML format. This can cause memory problems for the browser (particulrly Safari) and prevent javascript commands from functioning. It may also cause the browser to freeze and require restarting. Using the output format options which download results in a file can reduce the problem.

Another option for large requests is to use "Casjobs". Casjobs requires requesting a user name and password, and submitting queries in SQL, but it allows users to submit large search results and save them online. For Kepler, the link ito Casjobs is http://mastweb.stsci.edu/kplrcasjobs/. For other missions, check the Search_Retrieve page in the left gutter.

When queries are submitted as a web service, the default number of rows returned is 2,000, but any value is allowed when max_rows is specified as a query parameter. (See the MAST Web services page for more information.)

Records per page

This parameter controls the number of records displayed per web page. By default, 100 rows or records are displayed per HTML page. Therefore if 400 records are returned, links to 4 pages will be displayed at the top and bottom of the results table. The paging feature however uses javascript, so if javascript is turned off, paging won't work and only the rows shown on the first page can be displayed. This is one reason why increasing the default value may be helpful. The current limit is 1000 rows per page which is recommended for large requests. Note this value is ignored when output formats other than HTML are selected.

Make Rows Distinct

Selecting this checkbox will restrict the display to only rows in which every output column value for a given row is unique. This option is primarily useful when only a small number of columns are displayed (i.e., using the "remove" button to remove default output columns) and when the selected columns have duplicate values. Including columns which already have unique values (e.g. Kepler ID or Data ID) will make the "Distinct" option ineffective.

As an example, a IUE or FUSE user might want to create a list of unique target names for a specific object class/category. He or she would specify the desired object class, select only "object Class" and "target name" for the output columns, click the "Make Rows Distinct" button, then click "Search".

Note that for some missions, columns such as RA, Dec and Magnitude were frequently defined by the observer and often have different values for the same target. If columns such as these are chosen as output columns, there will often be more than one row listed per object name.

Output Format

You may choose any of the following formats for displaying/storing search results. If you are using a browser (e.g., Firefox, etc) to submit a query from a MAST web form, the formats labelled "File: ..." offer a way to download results directly to your local computer. Choose any of the other format options if you want to display the results in the browser, or if you are submitting the request from a program (see MAST web services).

HTML_Table (default) - results returned as a standard HTML table including various links for retrieving data, displaying previews, literature references, plotting spectra, etc.
File: Excel_Spreadsheet - results are stored as an Excel spreadsheet file. (Note: assumes users computer/browser provides support for Excel-format files). The default file name when downloaded is "mission"_search.xls where "mission" is the mission name (e.g., fuse_search.xls).
VOTable - an XML format adopted by the Virtual Observatory (VO) project and displayed in the user's browser. Note coordinates in VOTable format are always in decimal degrees rather than sexagesimal format. For searches returning results from more than one mission and/or target, multiple "resource" tags are created. Searches with a radius of 0 will return a VOTable file listing the output fields for that particular mission/catalog. (For more information on the XML file format, see VOTable documentation.)
Comma-separated values - a simple ASCII array containing column headings followed by rows of comma-separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries.
Space-separated values - a simple ASCII array containing column headings followed by rows of space-separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries. Note users may want to select coordinates in decimal rather than sexagesimal format to maintain the correspondence between column headings and entries although some column headings may contain blanks as well. (Semi-colons, pipes, or commas are probably safer delimiters.)
IRAF Space-separated values with INDEFs - Like the space-separated format above except empty fields are replaced with the string "INDEF". This format is useful for IRAF-compatibility.
Semi-colon separated values - a simple ASCII array containing column headings followed by rows of semi-colon separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries.
Tab-separated values - a simple ASCII array containing tabs for delimiters. May be useful for ingesting into Excel spreadsheets.
Pipe-separated values - a simple ASCII array containing column headings followed by rows of pipe or vertical bar separated values. (Note: not offered in file upload mode.)
Json format - Javascript Object Notation (json) is a simple machine and human-readable, name/value ASCII format supported by many programming languages. (Note: not offered in file upload mode.)
File: comma-separated values - a simple ASCII text file containing column headings followed by rows of comma-separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries. Rather than being displayed in the browser, the results are directly downloaded to the users computer using the file name "mission"_search.txt where "mission" is the mission name (e.g., hst_search.txt). Depending on the browser settings, the user may be prompted for a file location before the file is downloaded.
File: Space-separated values - a simple ASCII text file containing column headings followed by rows of space-separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries. Rather than being displayed in the browser, the results are directly downloaded to the users computer using the file name "mission"_search.txt where "mission" is the mission name (e.g., hst_search.txt). Depending on the browser settings, the user may be prompted for a file location before the file is downloaded.
File: IRAF Space-separated values with INDEFs - Like the space-separated format above except empty fields are replaced with the string "INDEF". This file format is useful for IRAF-compatibility.
File: Semicolon-separated values - a simple ASCII text file containing column headings followed by rows of semicolon-separated values. In file upload mode, a blank line is inserted between the search results to separate multiple target queries. Rather than being displayed in the browser, the results are directly downloaded to the users computer using the file name "mission"_search.txt where "mission" is the mission name (e.g., hst_search.txt). Depending on the browser settings, the user may be prompted for a file location before the file is downloaded.
File: Tab-separated values - a simple ASCII file containing tabs for delimiters. May be useful for ingesting into Excel spreadsheets.
File: Pipe-separated values - a simple ASCII text file containing column headings followed by rows of pipe-separated values. (Note: not offered in file upload mode.) Rather than being displayed in the browser, the results are directly downloaded to the users computer using the file name "mission"_search.txt where "mission" is the mission name (e.g., hst_search.txt). Depending on the browser settings, the user may be prompted for a file location before the file is downloaded.
File: Json format - Javascript Object Notation (json) is a simple machine and human-readable, name/value ASCII format supported by many programming languages. Rather than being displayed in the browser, the results are directly downloaded to the users computer using the file name "mission"_search.txt where "mission" is the mission name (e.g., hst_search.txt). Depending on the browser settings, the user may be prompted for a file location before the file is downloaded.
File: WGET Commands - This option is only available for certain missions. If selected, a shell script file is output which the user can execute from his own computer to download all the selected light curves or extracted spectral files with one command. The shell script file uses the "WGET" program which is available for most operating systems. Note for Kepler, two WGET options are available: "File: WGET LC commands" will create a script for downloading available light curves, "File: WGET TPF commands" will create a script for downloading target pixel files. For Swift, the WGET option will create a script for downloading the Sky coordinate images.

Note: If no entries are found for an entry a "no entries found" message is written in the selected format and the program continues. In all cases, error conditions will cause the database search to abort.