Appendix 2 HST Specific Information

A2.2 ASCII Catalogs

There are three ways to search for positional coincidences with observations in the HST catalog using StarView or MAST web. You can constrain the target RA and Dec, specifying a range of permissible values. This defines a search rectangle. You can also specify an RA, Dec, and search radius (i.e., a cone search). You can also specify a common name for the target, let SIMBAD or NED provide the target position, and use the default search radius.

Archival researchers may be interested in finding all (or most) HST observations of a given class of objects (e.g., observations of elliptical galaxies or Wolf Rayet stars). There are several ways to carry out such searches:

• Search on the target description.

• Search on proposal title, proposal keywords, or proposal abstract.

• Cross-correlate the contents of the HST catalog with a catalog or personally created list of sources on the basis of position.

Searches can be done on descriptions entered by the original observers during the Phase II (planning) process. These fields allow an archival researcher to search for HST observations of classes of objects. The target categories (aka Broad Category) appears on the Staview <Fixed Target> screen. See the StarView chapter for instruction on adding the target description (tak_keyword_test) from the target_keyword table. The MAST search form contains the target description and allows the user to add the broad category. The target description is the target category with additional descriptive words appended, so the target category is somewhat redundant.

To search for all observations of nova stars within our galaxy, use the <Fixed Target> screen, constrain "Target category" as star, and constrain "Target description" as *nova*. Note the use of the wild card delimiters (*), since the word "nova" will likely be embedded in a longer text. Alternatively, since the target description in StarView is the Phase II target category plus the Phase II target description - in that order - one could simply use the <Quick Search> and <General Search> screens and constrain the "Target description" field to *star*nova* or *star* & *nova*.

The VizieR/MAST or the NED/MAST Cross-correlation tools may be used to find classes of objects within the HST archive. There's also a File Upload Form option on the HST Search Form and the NED/MAST form.

To use the File Upload Form option on the HST Search Form, the user must have prepared a file, or downloaded a catalog, that contains RA and Dec, and/or target name and/or a data_id. The order of the information is not important, as long as it appears in one of the first 10 columns. The column delimiter must be one of Tab, |, comma or semi-colon. Clicking on the File Upload Form in the HST Search Form will bring up an HST Search Form with fields to provide the name of the file containing the list or catalog, the delimiter used in the file and the column(s) number(s) in the file that contains the coordinates or target name or data_id. The remainder of the form is the standard HST Search Form, which should be used to specify the search.

The VizieR/MAST Cross Correlation Search permits users to search for catalogs held by VizieR and then either search them internally or to use them as input for a MAST cross-mission search.

First search for a VizieR catalog. When searching for a catalog name, it is useful to search using a wild card (e.g. *Abell*). The result of the search is a list of the catalogs found at VizieR with two options to the left of each found catalog (S and CC).

• Click on the "S" to Search the catalog itself. This brings up a form with the same basic format as the mission searches. The top section permits entry of target name or coordinate criteria and the bottom section specifies output formats. The middle section gives you the option of qualifying the search using any of the fields present in the catalog. The field descriptions link brings up a page describing the column of the catalog as provided by VizieR.

• Click on the "CC" to cross-correlate the catalog with any of the MAST missions (except GALEX at this writing). This form has 4 sections. The top section is the usual Target Name/coordinate specification. The second section permits you to add qualifications concerning the catalog. Again use the Field Descriptions link to see the column/field descriptions from VizieR. The third section permits you to cross-correlate with data from specific instruments. The fourth section is the normal output selection criteria. The first 10 records found per catalog entry will be displayed. The total number of records found will be indicated.

The NED/MAST Cross Correlation Search permits users to search the NED catalog by object name, position, object category etc. and optionally cross-correlate results with MAST mission databases.

NED search results are retreived as a web service, and the results are either displayed directly, or the retrieved coordinates are used to display MAST mission observations that are within a specified search radius of the retrieved catalog entries.

While it is difficult to assess the quality of the data in a given HST exposure from the keyword information alone (i.e., without actually obtaining and analyzing the data), there are several keywords that are written to the HST catalog and displayed by StarView screens that can help you to get a feel for the quality of the data in a given dataset. Among these are the following (each of which are described in the following sections):

• The PDQ keywords and comments
• The exposure flag (expflag)
• The fine guidance system lock used to track the target (fgslock)
• Preview images
• Observation log files for most data taken after October 20, 1994

When interpreting the relevance of these parameters for a given observation, bear in mind that the importance of these parameters to data quality is relative. It depends both on the characteristics of the source being observed and the purpose to which you will put the data.

In addition to the complete science data set, the archive contains previews of all public data sets. The previews can be viewed with StarView or MAST web and thus allow the user to gain an idea of the quality of an image or spectrum by direct inspection. A discussion of the preview capability can be found in the MAST and StarView chapters.

In addition to the above methods of assessing data quality, for most data taken after October 20, 1994, you can also obtain observation logs for a given dataset. The observation log files contain detailed pointing information (including jitter information) as well as the information on the spacecraft position and attitude. This information is provided at high time resolution, and should be particularly useful in many cases for investigating problems that are revealed by an inspection of the data or data quality comments. The Observation Logs can be retrieved by clicking on the "Observation Log Files" box on the Retrieval Options Page.

The handling of data quality evaluation has changed several times since HST was launched in April, 1990. For the first few years, each science dataset was displayed and evaluated after it had been calibrated by the ground system (then PODPS, now OPUS). A keyword, QUALITY, was assigned to each dataset and a file, called the PDQ file, was produced to hold any comments the reviewer had. The PDQ file was archived as part of the dataset. The QUALITY keyword and 3 comments, called QUALCOMs, were placed in the HST catalog. The QUALITY and QUALCOMs can be viewed using StarView or MAST web.

At the time of the second servicing mission, manual assessment was replaced by an automated system that was used by OPUS staff to assess the procedural data quality of the data. The datasets were not displayed unless there was some indication that a problem had occurred either with the instrument, the FGSs or HST. PDQ files and QUALITY and QUALCOM keywords continued to be produced for all datasets. Albeit, most files and keywords were produced automatically and contain limited information on the quality of the data.

On July 1, 2002, the automated system used by OPUS personnel was replaced by a new software package, called PROMPT. Responsiblity for procedural data quality assessment had been removed from OPUS in March 2002. The PROMPT system continues to populate QUALITY and QUALCOM keywords, which continue to be placed in the HST catalog. However, PROMPT does not produce PDQ files. In addition, PROMPT primarily reports on failures, such as non-optimal guiding, loss of lock, and instrument or spacecraft safing. Comments on the actual data only occur, after the fact, if the PI reports a problem to the archive help desk.

In 2006, a project to regularize the QUALITY keyword and provide a numeric value for the quality was begun. A clean up of the existing QUALITY information is part of this project. As of this revision to the Archive Manual (2007), the QUALITY keywords have been revised. The clean up of the QUALITY information continues.

The PDQ quality keyword and comments can be displayed in StarView using the <General> screen (which displays the PDQ comments). When interpreting the relevance of the PDQ keyword and comments, bear in mind that:

• Data quality parameterization is inherently subjective
• The relevance of the keyword and comments depends on the nature of the target of the observations

Care must be taken to combine a knowledge of the characteristics of the target with the PDQ keyword and comments when evaluating data quality. For example, while an evaluation of no-source for an observation where the target was a bright star and the integration time sufficient that a clear detection was expected may well indicate poor data quality, the same evaluation for an observation where the target was a distant, faint cluster of galaxies (for which co-adding of many exposures is required) indicates little about the quality of these data.

A2.4.2 The PDQ Files

When it exists, the complete OPUS data quality report can be retrieved from the archive. Every PDQ file is archived with the same root name as the observation, a trailing character of 'c' and an extension of _pdq.fits (i.e., u6712709c_pdq.fits). In addition to the data quality keyword and comments, the complete PDQ file contains predicted as well as actual observation parameters extracted from the standard header and science header files. To obtain the PDQ files associated with particular datasets, be sure that the [Data Quality] box is selected on the <Retrieval Options> screen.

Observer comment (ocx) files may help an archive user determine the quality of an HST observation. These are ASCII text files prepared by OSS (later OPUS) personnel who, early in the mission, monitored the health of the telescope and science instruments, and provided observers with a real-time interface. Early ocx files contain updated mission information obtained at the time the observation was executed. Later ocx files primarily contain information related to spacecraft pointing adjustments made in support of Mode I Target Acquisitions. These files are not created for every observation executed. When they exist, the ocx files, like the PDQ files, can be retrieved by clicking the [Data Quality] box on the <Retrieval Options> screen.

Observation log files for most science datasets taken after October 20, 1994 have been produced in the pipeline and written to the HST archive. The observation log files contain a specialized set of pointing and engineering data associated with each science exposure (including the jitter information). The extent of this information has changed over the years, with more recent observations logs containing less instrument specific information. You can retrieve these log files by first marking the science dataset for retrieval and then selecting the observation log files option on the archive retrieval screen. It is not necessary to retrieve the science data; the observation logs may be retrieved independently. Remember, the observation log files are only available for those data taken after October 20, 1994.

Users who need the location of HST during their observations will find the information in the HST ephemeris file, also know as an ORB file or an orbit file. Each ephemeris file covers 3 days, with a resolution of 60 seconds.

The ephemeris files are archived and available to any user. First the user must locate the ephemeris file(s) of interest using StarView. Retrieval of files may be done via StarView or MASTWeb can be used.

This is a continuation of the above section on locating HST ephemeris files. It assumes the results screen from the location stage is available.

• In the Results portion of the data files screen, select the files to retrieve by clicking on the dataset name in the box on the left hand side of the screen, then clicking on the Mark or All button at the top of the screen. There are many ways to mark datasets of retrieval. Do not be afraid to experiment.

  A StarView retrieval screen will be displayed.

• In the StarView retrieval screen, confirm that all desired files are listed and have a check in the "Marked" column.

  Files may be added by returning to the results screen and selecting more dataset names.

• In the StarView retrieval screen, when all desired datasets are listed and marked, click on the Submit button at the bottom of the screen.

  The HST Retrieval Configurations Options screen is displayed.

• Fill in the fields, specifying either an archive account or anonymous (check the box)
  • Check the "Show override (Expert only)" box in select data type(s) for retrieval on the middle left hand side of the screen.

  • Enter POD in the Extensions box and check "Fetch only listed extensions." Both are at the bottom of the screen.

    The "Calibrated/OTFR" box should be automatically unchecked at this point. If it is not, uncheck it now.

  • On the right side of the screen, Choose a method of data delivery, select a method for data delivery. NB: Anonymous users may use FTP Delivery, but will be prompted to enter a username and password.

• Click the DONE button at the bottom of the screen

  A Submitting Request dialog box will be displayed. After the request has been delivered, that dialog box is dismissed and another dialog box, containing the Request ID, is displayed. Click OK to dismiss the Request ID dialog box.

The standard e-mails regarding submission and completion of the request will also be sent.

As archived, the HST ephemeris has a resolution of 1 minute of time. Some users need a finer resolution. Although we can not provide the HST ephemeris at a higher resolution than 1 minute, a technique does exist to interpolate between the discrete points.

Start with the HST state vector from the data headers and do a linear interpolation between HST ephemeris points.

The HST state vector, that is the X, Y and Z components of HST's position and velocity, in the header should be the state vector at the start of the exposure. For HST data the information is in the shf file for the exposures.

For example, proposal 10500 produced products named J9CX01010, J9CX01020, J9CX02010 and J9CX02020. The association file, J9CX01010_asn.fits, shows exposures J9CX01CEQ, J9CX01CHQ, J9CX01CKQ, J9CX01CNQ, J9CX01CQQ, J9CX01CTQ, J9CX01CZQ and J9CX01D5Q comprise the product. It is the shf file (e.g., J9CX01CQQ_shf.fits) for these exposures that contain the HST state vectors at the start of each exposure that makes up J9CX01010. Depending on what was requested when the data were retrieved, the shf file may or may not be present. Request uncalibrated data to get the individual member exposures, which will include their shf files.

Continuing with the same dataset, there will be 8 state vectors, one from each exposure that comprises J9CX01010. Since the data cover about 43 minutes of time, the ephemeris file should yield about 45 data points, with the first point at a time before the start of the first exposure and the last point from a time after the last exposure. A curve can be fit to the data.

The archived version of the HST ephemeris files are named to follow the engineering file conventions (ICD-19), with a first character of 'P' and an encoded time for the start time of the file. The syntax is PYMDhhmm, where P indicates an ephemeris file, Y is the year code, M is the month code, D is the day code, hh is the starting hour and mm is the starting minute. The codes start at 1 and increment through the single digits, then start into the alphabet. For Y, year 1 is 1980. For M, January = 1, December = C. For D, the first = 1, the tenth = A, the 31st = V. For example, dataset 'PNBP0000R' starts in year 'N' = 1980 + 23, month 'B' = November, day 'P' = 25, hh:mm = '0000'.