spacer link to MAST page spacer logo image spacer
 
link to STScI page


The Multimission Archive at STScI Newsletter


February 25, 2004 Space Telescope Science Institute Volume 14


The Multimission Archive at STScI (MAST) Newsletter disseminates information to users of the HST, FUSE, GALEX, IUE, Copernicus, EUVE, HUT, UIT, WUPPE, IMAPS, BEFS, TUES and VLA-FIRST data archives supported by MAST. Inquiries should be sent to archive@stsci.edu.


Index of Contents
Introducing the New Head of MAST/AB
MAST Hardware and Software Upgrades
New FTP Services at MAST
New tools at MAST
VizieR Catalog Cross-correlation and VOPlot
New additions to the Pointings Search Tool
GALEX Early Release Observations now available at MAST
MAST and the Virtual Observatory
MAST User Survey
MAST Users Group Meeting
High-Level Science Products at MAST


Introducing the New Head of MAST/AB

By Rick White

Greetings MAST users! I am the new head of the Archive Branch, taking over from Marc Postman who has moved on to the Community Missions Office. Many of you know me from my 20 years at STScI. I have served the Institute in many roles including High Speed Photometer instrument scientist, developer of image restoration and image compression algorithms, and head of the STSDAS/PyRAF software group. In many ways, though, the archive is the best match to my research interests. For the past 10 years most of my research has been based on the FIRST radio survey and the Sloan Digital Sky Survey. I have been doing multiwavelength astronomy for many years as well, with papers on topics from the radio to gamma rays.

These are stressful, uncertain times at STScI, but they are also exciting times for the archive. Advances in computer hardware capabilities and the Virtual Observatory software efforts promise to greatly improve both the power and usability of our archive, and we expect MAST to become an increasingly important tool for astronomical research over the new few years. I welcome your comments and suggestions about MAST -- please feel free to send email, or talk to me at AAS meetings.


MAST Hardware and Software Upgrades

By Faith Abney and Rachel Somerville

The placement of the Advanced Camera for Surveys on HST brought about wonderful new science, but the large size of ACS data files posed a new challenge to the MAST goal of rapid data delivery. To meet this challenge, MAST has implemented major hardware and software enhancements. These changes include the migration of HST and other MAST data from magneto-optical jukeboxes to a new 32 TB EMC `spinning disk' storage array, which is less subject to mechanical problems and which will provide more rapid delivery of requested files. We have also installed a new multi-CPU SunFire 15K server, with 68 CPUs and 272 GB of memory that can be dynamically allocated to be used by our development, test and operations systems. The SunFire now hosts the latest version of the Operation Pipeline Unified System (OPUS) software, greatly speeding up On The Fly Reprocessing (OTFR) and the Data Archiving and Distribution System (DADS) and allowing faster access to the HST/FUSE archive database. In addition to the performace improvement from the new hardware, it is also engineered to be fault-tolerant and highly available. This should reduce the frequency of hardware problems that had been increasing as our previous systems aged, contributing to long retrieval times.

We have also installed a redesigned version of the Data Archiving and Distribution System (DADS) software, which provides greater control over the request queue, and prevents bottlenecks from forming when the system encounters a problematic retrieval request.

We noticed a significant improvement in retrieval times in October, 2003, after the SunFire server was installed. Since the new version of the DADS software (DADS 10.2) was installed on December 2, 2003, retrieval times have dropped still further. In December 2003, HST and FUSE data retrieval requests frequently took under 2 hours to complete, and 50 percent completed in under 10 hours.

Histograms showing the number of hours needed to complete HST and FUSE data retreival requests
submitted in September 2003 and in January 2004.

In addition, DADS 10.2 also has several new options for users that have quickly become popular. For example, users can now request only files with certain extensions, or can request raw data only (by checking a button on StarView or on the www interface). An anonymous retrieval option is now available, so that it is no longer necessary to have a MAST account in order to retrieve non-proprietary HST or FUSE data (just enter 'anonymous' as your archive username and your email as your password). Users can now retrieve data with secure FTP, either by staging the data and pulling it over to their home machine, or by having the data automatically `pushed' to their home machine. To use SFTP, users must be running an SSH daemon on their system that supports sshv2. If not, the SFTP could time out frequently or the data transfer could be extremely slow.

We thank our users for their patience during the transition to the new system. We think you will agree that it has been well worth the wait!


New FTP Services at MAST

By Tim Kimball

MAST now has a new FTP server. The Covalent Enterprise FTP Server is a commercial product based on Apache. By using Covalent, we can bring many of the capabilities of a Web server to FTP.

The biggest change is that data staged from DADS is now protected. Previously, when you had data staged, it was accessible to you - and everyone else - through anonymous FTP. Now, the data will be staged in a directory that only you can see by logging into the FTP server with the same archive username and password that you use to retrieve data from DADS. Other users will not be able to see your data. While you can't store files in your directory, you can delete them after you're done downloading them, to keep your directory uncluttered.

Besides offering a little more privacy for retrievals, these protected staging directories will allow us to stage proprietary data. This new feature will become available after the next release of DADS late May and it will work for both HST and FUSE data.

We are also using Covalent to create areas on our server that let project teams exchange data with each other - but not the world - in preparation for publishing the data. Once the data are ready for publication we can open the area up to the public, even while maintaining restricted areas for the project teams.

With such a flexible server at our disposal, we are looking at other possibile ways to improve data transfer services in MAST and at STScI in general. More information about Enterprise FTP Server can be found on Covalent's web site (http://covalent.net).


New tools at MAST

By Randy Thompson, Rachel Somerville, and Karen Levay

We invite users to explore the new 'MAST Search Toolbox' at http://archive.stsci.edu/mast_search_toolbox.html. The toolbox contains several familiar tools, like the Scrapbook, Pointings Tables, Abstract Search, and Parallel search, as well as some new tools. We are constantly improving these tools and adding new ones. Here are some of the new features that we have added recently.

VizieR Catalog Cross-correlation and VOPlot

The new VizieR Catalog Search and Cross-correlation tool allows MAST users to search the entire set of 4,000 VizieR catalogs and cross correlate the results with any MAST mission. This web-based search tool accesses one of the several VizieR catalog servers (e.g. at CfA) as a web service and retrieves the search results in a machine-readable VOTable XML format.

Users can use the VizieR search form (http://archive.stsci.edu/vizier.php) to search for a particular catalog using catalog name, mission name, target name, wavelength range, author name or keywords (as in the VizieR search based at the Centre de Donnees astronomiques de Strasbourg (CDS)). A catalog of interest can be selected from the returned list. Users can then search within the catalog based on internal object parameters, and may cross-correlate the full catalog or the refined list of objects with any selected MAST mission. This provides a way for users to effectively search MAST by object class, where the object class is specified as a parameter in an object catalog (e.g., spectral type in the Hipparcos catalog). In the cross correlation mode, the search tool retrieves ra and dec values for each found VizieR catalog entry and performs a cone search using a specified search radius. Because this process is repeated for each VizieR catalog entry and each selected MAST mission, execution times can become rather long for large requests.

Users may also upload their own object catalog and cross-correlate it with MAST holdings (http://archive.stsci.edu/xcorr.php?form=fuf). Another recent improvement is that the WFPC2 Associations (see http://archive.stsci.edu/hst/wfpc2/) can now be accessed by the cross correlation search. Output from the cross-correlation search may now be requested in several different formats (including VOTable).

Another new feature is the ability to plot search results from any MAST mission or the VizieR catalog search with a new JAVA-based graphical display tool called VOPLot (http://vo.iucaa.ernet.in/~voi/voplot.htm). This new tool allows users to plot the search results from any numerical field returned in a search as a histogram, or as a scatter plot against any other field, as well as to display statistics on the results. VOPlot was developed as a part of the Virtual Observatory - India initiative by Persistent Systems (PSPL) and the Inter-University Centre for Astronomy and Astrophysics (IUCAA), in collaboration with CDS, and the European AVO project.

As a simple example, after using the VizieR search and selecting the Abell Catalog of Rich Clusters of Galaxies, voplot was used to plot magnitude vs redshift.

New additions to the Pointings Search Tool

The MAST `pointings' search (http://archive.stsci.edu/cgi-bin/point) allows users to determine which regions of the sky HST has observed, and to do sophisticated searches based on the number of observations, number of filters, time between observations, etc., for a given region. In addition to WFPC2, STIS (images), FOC, and NICMOS, the Advanced Camera for Surveys (ACS) and WFPC2 associations have now been added to the pointings database. Please see the webpage (http://archive.stsci.edu/cgi-bin/point) for details on how the pointings are defined for each different instrument.


GALEX Early Release Observations now available at MAST

By Myron Smith

The GALEX satellite (Galaxy Evolution Explorer) was launched on April 28, 2003, with the goal of mapping the full sky in the near-UV (1350-1750A) and far-UV (1750-2800A) in imaging mode and part of the sky in spectroscopic mode (resolution = 125-150). This project is being led by the Califonia Institute of Technology and is now in operation, surveying stars and galaxies out to redshift ~2". MAST is hosting GALEX data as its newest active mission.

On February 9, 2004, MAST opened its public website for the GALEX Early Release Observations. This opening coincided approximately with NASA's NRA for Cycle 1 of the GALEX Guest Investigator Program. The GALEX site may be found at http://galex.stsci.edu, or by going to the MAST home page and clicking on the GALEX mission page.

The MAST GALEX website, written using .NET technology, provides users with the capability of searching GALEX data in a variety of ways. For example, searches can be made on a region of the sky, a single object, or a class of objects using a familiar "MAST-style" or "SQL" form. The latter can be used to construct complex queries using the full set of database parameters, which may be consulted by the user. The results page includes direct images in the two bands and 1- and 2-dimensional spectra. The direct images are "active" and allow users to click on a neighboring object of the one initially selected. This generates a new results page centered on the newly selected object. As an aid to navigation, a tutorial is available. We invite interested users to explore the data from this exciting and productive mission. In the near future look for a cross-correlator tool that will enable searches of data from GALEX and other survey missions that use Virtual Observatory data formats.

GALEX observation of M31. See the GALEX Image Gallery for more information about this image and to see other pictures.


MAST and the Virtual Observatory

By Rachel Somerville

MAST is an important part of the emerging `Virtual Observatory'. We are continuing to implement standards and protocols as they are developed by the National Virtual Observatory (NVO) and the International Virtual Observatory Alliance (IVOA). For example, the MAST www interface can access information from other sites in the VOTable format, and can also return search results in VOTable. VOTable is a format standard, designed for astronomical applications by the NVO, which allows any kind of tabular data to be encoded in a standard machine-readable XML format. We use this format in communicating with the VizieR catalog database in the new VizieR search/cross-correlation tool.

VOTable is compatible with a growing number of newly-developed web-based tools, such as the VOPlot plotting tool that we have recently implemented at MAST (see above). We are in the process of making some of MAST's image data, particularly High Level Science Products, available through the Simple Image Access Prototype (SIAP) specification. Similarly, SIAP is just a standard for retrieving astronomical image data through a uniform interface. Both VOTable and SIAP files contain not only the actual data, but also any relevant metadata such as the World Coordinates, the bandpass, etc. Soon, popular HLSP such as the GOODS images and catalogs will be available via VOTable and SIAP at MAST. As similar standards for spectra become available, we will carry out a similar effort for MAST's spectral data holdings.

Check with our web page of MAST related VO activities to see what is new.


MAST User Survey

By Rachel Somerville

In September 2003, we conducted a user survey to help determine our users' priorities for future developments at MAST.

The response to the survey was excellent, with over 300 responses, and we thank all the users who took the time to send in the survey. A clear message from the survey was that decreasing retrieval times and downtime takes highest priority for many of our users. This is also our highest priority at MAST, and as described above, we are working hard to implement the hardware and software upgrades that will allow us to accomplish this goal.

Some other interesting highlights from the survey are:
  • 50 percent of users access MAST at least once a month
  • 61 percent felt it would be extremely useful to be able to search MAST holdings based on object class
  • 80 percent of our users prefer the WWW interface over StarView
  • the highest ranked ways to expand and improve MAST services were improved support for multi-wavelength data cross correlation and developing techniques to perform efficient queries on Tb scale object catalogs.

To browse the full results of the survey, go to http://archive.stsci.edu/s/survey.html. To see plots of survey results, see the Survey Results presentation made to the MAST Users Group.


MAST Users Group Meeting

By Karen Levay

The MAST Users Group (MUG) which plays an advisory role for the MAST archive, held their annual meeting on October 20, 2003. Present were committee members Thomas Ayres (Chair, Colorado), Luciana Bianchi (JHU), Aki Roberge (Carnegie Institution of Washington), Paula Szkody (Washington) and Ann Zabludoff (Arizona). The MUG's agenda included reports on ongoing archive activities, problems and solutions planned and already implemented. The presentation slides and the 2003 MUG report are available at http://archive.stsci.edu/mug/index.html.


High-Level Science Products at MAST

By Michael Corbin

This is a cutout from the GOODS Chandra Deep Field South field. Click on the image to see a larger section of the field
The MAST High-Level Science Products program is an increasingly important part of our activities. High-Level Science Products (HLSP) are fully reduced MAST data and ancillary products such as object catalogs and supplementary ground-based observations. HLSP currently available in MAST include the combined images and object catalogs from the Hubble Deep Field, as well as products from more recent programs including the HST portion of the Great Observatories Origins Deep Survey (GOODS) program. A listing of the current HLSP holdings can be found at http://archive.stsci.edu/prep_ds.html.

Most of the HLSP are located in the pub directory on archive.stsci.edu and are available via anonymous ftp. After connecting to the server, cd to /pub/hlsp. Some sets of HLSP (e.g. the Hubble Deep Field) are available directly from the /pub directory.

The DADS query results pages for these missions now also list any HLSP associated with a given dataset, e.g., if the dataset was used as a member of a combined image a link to the combined image is provided. We have created a separate search interface for HLSP, through which the user can search by several criteria including target name, project name (e.g. GOODS) or position. This interface can be found at http://archive.stsci.edu/hlsp/search.php. HLSP from a number of large and important programs will soon be added to our holdings. These include the Hubble Ultra Deep Field project (see http://www.stsci.edu/hst/udf), as well as the HST "COSMOS" Treasury program (see http://archive.stsci.edu/cgi-bin/proposal_search?mission=hst&id=9822). We will announce the availability of these and other HLSP on the MAST website when they have been delivered to us. We also welcome the contribution of HLSP derived from MAST missions. Our guidelines for such contribution are described at http://archive.stsci.edu/hlsp/hlsp_guidelines.html.


To Subscribe or Unsubscribe to the Newsletter

If you would like remove yourself from the mailing list send mail to archive_news-request@stsci.edu and put the single word unsubscribe in the BODY of the message. Information on the subject line will be ignored.

If you would like to subscribe to the mailing list to receive notification of future Archive Newsletters, send mail to archive_news-request@stsci.edu and put the single word subscribe in the BODY of the message. Information on the subject line will be ignored. You will be asked to confirm the subscription.