What’s New at MAST: Updated Archive Search Interfaces and New HST Data Products

The MAST Portal and HST archive teams have released significant software and data product updates.

Lee Quick Jonathan Hargis

This issue of the MAST Newsletter highlights several of the recently released archive projects by our STScI staff and scientists. These projects feature work by nearly everyone on the MAST team: project managers, software developers and testers, database engineers, archive scientists and analysts, and STScI astronomers. These project include:

The Hubble Source Catalog Version 2:

  • Four additional years of ACS source lists (i.e., through June 9, 2015). All ACS source lists go deeper than in version 1.
  • One additional year of WFC3 source lists (i.e., through June 9, 2015).
  • Cross-matching between HSC sources and spectroscopic COS, FOS, GHRS and ACS grim observations.
  • MAST Classic search forms are updated with the new HSC version 2, including flag values for spectral cross matches.
  • CasJobs is now updated to include HSC version 2
MAST Discovery Portal 2.7:

  • New Advanced Search interface for MAST observation searching
  • Hubble Source Catalog spectra results page and cross-matching available
  • SExtractor magauto values now available
  • Maximum number of sources displayed has increased from 10,000 to 50,000
The Hubble Legacy Archive 9.1:

  • Almost all the ACS and WFC3 source lists now have higher quality astrometry and photometry, especially for SExtractor catalogs in crowded fields.
  • The interactive display advanced HSC controls can be used to show both versions of the Hubble Source Catalog.
  • The Gaia DR1 catalog is available for overlay in the display.
  • New controls can change the catalog overlay region to load sources from only a small part of the image. This greatly improves performance in large images and crowded regions.

Any questions or comments about these projects can be directed to archive@stsci.edu.

Introducing the Advanced Search functionality for the MAST Discovery Portal

The MAST archive can now be explored using a wide range of observational metadata, including non-positional searches.

Geoff Wallace

Figure 1: Location of the Advanced Search hyperlink on the MAST Discovery Portal webpage.
A new feature in version 2.7 of the MAST Data Discovery Portal is the advanced search for MAST observations, available as a hyperlink below the search textbox at the top of the page (see Figure 1). Opening this window allows you to browse and filter on the values for all columns for every observation in the database. The primary advantage to searching via this feature is that it allows a user to apply non-positional criteria to a query, while optionally still also performing a basic cone search. For example, one might want to find all HST WFC3/IR observations that are particularly deep. This can be accomplished by selecting the HST WFC3/IR value in the “Instrument” filter panel, and adjusting the lower bound of the “Exposure Time” panel to be a large value, say 10000 seconds. This returns just over 150 observations from all over the sky. The resulting observations may then be loaded into the browser using the button in the top-left of this window (see Figure 2).
Figure 2: Buttons to load the Advanced Search results into the MAST Portal grid view (left) and download VO/CSV table of search results (right).

The advanced search loads a number of search filters by default, including the data product type, mission, instrument, and waveband. These filters can be removed by clicking appropriate red dash button, and additional filters can be added by selecting the green plus buttons (See Figure 3). Typing directly into the “Filter columns” text box, one can quickly search through the various filters and toggle on or off as appropriate.

Figure 3: Advanced search filters can be quickly added and removed using the red dash (remove filter) and green plus sign (add filter).
The advanced search includes the ability to autocomplete column names and search terms, similar to how Google works. This is particularly helpful for the individual filter panels such as “Filters” or “Target Classification” which contain a large number of categories. For example, typing “galax” into the box in the Target Classification filter panel, a scrollable list of all the types of galaxies classified by PIs pops-up (“GALAXY”, “CLUSTER OF GALAXIES”, “HIGH REDSHIFT GALAXY”, etc.). Clicking or highlighting one and hitting enter toggles the corresponding checkbox and updates.

Lastly, the advanced search allows you to download very large CSVs or VO Tables directly without trying to load the results in the browser. When there are fewer than half a million observations available, use the second button in the upper left-hand corner and choose a format to download (Figure 2). For example, one can download the metadata for every observation ever taken by SWIFT in about a minute and use desktop applications to do some heavier lifting than a browser can perform. The advanced search feature is currently limited to MAST observations, but we expect to extend this functionality to larger collections such as the Hubble Source Catalog soon. For a demo of advanced search in action or many of our other features and tools, please visit our YouTube channel at https://www.youtube.com/user/STScIMAST. Questions about the advanced search can be sent to archive@stsci.edu or posted on the MAST Forum.

New Releases of the Hubble Source Catalog (v2.0) and the Hubble Legacy Archive (9.1)

New and updated HST catalogs and images are now available through the MAST Portal, CasJobs, and web search forms

Lee Quick Brad Whitmore

Version 2.0 of the Hubble Source Catalog (HSCv2) was released on September 29, 2016. The primary improvements are inclusion of four additional years of ACS source lists from the Hubble Legacy Archive and spectroscopic cross matching with COS, FOS, GHRS, and ACS grism observations. An additional year of WFC3 data is also included. The number of sources roughly tripled, with about 300 million detections now available in the HSC. More details can be found in the HSC FAQ at http://archive.stsci.edu/hst/hsc/help/HSC_faq.html. The journal article by Whitmore et al. (2016) describes how the HSC was constructed and the quality of the data.

Figure 1: Examples of spectroscopic cross match results from HSC use case #9.

We invite you to try one of the use cases (found here), especially use cases #8 and #9. These demonstrate the use of the new spectroscopic cross matching. Figure 1 shows part of Use Case #9.

New releases of the Hubble Legacy Archive (HLA) and MAST Portal came out on the same day, primarily in support of HSCv2. Almost all the ACS and WFC3 source lists have been replaced. The new source lists have higher quality astrometry and photometry, especially for SExtractor catalogs in crowded fields where the parameters that were used previously led to errors in the segmentation map. The HLA interactive display advanced HSC controls can be used to show either the new HSCv2 catalog or the previous HSCv1 catalog. The Gaia DR1 catalog is also available for overlay in the display. For the MAST Portal, the main changes were the addition of new pages supporting the spectroscopic cross matching with the HSC, the availability of HSC/SExtrator magauto values, and the increase in the number of sources that can be displayed from 10,000 to 50,000. A new Advanced Search interface was also released and is described here.

What We Do at MAST: Archive Scientists

STScI is hiring three new Archive Scientists. Learn more about this opportunity to work with the MAST team.

Scott Fleming

Archive Scientists at MAST serve as liaisons between many different groups within the field of astronomy. Internal to STScI, we bridge the gap between the technical staff such as developers or engineers and the astronomy staff. We support active missions by coordinating between the MAST team and the mission team, whether based at STScI or at other institutions. We also engage with the astronomical community that relies on the MAST archive to conduct their research and teaching projects. STScI currently has an open job advertisement for two Archive Scientists (see AAS job posting here), and a third position will be advertised soon.

When working with software developers and engineers at MAST, we assist with the creation of search tools, preview software, website design/content, and database updates. Although we have PhDs in astronomy or a closely related field, we also have good technical skills -- programming, database, or web development -- allowing us to “speak” both languages. One major component to our jobs is to gather what new capabilities the astronomical community needs, and assist the technical staff in implementing them. We work closely with the technical staff to plan new tools and interfaces to search the archive.

For active missions like Hubble or K2, we work closely with the mission team to ensure new data are archived properly within MAST. This includes making changes to the MAST archive itself as data products evolve due to different pipeline versions, or as new types of observations are conducted. We also answer help desk questions about the data products. For missions that are under preparation for launch, like TESS and JWST, we work with the mission teams to prepare for the data that will come from these spacecraft as they design their pipelines, data products, and observing capabilities. We also develop major new data products for our missions, including for example the Hubble Source Catalog and the GALEX gPhoton database.

Finally, we work with the astronomical community to not only advertise what MAST offers now, but to listen to what tools and data products are needed to enable archival science projects. We attend professional conferences, work with astronomers to accept High Level Science Products they create for archiving at MAST, and stay informed on the latest research papers. We are also active researchers ourselves, publishing papers that often use data archived at MAST. Archive Scientist positions include the option to buy back up to 50% of your time for personal research, through the use of any external funding or grants you may receive.

The job requires a diverse skill set, from both the astronomy and technical side, as the day-to-day tasks are varied, and you work on all sub-fields within astronomy. It is an exciting position where flexibility, breadth of knowledge, and the ability to conduct multiple projects concurrently are rewarded.

Inside the Archive: Finding Moving Targets with Portal Advanced Search

Finding observations of solar system objects is now easier than ever using the Advanced Search functionality of the MAST Portal

Jonathan Hargis

Searches for MAST archival data of moving targets – typically solar system objects such as planets, comets, and asteroids – have been greatly simplified by the release of the Advanced Search functionality of the MAST Data Discovery Portal. Instead of searching on fixed sky positions (which clearly fails to find moving targets), the MAST Portal allows users to search on other observational metadata to find these objects.

Figure 1: Location of the Advanced Search hyperlink on the MAST Discovery Portal webpage.
The following example shows how to find all the MAST archival data for Jupiter. First, click on the “Advanced Search” hyperlink located directly below the search box at the top of the MAST Portal search page (Figure 1). After the initial load of the data finishes, locate the “Target Name” filter box, and enter “Jupiter” and hit Enter/Return to apply the filter (Figure 2).
Figure 2: Entering the search term "jupiter" into the "Target Name" search filter.

At the top of the page, you will see that the filter has been applied, resulting in 7,661 observations from all missions. The search could be further refined, for example, to only include ultraviolet or infrared imaging observations by selecting the UV and Infrared check boxes in the “Waveband” filter box as well as the “image” check box in the “Product Type” filter box.

We welcome your feedback on ways in which we can improve searches for solar system objects and moving targets. Please email us at archive@stsci.edu or post on the MAST Forum.

Addendum: Although the MAST Portal now provides an opportunity to search for moving targets, we would like to clarify several points. First, advanced searches using the “Target Name” filter (as described above) will only return data for those observations which the Principle Investigator has tagged with the actual target name. As a result, searches using the “Target Name” field will be incomplete, since a PI may have used abbreviations, suffixes, or a personal naming scheme. The MAST staff are currently working on developing specific database flags and search interfaces for moving objects which will provide more robust search results. We apologize for any misleading information in the original article, and we thanks to those in the MAST community for providing clarifying feedback on this issue.