Response to Comments and Questions from the 2010 MAST User Survey
Q: "Better linking with other VO services, particularly being able to cross-correlate any table accessible from VO against MAST products. Uploading and downloading large tables. Accessing data products using third-party applications (such as Topcat)."
A: As you can imagine, we are actively working on better integration with VO services and protocols. MAST is an active part of the National Virtual Observatory and we'd like to offer many of your suggestions to all our users. The hurdles with cross-correlation are still in the size of the correlations itself. For many large surveys we have already done an internal cross-correlation (GALEX, 2MASS, GSC2, SDSS, Kepler) and we are now working on exposing these data to the user in a meaningful interface. At the moment our only tool to upload and download large tables remains CasJobs (for GALEX and Kepler only at the moment). We are planning on expanding CasJobs to all missions in MAST.
Q: "There are no improvements that would be relevant to me but other individuals would desire upgrades and expansions. I guess I'm a simple guy in a hectic world."
A: And we are glad to be your archive of choice.
Q: "It would be useful to cross correlate a user's provided catalogue (e.g. AKARI) with the whole content of external catalogues (e.g. 2MASS). I mean no limitations on the search radius."
A: We are indeed considering this option, but for large catalogs (> million sources) this does not seem to be viable at the moment outside the database. If you have optical or UV data, we would welcome the opportunity to cross-match it again out holdings. Currently, we have cross-matched very large catalogs in house (GALEX, GSC2, SDSS, 2MASS) but we are still working on the interfaces to expose these matches. If your catalog is not too large, you will soon be able to use CasJobs and match it against GALEX and Kepler, which in turn have been matched with SDSS, GSC2 and 2MASS. Alternatively, we might be able to help you directly with the match in our databases.
A: Thanks for the response and your suggestions. We will take points (1) and (2) under consideration, particularly because they are indeed what others are asking. We have already initiate a SDSS and 2MASS cross-matching with all our data holdings. We hope to make them available soon. Persistence is also something we are closely looking at as part of our new portal re-design.
As for (3), we will definitely consider how we could implement this. We already work with Treasury and Large Programs who request this support, access to a collaborative ftp site where the proprietary nature of the data are preserved until they become public HLSP. The Teams control the data that is available in these sites. We also have several longer term projects planned such as changing from on the fly reprocessing for each data request to producing a static archive and reprocessing as needed. Once this is in place, it would make it much easier to provide the type of service you are suggesting.
Q: "Heavy user of CasJobs"
A: Thanks for responding to the survey. We are planning to expand the use of CasJobs to other MAST missions.
Q: "The danger in some of these is duplication of efforts with what NED provides."
A: At MAST we see our ultimate goal to serve users. As such we try to provide them with a comprehensive set of services to make the user experience more relevant to their research when they make use of MAST. In this spirit, we really don't see duplication of effort, or competition with NED. We want to be able to provide users with services in MAST that use NED and other data archives.
Q: "My main difficulty now is in downloading GALEX data. The archive structure is so clumsy that it is close to impossible to download large numbers of files."
A: The structure of the GALEX archive, and the complex directory tree, is something MAST receives from the Science Team at Caltech. MAST has no input on this structure, but you have some tangible suggestions as to how to make file selections more navigable and self-evident, we'd be glad to forward them to the GALEX Science Team. It is indeed complex.
To help you retrieve the data you need, we would be happy to work with you and either generate scripts or show you how to generate them.
Q: "Please, make the same advanced search options offered by MAST also available in the HLA. It is truly painful to select observations in HLA according to filter names, exposure times or or proposal IDs. "
A: Thanks for your input. You can search now for one or more filters or for proposal IDs using the advanced search form. But currently there is no option to search including a limit on exposure time. We agree that having the full search capability including many more of the database fields is important to add and will be working on that in the near future.
If you have more suggestions for specific types of searches you'd like to be able to perform, we would be interested to hear them. Thanks for taking the time to fill out the survey and send us comments.
Q: "Most of the mission description documentation is readily available; however, I have had problems finding documentation on how to format searches (e.g., entering ranges, excluding values, etc.) "
A: General information on formatting search queries is available by clicking the "Help" icon in the upper right corner of most MAST mission search forms. For Kepler, the information is at: http://archive.stsci.edu/kepler/help/search_help.html
The available search operators/options depend on the data type of the field being searched, so the descriptions are divided according to data type. The descriptions include not only string, numerical, and datetime fields but also special options for searching coordinates. (Note searching for null values is currently not supported.)
The link below Help, labeled "Field Descriptions", gives a short description of the available search fields along with either examples or ranges of values. The "column label" describes the form elements on the HTML search forms, while the actual "column names" would be specified in GET requests. The data type of each field is also listed which is linked to more information on search options.
More information on formatting search queries from a script or as a GET request can be found at: http://archive.stsci.edu/vo/mast_services.html. If you have further questions, please contact the help desk at firstname.lastname@example.org