ASDI Prototype Documentation
Back to main ASDI page
MAST developed the Archived Synthetic Data Initiative (ASDI) prototype service to demonstrate how a broader synthetic data service might operate. The ASDI concept is based on different "Project Types" each focused on a particular astrophysical topic. Within each Project Type, there might be multiple datasets provided by one or more contributors, where each dataset begins as a contribution to MAST as a High Level Science Product (HLSP). The ASDI prototype dataset HLSP was contributed to MAST by the FOGGIE Collaboration, and includes synthetic absorption spectra derived by ray-tracing through the gaseous halos of galaxy formation simulations. The synthetic absorption spectra were saved in FITS files containing a slew of other metadata that a user of these data may want to use as the basis to search and refine a list of useful products. The ASDI prototype uses a VO Table Access Protocol (TAP) service to expose the metadata database ingested from these data products and to provide links to the pristine synthetic spectra. In addition, the ASDI prototype includes a new MAST service which enables the user to create custom data simulations by applying the effects of resolution and noise. Expand the sections below to learn how to use the ASDI prototype, which is a pre-release Beta version. We welcome feedback, comments, and suggestions! See the main ASDI page for more information.
TAP Service Introduction
Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic / spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns returned from cone, image, and spectral services.
Every TAP service has a "Base URL" plus associated endpoints for synchronous and asynchronous queries, as well as status and capability information, and sometimes service-provided sample queries. The endpoints are predefined in the TAP standard, so clients can infer them using the base. We therefore only have to provide astroquery that base. The ASDI Demo Notebook utilizes the astroquery TapPlus library as the client to the search service, whose base URL is below:
from astroquery.utils.tap.core import TapPlus
TAP_URL = "http://vao.stsci.edu/ASDI/tapservice.aspx"
TAP_service = TapPlus(url=TAP_URL)
Browsing the Schema
TAP gives us access to descriptive metadata for this catalog. We can use this to narrow searches and filter our results. For the current catalog, there is only one table, with columns for the project name, nullable fields relevant to various projects, and a public path for downloading associated files.
table_descriptions = TAP_service.load_tables()
print('\n')
for table in table_descriptions:
if(not table.name.startswith('tap_schema')):
print('TAP table: ' + table.name)
print(table.description)
print('\n')
for i, column in enumerate(table.columns):
print(column.name)
print(column.description,'\n')
Access the ASDI Demo Catalog
The ASDI CGM Spectra metadata catalog contains information about the individual absorption lines through sightlines in simulated galaxies' halos. The database table is accessed by the name dbo.ASDISpectra1DCGM as in the example below. For the ASDI Prototype, we have only one available project, contributed by the FOGGIE collaboration.
# Launch TAP query
job = TAP_service.launch_job("""
SELECT *
FROM dbo.ASDISpectra1DCGM
WHERE projectName = 'foggie'
""")
# Obtain results
foggie_results = job.get_results()
# Print available columns in resulting catalog
for c in foggie_results.columns: print(c)
# Display example entires in resulting catalog
foggie_results[['lineName','redshift','totalColumn','impact','publicPath']][0:10]
For further information, see: Peeples et al. (2019)
Line of Sight and Spectral Line Properties |
Available Values |
Description |
||
---|---|---|---|---|
lineName |
'C IV 1548' 'H I 1216' |
'H I 919' 'O VI 1032' |
'Si II 1260' 'Si IV 1394' |
Name of the transition associated with this spectral feature. |
restWave |
1548.205 1215.67 |
919.3514 1031.9261 |
1260.4221 1393.755 |
Rest-frame wavelength in Angstroms. |
redshift |
2.0, 2.5 |
Redshift of the galaxy simulation. |
||
totalColumn |
5.64 < totalColumn < 20.37 |
Total column number density of this ion. Units are log cm^-2. |
||
projectName |
'foggie' |
Name of the simulation suite. |
||
physicsName |
'nref11n-nref10f' |
Name of the physics set assumed for these simulations. |
||
seriesName |
'halo008508' |
Name of the simulation series (for example, name of the halo evolved over time). |
||
galaxyName |
rd0018, rd0020 |
Name of the galaxy simulation (for example, the name of a timestep in a series). |
||
mstar |
1.1e9, 2.3e9 |
Stellar mass of the galaxy simulation in solar masses. |
||
sfr |
77, 49 |
Star formation rate of the galaxy simulation in solar masses per year. |
||
impact |
0.97 < impact < 60.2 |
Impact parameter, or distance from galaxy center, for this line of sight, in kiloparsecs. |
||
direction |
'axx', 'axy', 'axz' |
String indicating the axis along which the line of sight travels. |
||
pos1 |
5.6 < pos1 < 90.5 |
Position along first other axis. For example, with direction 'axx', pos1 gives position in the y dimension in kpc. |
||
pos2 |
4.3 < pos2 < 90.8 |
Position along second other axis. For example, with direction 'axz', pos2 gives position in the y dimension in kpc. |
||
numLines |
6 |
Number of spectral lines along this line of sight. |
||
physIonDensity |
1.03x10-28 < physIonDensity < 1.009x10-24 |
Ionic column-density-weighted density, in g cm-3. |
||
physIonTemperature |
3381 < physIonTemperature < 583350 |
Ionic column-density-weighted temperature, in degrees Kelvin. |
||
physIonMetallicity |
1.63 < physIonMetallicity |
Ionic column-density-weighted metallicity, in units of solar metallicity. |
||
creationDate |
example: 2018-08-22T13:27:19.793136 |
Date and time at which the simulated sightline was created. |
||
extName |
'C IV 1548' 'H I 1216' |
'H I 919' 'O VI 1032' |
'SI II 1260' 'SI IV 1394' |
Extension name in line of sight FITS file containing the absorption spectrum of this ion. |
extNum |
2-7 |
Extension number in line of sight FITS file containing the absorption spectrum of this ion. |
||
publicPath |
example: /hlsps/misty/foggie/nref11n-nref10f/halo008508/rd0020/axz/los/hlsp_misty_spectra1d_cgm_foggie_nref11n-nref10f-halo008508-rd0020-axz-dx067.4-dy032.5_v1_pristine-los.fits |
Path to the FITS file containing pristine spectra and physical information vectors. Prepend 'https://archive.stsci.edu' to the path entry to locate the file. |
The ASDI data simulation prototype uses an application programming interface (API) to expose a service that applies STIS instrumental effects to the pristine input spectra, and stream the results back to the user. For this beta version, the API is accessed from the base URL: "https://masttest.stsci.edu/asdi/api/v0.1/addsignature".
To access the API, we can construct Get requests in Python or other languages. The prototype service has only a few options for input parameters to specify the target pristine spectrum, as well as the desired output data simulation properties. Several of these parameters can be obtained by utilizing the ASDI TAP search service described above.
- instrument - Instrument for which we want to simulate data, currently only the 'STIS' option is available.
- element1 - Instrument element, currently only the 'E140H' option is available.
- line_name - Name of the spectral line to simulate, returned by ASDI TAP query.
- ext_num - Extension number of the pristine spectrum FITS file, returned by ASDI TAP query.
- public_path - Path to the pristine input spectrum FITS file, returned by ASDI TAP query.
- snratioresel - Signal to noise ratio per resolution element of the desired data simulation.
- override_observed_wavelength - Center the absorption line at this wavelength, for customizing the mock observation to explore differences in absorber redshift.
Parameter choices are passed to the service by augmenting the URL with strings, which is handled as in the example below using the Python requests library. The beta service can handle one spectrum at a time.
import requests
#use the foggie_results catalog returned from our VO TAP query above
example_result = foggie_results[0]
example_line_name = example_result['lineName']
example_ext_num = example_result['extNum']
example_path = example_result['publicPath']
#the ASDI prototype is accessed through this URL
inst_sim_url = "https://masttest.stsci.edu/asdi/api/v0.1/addsignature"
#construct a dictionary of input parameters
PARAMS = {'instrument':'STIS',
'line_name': example_line_name,
'ext_num': example_ext_num,
'public_path': example_path,
'snratioresel': 25,
'override_observed_wavelength': 1500.0}
#pass the service URL and parameters to Python requests call
r = requests.get(url = inst_sim_url, params = PARAMS, allow_redirects = True)
This last command passes the parameters to the MAST service, which carries out the requested data simulation and returns a response to the user. The response is a JSON object which carries status and (if successful) the resulting simulated spectrum and associated FITS metadata, which allows us to save the result into a new FITS file.
#specify the FITS file in which to save the results
stis_file_path=os.path.join(save_location,STIS_sim_filename)
#response is returned as a JSON object which can be inspected for status and content.
if r.status_code != 200: #show what went wrong
#print(r.headers)
print(r.text)
else:
#if successful, write the content to a FITS file
open(stis_file_path, 'wb').write(r.content)
With the response saved into a FITS file, we can now inspect the new file and begin to use it:
fo=fits.open(stis_file_path)
print(fo.info())
print(fo['SyntheticData'].data.columns)
See the Demo Notebook at the link below for a more comprehensive demonstration.