THE INITIAL DATA PRODUCTS FROM THE EUVE SOFTWARE PIPELINE -- 
          A PHOTON'S JOURNEY THROUGH THE END-TO-END SYSTEM
                              Behram Antia
           Center for EUV Astrophysics, 2150 Kittredge St.,
      University of California, Berkeley, California 94720, USA


ABSTRACT

   The End-to-End System (EES) is a unique collection of software modules
created for use at the Center for EUV Astrophysics.  The "pipeline" is a shell
script which executes selected EES modules and creates initial data products:
skymaps, data sets for individual sources (called "pigeonholes") and catalogues
of sources.  This article emphasizes the data from the all-sky survey, conducted
between July 22, 1992 and January 21, 1993.  A description of each of the major
data products will be given and, as an example of how the pipeline works, the
reader will follow a photon's path through the software pipeline into a pigeon-
hole.  These data products are the primary goal of the EUVE all-sky survey
mission, and so their relative importance for the follow-up science will also
be discussed.


1. INTRODUCTION

   The End-to-End System (EES) is a constantly evolving software system that
provides the computer environment for the reception, handling, archiving, and
reduction of data from EUVE.  These modules are written in C and run on a Unix
network of servers and Sparcstations.  The pipeline script uses a subset of
these modules for the reduction of data from EUVE and the creation of useful
scientific data products.  The pipeline team is responsible for running the
pipeline, troubleshooting, strategy of operation, error-checking of data pro-
ducts, and a host of other related tasks.  The team produces data products and
then assists with many aspects of the data analysis.
   The pipeline script is separated into two distinct phases of operation:

 (1) the EUVE Science Operations Center (ESOC) pipeline and
 (2) the Science Data Analysis Facility (SDAF) pipeline.

   Each pipeline module has an associated parameter file which allows the user
to customize its operation; these parameter files contain instrument and soft-
ware parameters and data product formats.  A module configures itself based on
this file at each initiation; usually just the default settings are sufficient.
   The ESOC pipeline is the initial processing stage.  Preliminary engineering
and science products are created and passed along to the analysis (SDAF) pipe-
line.  The products that are created daily are combined, so that by the end of
the survey, complete pigeonholes and skymaps were created and ready for further
analysis.  The science products are created only from data obtained during
nighttime portions of the orbit.  Daytime data are not processed because the
detectors are turned off due to high background.  This is the pipeline proces-
sing concept:  process the data, make skymaps and pigeonholes, detect sources,
and reprocess the data for the newly discovered sources.


2. ESOC PIPELINE

2.1 Decommutate

   When telemetry files (or messages) are received in Berkeley, several pro-
cesses are automatically initiated:  the incoming message is logged and ar-
chived, and the pipeline is started.  The first and most important module in
the pipeline is the decommutate module.  This module separates the telemetry
stream into various binary format data files known as slp (pronounced "slip",
standing for "self-labeled packet") files, which allow the combination of se-
veral segments of data in one file.
   The decommutate module produces one detector slp file for each of the EUVE
detectors (three Scanners, short, medium, and long wavelength spectrometers and
the Deep Survey, for a total of seven) and an engineering data slp file.  The
detector slp files (det[1-7].slp) contain the position of each photon event in
detector coordinates and timing information from the spacecraft clock, while
the engineering slp (eng.slp) file contains the on-board computers (OBC) report
of satellite status.  Thus, from these 8 slp files we can perform a wide vari-
ety of hardware and science analysis tasks; these are the essential files for
all further work.

2.2 Engineering History Database

   To properly understand our instrument's performance, we keep a record of the
engineering monitors in a database.  The engineering slp file contains all the
relevant hardware information, including values for approximately 850 monitors
on the spacecraft and instrument.  This database module stores values for the
monitors in a commercial database (Sybase), so that trends of instrument para-
meters like detector current and temperature can be followed easily.  Sybase is
also used for a database containing time intervals in the telemetry that have
been processed, so that duplication of photon events does not occur.  It should
be mentioned that this particular module has been the most troublesome and most
difficult to implement properly.

2.3 Aspect, Remapping and Pigeonholing

   The next module of importance extracts the aspect information.  The space-
craft aspect is a part of the OBC report, and the decommutate module puts it
in the eng.slp file.  The aspect module then creates an ascii, time-sorted file
with time and the four quaternions representing spacecraft orientation.
   The last two modules, essential for analysis, are remapping and pigeonhol-
ing.  The remap module transforms a slp file containing every photon's detector
coordinates to a slp file with every photon's celestial coordinates, using the
det.slp files and the aspect file.  A pigeonhole is basically a data file cor-
responding to a circular region on the sky centered at a point listed in a ca-
talogue.  The catalogue is a list of possible EUV sources compiled before launch
from various astronomical databases.  Each pigeonhole thus constitutes a data
set for a 10 arc-minute circular region with all the photons from a particular
object contained in this set when they are properly remapped.
   The pigeonholing module will then look at the input catalogue, which con-
tains a set of right ascension (R.A.) and declination (decl.) for possible
sources.  If photon events detected by EUVE fall within the given coordinates,
a pigeonhole is updated with this photon detection information.
   There are many other modules for producing plots, stripcharts, and various
other types of products.  The products described above are the critical ones,
however, and all rely on the successful completion of the decommutate module.
The pigeonholing process is essentially the end of the SOC pipeline.  Each day's
data are merged together, and the set is then processed by the analysis (SDAF)
pipeline.


3. SDAF PIPELINE

   The SDAF pipeline is a distinctly different set of modules, which works on
the final products from the SOC pipeline.  The data files of importance here
are the aspect, a slp file of all photons (sky_phot.slp), and the pigeonholes.
From these, we make maps of the sky in each filter, compile catalogues of
sources, and accumulate pigeonholes.
   During the survey, we tried to update exposure maps, binned photon maps,
and pigeonholes on a daily basis.  Now that the survey is over, we reprocess
data in larger contiguous sections.  The photon information in the sky_phot.slp
file is collected for each filter and binned in skymaps at a defined resolution
so that, for each wavelength region covered by the detectors, we have a raw
photon map of the sky.  We also compute exposure times and accumulate separate
exposure maps for each filter.  In addition, pigeonholes are updated from in-
dividual pipelines.  These products form the basis for the science we are able
to perform from the survey data-set.
   Source detection, the most important module in the SDAF pipeline, is des-
cribed more fully elsewhere in this issue (Lewis 1993).  In brief, this module
"looks" at skymaps for each filter and finds statistically significant possible
EUV sources.  Once this module has located possible sources and supplied their
corresponding positions, the input catalogue is updated and the entire process
starts again -- very much an iterative processing process! With possible de-
tection positions put into a revised catalogue, processing from the ESOC pipe-
line stage starts, and photons for these "new" pigeonholes are accumulated.
Thus, the detect module determines which pigeonholes we examine for sources:
old (in original catalogue) and new (newly discovered sources).
   Some examples of the products are shown in figs. 1 and 2.  Fig. 1a is the
Lexan/boron portion of a pigeonhole containing a detected source.  The pigeon-
hole contains the photon events for each bandpass: Lexan/boron (50-180 A),
Al/Ti/C (160-240 A), Al/Ti/Sb/Ti (345-605 A) and Sn/SnSiO (500-740 A) and can
be readily subdivided.  For the cases where a pigeonhole may contain a source
but it is not obvious to the eye, we convolve the data with a point-spread
function derived from an in-orbit observation; the result becomes more apparent
visually: fig. 1b.
   Fig. 2 shows a skymap for the Lexan/boron filter.  Some gaps are present due
to calibration pointings taken during the survey; these are being filled and
the maps are updated periodically.  Similar maps are produced for each bandpass
and the Deep Survey instrument; the detection algorithm runs on each separately,
in order to find sources in each bandpass.  On this scale, only the very bright-
est sources are visible as darker dots.  The resolution of the maps is approxi-
mately 1.3 arc-minutes, and so we can examine these maps in much greater detail
with various software tools such as IRAF and IDL.  The challenges in producing
these maps are numerous, and this example (fig. 2) shows several of the diffi-
culties we faced.  Some pointed data was processed as survey data and included
in the map and several small gaps are present due to bad aspect data for several
orbits.  These and other problems make source detection more difficult.
   The skymaps are intrinsically interesting as well; we have identified ex-
tended sources (larger than current pigeonholes allow) such as the Vela and
Cygnus supernovae remnants.  The beauty of this system is the ease with which
the data products can be subsequently analysed.  A given pigeonhole has the
photons associated with a particular source, so all the data are readily sub-
divided, yet larger or full-scale maps are also created for other types of
study.  We have found that this parallel effort is quite complementary to and
essential for analysis of the EUVE all-sky survey data.


4. A PHOTON'S JOURNEY

   It would be instructive at this point to see an actual photon event as it
moves through the pipeline from a telemetry file to a pigeonhole.  The decom-
mutate module has created the binary data files called det[1-7].slp.  If we
convert these to ascii, we get entries that include the following information:

            +----------------------------------------+
            |    time (sec)       x       y    det#  |
            |  774683832.680     356     811     1   |
            +----------------------------------------+

   There are typically many thousands of entries per file, but the first is
representative.  So far, we have photon time and location on the detector (the
detector is 2048x2048 pixels).  The time of this event is based on an offset
of 00:00:00 GMT on May 24, 1968.  Thus, this photon hit our detector about
774683832 seconds after the zero date (which is approximately 05:55:50 GMT on
Dec. 10, 1992).  To remap this event to the sky properly, we use the aspect
file that was extracted from the engineering data.  The aspect file has entries
of the form:

     +-----------------------------------------------------------+
     |    time (sec)       q1        q2        q3         q4     |
     |  774683832.723   0.83997   0.30265   -0.22039   -0.39277  |
     +-----------------------------------------------------------+

where time is in the same format as above and q1 to q4 are the four components
of the quaternion that represents spacecraft orientation.  The remap module will
use both of the above files to create another slp file that contains the cal-
culated right ascension and declination for each photon event (aspect can be
interpolated).  Thus the photon we are tracing will now have the following form:

          +------------------------------------------------+
          |   time (sec)       R.A. (deg)     decl. (deg)  |
          |  774683832.680     40.5280715     -35.3940232  |
          +------------------------------------------------+

   The pigeonholing module will then look in the input catalogue and update
pigeonholes as described previously.  The photon we are tracing is found in fig.
2a in the Lexan/boron filter image.  These remapped photons are also placed in
the raw photon map of the sky for this filter (fig. 3) as they accumulate.  So,
in general, after each portion of data from the survey is processed through
both the ESOC and SDAF pipelines, both skymaps and pigeonholes are updated or
appended with the new photon detections.  This scheme of putting photons in
pigeonholes is excellent for two reasons: the compression of information for
a source makes the physical handling of the data files tractable, and it also
makes reprocessing data for individual sources a task that can be accomplished
within a few hours.


5. RESULTS AND CONCLUSIONS

   So, how useful are the pigeonholes and skymaps? The data in the pigeonholes
have revealed a great deal about the sources they contain.  We have been able
to derive count rates for many of the objects and thereby infer some previously
untestable physical properties.  We have also obtained light curves, which has
shown variability of some sources.  These pigeonholes play an important part
both in the detection of sources and in their analysis.
   The distribution of EUV-emitting sources is also revealing information about
what is not seen -- the interstellar material (ISM).  So what we see and what
we do not are both very important results.  One of the major products of the
mission will be  a map of the distribution of the local ISM.  Because of its
pioneering nature, this all-sky survey will form a basis for more detailed in-
vestigation.  Some sources detected in the survey will become targets of study
by guest observers with EUVE's spectrometer.  The spectrometer results will
yield a great deal more detailed information about these sources.
   To date, the initial processing of the survey data-set has been completed,
and reprocessing for new sources is also complete.  The initial release of the
EUVE Bright Source List was made with the second NASA research announcement
in June 1993.  Efforts are continuing, and will continue for some time, with
respect to reprocessing the data, with the aim of  incorporating the knowledge
gained on these initial efforts.  We look forward to continually improving the
quality of our data products.
   These results, and many others, were presented at the 182nd meeting of the
American Astronomical Society (AAS), in a special session of EUV Astronomy.
The sessions included 33 poster and 18 oral presentations, indicating the huge
advances in the field since the launch of EUVE.  Many papers are also in various
stages of publication in refereed journals.  The field has exploded with new and
exciting results, in large part because of the pipeline and its data products.
   The smooth operation of the pipeline and the high quality data products that
have been produced so rapidly have played a large part in the success of the
mission to date.  Our current and future reprocessing efforts will result in
higher quality skymaps and pigeonholes, and more detections of EUV-emitting
sources as we lower our significance threshold.  In the meantime, spectroscopy
for guest observers continues and will continue for the satellite's lifetime
-- which, we hope, will be a very long and productive lifetime.


FIGURE CAPTIONS

Figure 1:  (a) Example of pigeonhole data for a source in the Lexan/boron
	bandpass.  (b) Pigeonhole data convolved with a point-spread function.
Figure 2:  A map of the sky (equal area Aitoff projection) for Lexan/boron
	bandpass.  The centre of image is 0, 0 (R.A., decl.) and left side is
	180 degrees.