Building upon this work, ISO has been developing plans for a one-shift to zero-shift (1:0) transition. The goal of 1:0 is to eliminate all routine shift work, which is defined to be those tasks that must be done on a specific time basis (e.g., daily); those routine tasks that are not time constrained and that can be done as they best fit into one's schedule, are not considered shift work. Therefore, under our definition shift work mainly includes the daily monitoring of real-time passes ("console duty") and the associated console support activities.
The reasons for transitioning to a zero-shift scenario are many, the most obvious one being as a means to further reduce operations costs. Although this is important, a 1:0 transition is also in line with NASA's goals for innovation and with its recent EUVE out-sourcing decision, which was made as a direct result of CEA's previous innovation activities (including 3:1). In going to zero shifts we will have demonstrated a "proof of concept" that other future missions can apply to their own operations. The 3:1 transition changed the mission operations paradigm for NASA and, in the process, provided CEA with many opportunities for innovation and collaboration; as the next logical step in the extension of this paradigm, 1:0 is expected to do the same.
The basic operations requirements for the 1:0 transition, as follows, are simple and straightforward:
This phased approach allows us to progress towards the full
zero-shift scenario one step at a time, learning from any mistakes
along the way. Each phase progressively provides for significantly
increased levels of autonomous operation and degrees of innovation.
This plan allows CEA to reach zero-shift operations in a reasonable
time period and it provides reasonable periods of development between
each succeeding phase. Additionally, since budgetary constraints may
affect our ability to reach Phase 3, this approach has been designed
so that Phase 2 represents a true zero-shift scenario with regards to
science payload H&S.
The major routine on-shift work duties include the following:
In addition to the above routine shift work, PCs and/or PCAs are
also responsible for the following:
The major goal of the 1:0 transition is to eliminate via
automation all of the routine shift work as outlined above. The
following sections describe in detail the "needs" and "wants" of the
various phases of the 1:0 transition. Each issue is categorized into
one of the following:
We also recommend the implementation of the following "wants" for
Phase 1:
The roles of PCs/PCAs in Phase 1 zero-shift operations will remain
similar to the current single-shift operations. Since the daily
console support duties will remain at this point, personnel will
continue to remain available to support a single shift seven days a
week. As tasks are automated to free up personnel resources, the
roles of the Science Instrument Support team members will continuously
be reallocated to support development and test-bed activities.
Due to the minimal software development required for the Phase 1
transition, we estimate that it can be accomplished by early 1996.
We also recommend that the following "wants" be addressed for Phase
2:
Although no shift work remains in Phase 2 zero-shift operations, a
duty PC will remain available during regular business hours (M-F,
9am-5pm) to attend to periodic (bi-weekly?) checks in the ESOC.
These checks would include attending to command planning and
specialized commanding, software and procedure maintenance, anomaly
response, the various weekly duties, and checks of the ground and
software systems to verify that everything is working properly. The
PCAs will be available during this time to assist the duty PC with
these activities. All remaining PC/PCA time will be reallocated to
development and test-bed activities. The ACE will continue to
repspond to all anomalies that occur during non-business hours.
Due to the significant software development required for the Phase
2 zero-shift transition, it is expected that it will not occur until
at least mid-1996.
In Phase 3 operations a duty PC will conduct no shift work, but
will remain available during regular business hours (M-F, 9am-5pm).
Their duties will remain similar to those in Phase 2, although some
tasks will have been simplified or eliminated. The PCAs will be
available during this time to support all of the above activities. In
general, PCs/PCAs will be able to spend most of their time on
development and/or test-bed activities. As always, the ACE will
continue to deal with all anomalies that occur during non-business
hours.
Full implementation of Phase 3 zero-shift operations will involve
significant software development and is not expected to be completed
until at least early- to mid-1997.
In such a scenario the general operational duties of the PC/PCA
will consist of no routine shift duties, only periodic checks in the
ESOC including command scheduling and specialized commanding, software
and procedure maintenance, anomaly response, checks to verify proper
operation of all autonomous systems, and handling of any remaining
weekly duties. The rest of their time would be in support of
development and/or test-bed activities.
3.1 Current One-Shift Operations
During current one-shift operations the ISO Science Instrument
Support team has personnel working in the ESOC every day of the week
(including weekends) from 8am-5pm. The goal has been to have the PCAs
handle for the PCs as much of the routine shift work as possible.
PCAs support weekend shifts by themselves and, as school schedules
allow, fill in for the PC on duty during weekday shifts; all remaining
routine duties, as well as the non-routine ones, are handled by the
duty PC. The ACE responds to all off-shift anomalies via pages from
the software, from the message center (e.g., from PACOR), and from the
PCAs during the weekend shifts.
PCAs generally perform all of the routine daily console work and
console support duties. In addition to supervising the work of the
PCAs, PCs generally attend to all the routine commanding duties.
3.2 Phase 1 Zero-Shift Operations
The primary and secondary "needs" of the Phase 1 zero-shift
transition are as follows:
Primary Needs
Secondary Needs
We call Phase 1 a "minimal" zero-shift operations scenario because
at this point personnel are no longer required to sit at the console
screen for the purposes of visually monitoring the incoming H&S
telemetry. The automating of all essential H&S telemetry monitoring
will free up at least two hours/day of PC/PCA time each and every day,
along with a significant amount of time (based on the 3:1 experience)
dealing with unnecessary paging. Primary Wants
These additional capabilities will greatly reduce the effort required
for software installations or patches, and will streamline the anomaly
resolution process. 3.3 Phase 2 Zero-Shift Operations
The major goals of Phase 2 are to automate all of the routine daily
console support duties and science payload commanding activities. To
do this requires addressing the following:
Primary Needs
Secondary Needs
We call Phase 2 the "operational" zero-shift scenario because it is at
this point that all routine manual shift work has been eliminated or
automated. The above efforts will free up ~8 hours/day, seven
days/week, of PCA/PC time, and will eliminate the need for any weekend
support for nominal operations. Primary Wants
Secondary Wants
The major impacts of implementing the above "wants" will be to
simplify and streamline the network needs (hardware and facilities),
and to provide the ground software with some context of each
observation, which will allow it to change its rules accordingly to
accomodate both H&S and science. 3.4 Phase 3 Zero-Shift Operations
The goals of Phase 3 will be to implement the following:
Primary Needs
Primary Wants
Secondary Wants
We call Phase 3 the "full" zero-shift scenario because it is in this
phase that we truly reach full automation of all nominal payload
operations, both in terms of H&S and science. 3.5 A Realistic Zero-Shift Operations Scenario at CEA
Given the stage of the EUVE mission, the goals of CEA, and the
available programming resources, we expect that a reasonable and
realistic zero-shift payload operations scenario for CEA would be a
Phase 2 implementation in which all the above-mentioned "needs" have
been met. Such a scenario will ensure instrument H&S, but not
necessarily the maximal return of high quality science data.
4.0 "Needs" vs. "Wants" for 1:0
The following sections list out in detail the primary and secondary
"needs" and "wants" for the 1:0 transition. For each issue below is
indicated the following:
----------------------------------------------------------------
ISSUE: The need or want to be addressed.
PR/PH: The priority (Need/Want) and transition phase (1/2/3).
CURNT: Description of current implementation.
CCOST: Approximat FTE cost of current implementation.
RECOM: Recommended action.
RADAR: Relevant radar entry(ies).
IMPCT: Health and safety impact if issue is not addressed.
----------------------------------------------------------------
4.1 Primary Needs
Definition: A primary need is defined as an issue that, if not
addressed, could have negative effects on instrument H&S during a
zero-shift operations scenario.
----------------------------------------------------------------
ISSUE: Eliminate the need for all daily routine operator
console duty (i.e., visual H&S monitoring of real-time
telemetry passes).
PR/PH: N/1
CURNT: PCs/PCAs visually monitor at least three realtime passes
during each day shift using soctools, which looks at
all payload engineering monitors while eworks only sees
a subset (the ones deemed most critical for ONE-SHIFT operations).
CCOST: 0.26 FTE (0.5 hr/pass * 3 pass/day / 8 hr/day * 1.4)
RECOM: Expand eworks to include additional monitors to ensure
H&S during a zero-shift scenario in which we will not be looking
at the science payload at all on a routine basis; the current
eworks setup was designed for 3:1. Eliminating the
need for human monitoring of H&S telemetry will free up
personnel resources from visual monitoring to do other things.
We need to work with the DSs (e.g., John Vallerga) to identify
any additional monitors/rules to build into eworks for
zero-shift operations.
RADAR: 3105, TBD
IMPCT: Instrument problems may go unnoticed using the current eworks
setup, which only monitors a subset of the full compliment of
engineering monitors.
----------------------------------------------------------------
ISSUE: Review the ACE system.
PR/PH: N/1
CURNT: The current ACE system was designed for one-shift operations
and works well in that scenario. There will undoubtedly be
adjustments that need to be made to ensure it is properly set
up for zero-shift operations.
CCOST: n/a
RECOM: We need to review all aspects of the current ACE system -- roles,
responsibilities, memebers, levels of training, etc. -- to ensure
it meets the needs and requirements of zero-shift operations.
RADAR: TBD
IMPCT: There could be significant unknown instrument H&S risks should
we use continue to use the current ACE system and it proves
inadequate to zero shift operations.
----------------------------------------------------------------
ISSUE: Automate the semi-manual daily process of ensuring that all
critical ground systems and processes are actively functioning
at all times.
PR/PH: N/2
CURNT: There is a PCS routine called status_checker that currently
monitors the most critical areas at a minimal level.
CCOST: 0.1 FTE (~4 hrs/wk / 40 hrs/wk)
RECOM: Implement an automated ground system monitoring system. This
would ensure that all necessary needs of automated data acquisition
system are met. This would include the following: disk space,
eworks/epage processes, GSFC data link (X.25), swap
space, queues (printers, eworks, pipeline, etc.),
kb.out alerts/warnings. The full list of critical items needs
to be reviewed for completeness. Such a system could use
status_checker as a basis or starting point. At a
minimum, we must be able to page based on the 20-hour rule.
It would be great if such a system would also fix problems on
its own (e.g., reboot machines, etc.).
RADAR: TBD
IMPCT: Could result in a potential loss of monitoring knowledge for
extended periods of time, violating the 20-hour rule.
----------------------------------------------------------------
ISSUE: Eliminate the daily manual review of limit transitions.
PR/PH: N/2
CURNT: The duty PC/PCA currently visually reviews hardcopies of limit
transition logs produced from the processing of the previous
days's telemetry.
CCOST: 0.4 FTE (0.25 hr/day / 8 hr/day * 1.4)
RECOM: Limit transitions seem to be the easiest available means to identify
potential anomalies and we should continue to use this information
in some way. At this point we have no specific recommendation for a
means to accomplish this, although one could consider the following:
* review and/or change our current monitor limits and the way we
use them
* use SELMON or another such test-bed system
* fold this in as part of the payload configuration (see below)
Alternatively, maybe this be added to the periodic system
checks if it only needs to be done once or twice a week.
RADAR: TBD
IMPCT: This would result in loss of H&S knowledge regarding potential
monitor problems.
----------------------------------------------------------------
ISSUE: Eliminate the manual daily review of command echos.
PR/PH: N/2
CURNT: The duty PC/PCA currently visually reviews hardcopies of the
command echo logs from the processing of the previous days's
telemetry. The command echoes come from both uploaded
commands (e.g., for observation reconfigurations) and on-board
macros (e.g., dawn/day/dusk/night).
CCOST: 0.4 FTE (0.25 hr/day / 8 hr/day * 1.4)
RECOM: It is extremely important that we in some way verify that all
payload commands executed as expected. Since we are not guaranteed
that all command echoes even get into the telemetry stream, this
is an extremely difficult problem to handle in an automated fashion.
However, we offer the following possibilities:
* use the fact that the proper payload configuration (see below)
is in effect; if it is, one could assume that all
commands executed properly
* add this to the periodic system checks if it only needs to
be done once or twice a week.
RADAR: TBD
IMPCT: Improper execution of a command would indicate bugs/problems in
the on-board software that could negatively affect both H&S and
science.
----------------------------------------------------------------
ISSUE: Eliminate the daily manual review of detector plots.
PR/PH: N/2
CURNT: The duty PC/PCA is currently responsible for reviewing these
hardcopies on a daily basis, searching for any anomalies
(shape, hotspots, etc.) that may indicate degraded detector
health. All anomalies are passed on to the DS for investigation.
CCOST: 0.04 FTE (0.25 hr/day / 8 hr/day * 1.4)
RECOM: John Vallerga says that visual review of detector plots
provides the simplest and most effective means for verifying
detector health. Things like gain changes and hotspots are
readily apparent to the eyes, checks that may be extremely
difficult (if not impossible) to implement in software. He
wants to continue doing this on a daily basis. If nothing
else, he said that they should daily be routed to his desk and
he'll review them.
This issue makes zero-shift operations difficult, since it is
something that needs to be done every day and is best done by a
human. However, the following options should be considered:
* maybe gain changes can be monitored in software using pulse
heights or stimpins
* farm out task (along with the general DS report activities)
to the Engineering and Science Support group
RADAR: TBD
IMPCT: Without this daily review we won't be able to easily monitor
detector health. This must continue to be done on a daily basis,
either by humans or by software.
----------------------------------------------------------------
ISSUE: Eliminate the manual daily check for hight count rate areas on
the detectors.
PR/PH: N/2
CURNT: This is currently only done in a partial manner for the Deep
Survey detector. The arborday script checks the entire
Lex/B section quadrant as well as directly at the boresight,
reporting any high count rate areas in order to alert a
response to prevent deadspots and/or general detector gain loss.
CCOST: negligible
RECOM: If this need only be done once or twice a week it could be
added to the periodic system checks.
RADAR: 3065
IMPCT: Ignoring such a check could lead to potential gain loss or deadspots
on the detectors (e.g., for bright hotspot or source). This
could also affect the science data via increased primbsching.
----------------------------------------------------------------
ISSUE: Eliminate the manual daily review of "alert" conditions.
These are non-critical anomalies that are serious enough to
require action but can tolerate slower reponse times.
PR/PH: N/2
CURNT: Alerts are currently investigated during the daily shift by the
duty PC/PCA.
CCOST: negligible
RECOM: Add to eworks the necessary rules to page based on "alert"
conditions (paging could be limited to normal business hours
or could be tied in with a coding scheme as outlined below).
This capability could be integrated with a ground system
monitoring package (see below) and could make use of the
watchd facility. This "need" is based on the assumption
that these alerts need to be tended to in a timely fashion; if
they can wait a few days, then this issue would become a
"want" and could become part of the periodic system checks by
the duty PC/PCA.
RADAR: TBD
IMPCT: Could go for days before notification of a problem (i.e., may
only find out about an underlying "alert" condition when it's
become a larger problem).
----------------------------------------------------------------
ISSUE: Ensure a minimal backup ground (computer and communications)
system.
PR/PH: N/2
CURNT: The current backup system uses sdp2, which does not currently
support paging capabilities (a modem is being hooked up to
hyperion to fill this gap). Changing over to the backup
system is currently a difficult manual process, and overall it
requires a large amount of maintenance.
CCOST: 0.02 FTE (2-3 times/yr, ~10-15 hrs/time)
RECOM: Establish a stand-alone machine on the ESOC network to provide
a backup system with dial-out modem capability for paging (it
appears that the RAID box will form a basis for the future
ESOC network, but we need something in the event the RAID box
itself fails). Although the min-ESOC would be available, we
don't want to incur the expense of a flight to GSFC except for
cases where the CEA systems remain down for extended periods
of time.
RADAR: TBD
IMPCT: Loss of H&S information if machines/network go down.
----------------------------------------------------------------
ISSUE: Eliminate the manual daily shift report.
PR/PH: N/2
CURNT: At the end of each day's shift the duty PC/PCA currently
writes up and distributes a report that summarizes the activities
for that day. This summary contains information about the current
target, activities and anomalies (e.g., DDLs) during the day, and
upcoming activities. It's main use is to keep other controllers
informed of the status of the ESOC and payload; this has been
particularly effective for personnel duty changeovers.
CCOST: 0.9 FTE (0.5 hr/day / 8 hr/day * 1.4)
RECOM: Unless an easy way is found to automate this (probably not), then
the PC schedule could be rearranged to provide sufficient
continuity to allow this to be done on a less frequent basis (e.g.,
bi-weekly) and, therefore, added to the periodic system checks.
RADAR: TBD
IMPCT: Could result in both H&S and science impacts if the duty PC is
not properly advised of the recent situation.
----------------------------------------------------------------
ISSUE: Ensure for adequate simulations, which are necessary for validation
of software, training of personnel, etc., before the actual
transition is made.
PR/PH: N/ALL
CURNT: n/a
CCOST: n/a
RECOM: Adequate resources, time, etc. need to be allocated to ensure
for proper simulation planning and implementation. On a
longer term basis, as PCs/PCAs are removed from daily payload
operations they will tend to get "rusty". Periodic
simulations and/or recertifications may be necessary to ensure
that all operators retain the skills and knowledge required to
respond appropriately to anomalies.
RADAR: TBD
IMPCT: Could be significant H&S impacts if simulations don't properly
validate that all such needs are met before we actually
transition to zero-shift operations.
----------------------------------------------------------------
ISSUE: Update and augment all ESOC operations procedures for
zero-shift.
PR/PH: N/ALL
CURNT: The procedures are currently in use. However, they are
written at a level commensurate with a certified PC/PCA.
CCOST: 0.1 FTE (4 hr/wk / 40 hr/wk)
RECOM: Put all procedures under revision control in an organized
release hierarchy. Then review them to identify those that
need to be changed or augmented. New ones should be created
as necessary. The procedures should always reflect the
current operating scenario.
More detail needs to be added to support a lower level of
cognizance and training by those using them. As things get
more and more automated it is inevitable that users will need
increased reminders on how to respond appropriately to
situations. The goal is to have the procedures accurately
reflect in detail the necessary response to known situations
so that someone with minimal training could execute them
properly.
RADAR: 3086, TBD
IMPCT: Outdated and/or less detailed procedures could lead to
improper or insufficient response, which could result in
increased risk to the instrument health and safety.
----------------------------------------------------------------
4.2 Secondary Needs
Definition: A secondary need is defined as an issue that, although
not affecting instrument H&S, if addressed would yield significant
operations cost savings.
----------------------------------------------------------------
ISSUE: Eliminate "bad data" in the monitoring and paging processing.
PR/PH: N/1
CURNT: None. Since PACOR strips out the quality information on realtime
data, the eworks monitoring of such data often leads to
pages due to "bad data", which result in unnecessary ACE response.
CCOST: TBD
RECOM: Replace monitoring of real-time data with that of post-pass
real-time data, which has the quality information included so
that any "bad data" is marked as such and ignored instead of
allowed to generate pages. Work has already been completed at
GSFC so that we are now receiving this data; work remains on
our end to have this data monitored. Once implemented this
should allow us to respond 4-5 hours earlier to some problems
since we'll no longer have to wait for production data to
confirm an anomalous situation; this will also allow command
echoes to be monitored for this data.
RADAR: 3120
IMPCT: None.
----------------------------------------------------------------
ISSUE: Eliminate all unnecessary paging.
PR/PH: N/1
CURNT: None. In addition to for real anomalies, PCs/ACEs are now
routinely paged for the following:
1. bad data
2. faulty eworks logic
3. known conditions/attitudes
4. multiple pages for single anomaly
All of the above conditions (in order of severity) result in
people responding unnecessarily. This response has proved
costly during initial one-shift operations and will
undoubtedly occur again for zero-shift.
CCOST: TBD
RECOM: Implement user configurable control of paging (e.g., via page
screening and/or disabling of eworks logic).
RADAR: 3067
IMPCT: None; unnecessary pages yield unnecessary response. However, based
on history, the cost of this in terms of money and morale will be
high.
----------------------------------------------------------------
ISSUE: Automate the testing and validation of the eworks software.
PR/PH: N/1
CURNT: The testing and validation of eworks rules is currently
a very human intensive and time consuming activity. The
validation of the rules for one-shift operations took multiple
person months.
CCOST: TBD
RECOM: Implement an automated validation and testing scheme for all
future eworks installs. It is expected that by
spending the time (~1 person month) to automate this task we
will save months of manual work for each future release/patch.
RADAR: 3021
IMPCT: No controlled tests to ensure software validation could
potentially result in H&S risks.
----------------------------------------------------------------
ISSUE: Automate all routine commanding activities.
PR/PH: N/2
CURNT: All commanding is generally done by the duty PC. This
includes routine commanding (e.g., daily memory dumps,
observation configurations) and specific anomaly response
commanding. In general, most commanding activies are routine.
CCOST: 0.2 FTE (1 hr/day / 8 hr day * 1.4)
RECOM: Automate the routine commanding activities (e.g., via ATCs). This
will free up personnel resources for other CEA activities. It
will also provide greater flexibility in reconfiguring the payload,
which will minimize the negative impacts on observation scheduling
efficiency.
RADAR: 3027
IMPCT: None. Without this capability we would continue to need regular
manual PC support for routine commanding (e.g., ramcheck).
It will also result in loss of scheduling efficiency since
commanding is only done during normal business hours, which
puts an additional constraint on science 2planning.
----------------------------------------------------------------
ISSUE: Eliminate the manual daily tape backup of PACOR telemetry.
PR/PH: N/2
CURNT: This is currently done manually on a daily basis, with telemetry
written to tape in TAR format. This serves as a redundant
backup to the copy on the tape carousel (and, therefore, those
on the optical disks), should it prove defective.
CCOST: 0.09 FTE (0.5 hr/day / 8 hr/day * 1.4)
RECOM: Since most of those involved see this to be a useful service
(overall, it is much easier than going back to PACOR), we will
continue doing this until we transition to Phase 2. If the RAID
box is used for direct telemetry archival, this would eliminate
the need for this task (as well as for the tape carousel); if
this doesn't happen then this service will be eliminated upon
the Phase 2 transition.
RADAR: TBD
IMPCT: Little, if any. May result in slower response to data review
should carousel (and, therefore, optical disk) copy of telemetry
be corrupted, which doesn't occur very often.
----------------------------------------------------------------
4.3 Primary Wants
Definition: A primary want is defined as an issue that, if addressed,
would provide minor cost savings and/or would yield science benefits.
----------------------------------------------------------------
ISSUE: Streamline and automate the reporting and tracking system for
DDLs.
PR/PH: W/123
CURNT: DDLs are currently filed via a paper system, many of which end up
as radar entries.
CCOST: TBD
RECOM: Implement an on-line DDL filing system. An on-line system would
simplify and streamline the current paper system, reducing the time
people spend documenting/resolving problems (e.g., allows one to
read in files, cut/paste, reference search, etc.). It would also
provide a mechanism that software could use in the future to
automatically file DDLs.
RADAR: 3066
IMPCT: Little. The issue here is convenience as the current manual paper
system requires more time and effort for tracking things down.
However, as ESOC operations become more automated and
personnel less cognizant of all the various activities, an
on-line DDL system would facilitate anomaly response.
----------------------------------------------------------------
ISSUE: Streamline the anomaly response process by providing ACEs with
remote access to the ESOC network.
PR/PH: W/123
CURNT: ACEs currently do not have remote access to the ESOC network.
When paged they must physically come in to the ESOC to verify
the anomaly (i.e., is it real?) and then to deal with it.
CCOST: TBD
RECOM: Provide ACEs with remote access. This would provide provide
for more rapid initial anomaly response, and could save the
project money by only requiring that the ACE come in for real
anomalies. Bogus anomalies and some simple actions (e.g.,
clearing out disk space) could be dealt with remotely, which
would result in cost (time, travel) savings; additionally, the
ACE would be able to acknowledge a page immediately so that
others aren't also paged. A procedure would be set up outlining
those types of anomalies that could be addressed remotely
(e.g., not commanding). On-line remote access to procedures
would also be available. Overall, remote access will decrease
the initial resonse time to an anomaly; it will also be a
major morale boost for the ACEs.
RADAR: TBD
IMPCT: None. Initial response time to an anomaly condition would be
decreased.
----------------------------------------------------------------
ISSUE: Eliminate machine dependency on the ESOC network.
PR/PH: W/23
CURNT: At the heart of the current setup are two Solbourne servers --
soc2 and sdp2. Various workstations are then dependent on one
or the other servers. Certain workstations then act as servers
themselves for various peripherals (e.g., printers, SYBASE).
When a unit crashes or locks up it often affects the other devices
attached to it. When soc2 goes down, we must switch over to sdp2.
This is not only problematic, but requires that sdp2 be a virtual
clone of soc2 in terms of software, etc. This is a large
maintenance issue.
CCOST: TBD
RECOM: Implement the RAID box on the ESOC network and then move the
relevant software and accounts to it. Some reconfiguration of
the software and/or network would be required.
Such an arrangement would simplify the network configuration
by providing a single area for executables and user accounts (i.e.,
all ESOC-net ones). It would also provide redundancy in a single
system instead of the current mirror systems, would minimize
network lockups, and would greatly simplify software and systems
maintenance problems. However, a seperate minimal backup system
(see above) would still be required should the RAID system
fail for any reason.
RADAR: TBD
IMPCT: Since operators aren't available on a daily basis during zero
shift, we could lose monitoring information for days if the network
locked up or went down. The current system also requires
higher software and systems maintenance costs.
----------------------------------------------------------------
ISSUE: Automate the manual verification of proper payload
configuration for H&S and science.
PR/PH: W/23
CURNT: The duty PC/PCA currently visually verifies the proper
payload observation configuration.
CCOST: TBD
RECOM: Expand the capabilities of eworks so that it can change its
reference rules based on the current observation configuration.
For example, a Moon observation requires a different configuration
and, hence, a different rule set than does a normal observation.
One could define a series of standard observation configurations
and then code up the relevant rules for each; eworks would
then refer to the appropriate set.
Such a scheme would provide eworks some context for an
observation, so that it could change its rules based on the
payload configuration in effect (e.g., normal observation, Moon,
etc.) for a given period of time. The benefits of such a capability
are numerous. This could solve the current problems of being paged
for violating rules when in a known safe state (e.g., Scanner C
off) or at a known attitude (e.g., "hot" thermal orientations).
It could also solve the problems mentioned above regarding verifying
proper command and macro execution.
RADAR: TBD
IMPCT: None. Would continue to get paged for unnecessary reasons mentioned
above. There could also be a science hit if the wrong configuration
was in use and there's nobody there to verify it.
----------------------------------------------------------------
ISSUE: Automate the manual verification of proper payload day/night
configuration.
PR/PH: W/23
CURNT: The duty PCs/PCAs currently perform a visual check of the
pipeline output to determine that the proper detectors were
on/off at the proper times. The current implementation of
eworks doesn't really know the difference between night and
day; it only checks voltage levels for full or half and, if
true, assumes everything is OK.
CCOST: 0.1 FTE (~0.5 hr/day / 8 hr/day * 1.4)
RECOM: Redefine the rampmacros to have a mask that would indicate
which detectors should be on during day, so that eworks could
check the mask and alter its rules accordingly. Such a scheme
would notify us if a detector is on/off when it should not be.
RADAR: TBD
IMPCT: Little. There is a potential for gain loss should detectors
be on during the day portion of the orbit; conversely, there
would be a loss of science data should detectors not be a
their full level during night. Both of these scenarios, which
presume that "rampmacros" doesn't work correctly (it always
has), would not be noticed in the present configuration until
a periodic system check was made (which could be days later).
----------------------------------------------------------------
ISSUE: Automate the verification of proper payload pointing for
science.
PR/PH: W/3
CURNT: The duty PC/PCA currently uses a visual check to verify proper
observation pointing. This is only effective when the source
is actually visible in the detector plots (~50% of the time);
otherwise, it is assumed to be correct (and almost always is).
CCOST: negligible
RECOM: Implement software to periodically check for accurate payload
pointing by comparing spacecraft attitude with the science plan
(e.g., patch Keh-Cheng's timeline database code into the
pipeline).
RADAR: TBD
IMPCT: None. Could result in loss of science observing efficiency.
----------------------------------------------------------------
4.4 Secondary Wants
Definition: A secondary want is defined as an issue that is of a
purely streamlining nature that may not, at this stage of the mission,
make fiscal sense to implement; it may, however, may make sense for
other reasons (e.g., degree of innovation, etc.).
----------------------------------------------------------------
ISSUE: Minimize and streamline the network configuration and hardware
needs.
PR/PH: W/23
CURNT: The current ESOC network and hardware configuration is large
and complicated. It consists of multiple servers, workstations,
tape drives, and printers; a tape carousel; and a SYBASE server.
There are also numerous phone lines, modems, a SCAMA
connection, and the X.25 line to GSFC. There are also
workstations attached to the SDAF network. Communication
between the two networks is virtually non-existent (a security
feature), except via tape drives and floppy disks.
CCOST: TBD
RECOM: Review the network and streamline it by moving out unnecessary or
non-critical equipment. A streamlined system would also require
less facilities space and could be moved into a smaller area (e.g.,
Pat's office), freeing up the traditional ESOC space for other
uses.
Some suggested changes include the following:
* minimize computer needs -- servers, workstations, modems,
printers, etc.
* minimize support equipment -- phones, etc.
* physically separate ESOC vs. SDAF network hardware
* integrate with RAID box capabilities; recommend using full
implementation (e.g., for direct t/m transfer for
archival; would eliminate carousel)
* implement a (minimal) buffer (e.g., on the RAID box) to allow
for direct network to network communication
* replace Solbournes -- long-term maintenance issue
RADAR: TBD
IMPCT: None. This is purely a streamlining move that, in our current
thinking, is heavily dependent on the RAID box. However, such
a streamlined system would require much less in terms of
maintenance and is more in tune with today's hardware technology.
----------------------------------------------------------------
ISSUE: Expand the mechanism for information exchange between computer
networks.
PR/PH: W/23
CURNT: A minimal implementation of this mechanism currently exists that
provides for ESOC network backups (i.e., via edaf) to the
jukebox.
CCOST: n/a
RECOM: Expand this data transfer buffer (e.g., on the RAID box) to allow
for direct telemetry archival. This would free up the tape carousel
for other uses (e.g., in GO/GI support). It would also eliminate
the need for the weekly carousel maintenance and for the daily
PACOR backup.
RADAR: TBD
IMPCT: None. This is purely a streamlining activity.
----------------------------------------------------------------
ISSUE: Eliminate the manual weekly report to NASA.
PR/PH: W/2
CURNT: A weekly report to NASA is currently compiled manually.
CCOST: 0.01 FTE (0.5 hr/wk / 40 hr/wk)
RECOM: The request for this report is from NASA, not from our end and
includes very minimal information (this could be automated),
basically just targets observed and scheduled. It would take
NASA's agreement to cut this task from the daily list. On the
other hand, since this report reaches some fairly powerful
NASA officials, it could be expanded at a very low cost to
further promote CEA activities.
RADAR: TBD
IMPCT: None. This is purely a streamlining activity.
----------------------------------------------------------------
ISSUE: Eliminate the manual weekly tape carousel maintenance
activites.
PR/PH: W/23
CURNT: The duty PC/PCA is currently required to maintain the tape
carousel on a weekly basis. This basically means changing in
and out tapes to ensure there are enough blank media for the
archival of the upcoming week's telemetry, and cleaning the
tape heads.
CCOST: 0.05 FTE (2 hr/week / 40 hr/week)
RECOM: We can continue to do this as is currently since it is not
considered "shift work" under our definitions. It could (and
should?), however, be farmed out to central since it is a hardware
maintenance activity. If the RAID box implementation allows for
the telemetry to be written directly to it, then the carousel
may not be needed at all. In such a scenario it could be moved
into the GO/GI processing loop, which would speed our data
delivery efforts (of course, this would just move maintenance
to someone else since the unit would still be used). The tape
carousel is currently the weak link in the archiving chain.
RADAR: TBD
IMPCT: None. This is purely a streamlining activity.
----------------------------------------------------------------
ISSUE: Automate the scheduling of all routine commands based on standard
observation configuration files.
PR/PH: W/3
CURNT: n/a
CCOST: n/a
RECOM: Automatically schedule (e.g., via HSTS) routine payload ATC
command loads.
RADAR: TBD
IMPCT: None. This is purely a streamlining activity that would free up
personnel for other activities.
----------------------------------------------------------------
ISSUE: Streamline the pager response process by coding all pages with
anomaly identification and/or categorization information.
PR/PH: W/3
CURNT: Current paging codes only indicate status of response to page.
There is no information included regarding the type of anomaly.
CCOST: n/a
RECOM: No specific recommendations on how to implement this.
RADAR: TBD
IMPCT: None. This is a convenience measure that would give the ACE more
information in a more rapid fashion than is currently done.
----------------------------------------------------------------
4.5 Miscellaneous Tasks
The following tasks are all currently part of the daily checklist
and could be handled as follows: