3 Data Quality Flag Description

 

For the creation of the IUE Final Archive, data quality ($\nu$) flags are provided on a pixel-by-pixel basis for the two-dimensional (2-D) photometrically corrected (LI) and geometrically resampled (SI) output files, and as a function of wavelength in the one-dimensional merged extracted image (MX) file. These quality flags denote exceptional conditions in the data which can range from fairly minor situations of telemetry dropouts in the background regions to quite serious conditions of telemetry dropouts in extracted spectral regions. The $\nu$ flag values have been apportioned such that the more serious conditions have more negative values in order to provide an immediate indication as to the severity of the problem condition. In contrast to the method of ``epsilon'' flagging as implemented in IUESIPS, where only one problem condition is noted (i.e., the most severe), the $\nu$ flags are encoded to indicate all problem conditions associated with each pixel or wavelength bin.

The flexibility of the $\nu$ flags is derived from the bit-encoding of the individual problem conditions. Using a total of 16 bits, the flags are stored as negative values in two's complement form (bit 16 contains the sign), and the remaining 15 bits are utilized to represent each defined problem condition. Table 3.1 describes the problem conditions and defines the corresponding $\nu$ flag values. Once all problem conditions are identified for each pixel or wavelength bin, the individual flag values are added together to produce a total value which is a unique combination of its components. Since additional $\nu$ flags are available for the Final Archive dataset, and some problem conditions may have an altered definition when compared to the IUESIPS $\epsilon$flags, a more detailed definition of the $\nu$ flags is in order. The descriptions of the $\nu$ flags in this chapter are ordered according to their generation in NEWSIPS processing rather than by severity. Flags produced by the raw image screening process ( RAW_ SCREEN ) are mentioned first, while the flags introduced during the extraction phase (i.e., SWET or BOXCAR ) are discussed last.

 

 

$$\nu$$ Condition BIT

Pixels not photometrically corrected -16384 15

Missing minor frame in extracted spectrum -8192 14

Reseau (in the ITF) -4096 13

Permanent ITF artifact -2048 12

Saturated pixel (w.r.t. ITF pixel DN) -1024 11

Warning track (near edge of PHOTOM region) -512 10

Positively extrapolated ITF -256 9

Negatively extrapolated ITF - far below ITF level 1 -128 8

RAW_ SCREEN cosmic ray/bright spot -64 7

SWET cosmic ray (low disp. only) -32 6

Microphonics (LWR only) -16 5

Potential DMU corrupted pixel -8 4

Missing minor frame in extracted background -4 3

Uncalibrated data point (MX data only) -2 2

No known problem condition 0  1

The RAW_ SCREEN  cosmic ray/bright spot detection algorithm (Chapter 4.1 ) is an application of a median filtering technique and identification of the spots is based upon their limited spatial extent and unusual brightness. This flag (-64) is differentiated from the low-dispersion extraction (SWET) cosmic ray flag (-32) which is generated by a sigma rejection criterion discussed below.

While the NEWSIPS algorithms used to detect cosmic rays/bright spots and microphonics (-16) are similar to the IUESIPS versions, the flagging of missing minor frames (MMFs) is a new implementation. Clearly, it is important to know of the existence of a MMF in the vicinity of spectral information in order to evaluate the extent to which the MMF will affect the data. All telemetry dropouts are flagged in 2-D images as MMFs, but it is important to translate this information into how these dropouts affect the extracted spectral data and background. As a result, MMF pixels can be assigned either of two $\nu$ flag values depending on their proximity to the spectral data. The first flag (-8192) is utilized to label missing pixels in the spectral data. The second flag (-4) denotes missing pixels in the extracted background. Since the background is fit with a Chebyshev polynomial, missing pixels in the extracted background potentially play a very small role in the computation of the net flux. The much smaller absolute value of the second flag reflects the less crucial nature of the condition. The MMF detection algorithm is described in Chapter 4.4.

A recent addition to the $\nu$ flags is the one that specifies pixels which are most likely affected by DMU corruption. This flag (-8) is allotted to pixels via a statistical process in ( RAW_ SCREEN ) and only applies to images taken after October of 1994. Information related to the detection of DMU corrupted pixels can be found in Chapter 4.5.

During the photometric correction (PHOTOM) stage of processing, extrapolations required for the conversion from data number (DN) to flux number (FN) are appropriately flagged as being either positive (DN above ITF Level 12) or negative (DN far below ITF Level 1). Positively extrapolated pixels are flagged with -256 while pixels with excessive negative extrapolations are given a flag of -128. Refer to Chapter 6.3.2 for details concerning the definition of what constitutes extrapolated data.

Some of the flags, or more properly, the pixel locations of the flags, have been pre-defined based upon the location of the condition in the ITF images. For example, the positions of all permanent artifacts (-2048) and reseaux (-4096) are defined according to their positions in the ITF images and are flagged after the science image and the ITF are properly registered. The PHOTOM region is based upon the shape of the illuminated target region of the ITF; this region is nearly a circle for the SWP camera, but resembles more of a flattened circle for the long wavelength cameras. Pixels located outside the PHOTOM region are given a $\nu$ flag value of -16384. The low-dispersion PHOTOM swath is a band contained within this pseudo-circular region, optimally positioned about the low-dispersion spectrum. In most instances, the photometric correction of an individual pixel depends somewhat upon its neighbors. As the edge of the PHOTOM region is approached, there can be wild fluctuations in the DN (and hence the resulting FN) values, so that the photometric correction may be less reliable in these regions. Therefore, a warning track flag (-512) is defined to designate these less reliable flux values. The warning track is an approximately five-pixel-deep buffer zone on the inside edge of the PHOTOM region.

The saturation level for each pixel within the PHOTOM region has been obtained by examining the ITF curve of each pixel and determining the DN associated with any substantial leveling off of the ITF curve (DN vs. exposure time) with increased intensity. According to the analysis of the ITF images, there are a number of pixels which ``saturate'' at values much less than 255 DN. Ergo, many more pixels may be considered saturated by the NEWSIPS system than IUESIPS, since in IUESIPS only pixels with a DN of more than 250 are deemed to be saturated. Saturated pixels are issued a $\nu$ flag value of -1024.

SWET examines the fitted profile values in the cross-dispersion direction in each wavelength bin and compares the actual data to the profile, rejecting those values deviating by more than ``N'' sigma and assigning a flag value of -32. The SWET cosmic ray detection is not applicable for high-dispersion data, as a boxcar extraction is applied in that case. A description of the SWET cosmic ray removal process is given in Chapter 9.5.

The uncalibrated data flag (-2) is found only in the final extracted spectrum and indicates wavelength regions which have been extracted from the resampled image, but whose absolute flux calibration has not been defined.

Because the $\nu$ flags are stored as negative integers, a warning on the correct interpretation of these data is necessary. Negative integers are stored in two's complement form by most computer systems. This means that the internal bit settings for negative values are calculated by subtracting the desired value from 2¹⁵ or 2³¹, for 2 or 4 byte integers respectively. Accordingly, many more bits are turned on (set to 1) to represent the negative integer than just the bits for the actual digit. Consequently, if intrinsic bit decoding functions are used to decode the $\nu$ flag values as is, the wrong answer will be derived. It is most straightforward to first take the absolute value of the $\nu$flag image or vector before attempting to interpret the data.