spacer link to MAST page spacer logo image spacer
 
link to STScI page


HST Processing Status

This page provides our users access to information about HST data reprocessing activities. This page will document large reprocessing efforts that we expect will take more than a few days. Small sets that we expect to be completed in a few hours will not be documented below.

Reprocessing efforts are almost always triggered by changes to the reference files. Information about updated reference files and the reasons and categories for selection of datasets for reprocessing are provided by the instrument teams as part of the CRDS system that can be found here.

Current Reprocessing Status

In general, HST data available from the archive are up to date with the latest pipeline software deliveries and calibration reference files. During times when significant amounts of data need to be reprocessed to take advantage of new deliveries in future, the reprocessing status will be indicated here.

Pipeline software releases

Information is presented below about the current software being used in the processing pipelines to process HST data. This includes summaries about new features, bug fixes, and other changes from previous versions of the software.

More details about these software versions, including instructions for downloading and running the pipeline software, are available at the AstroConda Channel. For questions please contact the Archive Hotseat at: archive@stsci.edu.

Release: 2017.2aInstalled: 2017-08-02 (current release)

HSTCAL Version 1.3.0

The following updates have been included with this release.

ACS

  • CALACS updates to version 9.2.0 with this release.
  • New BLEVCORR algorithm that uses new OSCNTAB, which now correctly calculates bias level for AMP B and D in subarrays.
  • New SINKCORR step to apply new SNKCFILE in order to flag sink pixels. See ACS ISR 2017-01.
  • New pixel based PCTECORR algorithm implemented (CTE_NAME= 'PixelCTE 2017').
    • The code defaults to this new algorithm, however, the old one can still be executed using the command line option --ctegen <1|2>. Note: The correct accompanying PCTETAB file must also be used.
    • The user can now override the PCTETAB file used (normally specified in the image header) with the command line option --pctetab <file path>.
    • The number of threads used for computing the PCTECORR can now be specified with the command line option --nthreads <N>. Note: OPENMP is required for N > 1.
  • Fixed uninitialized variable in prinfo.c.
  • Corrected --version reporting and added -r to report version string that was previously reported by --version.

  • [x] Affects old data.
  • [x] Users should re-retrieve affected datasets from the archive.
  • [x] Updated reference files delivered by science team.

Notes

  • Due to the quick succession of this release from 1.2.0, please refer to the following release notes for other recent updates.

Additional software included in this update

  • nictools 1.1.3
  • stistools 1.1
  • stsci.distutils-0.3.8
  • stsci.image-2.2.0
  • stsci.imagemanip-1.1.2
  • stsci.imagestats-1.4.1
  • stsci.ndimage-0.10.1
  • stsci.numdisplay-1.6.1
  • stsci.skypac-0.9.4
  • stsci.sphere-0.2
  • stsci.sphinxext-1.2.2
  • stsci.stimage-0.2.1
  • stwcs 1.3.2

Release: 2017.2Installed: 2017-06-15

acstools 2.0.7

  • Minor fix for compatibility with Numpy 1.12 or later.

crds 7.1.1

  • JWST B7.1 first quarter, HSTDP-2017.2, new HST + JWST types

drizzlepac 2.1.14

  • Supressing aggravating astropy.wcs INFO messages

fitsblender 0.3.1.dev2

  • bugfix release

HSTCAL 1.2.0rc2

The following updates, specified in separate sections for each instrument, have been included with this release.

HSTCAL package updates

  • waf installer upgraded to 1.9.9 to support Python 3.6.
  • Added --O3 and --release-with-symbols options in build configuration.
  • Enhanced platform check by installer.
  • HSTCAL error codes were consolidated into a single global include file.

ACS

  • Fixed uninitialized variable in prinfo.c.
  • Corrected --version reporting and added -r to report version string that was previously reported by --version.

STIS

  • calstis version 3.4.1
  • fixed a bug that garbled some FITS HISTORY keywords
  • fixed a bug in the bar finding algorithm
  • fixed a bug in processing highres echelle data in which the error and dq arrays were incorrectly calculated

    No keyword changes.

    The only science changes will be to longslit spectroscopic data and high resolution echelle data.

  • [x] Affects old data
  • [x] Users should re-retrieve affected datasets from the archive
  • [x] updated reference files delivered by science team

WFC3

  • calwf3 updates to version 3.4.1 with this release

Photometry Keyword Updates

  • There are default description strings in calwf3 which are used when the keyword is being added to a header where it doesn't already exist. This update makes the text consistent among all UVIS headers, including the global header keywords that calwf3 adds for user convenience that are not populated in the raw headers during conversion. See issue #67 for more details. The table below details the changes to the photometry header keywords:
Detector Extension Keyword Previous Updated
UVIS 0,1,4 PHOTCORR populate photometric header keywords NO CHANGE
UVIS 0,1,4 FLUXCORR convert to absolute flux units Scale UVIS2 to match UVIS1 using PHTRATIO
UVIS 0 PHOTFLAM inverse sensitivity Inverse sensitivity, ergs/cm2/A/e-
UVIS 1,4 PHOTFLAM Inverse sensitivity, ergs/cm2/A/e- NO CHANGE
UVIS 0 PHOTZPT zero point ST magnitude zero point
UVIS 1, 4 PHOTZPT zero point ST magnitude zero point
UVIS 0 PHTFLAM1 photometry scaling for chip 1 Ch1 Inv Sens, use PHOTFLAM for UV filters
UVIS 1,4 PHTFLAM1 PHOTFLAM on chip1 when scales are not uniform Ch1 Inv Sens, use PHOTFLAM for UV filters
UVIS 0 PHTFLAM2 photometry scaling for chip 2 Ch2 Inv Sens, use PHOTFLAM if FLUXCORR=COMPLETE
UVIS 1,4 PHTFLAM2 PHOTFLAM on chip2 when scales are not uniform Ch2 Inv Sens, use PHOTFLAM if FLUXCORR=COMPLETE
UVIS 0 PHTRATIO photometry scaling for chip2 to chip 1 PHTFLAM2/PHTFLAM1 ratio
UVIS 1,4 PHTRATIO Ratio of PHTFLAM1/PHTFLAM2 PHTFLAM2/PHTFLAM1 ratio
UVIS 1,4 PHOTMODE observation con Observing configuration
UVIS 1,4 PHOTFNU inverse sensitivity, Jy*Sec/electron Inverse sensitivity, Jy*sec/e-
UVIS 1,4 PHOTPLAM pivot wavelength NO CHANGE
UVIS 1,4 PHOTBW RMS bandwidth NO CHANGE
  • [x] Affects old data, all UVIS data will be affected since the keywords are commonly populated
  • [x] Users should re-retrieve affected datasets from the archive for updated header keywords
  • [x] Affects full-frame and subarray UVIS datasets for all filters

  • Code cleanup specific to issue #55 and issue #63; these do not change any data results

Updated reference files which will be part of the reprocessing

This software update itself does not affect data values, just header keyword descriptions. However, the team will be delivering updated reference files at the same time which will be used during reprocessing with calwf3 version 3.4.1. The following reference files will be delivered and used by the archive for reprocessing after this software change has been installed.

Keyword Description Applied Date Range for the new file (USEAFTER) selection keywords Content update or format change?
IMPHTTAB photometry table used by calwf3 All UVIS data DETECTOR = UVIS, PHOTCORR = PERFORM Update
BIASFILE bias reference file All UVIS data DETECTOR = UVIS, BIASCORR = PERFORM Update
FLSHFILE post-flash reference file July 2012 - present DETECTOR = UVIS Update

Reprocessing will begin after the delivery of the reference files and those will affect header keyword values and data for the following:

  • IMPHTTAB: only affects UVIS data which has PHOTCORR = PERFORM, this excludes calibration files like darks and biases
  • BIASFILE: only affects UVIS data which is corrected for bias, including subarrays, except for charge injected data, data which has been read out using a single amplifier, or data from modes which are no longer supported. The science team also plans to begin delivering one bias per year.
  • FLSHFILE: applies to all data taken with post-flash

pysynphot 0.9.8.7

  • Fixed tagging issue for "git describe" and failed tests.
  • No change to user-facing interface otherwise.

reftools 1.7.4

  • bugfix release

stsci.tools 3.4.9

  • Multiple compatibility improvements

stsci.convolve-2.2.1.dev0

  • bugfix release

Additional software included in this update

  • calcos 3.2.1
  • nictools 1.1.3
  • stistools 1.1
  • stsci.distutils-0.3.8.dev0
  • stsci.image-2.2.0.dev0
  • stsci.imagemanip-1.1.2.dev0
  • stsci.imagestats-1.4.1.dev0
  • stsci.ndimage-0.10.1.dev0
  • stsci.numdisplay-1.6.1.dev0
  • stsci.skypac-0.9.4.dev0
  • stsci.sphere-0.2.dev0
  • stsci.sphinxext-1.2.2.dev0
  • stsci.stimage-0.2.1.dev0
  • stwcs 1.3.2rc1

Release: 2017.1Installed: 2017-03-16

HSTDP delivery 2017.1 contained no updates to calibration software, this was an internal update from RHEL5 to RHEL 7

Release: 2016.2Installed: 2016-10-26

ACSTOOLS v2.0.0

The following changes were made since the last release in April 2014:

  • Removed standalone PixCteCorr task. Use ACSCTE step from HSTCAL instead to have parallelization and consistent results with the CALACS pipeline.
  • Removed runastrodriz task (moved to drizzlepac).
  • Added masking, post-flash processing, iterative cleaning, and alternative statistics to acs_destripe task. Masking is particularly useful for polarized data.
  • Fixed bugs in acs_destripe task, including proper error handling and pixel weighting.
  • Added new acs_destripe_plus task, which is particularly useful for destriping subarrays with acs_destripe in the middle of CALACS processing (between ACSCCD and ACSCTE steps).
  • Added new satdet module, which can detect and flag satellite trails.
  • Added Python 3 support (untested).

HSTCAL 1.1.0 Release Notes

The following changes were made to the HST calibration software sent to HST Data Processing for build 2016.2:

General

  • waf upgraded to 1.9.4, which enables building HSTCAL in Python 3, as well as Python 2.

Packages

  • WFC3 (increases to version 3.4)
    • CTE subarrays are now enabled for subarrays which include physical overscan pixels.
    • Sink Pixel flagging for all subarray images has been enabled.

Drizzlepac 2.1.3

This version represents the same code, with a few additional enhancements, as the code that was installed for operational calibration of HST data as of 23 Feb 2016. This represents a major update to the DrizzlePac software, and also the last version with significant new features. Future releases will focus primarily on bug fixes that affect operational use of this code. Full support for this package can be found at http://drizzlepac.stsci.edu.

A brief overview of the new features and major bug fixes found in this version (relative to what was released as part of v1.1.16) includes:

  • Full Python 2.7 and 3.5 support
  • Built-in support for automatic mosaic creation
    • Tweakreg now aligns all images in a mosaic into a single undistorted output frame even if some images do not overlap others in the mosaic
  • Improved sky matching
    • Produce seamless mosaics using new sky matching techniques. More details can be found in an example where these techniques are compared.
  • Support for the improved time-dependent ACS distortion model
    • Supports the new ACS distortion calibration: Only DrizzlePac 2.0 is able to interpret and apply the latest ACS/WFC distortion reference files
  • Support for alignment of data from different HST cameras
    • Specify separate source finding parameters for input and reference images to optimize source detection from images taken with different HST cameras.
  • Support for use of inclusion/exclusion regions in image alignment
  • Improved support for WFPC2 data
    • resolved problems processing WFPC2 data which had DGEOFILEs specified.
    • now requires user to run 'updatewcs' task on WFPC2 data to enable astrodrizzle and tweakreg to work with them seamlessly
  • [API change] Use of 'updatewcs' removed from TEAL interfaces
    • User and pipeline will need to run this task independently prior to running astrodrizzle or tweakreg
    • Python scripts calling astrodrizzle and tweakreg can still set the 'updatewcs' parameter and have it run as part of those tasks (presumably, the user understands when this task will wipe out previous updates in their own script)
  • [API change] The user interfaces to all 3 coordinate transformation tasks now use 'coordfile' as the input file of coordinates to transform. The use of 'coords' has been deprecated, but still can be used if needed. However, use of 'coordfile' will always override any input provided simultaneously with 'coords' parameter. Help files have been updated to document this as clearly as possible for users.
  • Now relies on astropy for WCS, coordinate specification and I/O libraries
  • A full 6-parameter general linear fit can now be performed using tweakreg, in addition to shift and rscale
    • WCS keywords updated by tweakreg will result in an undistorted output frame with NO residual skew (as represented in the CD matrix keywords)

The full set of Release Notes can be found at http://ssb.stsci.edu/doc/stsci_python_x/drizzlepac.doc/html/release_v2_0_0_notes.html.

fitsblender 0.2.6

This version of fitsblender supports the release of drizzlepac v2.1.3 as used in the operational HST calibration pipeline and archive as of 23 Feb 2016. It primarily includes bug fixes; namely,

  • Fixed problem with random results from fitsblender by replacing use of dict with OrderedDict. This problem resolves issues with which image was interpreted as first and last when looking for values to use to populate the combined header.
  • Default pipeline processing rules files for all instruments amended to reset FLASHCUR header value to 'multi' instead of first.
  • Simple update to insure that any keyword deletion works cleanly with astropy by trapping any KeyError? exceptions explicitly.
  • Replace use of pyfits with astropy.io.fits.
  • Now works as-is under Python 2.7 and Python 3.5

pysynphot 0.9.8.2

The following changes were made since the last PyPi release (v0.9.7) in October 19, 2015:

  • Updated spectra data including ACS wavecat and Vega reference spectrum.
  • Replaced PyFITS dependency with astropy.io.fits.
  • Added a lot of documentation and tutorials.
  • Bug fixes.
  • Python 3 support (untested).

wfpc2tools 1.0.3

The following changes were made since the last release in April 2014:

  • Replaced old stsci.* dependencies with SciPy.
  • Added Python 3 support (untested).

Other software included in this release:

  • calcos 3.1.8
  • costools 1.2.1
  • crds 7.0.1
  • nictools 1.1.3
  • wfc3tools 1.3.1
  • wfpc2tools 1.0.3
  • stsci.distutils-0.3.8.dev0
  • stsci.image-2.2.0.dev0
  • stsci.imagemanip-1.1.2.dev0
  • stsci.imagestats-1.4.1.dev0
  • stsci.ndimage-0.10.1.dev0
  • stsci.numdisplay-1.6.1.dev0
  • stsci.skypac-0.9.4.dev0
  • stsci.sphere-0.2.dev0
  • stsci.sphinxext-1.2.2.dev0
  • stsci.stimage-0.2.1.dev0