SIMPLE-WIRCAM:
SIMPLE Imaging and Mosaicking PipeLinE for WIRCAM
Wei-Hao Wang
(Academia Sinica Institute of Astronomy and Astrophysics)
SIMPLE-WIRCAM version:  1.3
Document version: 2.0
bug report and feedback:  whwang at asiaa. sinica. edu. tw

1. Introduction


The SIMPLE Imaging and Mosaicking Pipeline (SIMPLE) is an IDL based system designed for galactic/extragalactic imaging where there are no extended objects (i.e., blank fields).  It generates reasonably good astrometry and photometry.  It is also capable of making large mosaics of many sensor areas, while still maintaining the astrometry accuracy.  SIMPLE can be used on different data after simple modifications.  This particular distribution (SIMPLE-WIRCAM) is optimized for WIRCAM on the CFHT.  Detailed descriptions about the algorithms used in SIMPLE can be found in this document.

Running this package requires the most recent version of the IDL Astronomy User's Library, the package SExtractor, match, and funpack:

Recent x86 and SPARC CPUs with GHz speeds are all fast enough.  However, this package uses lot of memory to speed up the processing.  At least 2 GB of RAM is recommended for basic reduction.  For making large mosaics and cleanly removing cosmic rays, more than 4 GB of RAM and thus a 64 bit system are required.

1.1 Flat Field and Background Subtraction

Sky flat is generated from median-combining the dithered images.  By masking detected objects and with a better treatment to frame-to-frame fluctuation of sky background, artifacts caused by bright objects is minimized.  Furthermore, each image has its own flat-field image, created from other images (except itself) in the dither sequence.

It is also possible to use dome flat or twilight flat.  The standard background subtraction in SIMPLE-WIRCAM is to subtract a smooth surface.  However, SIMPLE-WIRCAM is also equipped with median sky background subtraction as an option.

1.2 Distortion Correction and Absolute Astrometry

Reduced images are warped and resampled to project onto a sky plane with sub-pixel accuracy.  Distortion correction uses all detected objects in the dithered images, and does not rely on any external information.  This is important for WIRCAM since its small FOV does not always have enough cataloged stars uniformly distributed across the FOV, especially at high galactic latitude.  Absolute astrometry is obtained by comparing the image with a reference catalog (can be from NOMAD or SDSS or 2MASS), and only a relatively small number of objects are required.  For data taken under <0".4 seeing and with a dense source catalog of good astrometric accuracy, the final astrometric accuracy across one WIRCAM FOV can be as good as 0".08 rms relative to the input catalog.  Throughout the reduction, astrometry in the header follows the FITS standard described in Calabretta & Creisen (2002, A&A, 395, 1077).


1.3 Photometry

WIRCAM images are usually calibrated with 2MASS stars in the FOV (see 3.1.2).  The WIRCAM photometry is consistent with independently reduced Subaru photometry across 200 arcmin2 in the GOODS-N.  The current version does not keep records of saturated pixels, so aperture photometry on bright stars are not reliable.  The current version has functions for handling data taken under nonphotometric conditions.

1.4 Wide-Field Mosaicking

User can specify the coordinate system (projection center on the sky, rotation angle, and plate scale) for the reduced/combined images.  A tangential projection is adopted, but other projections can be implemented by the users easily.  Images with the same coordinate system can easily be combined to form even larger mosaics accurately.  The final size of the mosaic is only limited by available RAM.  Exposure time on each pixel is always traced and recorded so the map can be deepened or widened easily.

1.5 Cosmic Ray Removal

Two methods are provided for cosmic ray removal.  The brightest and most obvious cosmic ray hits are removed from individual exposures with a sigma filter of 5x5 pixels.  Fainter cosmic ray hits are removed by applying a sigma filter to pixels that have the same sky position in a dither set.

1.6 Image Weighting

Basic image weighting in SIMPLE-WIRCAM is to weight each image according to its exposure time.  An optional weighting method is provided to weight each pixel according to sky transparency (cloud extinction and airmass effect), pixel quantum efficiency, sky brightness, and exposure time.


2. Important Procedures

Below highlights the main procedures in this package that are required to complete a full reduction, several procedures that may need some manual tweak, and other useful procedures.  Please see the explanations in the procedure files for details.

2.1 reduce_wircam

The main procedure in SIMPLE-WIRCAM is reduce_wircam.  It deals with images within a dither set from the same chip.  It produces a flattened, sky subtracted, defringed, cosmic ray removed, and distortion and astrometrically corrected mosaic image, as well as an exposure time map.  The reduced images and the exposure time maps can be used later to form large mosaics.  This procedure is controlled by several keywords that need to be provided in the IDL command line, and by external parameters that are stored in an ASCII parameter file (APF).  An example of APF is provided as the file HDFN.para under the pipeline directory.

2.2 chip_offset

This procedure derives the sensitivity offset between the two chips.  It is recommended to run this procedure at least once per observing run, although the chip offset does not seem to change significantly from run to run.  See field #9 in the APF for reduce_wircam.

2.3 calibrate

When there are enough objects in an image with known photometry, this procedure can be used to calibrate the image with the existing photometric catalog.  This is integrated into the standard pipeline reduce_wircam and the mosaicking function mosaic_wide, but can be also used as a standalone function.

2.4 mosaic_wide

Images reduced by reduce_wircam can be combined to form large mosaics or deeper images with mosaic_wide.  In order to make the images combinable by mosaic_wide, they have to have the same WCS.  This is achieved by using identical projection parameters in fields #4 through #7 in the APF of reduce_wircam.  Photometric calibration can be applied at this stage, using an existing photometric catalog (most likely 2MASS).  It handles images taken under non-photometric conditions.  SIMPLE-WIRCAM uses Jy (or similar linear units) as the default zero point.  For example, a zero point of 1 uJy per data unit corresponds to 23.9 magnitude in AB system.

2.5 read_wircam

This is the subroutine called by the main procedure that reads in a set of dithered WIRCAM images.  In the past, this procedure was sometimes useful for trouble shooting. In the current version of SIMPLE, it can be just considered as a hidden subroutine.

2.6 fz2fits_wircam

The SIMPLE-WIRCAM pipeline starts with compressed raw data provided by CFHT.  fz2fits_wircam is the first procedure in the pipeline, to uncompress the raw data.  Many processes in the pipeline rely on the particular format and information generated by fz2fits_wircam.  So please let fz2fits_wircam help you and do not uncompress the files by yourself. Despite the name fz2fits, it also deals with gzipped WIRCAM data. In earlier versions, it requires the installation of CFITSIO to deal with the RICE compression adopted by CFHT. Now the users can just download the funpack binary to the pipeline directory without worrying about CFITSIO.

2.7 summary_wircam

This procedure can generate a simple observing log.  It may help you to decide what files should be grouped together and feed to the main procedure.

2.8 chk_astrometry

This procedure tells you the accuracy of astrometry in a FITS image, useful for verifying astrometry.

2.9 compare_images

This procedure compares fluxes of detected objects in two FITS images, useful for verifying photometry.

2.10 remove_satellite_wircam

If there are satellite tracks, this procedure can remove them from the raw image. 




3. Reduction Steps and Trouble Shooting

After the necessary preparation, the reduction can be subdivided into two major steps: reduction of the target images, and combining multiple reduced images into a large mosaic.  An example script for a set of standard reduction is provided in the file script.example.  If successfully executed, the example script should generate an image like this.  Below I describe some details and the associated trouble shooting.

3.1 Preparation

3.1.1 Astrometric Reference Catalog

In order to obtain absolute astrometry, one needs to prepare a reference catalog that contains enough objects with good astrometry.  This can be a NOMAD catalog (or USNO-B1), which provides minimum astrometric accuracies and numbers of stars.  The procedure recompile_nomad.pro can be used to convert a NOMAD catalog into the required format.  See recompile_nomad.pro for details.  Additional to NOMAD, another good source is SDSS.  Indeed, users can use any source catalog (2MASS, Chandra, VLA, Spitzer, another deep ground-based image that has good astrometry, or whatever) and SIMPLE will force the reduced image to match the provided catalog with high accuracies.  (This also means that if the provided catalog is problematic, the results of the reduction will also be problematic.)  In general, on average more than 10 sources per WIRCAM chip are required for OK astrometry calibration.  As many as 1000 per chip will not hurt.  It is also important to make sure the distribution of the sources in the reference catalog is as uniform as possible.  See fit_distortion.pro or cross_images.pro for the format of the reference catalog.

3.1.2 Photometric Reference Catalog

To calibrate photometry, one needs to provide a photometric catalog that contains a sufficient number of stars with known fluxes in the field of view of the observations.  Usually, this is the 2MASS catalog.  See calibrate.pro for the format of this catalog.  Note that the useful magnitude range in 2MASS could be very small.  In the 2MASS bright end, the WIRCAM can be nonlinear or even saturated.  In the faint end, the 2MASS magnitude is biased by selection effect.  My experience shows that the useful magnitude range is 14--16 (Vega) for J and 12.5--14.5 for Ks.  The users may want to double check this by themselves. 

By the way, SIMPLE does not like magnitudes.  The reference catalog should use linear units for flux (e.g., uJy or Jy) instead of magnitude.  See explainations in calibrate.pro.  In the example directory HDFN/CATALOG, the procedure recat_2mass.pro shows how I convert a 2MASS catalog to the flux calibration catalog required by SIMPLE.

3.1.3 Reduction Parameter

The next preparation will be an ASCII file that stores the reduction parameters (the APF), such as coordinate system, methods of flat-fielding and background subtraction, treatment of cosmic ray, and many others.  The filenames of the astrometric and photometric reference catalogs mentioned above will be entered in the APT.  An example and explanations are provided in the file HDFN.para.  Note that if you wish to combine images from different reductions to form a big mosaic, the astrometry related fields (4 through 7) should be kept identical in all reductions.

3.1.4 File List

There have to be ASCII files that tell the reduction program what files to process.  The reduction procedure is designed to process images taken by the same chip within one dither sequence at once.  A file list contains the filenames of the first exposure and the last exposure, and reduce_wircam reduces all files between these two at once, chip by chip.  See read_wircam.pro for more details about the file list.  Example file lists are provided under the directory lists.  The procedure summary_wircam (§2.7) may help you preparing the file lists. 

When preparing the file lists, it is highly recommended to only reduce sequences of images taken within ~0.5 hr together.  This is because the color of the near-IR sky varies with time and usually only images taken within ~0.5 hr have sufficiently similar background color.  Keeping each reduction sequence shorter than 0.5 hr helps to get good flat field and background subtraction.  For example, the actual observation may contain one 50-minute dither sequence of 20 exposures.  In this case, it is better to prepare two file lists, each containing only 10 expousres, and execute reduce_wircam on each of the file list.  On the other hand, if the dither sequence is too short, it may not contain enough number (> 5 is highly recommanded) of exposures to generate good sky flats. In such a case, one may want to consider using dome flat or twilight flat.

3.1.5 Chip Sensitivity

This is optional.  There is a weight keyword in the main reduction routine reduce_wircam.pro.  This keyword allows the users to weight each pixel according to its quantum efficiency (and other factors such as sky brightness).  To derive the relative QE of the four chips (the relative QE of the pixels will be taken care by the flat field), put 0 in field 9 in the APF, and reduce a set of dithered images on all four chips.  (The results of the reduction is not needed and can be deleted later.)  Then use the procedure chip_offset (§2.2) to derive the sensitivity offset of the four chips, and put back the chip offset filename in field 9 in the APF. 

Note that the users can still turn on the weight keyword without measuring the relative sensitivity of the four chips.  In this case, data from the four chips will be weighted equally.  This does not affect flux calibration.


3.2 Target Reduction

Once the above preparations are done, reducing the images should be fairly easy and automatic, using the main procedure reduce_wircam (§2.1).   Examples are provided in the script script.example.  Occasionally there are errors.  The common problems at this stage are:  wrong astrometry, imperfect background subtraction, and crosstalk.  It is always good to set the keyword individual=1 in the main procedure and this should help the diagnosis in most cases.  (You can remove the unwanted files later.)  Below describes each case.

3.2.1 Problem in Astrometry

How do we know there is a problem in astrometry?  During the reduction, the main procedure will tell you what is the rms astrometry error relative to the reference catalog.  It is a function of seeing and brightness of objects in the field, as well as the quality of the astrometry reference catalog.  Usually the error should be well within 0.5 pixel, sometimes around 0.5 pixel if the seeing is bad or the quality of the reference catalog is not too great.  Any rms error larger than that indicates something might be wrong in astrometry.  If the reduced image is warped into a very strange shape, something is definitely wrong.  (This is rare now.) You can also write the reference catalog into a ds9 reg file and overlay the catalog objects on the reduced image in ds9.  (This could be done with the procedures nomad2ds9 or write_ds9reg provided with this package.  See details therein.)  If the catalog objects do not match the objects in the reduced images, something is wrong.

In a few cases, astrometry problems are caused by the low quality of the reference catalog, such as not having enough stars.  This can usually be solved by decreasing the degree of fitting in determining the absolute astrometry, by setting the keyword fit_order=2 or even 1 (default is 3) in reduce_wircam.   One should try this first.  

There may be other problems.  Always use the individual=1 keyword in the main procedure and look at the reduction of individual exposures.  Chance is that you will spot a problem there.

3.2.2 Imperfect Background Subtraction

Sometimes you will see residual background in the reduced image, especially around the image corners.  Set individual=1 to inspect individual images and identify the frames with background subtraction problems.  Then use the keyword more_bg_subtract in the main procedure to specify these frames and to perform a more aggressive background subtraction.  This should give you a very flat image background.  This will also produce artifacts on very bright and saturated stars.  (Who care about these stars anyway?)  The real caveat is, the entire package is designed to reduce images that only contain faint galaxies.  So are the background subtraction routines.  The more_bg_subtract keyword is very aggressive and is very likely to produce inaccurate photometry on large galaxies (larger than ~5").  This will be a problem for observations of low-redshift clusters, for example.  For high-redshift blank-field imaging, there is less to worry about. Anyway, the suggestion is to use this keyword only if you really feel it is necessary. There are cases where the background look really bad in a few exposures. This can be caused by some rapid varying sky condition. The more_bg_subtract keyword may help to reduce the problem, but I would suggest to just leave out these images (remove their raw files).

3.2.3 Crosstalk (xtalk)

WIRCAM chips have xtalk between each of the readout channels, on data taken before early 2008.  Set dxtalk=1 (default) in the reduction routine will remove xtalk quite cleanly.  The de-xtalk feature in SIMPLE removes the 32-channel xtalk and guidebox xtalk.  It only removes the 8-channel xtalk if a list of bright stars (that produce observable 8-channel features in deep integrations) is provided in field 14 in the APF.  If the 8-channel xtalk features appear in very deep integrations, more aggressive de-xtalk can be used by setting dxtalk=2 or even 3.  This usually removes xtalk cleanly even in images as deep as ~30 hr. Recent (post-2009) WIRCAM images are relatively clean in xtalk.

3.4 Flux Calibration and Mosaicking

The final step is to put all single-chip, SIMPLy reduced images together, to form a larger or deeper mosaic image, using the procedure mosaic_wide (§2.3).  To achieve this, there are two requirements.  First, all images to be combined have to have identical world coordinate systems, including center of projection, platescale, and rotation.  This can be done by keeping fields 4 through 7 unchanged in the APF throughout the SIMPLE reduction.  Second, the zeropoint difference between the images have to be taken into account.  This is done automatically by mosaic_wide if you can provide it the zeropoint of each image through the flux_conv keyword.  If the images are already calibrated in the main reduction stage (see 3.1.2), then there is no need to worry about this here.


4. Changes in v1.3

Major changes in v1.3 are astrometry and flat fielding. The new version is much more robust and more accurate in astrometry. Failure in astrometry occurs much less often, although when there is a failure, it is quite difficult to correct. In the past, a flat field image is generated by median-combining all images in a dither sequence. In this version, N flat field images are generated for a dither sequence of N exposures. Each image has its own flat field image, created by median-combining all the other images in the dither sequence. This reduces the sysmatics in the photometry of faint objects. There are minor changes in v1.3 throughout the pipeline, all contributing to its robustness.

5. Limits and Possible Future Improvements

The current version of SIMPLE-WIRCAM uses tangential projection for the reduced maps.  However, it should be fairly easy to include other kinds of projection, if needed. 

The main procedure is specifically designed for WIRCAM in many ways.  However, many SIMPLE subroutines are written as general as possible.  It is possible to modify this package for other near-IR mosaic cameras.  With appropriate treatment for flat fielding and defringing, I believe it can be used on CCD cameras as well.  SIMPLE has not been tested for cameras with large distortions, such as the SuprimeCam on Subaru.  This will be the next SIMPLE development.

Currently SIMPLE cannot drizzle images.  A drizzle function can be easily written for SIMPLE but I do not feel a need for this.  If I do receive considerable amount of request from users, I may provide this function in the future.

6. Acknowledgment

When publishing data processed by SIMPLE, please refer to the paper Wang et al. (2010, ApJS, 187, 251).