IDL MOIRCS Reduction Pipeline

SIMPLE-MOIRCS:
SIMPLE Imaging and Mosaicking PipeLinE for MOIRCS

Wei-Hao Wang
(National Radio Astronomy Observatory)

SIMPLE-MOIRCS version: 1.1
Document version: 1.2

bug report and feedback: whwang at aoc. nrao. edu

The SIMPLE Imaging and Mosaicking Pipeline (SIMPLE) is an IDL based system designed for galactic/extragalactic imaging where there are no extended objects (i.e., blank fields). It generates reasonably good astrometry and photometry. It is also capable of making large mosaics of many sensor areas, while still maintaining the astrometry accuracy. SIMPLE can be used on different data after simple modifications. This particular distribution (SIMPLE-MOIRCS) is optimized for the MOIRCS camera on the Subaru Telescope. Detailed descriptions about the algorithms used in SIMPLE can be found in this paper.

To run this package, the most recent version of the IDL Astronomy User's Library and the package SExtractor are required to be pre-installed. The SExtractor will be called from command line with the command "sex." Recent x86 and SPARC CPUs with GHz speeds are all fast enough. However, this package uses lot of memory to speed up the processing. At least 2 GB of RAM is recommended for basic reduction. For making large mosaics and cleanly removing cosmic rays, more than 4 GB of RAM and a 64 bit system are required. The 32 bit IDL cannot locate more than 2 GB of RAM, so the 64-bit requirement is not just to speed up the processing.

1.1 Flat Field and Background Subtraction

Sky flat is generated from dithered broad-band images. By masking detected objects and with a better treatment to frame-to-frame fluctuation of sky background, artifacts caused by bright objects is minimized. It is also possible to use dome flat or twilight flat. The standard background subtraction in SIMPLE-MOIRCS is to subtract a smooth surface in each quadrant of the HAWAII array. However, SIMPLE-MOIRCS is also equipped with median sky background subtraction.

1.2 Distortion Correction and Absolute Astrometry

Reduced images are warped and resampled to project onto a sky plane with sub-pixel accuracy. Distortion correction uses all detected objects in the dithered images, and does not rely on any external information. This is important for MOIRCS since its small FOV does not always have enough cataloged stars uniformly distributed across the FOV, especially at high galactic latitude. Absolute astrometry is obtained with objects in the NOMAD catalog or any source catalog, and only a relatively small number of objects are required. For data taken under <0".4 seeing and with a dense source catalog of good astrometric accuracy (such as the GOODS ACS catalog), the final astrometric accuracy across one MOIRCS FOV can be as good as 0".03 rms relative to the input catalog. With NOMAD catalogs, the typical astrometric accuracy is 0".06 to 0".1 rms, which is the limit of the NOMAD catalog itself. Throughout the reduction, astrometry in the header follows the FITS standard described in Calabretta & Creisen (2002, A&A, 395, 1077).

1.3 Photometry

A standard-star procedure can be used to generate zero points from standard star observations. The MOIRCS photometry seems consistent with independently reduced WIRCAM photometry across 200 arcmin² in the GOODS-N. The current version does not keep records of saturated pixels, so aperture photometry on bright stars are not reliable. The current version has functions for handling data taken under nonphotometric conditions.

1.4 Wide-Field Mosaicking

User can specify the coordinate system (projection center on the sky, rotation angle, and plate scale) for the reduced/combined images. A tangential projection is adopted, but other projections can be implemented by the users easily. Images with the same coordinate system can easily be combined to form even larger mosaics accurately. The final size of the mosaic is only limited by available RAM. Exposure time on each pixel is always traced and recorded so the map can be deepened or widened easily.

1.5 Cosmic Ray Removal

Two methods are provided for cosmic ray removal. The brightest and most obvious cosmic ray hits are removed from individual exposures with a sigma filter of 5x5 pixels. Fainter cosmic ray hits are removed by applying a sigma filter to pixels that have the same sky position in a dither set.

1.6 Image Weighting

Basic image weighting in SIMPLE-MOIRCS is to weight each image according to its exposure time. An optional weighting method is provided to weight each pixel according to sky transparency (cloud extinction and airmass effect), pixel quantum efficiency, sky brightness, and exposure time.

2. Important Procedures

Below highlights the main procedures in this package that are required to complete a full reduction, several procedures that may need some manual tweak, and other useful procedures. Please see the explanations in the procedure files for details.

2.1 reduce_moircs

The main procedure in SIMPLE-MOIRCS is reduce_moircs. It deals with images within a dither set from the same chip. It produces a flattened, sky subtracted, defringed, cosmic ray removed, and distortion and astrometrically corrected mosaic image, as well as an exposure time map. The reduced images and the exposure time maps can be used later to form large mosaics. This procedure is controlled by several keywords that need to be provided in the IDL command line, and by external parameters that are stored in an ASCII parameter file (APF). An example of APF is provided as the file GOODSN.para.

2.2 chip_offset

This procedure derives the sensitivity offset between the two chips. It is recommended to run this procedure at least once per observing run, although the chip offset does not seem to change over the past 1.5 year (of course only before the scientific grade chip 1 breaks). See field #9 in the APF for reduce_moircs.

2.3 reduce_std_moircs

Standard star observations are reduced with this procedure. Zero points will be automatically calculated if the star is a UKIRT Faint Standard (FS) Star. Zero points can be applied to images to change the image unit in the mosaic_wide procedure.

2.4 mosaic_wide

Images reduced by reduce_moircs can be combined to form large mosaics or deeper images with mosaic_wide. In order to make the images combinable by mosaic_wide, they have to have the same astrometry system. This is achieved by using identical projection in fields #4 through #7 in the APF of reduce_moircs. Photometric calibration can be applied at this stage, using zero points provided by reduce_std_moircs. It handles images taken under non-photometric conditions. SIMPLE-MOIRCS uses Jy (or similar linear units) as default zero point. For example, a zero point of 1 uJy per data unit corresponds to 23.9 magnitude in AB system.

2.5 read_moircs

This is the subroutine called by the main procedure that reads in a set of dithered MOIRCS images. When there are problems with the file/header format, or telescope pointing offsets, or change in platescale, modification to this procedure is needed. Sometimes this procedure is useful for trouble shooting as well.

2.6 defringe_moircs

This procedure is called by the main procedure. It removes circular fringe patterns in MOIRCS images. For chip 2, the center of the circles always moves around and the user needs to find out its coordinates. The method of determining the coordinates is explained in the procedure. An example script is provided in the file script.fringe for this purpose. This is usually the most time consuming part of the entire SIMPLE reduction. After MOIRCS upgrades its filter (hopefully soon), this procedure may not be needed any more.

2.7 summary_moircs

If you are too lazy to keep logs during the observations, or if you are using images from the archive that were not taken by yourself, this procedure may help you.

2.8 chk_astrometry

This procedure tells you the accuracy of astrometry in a FITS image, useful for verifying astrometry.

2.9 compare_images

This procedure compares fluxes of detected objects in two FITS images, useful for verifying photometry.

2.10 remove_satellite

If there are satellite tracks, this procedure can remove them from the raw image.

2.11 recompile_nomad

This can rewrite a NOMAD catalog into the format required by SIMPLE.

2.12 make_moircs_flat

This produces dome flats or twilight flats.

2.13 make_moircs_dark

This produces dark files.

3. Reduction Steps and Trouble Shooting

After the necessary preparation, the reduction can be subdivided into three major steps: reduction of the target images, reduction of the standard stars and calculating zeropoints, and combining multiple reduced images into a large mosaic. An example script for a set of standard reduction is provided in the file script.example. If successfully executed, the example script should generate an image like this. Below I describe some details and the associated trouble shooting.

3.1 Preparation

In order to obtain absolute astrometry, one needs to prepare a reference catalog that contains enough objects with good astrometry. Usually this is the NOMAD catalog (or USNO-B1 and equivalents), which provides minimum astrometric accuracies and numbers of stars. The procedure recompile_nomad.pro (§2.11) can be used to convert a NOMAD catalog into the required format. See recompile_nomad.pro for details. Rather than using a NOMAD catalog, it will be much better if there is already a deep and astrometrically correct image/catalog in the observing fields (such as the ACS catalogs of the GOODS fields). One can use that image/catalog to generate a deeper reference catalog. The reduced image will be registered to that image/catalog with high accuracies. See fit_distortion.pro or cross_images.pro for the format of the reference catalog.

The next preparation will be an ASCII file that stores the reduction parameters (the APF), such as coordinate frame, methods of flat-fielding and background subtraction, treatment of cosmic ray, and many others. An example and explanations are provided in the file GOODSN.para. Note that if you wish to combine images from different reductions to form a big mosaic, the astrometry related fields (4 through 7) should be kept identical in all reductions.

Third, there have to be ASCII files that tell the reduction program what files to process. The reduction procedure is designed to process images taken by the same chip within one dither sequence at once. Therefore two file lists per dither sequence are needed, one for chip 1 and the other for chip 2. Each file contains the filename of the first exposure and the last exposure. See read_moircs.pro for more details. As for reducing standard stars, only one file list is required for a dither sequence rather than two files, and all exposures should be included rather than just the first one and the last one. Example file lists are provided under the directory lists. The procedure summary_moircs (§2.7) may help you preparing the file lists.

Forth, for SExtractor to work, you will need to modify the file sexfind.default. Change the file paths to match your system.

Finally, the sensitivity offset between the two chips has to be derived first before photometrically accurate images can be produced. To do this, put 0 in field 9 in the APF, and reduce a full set of dithered images for both chips. Then use the procedure chip_offset (§2.2) to derive the sensitivity offset between the two chips, and put back the chip offset filename in field 9 in the APF. Now you are ready to rock.

3.2 Target Reduction

Once the above preparations are done, reducing the images should be fairly easy and automatic, using the main procedure reduce_moircs (§2.1). Examples are provided in the script script.example. Occasionally (or quite often?) there are errors. The common errors at this stage are: wrong astrometry, imperfect background subtraction, circular fringes, and flat-field artifacts caused by bright stars. It is always good to set the keyword individual=1 in the main procedure and this should help the diagnosis in most cases. Below describes each case.

3.2.1 Problem in Astrometry

How do we know there is a problem in astrometry? During the reduction, the main procedure will tell you what is the rms astrometry error relative to the reference catalog. It is a function of seeing and brightness of objects in the field. Usually the error should be sub-pixel, sometimes around one pixel if the seeing is bad or the quality of the reference catalog is not too great. Any rms error significantly larger than that indicates something might be wrong in astrometry. If the reduced image is warped into a very strange shape, something is definitely wrong. You can also write the reference catalog into a ds9 reg file and overlay the catalog objects on the reduced image in ds9. (This could be done with the procedures nomad2ds9 or write_ds9reg provided with this package. See details therein.) If the catalog objects do not match the objects in the reduced images, something is wrong.

In a few cases, astrometry problems are caused by the low quality of the reference catalog, such as not having enough stars. This can usually be solved by decreasing the degree of fitting in determining the absolute astrometry, by setting the keyword fit_order=2 or even 1 (default is 3) in reduce_moircs. One should try this first. If it does not work, then try what is in the next paragraph.

In many cases, especially the worst cases, errors are caused by image platescales that do not match what is in the header. This reduction package is quite tolerant to pointing errors. Even if the Subaru pointing is off by several arcsec, it still works. On the other hand, it is not tolerant to even 2% of platescale errors. One should use the individual=1 keyword in the main procedure to look at individual images. Identify pair of objects in the image and in the catalog, and compare the angular separation between the objects suggested by the header and that in the catalog. This should tell you what the real platescale is, to within 1% or so. In the source code of the procedure read_moircs, identify a line that contains "; ps_factor = xxxx." Uncomment this line, put the relative platescale offset in it, and recompile (.run) this procedure. If you do this correctly, it should solve the problems.

There may be other problems. Always use the individual=1 keyword in the main procedure and look at the reduction of individual exposures. Chance is that you will spot a problem there.

3.2.2 Imperfect Background Subtraction

Sometimes you will see residual backgrounds in the reduced image. Changing field 10 in the APF from 0 to 1 should solve most problems. (This is my default mode.) Occasionally there are still residual backgrounds, especially around the image corners. In this case, use the individual=1 keyword to inspect individual images and identify the frames with background subtraction problems. Then use the keyword more_bg_subtract in the main procedure to specify these frames and to perform a more aggressive background subtraction. This should give you a very flat image background. This will also produce artifacts on very bright and saturated stars. (Who care about these stars anyway?) The real caveat is, the entire package is designed to reduce images that only contain faint galaxies. So are the background subtraction routines. The more_bg_subtract keyword is very aggressive and is very likely to produce inaccurate photometry on large galaxies (larger than ~5"). This will be a problem for observations of low-redshift clusters, for example. For high-redshift blank-field imaging, there is less to worry about.

3.2.3 Fringes

MOIRCS sometimes produces nearly circular fringes. To remove the circular fringes, an image is first transformed to a polar coordinate and a model of the fringes is derived there. Then the model is transformed back to Cartesian coordinate and is subtracted from the image. The only question here is that the defringe procedure cannot automatically determine the center of the circular fringes, and for chip 2 the center of the fringes moves slowly, by a time scale of roughly an hour. Thus, images taken in different dither sets may have different fringe centers and the user needs to find out where the center is. Fortunately this is not too difficult. The first step is to use the individual=1 keyword in the main procedure without defringing. Then pick up a reduced individual-frame image that has strong fringes, and use the write_transform=1 keyword in the defringe_moircs procedure to check if the fringes in polar coordinate are perfectly vertical straight lines. If not, change the center of fringe until they are vertical and straight (as much as possible). An example script of how to do this is provided in the file scripts.fringe. This is only necessary for chip 2 for most cases. The fringe center on chip 1 rarely moves. Hopefully after MOIRCS upgrades its filter, there won't be any fringe problems.

3.2.4 Artifacts around Bright Stars

When there are very bright stars in the field, the flat-fielding can be screwed and there will be artifacts around the bright stars with a shape identical to the dither pattern. When this happens, changing the flat method in field 2 in the APF from 1 to 0 usually reduces the problem. If your computer is fast enough, always using 0 is not a bad idea. Also see field 14 in the APF.

3.3 Standard Star Reduction

At this moment, the procedure reduce_std_moircs (§2.3) is specifically designed for my standard star observations. In my case, I obtain at least two (usually three or more) dithered exposures of the standard star on each of the chip. I can easily imagine different people do this differently (for example, using one chip only), and the procedure may not work. In case it doesn't, you need to reduce the standard star observations manually. Fortunately this is not difficult. A very simple example in script.standard shows you how to do this. If you want, you can even use defringe_moircs (§3.2.3 and §2.6) to remove fringes, and/or use subtract_median_sky to remove residual background pattern. If you do follow the example script and use read_moircs and flat-field images generated in the target reduction, both the image counts and the flat-field image will be normalized properly. The zeropoint you derived from such standard-star frames can be directly applied to any of the reduced images, without correcting for the sensitivity difference between the two chips.

When reduce_std_moircs works, things are much easier. If the standard star is a UKIRT faint standard, it will automatically calculate the zeropoint for you. Good Luck.

3.4 Flux Calibration and Mosaicking

The final step is to put all single-chip, SIMPLY reduced images together, to form a larger or deeper mosaic image, using the procedure mosaic_wide (§2.3). To achieve this, there are two requirements. First, all images to be combined have to have identical world coordinate systems, including center of projection, platescale, and rotation. This can be done by keeping fields 4 through 7 unchanged in the APF throughout the SIMPLE reduction. Second, the zeropoint difference between the images have to be taken into account. This is done automatically by mosaic_wide if you can provide it the zeropoint of each image through the flux_conv keyword. In the example script, I choose to use uJy as the final map unit. This corresponds to an AB mag zeropoint of 23.9.

3.5 New Detector in Channel 1

Between Oct 2007 and June 2008, chip 1 is replaced with an engineer grade one. After that, a new science grade chip was installed. At this moment, SIMPLE can only deal with data taken with the engineer grade chip (and the old sceince grade one, of course), but not yet the new science grade chip. SIMPLE can tell whether the data were taken with the engineer grade chip by looking at the date in the header. Users do not need to do anything. A revision that accounts for the new science grade chip will be provided later.

4. Limits and Possible Future Improvements

The current version of SIMPLE-MOIRCS uses tangential projection for the reduced maps. However, it should be fairly easy to include other kinds of projection, if needed.

The main procedure is specifically designed for MOIRCS in many ways. However, many SIMPLE subroutines are written as general as possible. It is possible to modify this package for other near-IR mosaic cameras. With appropriate treatment for flat fielding and defringing, I believe it can be used on CCD cameras as well. SIMPLE has not been tested for cameras with large distortions, such as the SuprimeCam on Subaru. This will be the next SIMPLE development.

Currently SIMPLE cannot drizzle images. A drizzle function can be easily written for SIMPLE but I do not feel a need for this. If I do receive considerable amount of request from users, I may provide this function in the future.

5. Acknowledgment