IDL WIRCAM Reduction Pipeline

SIMPLE-WIRCAM:
SIMPLE Imaging and Mosaicking PipeLinE for WIRCAM

Wei-Hao Wang
(National Radio Astronomy Observatory)

SIMPLE-WIRCAM version: 1.0
Document version: 1.0

bug report and feedback: whwang at aoc. nrao. edu

The SIMPLE Imaging and Mosaicking Pipeline (SIMPLE) is an IDL based system designed for galactic/extragalactic imaging where there are no extended objects (i.e., blank fields). It generates reasonably good astrometry and photometry. It is also capable of making large mosaics of many sensor areas, while still maintaining the astrometry accuracy. SIMPLE can be used on different data after simple modifications. This particular distribution (SIMPLE-WIRCAM) is optimized for WIRCAM on the CFHT. Detailed descriptions about the algorithms used in SIMPLE can be found in this paper.

To run this package, the most recent version of the IDL Astronomy User's Library, the package SExtractor, and the CFITSIO library are required to be pre-installed. The SExtractor will be called from command line via "sex," and the CFITSIO will be called via "imcopy." (CFITSIO is not needed if all your data are gzipped instead of fzipped.) Recent x86 and SPARC CPUs with GHz speeds are all fast enough. However, this package uses lot of memory to speed up the processing. At least 2 GB of RAM is recommended for basic reduction. For making large mosaics and cleanly removing cosmic rays, more than 4 GB of RAM and a 64 bit system are required. The 32 bit IDL cannot locate more than 2 GB of RAM, so the 64-bit requirement is not just to speed up the processing.

1.1 Flat Field and Background Subtraction

Sky flat is generated from dithered broad-band images. By masking detected objects and with a better treatment to frame-to-frame fluctuation of sky background, artifacts caused by bright objects is minimized. It is also possible to use dome flat or twilight flat. The standard background subtraction in SIMPLE-WIRCAM is to subtract a smooth surface in each quadrant of the HAWAII array. However, SIMPLE-WIRCAM is also equipped with median sky background subtraction.

1.2 Distortion Correction and Absolute Astrometry

Reduced images are warped and resampled to project onto a sky plane with sub-pixel accuracy. Distortion correction uses all detected objects in the dithered images, and does not rely on any external information. This is important for WIRCAM since its small FOV does not always have enough cataloged stars uniformly distributed across the FOV, especially at high galactic latitude. Absolute astrometry is obtained with objects in the NOMAD catalog or any source catalog (SDSS), and only a relatively small number of objects are required. For data taken under <0".4 seeing and with a dense source catalog of good astrometric accuracy (such as the GOODS ACS catalog), the final astrometric accuracy across one WIRCAM FOV can be as good as 0".08 rms relative to the input catalog. With NOMAD catalogs, the typical astrometric accuracy is 0".1 rms, which is the limit of the NOMAD catalog itself. Throughout the reduction, astrometry in the header follows the FITS standard described in Calabretta & Creisen (2002, A&A, 395, 1077).

1.3 Photometry

WIRCAM images are usually calibrated with 2MASS stars in the FOV (see 3.1.2). The WIRCAM photometry is consistent with independently reduced MOIRCS photometry across 200 arcmin² in the GOODS-N. The current version does not keep records of saturated pixels, so aperture photometry on bright stars are not reliable. The current version has functions for handling data taken under nonphotometric conditions.

1.4 Wide-Field Mosaicking

User can specify the coordinate system (projection center on the sky, rotation angle, and plate scale) for the reduced/combined images. A tangential projection is adopted, but other projections can be implemented by the users easily. Images with the same coordinate system can easily be combined to form even larger mosaics accurately. The final size of the mosaic is only limited by available RAM. Exposure time on each pixel is always traced and recorded so the map can be deepened or widened easily.

1.5 Cosmic Ray Removal

Two methods are provided for cosmic ray removal. The brightest and most obvious cosmic ray hits are removed from individual exposures with a sigma filter of 5x5 pixels. Fainter cosmic ray hits are removed by applying a sigma filter to pixels that have the same sky position in a dither set.

1.6 Image Weighting

Basic image weighting in SIMPLE-WIRCAM is to weight each image according to its exposure time. An optional weighting method is provided to weight each pixel according to sky transparency (cloud extinction and airmass effect), pixel quantum efficiency, sky brightness, and exposure time.

2. Important Procedures

Below highlights the main procedures in this package that are required to complete a full reduction, several procedures that may need some manual tweak, and other useful procedures. Please see the explanations in the procedure files for details.

2.1 reduce_wircam

The main procedure in SIMPLE-WIRCAM is reduce_wircam. It deals with images within a dither set from the same chip. It produces a flattened, sky subtracted, defringed, cosmic ray removed, and distortion and astrometrically corrected mosaic image, as well as an exposure time map. The reduced images and the exposure time maps can be used later to form large mosaics. This procedure is controlled by several keywords that need to be provided in the IDL command line, and by external parameters that are stored in an ASCII parameter file (APF). An example of APF is provided as the file HDFN.para.

2.2 chip_offset

This procedure derives the sensitivity offset between the two chips. It is recommended to run this procedure at least once per observing run, although the chip offset does not seem to change significantly from run to run. See field #9 in the APF for reduce_wircam.

2.3 calibrate

When there are enough objects in an image with known photometry, this procedure can be used to calibrate the image with the existing photometric catalog. This is integrated into the standard pipeline reduce_wircam and the mosaicking function mosaic_wide, but can be also used as a standalone function.

2.4 mosaic_wide

Images reduced by reduce_wircam can be combined to form large mosaics or deeper images with mosaic_wide. In order to make the images combinable by mosaic_wide, they have to have the same WCS. This is achieved by using identical projection in fields #4 through #7 in the APF of reduce_wircam. Photometric calibration can be applied at this stage, using an existing photometric catalog (e.g., 2MASS). It handles images taken under non-photometric conditions. SIMPLE-WIRCAM uses Jy (or similar linear units) as the default zero point. For example, a zero point of 1 uJy per data unit corresponds to 23.9 magnitude in AB system.

2.5 read_wircam

This is the subroutine called by the main procedure that reads in a set of dithered WIRCAM images. When there are problems with the file/header format, or telescope pointing offsets, or change in platescale, modification to this procedure is needed. Sometimes this procedure is useful for trouble shooting as well.

2.6 fz2fits_wircam

The SIMPLE-WIRCAM pipeline starts with compressed raw data provided by CFHT. fz2fits_wircam is the first procedure in the pipeline, to uncompress the raw data. Many processes in the pipeline rely on the particular format and information generated by fz2fits_wircam. If one does not start from compressed raw data generated by this procedure, chance is high that something will go wrong later. Despite the name fz2fits, it also deals with gzipped WIRCAM data.

2.7 summary_wircam

This procedure can generate a simple observing log. It may help you to decide which files should be grouped together and feed to the main procedure.

2.8 chk_astrometry

This procedure tells you the accuracy of astrometry in a FITS image, useful for verifying astrometry.

2.9 compare_images

This procedure compares fluxes of detected objects in two FITS images, useful for verifying photometry.

2.10 remove_satellite_wircam

If there are satellite tracks, this procedure can remove them from the raw image.

2.11 recompile_nomad

This can rewrite a NOMAD catalog into the format required by SIMPLE.

3. Reduction Steps and Trouble Shooting

After the necessary preparation, the reduction can be subdivided into two major steps: reduction of the target images, and combining multiple reduced images into a large mosaic. An example script for a set of standard reduction is provided in the file script.example. If successfully executed, the example script should generate an image like this. Below I describe some details and the associated trouble shooting.

3.1 Preparation

3.1.1 Astrometric Reference Catalog

In order to obtain absolute astrometry, one needs to prepare a reference catalog that contains enough objects with good astrometry. Usually this is the NOMAD catalog (or USNO-B1), which provides minimum astrometric accuracies and numbers of stars. The procedure recompile_nomad.pro (§2.11) can be used to convert a NOMAD catalog into the required format. See recompile_nomad.pro for details. Additional to NOMAD, another good source is SDSS. Indeed, users can use any source catalog (2MASS, Chandra, VLA, Spitzer, another deep ground-based image that has good astrometry, or whatever) and SIMPLE will force the reduced image to match the provided catalog with high accuracies. (This also means that if the provided catalog is problematic, the results of the reduction will also be problematic.) In general, on average at least several tens of sources per WIRCAM chip are required for OK astrometry calibration. As many as 1000 per chip will not hurt. It is also important to make sure the distribution of the sources in the reference catalog is as uniform as possible. See fit_distortion.pro or cross_images.pro for the format of the reference catalog.

3.1.2 Photometric Reference Catalog

To calibrate photometry, one needs to provide a photometric catalog that contains a sufficient number of stars with known fluxes in the field of view of the observations. Usually, this is the 2MASS catalog. See calibrate.pro for the format of this catalog. Note that the useful magnitude range in 2MASS could be very small. In the 2MASS bright end, the WIRCAM can be nonlinear or even saturated. In the faint end, the 2MASS magnitude is biased by selection effect. My experience shows that the useful magnitude range is 14--16 (Vega) for J and 12.5--14.5 for Ks. The users may want to double check this by themselves.

By the way, SIMPLE does not like magnitudes. Everything needs to be in linear units. See explainations in calibrate.pro. In the example directory HDFN/CATALOG, the procedure recat_2mass.pro shows how I convert a 2MASS catalog to the flux calibration catalog required by SIMPLE.

3.1.3 Reduction Parameter

The next preparation will be an ASCII file that stores the reduction parameters (the APF), such as coordinate frame, methods of flat-fielding and background subtraction, treatment of cosmic ray, and many others. The filenames of the astrometric and photometric reference catalogs mentioned above will be entered in the APT. An example and explanations are provided in the file HDFN.para. Note that if you wish to combine images from different reductions to form a big mosaic, the astrometry related fields (4 through 7) should be kept identical in all reductions.

3.1.4 File List

There have to be ASCII files that tell the reduction program what files to process. The reduction procedure is designed to process images taken by the same chip within one dither sequence at once. A file list contains the filenames of the first exposure and the last exposure, and reduce_wircam reduces all files between these two at once, chip by chip. See read_wircam.pro for more details about the file list. Example file lists are provided under the directory lists. The procedure summary_wircam (§2.7) may help you preparing the file lists.

When preparing the file lists, it is highly recommended to only reduce sequences of images taken within ~0.5 hr together. This is because the color of the near-IR sky varies with time and only images taken within ~0.5 hr have sufficiently similar background color. Keep each reduction sequence shorter than 0.5 hr helps to get good flat field and background subtraction. For example, the actual observation may contain one 50-minute dither sequence of 20 exposures. In this case, it is better to prepare two file lists, each containing only 10 expousres, and execute reduce_wircam on each of the file list.

3.1.5 SExtractor Setup

For SExtractor to work, you will need to modify the file sexfind.default. Change the file paths to match your system.

3.1.6 Chip Sensitivity

This is optional. There is a weight keyword in the main reduction routine reduce_wircam.pro. This keyword allows the users to weight each pixel according to its quantum efficiency (and other factors such as sky brightness). To derive the relative QE of the four chips (the relative QE of the pixels will be taken care by the flat field), put 0 in field 9 in the APF, and reduce a set of dithered images on all four chips. (The results of the reduction is not needed and can be deleted later.) Then use the procedure chip_offset (§2.2) to derive the sensitivity offset of the four chips, and put back the chip offset filename in field 9 in the APF.

Note that the users can still turn on the weight keyword without measuring the relative sensitivity of the four chips. In this case, data from the four chips will be weighted equally. This does not affect flux calibration.

3.2 Target Reduction

Once the above preparations are done, reducing the images should be fairly easy and automatic, using the main procedure reduce_wircam (§2.1). Examples are provided in the script script.example. Occasionally (or quite often?) there are errors. The common errors at this stage are: wrong astrometry, imperfect background subtraction, crosstalk, and flat-field artifacts caused by bright stars. It is always good to set the keyword individual=1 in the main procedure and this should help the diagnosis in most cases. (You can remove the unwanted files later) Below describes each case.

3.2.1 Problem in Astrometry

How do we know there is a problem in astrometry? During the reduction, the main procedure will tell you what is the rms astrometry error relative to the reference catalog. It is a function of seeing and brightness of objects in the field. Usually the error should be well within 0.5 pixel, sometimes around 0.5 pixel if the seeing is bad or the quality of the reference catalog is not too great. Any rms error larger than that indicates something might be wrong in astrometry. If the reduced image is warped into a very strange shape, something is definitely wrong. You can also write the reference catalog into a ds9 reg file and overlay the catalog objects on the reduced image in ds9. (This could be done with the procedures nomad2ds9 or write_ds9reg provided with this package. See details therein.) If the catalog objects do not match the objects in the reduced images, something is wrong.

In a few cases, astrometry problems are caused by the low quality of the reference catalog, such as not having enough stars. This can usually be solved by decreasing the degree of fitting in determining the absolute astrometry, by setting the keyword fit_order=2 or even 1 (default is 3) in reduce_wircam. One should try this first. If it does not work, then try what is in the next paragraph.

In many cases, especially the worst cases in which the reduction pipeline unexpectedly quits, errors are caused by image rotation or platescale that do not match what is in the header.

This reduction package is quite tolerant to pointing errors. Even if the telescope pointing is off by more than 10 arcsec, it still works. On the other hand, it is not tolerant to even 2% of platescale errors. One should use the individual=1 keyword in the main procedure to look at individual images. Identify pair of objects in the image and in the catalog, and compare the angular separation between the objects suggested by the header and that in the catalog. This should tell you what the real platescale is, to within 1% or so. In the source code of the procedure read_wircam, identify a line that contains "; ps_factor = xxxx." Uncomment this line, put the relative platescale offset in it, and recompile (.run) this procedure. If you do this correctly, it should solve the problems.

There may be other problems. Always use the individual=1 keyword in the main procedure and look at the reduction of individual exposures. Chance is that you will spot a problem there.

3.2.2 Imperfect Background Subtraction

Sometimes you will see residual backgrounds in the reduced image, especially around the image corners. Set individual=1 to inspect individual images and identify the frames with background subtraction problems. Then use the keyword more_bg_subtract in the main procedure to specify these frames and to perform a more aggressive background subtraction. This should give you a very flat image background. This will also produce artifacts on very bright and saturated stars. (Who care about these stars anyway?) The real caveat is, the entire package is designed to reduce images that only contain faint galaxies. So are the background subtraction routines. The more_bg_subtract keyword is very aggressive and is very likely to produce inaccurate photometry on large galaxies (larger than ~5"). This will be a problem for observations of low-redshift clusters, for example. For high-redshift blank-field imaging, there is less to worry about.

3.2.3 Crosstalk (xtalk)

WIRCAM chips have xtalk between each of the readout channels, on data taken before early 2008. Set dxtalk=1 (default) in the reduction routine will remove xtalk quite cleanly. The de-xtalk feature in SIMPLE removes the 32-channel xtalk and guidebox xtalk. It only removes the 8-channel xtalk if a list of bright stars (that produce observable 8-channel features in deep integrations) is provided in field 14 in the APF. If the 8-channel xtalk features appear in very deep integrations, more aggressive de-xtalk can be used by setting dxtalk=2 or even 3. This usually removes xtalk cleanly even in images as deep as ~30 hr.

3.2.4 Artifacts around Bright Stars

When there are very bright stars in the field, the flat-fielding can be screwed and there will be artifacts around the bright stars with a shape identical to the dither pattern. When this happens, changing the flat method in field 2 in the APF from 1 to 0 usually reduces the problem. If your computer is fast enough, always using 0 is not a bad idea. Also see field 14 in the APF.

3.4 Flux Calibration and Mosaicking

The final step is to put all single-chip, SIMPLY reduced images together, to form a larger or deeper mosaic image, using the procedure mosaic_wide (§2.3). To achieve this, there are two requirements. First, all images to be combined have to have identical world coordinate systems, including center of projection, platescale, and rotation. This can be done by keeping fields 4 through 7 unchanged in the APF throughout the SIMPLE reduction. Second, the zeropoint difference between the images have to be taken into account. This is done automatically by mosaic_wide if you can provide it the zeropoint of each image through the flux_conv keyword. If the images are already calibrated in the main reduction stage (see 3.1.2), then there is no need to worry about this here.

4. Limits and Possible Future Improvements

The current version of SIMPLE-WIRCAM uses tangential projection for the reduced maps. However, it should be fairly easy to include other kinds of projection, if needed.

The main procedure is specifically designed for WIRCAM in many ways. However, many SIMPLE subroutines are written as general as possible. It is possible to modify this package for other near-IR mosaic cameras. With appropriate treatment for flat fielding and defringing, I believe it can be used on CCD cameras as well. SIMPLE has not been tested for cameras with large distortions, such as the SuprimeCam on Subaru. This will be the next SIMPLE development.

Currently SIMPLE cannot drizzle images. A drizzle function can be easily written for SIMPLE but I do not feel a need for this. If I do receive considerable amount of request from users, I may provide this function in the future.

5. Acknowledgment