MSUM python toolbox¶

Classes for managing a collection of FITS files¶

The class ImageFileCollection provides a table summarizing the values of FITS keywords in the files in a directory and provides easy ways to iterate over the HDUs, headers or data in those files. As a quick example:

>>> from msumastro import ImageFileCollection
>>> ic = ImageFileCollection('.', keywords=['imagetyp', 'filter'])
>>> for hdu in ic.hdus(imagetype='LIGHT', filter='I'):
...     print hdu.header, hdu.data.mean()

does what you would expect: it loops over all of the images in the collection whose image type is ‘LIGHT’ and filter is ‘I’. For more details see Image Management.

The TableTree constructs, from the summary table of an ImageFileCollection, a tree organized by values in the FITS headers of the collection. See Image Management for more details and examples.

Header processing of images from the Feder Observatory¶

This software¶

pip install msumastro

pip install astropysics

git clone https://github.com/mwcraig/msumastro.git

python setup.py develop

Dependencies¶

pip install pyfits

Overview¶

The scripts described here are intended primarily to be run in an automated way whenever new images are uploaded to the physics server. Each can also be run manually, either from the command line or from python.

Both these ways of invoking the script (from the command line or from python) is are wrappers around the python functions that do the real work. References to those functions, which provide more control over what you can do at the expense of taking more effort to understand, are provided below where appropriate.

The purpose of the header processing is to:

• Modify or add keywords to the FITS header to make working with other software easier:
  • Standardize names instead using the MaxIm DL defaults (e.g. RA and Dec instead of OBJCTRA and OBJCTDEC)
  • Set IMAGETYP to IRAF default, i.e. “BIAS”, “DARK”, “FLAT” or “LIGHT”
  • Add convenient keywords that MaxImDL does not always include (e.g. AIRMASS, HA, LST)
  • Add keywords indicating the overscan region, if any, in the image.
• Add astrometry to the FITS header so that RA/Dec can be extracted/displayed for sources in the image.
• Add object names to the HEADER where possible.
• Identify images that need manual action to add any of the information above.
• Create a table of images in each directory with columns for several FITS header keywords to facilitate indexing of images taken at Feder.

Intended workflow¶

The three primary scripts are, in the order in which they are intended (though not required, necessarily) to run:

• run_patch to do a first round of header patching that puts site information, LST, airmass (where appropriate) and RA/Dec information (where appropriate) into the files. See Header patching: run_patch.py for details.
• run_triage (details at Find problems and create summary: run_triage.py) to:

• generate a table summarizing properties of images in a directory. Each image is one row in the table and the columns are keywords from the FITS headers.
• create files with lists of images missing key information.

• Fix any problems identified by run_triage. The script quick_add_keys_to_file may be useful for this; it is an easy way to add/modify the values of keywords in FITS headers from the command line; see details at Manual intervention: quick_add_keys_to_file.py. After fixing these problems you may need to re-run patch, particularly if you have added pointing information or changed the IMAGETYP of any of the images.
• run_astrometry to use astrometry.net to add WCS (astrometry) information to the file. See Adding astrometry: run_astromtery.py for details. Note that this requires a local installation of astrometry.net.
• If desired, run_triage again to regenerate the table of image information.

The intended workflow will not work when...¶

The workflow above works great when the images that come off the telescope contain pointing information (i.e. RA/Dec), filter information and the image type in the header matches the actual image type.

Manual intervention will be required in any of these circumstances:

• There is no pointing information in the files. In files that are produced at Feder Observatory the pointing information is contained in the FITS keywords OBJCTRA and OBJCTDEC. If there is no pointing information, astrometry will not be added to the image headers. astrometry.net can actually do blind astrometry, but it is fairly time consuming. Alternatives are suggested below.
  • How to identify this case: There are two ways this problem may be noted. If run_triage has been run (and it is run in the standard workflow) then a file called “NEEDS_POINTING_INFO.txt” will be created that lists all of the light files missing pointing information. In addition, one file with suffix .blind will be created for each light file which contains no pointing information.
• Filter information is missing for light or flat images. All of the data preparation will occur if the FILTER keyword is missing from the headers for light or flat images, but the filter needs to be added to make the images useful.
  • How to identify this case: A file called “NEEDS_FILTER.txt” will be created as part of the standard workflow that lists each file that needs filter information.
• Incorrect image type set for image. If the incorrect image type is set it can prevent some data preparation steps to be omitted that should actually occur or cause steps to be attempted that shouldn’t be. For example, if an image is really a LIGHT image but is labeled in the header as a FLAT, then no attempt will be made to calculate an apparent position (Alt/Az) for the frame or to add astrometry. If the mistake is reversed, with a FLAT image labeled as LIGHT an attempt will be made to add astrometry which will fail.
  • How to identify this case: Manual inspection of affected images is the only reliable way to do this. A good place to start looking is at light files for which adding astrometry failed, file names whose name implies a different type than its IMAGETYP in the FITS header (e.g. a file with IMAGETYP = LIGHT whose name is flat-001R.fit)
• The object being observed is not in the master object list. The standard workflow has run but object names have not been add to all of the light files. This occurs when the object of the image was not in the list of objects used by run_patch.py or the object’s RA/Dec was too far from the center of the image to be matched.
  • How to identify this case: The script run_triage.py, part of the standard workflow, will produce a file called “NEEDS_OBJECT.txt” with a list of light files for which there is no object.

Fixes for cases that require intervention¶

The discussion below is deliberately broad. For some concrete examples see Manual header processing

• Adding pointing information: There are a few options here:

  • Use quick_add_keys_to_file to add the OBJECT keyword to the header, then add_ra_dec_from_object_name() to add pointing information, then run_astrometry to add astrometry to the images.
  • Use quick_add_keys_to_file to add the OBJECT, RA, DEC, OBJCTRA and OBJCTDEC to the headers, then run_astrometry to add astrometry to the images. This route is not recommended because it is easy to use a format for RA/Dec that isn’t understood (or is misinterpreted) by the code that adds astrometry.
  • Do blind astrometry to add pointing information, then use add_object_info() to add object names. There are no inherent with with this approach, though it may be simpler to add the astrometry then re-run the standard processing workflow to add any missing keywords than it is to manually use add_object_info()

• Adding filter information: The hard part here is not adding the filter keyword, it is figuring out what filter was used when the image was taken. You are on your own in figuring out that piece. Once you know what the filter should be, use quick_add_keys_to_file to add the keyword FILTER to the relevant files.

• Adding filter information: As with adding filter information, the hard part is figuring out what the image type should be. In practice most cases of this are light images misidentified as flat and vice versa and it ought to be easy to determine which of those an image is at a glance (arguably, if you can’t tell at a glance then the image is probably useful as neither a light nor a flat image). Once you know what the image type should be, use quick_add_keys_to_file to set the keyword IMAGETYP to the appropriate value in the relevant files. Allowed values for IMAGETYP are “BIAS”, “DARK”, “FLAT” or “LIGHT”.

• Adding object information: Assuming pointing information is already in the header for the images that need object information this is fairly straightforward. One way to do it is to add the object to the master object list and run add_object_info() (or even just re-run run_patch, which will end up re-doing some of the keyword-patching work). Another way to approach is to use quick_add_keys_to_file to set the OBJECT keyword directly. Either way you are encouraged to upadte the master object list.

Detailed list of keywords changed¶

OBJECT
JD
JD-HELIO
OBJCTALT
OBJCTAZ
OBJCTHA
AIRMASS

IMAGETYP: Type of image
LATITUDE: [degrees] Observatory latitude
LONGITUD: [degrees east] Observatory longitude
ALTITUDE: [meters] Observatory altitude
LST: Local Sidereal Time at start of observation
JD-OBS: Julian Date at start of observation
MJD-OBS: Modified Julian date at start of observation
BIASSEC: Region of the image useful for subtracting overscan
TRIMSEC: Region to which the image should be trimmed after removing overscan

RA: Approximate RA at EQUINOX
DEC: Approximate DEC at EQUINOX
OBJECT: Target of the observations
HA: Hour angle
AIRMASS: Airmass (Sec(Z)) at start of observation
ALT-OBJ: [degrees] Altitude of object, no refraction
AZ-OBJ: [degrees] Azimuth of object, no refraction

Reference/API¶

usage: run_standard_header_process.py [-h]
                                      (--dest-root DEST_ROOT | --overwrite-source)
                                      [--scripts-only] [-r {a,t,p}]
                                      [--no-blind] [-o OBJECT_LIST]
                                      [--quiet-console] [--silent-console]
                                      [--debug] [--quiet-log]
                                      source_root

usage: run_patch.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                    [--quiet-console] [--silent-console] [-o OBJECT_LIST]
                    [--overscan-only]
                    dir [dir ...]

run_patch.py /my/folder/of/images

from msumastro.scripts import run_patch
run_patch.main(['/my/folder/of/images'])

run_patch.py --object-list path/to/list.txt dir1 dir2 dir3

from msumastro.scripts import run_patch
run_patch.main(['--object-list', 'path/to/list.txt',
               'dir1', 'dir2', 'dir3'])

usage: run_astrometry.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                         [--quiet-console] [--silent-console] [-b] [-c]
                         [-o ODDS_RATIO]
                         [--astrometry-config ASTROMETRY_CONFIG]
                         dir [dir ...]

usage: run_triage.py [-h] [--debug] [-v] [-d DESTINATION_DIR] [-n]
                     [--quiet-console] [--silent-console] [-k KEY] [-l] [-a]
                     [-t TABLE_NAME] [-o OBJECT_NEEDED_LIST]
                     [-p POINTING_NEEDED_LIST] [-f FILTER_NEEDED_LIST]
                     [-y ASTROMETRY_NEEDED_LIST]
                     [dir [dir ...]]

usage: quick_add_keys_to_file.py [-h] [--file-list FILE_LIST]
                                 (--key-file KEY_FILE | --key-value KEY_VALUE KEY_VALUE)
                                 [files [files ...]]

usage: sort_files.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                     [--quiet-console] [--silent-console] [--move]
                     dir [dir ...]

Overview¶

Sometimes the standard data preparation will fail at one stage or another, most often because pointing information is missing for an image or because no object was found matching the RA/Dec of the image. Your tool of choice in such cases, either to add pointing information or to add object names is quick_add_keys_to_file. A broad discussion of using it is at Fixes for cases that require intervention.

This document provides some examples of using quick_add_keys_to_file from the command line. See the documentation for add_keys() for use from python scripts.

Examples¶

quick_add_keys_to_file.py --key-value object "EY UMa" image.fits

quick_add_keys_to_file.py --key-value object "EY UMa" ey-uma*.fits

Keyword,Value
OBJECT,"EY UMa"
RA,"09:02:20.79"
DEC,"+49:49:09.7"

File
MyFirstFile.fit
another_fits_file.fits
/or/even/the/full/path/to/a/fits/file.fit

quick_add_keys_to_file.py --key-file keys.txt --file-list files.txt

quick_add_keys_to_file.py --key-file keys.txt image1.fit image2.fit

quick_add_keys_to_file.py --key-value my_key "some value" --file-list files.txt

Working with a directory of images¶

For the sake of argument all of the examples below assume you are working in a directory that contains FITS images.

The class ImageFileCollection is meant to make working with a directory of FITS images easier by allowing you select the files you act on based on the values of FITS keywords in their headers.

It is initialized with the name of a directory containing FITS images and a list of FITS keywords you want the ImageFileCollection to be aware of. An example initialization looks like:

>>> from msumastro import ImageFileCollection
>>> keys = ['imagetyp', 'object', 'filter', 'exposure']
>>> ic1 = ImageFileCollection('.', keywords=keys) # only keep track of keys

You can use the wildcard * in place of a list to indicate you want the collection to use all keywords in the headers:

>>> ic_all = ImageFileCollection('.', keywords='*')

Most of the useful interaction with the image collection is via its .summary property, an `~astropy.table.Table`_ of the value of each keyword for each file in the collection:

>>> ic1.summary.colnames
['imagetyp', 'object', 'filter']
>>> ic_all.summary.colnames
# long list of keyword names omitted

>>> matches = (ic1.summary['filter'] == 'I' & ic1.summary['exposure'] < 60)
>>> my_files = summary['file'][matches]

>>> my_files = ic1.files_filtered(filter='I', exposure=30)

>>> for hdu in ic1.hdus(imagetyp='LiGhT', filter='I', exposure=30):
...     hdu.header['exposure']
...     new_data = hdu.data - hdu.data.mean()

>>> for hdu, fname in ic1.hdus(return_fname=True,
...                            imagetyp='LiGhT', filter='I', exposure=30):
...    hdu.header['meansub'] = True
...    hdu.data = hdu.data - hdu.data.mean()
...    hdu.writeto(fname + '.new')

 >>> for hdu in ic1.hdus(save_with_name='_new',
 ...                     imagetyp='LiGhT', filter='I', exposure=30):
 ...    hdu.header['meansub'] = True
 ...    hdu.data = hdu.data - hdu.data.mean()

It saves, in the ``location`` of the image collection, a new FITS file with
the mean subtracted from the data, with ``_new`` added to the name; as an
example, if one of the files iterated over was ``intput001.fit`` then a new
file, in the same directory, called ``input001_new.fit`` would be created.

 >>> for hdu in ic1.hdus(save_location='other_dir',
 ...                     imagetyp='LiGhT', filter='I', exposure=30):
 ...     pass

This option can be combined with the previous one to also give the files a
new name.

>>> for hdu in ic1.hdus(overwrite=True,
...                     imagetyp='LiGhT', filter='I', exposure=30):
...    hdu.header['meansub'] = True
...    hdu.data = hdu.data - hdu.data.mean()

Turning an image collection into a tree¶

The class TableTree turns an Astropy Table into a tree based on the values in a particular column or columns.

Introduction¶

The msumastro.header_processing subpackage contains classes and functions that do the work of modifying headers. There are several scripts for Automated header processing and Manual header processing provided to carry out the most common types of headers processing.

Reference/API¶