Shape Measurement Fisher Formalism Package¶
Fast simulations and analysis for the Weak Lensing Working Group of the LSST Dark Energy Science Collaboration.
This software was primarily developed to study the effects of overlapping sources on pixel-noise bias estimation using the Fisher Formalism.
The code is hosted on github. Please use the issue tracker to let us know about any issues you have with installing or running this code, or to request new features.
This work was conducted by Ismael Mendoza, in collaboration with Pat Burchat and Josh Meyers, at Stanford University in 2015-2017. Work was supported by the National Science Foundation and by LSST.
Installation¶
Install with GIT¶
The code is hosted on github so the easiest method to perform an initial installation is with git:
git clone https://github.com/ismael2395/ShapeMeasurementFisherFormalism.git
This will create a new subdirectory ShapeMeasurementFisherFormalism containing the latest stable version of the complete package.
Experts who already have a correctly configured github account might prefer this alternative:
git clone git@github.com:ismael2395/ShapeMeasurementFisherFormalism.git
Update with GIT¶
You can update your local copy of the package at any time using:
cd ShapeMeasurementFisherFormalism
git update
Getting Started¶
Programs can be run directly from the top-level directory as long as you have the required packages already installed, e.g.:
cd ShapeMeasurementFisherFormalism
python generate.py --help
For an introduction to the available programs, see here and for examples of running these programs see here.
Required Packages¶
The following python packages are required by this package:
Note that numpy is available in recent anaconda or enthought canopy distributions. Installing GalSim is a more involved process, but well worth the effort. The lmfit package is only required if you will be running your own fits using the fitting.
Examples¶
Command-line Options¶
Print out usage info for command-line options:
python generate.py --help
python fitting.py --help
Quick Simulation Demo¶
The first thing to do is to generate a desired galaxy model (with an optional PSF) using the generate.py file:
python generate.py -p project -gal 1 --galaxy-model gaussian --psf_model psf_gaussian --g1 0.2 --g2 0.2 --y0 0. --x0 0. --flux 1. --psf_flux 1. --hlr 0.5 --psf_fwhm 0.7 --snr 20.0
Display partial derivatives and save it to the project folder just created with generate.py:
python display.py -p project --partials --snr 20.
Display a fisher matrix elements without showing it when the command is executed:
python display.py -p project --fisher --snr 20. --hide
It is important to always specify the project that is being run as well as the signal to noise ratio to be used. . Finally one can also save all the possible outputs to the current project folder with:
python display.py -p project --all --snr 20.
This command hides the output by default and saves all files in a .pdf format.
More complicated examples¶
It is not necessary to use the display module to display the images (and it can be a little limiting in terms of formatting the images
to your liking). It is also possible to access the images directly from the analysis.fisher.fisher
objects and use them.
Please refer to the tutorial notebooks for tutorials on how to do this. The notebooks also contain other more complicated
examples of how the package can be used.
Programs¶
All available programs are contained in the top-level directory where this package is installed. Programs are written in python and use the ‘.py’ file extension.
All programs are configured by passing command-line options. Pass the –help option to view documentation for these options, e.g.:
python generate.py --help
Executable Programs¶
generate¶
The generate program writes to a CSV file that can be later be read by the analysis.galfun
or by analysis.fisher
modules to generate galaxy images and their analysis. Fast image rendering with galsim.
The user specifies the arguments of the galaxy and the PSF in the following way:
python generate.py -p PROJECT_NAME -gal 1 --galaxy-model MODEL_NAME --snr VALUE --PARAM1_NAME PARAM1_VALUE ...
For a given galaxy model at least the following parameters should always be specified:
- x0
- y0
- flux
- measure of size (sigma,hlr,fwhm)
- measure of ellipticity ((g1,g2),(e1,e2))
You can look up what the different parameters mean exactly in this section. The project name, the galaxy model, the snr value, and the galaxy ID (in the case when you draw 1 you only need one galaxy ID) should also always be specified. One can optionally add a PSF for the galaxy to be convolved with:
python generate.py -p PROJECT_NAME -gal 1 --galaxy-model MODEL_NAME --snr VALUE --PARAM1_NAME PARAM1_VALUE ... --psf_model PSF_MODEL_NAME --PSF_PARAM1 PSF_PARAM1_VALUE ...
A PSF always needs a model name, a measure of size, and a flux (which should be 1). The PSF and the galaxy are treated differently in the package. If you want to generate two galaxies just use the procedure above changing the id:
python generate.py -p PROJECT_NAME -gal 2 --galaxy-model MODEL_NAME --snr VALUE --PARAM1_NAME PARAM1_VALUE ... --psf_model PSF_MODEL_NAME --PSF_PARAM1 PSF_PARAM1_VALUE ...
Then the .csv file should contain information about both galaxies (one row per galaxy). It is recommended you use specify the same PSF for both galaxies.
display¶
The display.py program displays simulated images and analysis results stored in the CSV file generated by the generate.py program. It will display the images when the program is executed produced unless the –hide argument is used when executed. Please run:
python display.py --help
For the different images that can be produced and some explanations for them. More information on how to use the module can also be found in here.
fitting¶
The fitting program runs fits using lmfit. as an alternative way of estimating the noise bias and error in order to compare it to the Fisher Formalism.
In order to run a certain number N of fits, make sure you already created a galaxy with the generate program and then run:
fitting.py -p EXISTING_PROJECT_NAME -rf -n N --snr SNR_VALUE
If you are Stanford SLAC affiliated and want send N batch jobs to SLAC, then log in to SLAC and then run:
fitting.py -p EXISTING_PROJECT_NAME -rfs short -n N --snr SNR_VALUE
This fitting module internally calls another module named runfits N times which produces the results from a single fits
and then writes them to a .csv file. The correct way to read the data produced from this process is to use the analysis.galfun.read_results()
function which returns the results in a convenient format. For more specific information about the output of this function please consult the
docstring. To see how these process can be used to produce interesting fits and their plots please take a look at the fits_anlaysis tutorial notebooks.
Other Modules¶
models¶
The analysis.models
module contains the galaxy and PSF models that can be produced by generate and
analyzed by analysis.fisher
. It specifies the commands that galsim
should use to produce them.
fisher¶
The analysis.fisher
module contains the analysis by the Fisher Formalism which is made on the galaxy that generate produced.
Please refer to the tutorials in tutorial notebooks for specific instructions on how to use this module efficiently
in conjunction with analysis.galfun
module.
galfun¶
The analysis.galfun
is a multipurpose module that contains important functions ranging from managing parameters of
generated galaxies to extracting information from relevant files.
defaults¶
The analysis.defaults
module stores defaults parameter values for different parts of the package.
Models and Parameters¶
This document describes the galaxy parameters and models that are implemented in the generate.py program. For more detailed information
take a look into analysis.models
and the galsim documentation.
Galaxy Models¶
Name | Description |
---|---|
gaussian | Gaussian galaxy profile, galsim.Gaussian . |
exponential | Exponential galaxy profile, galsim.Exponential . |
deVaucouleurs | DeVaucouleurs galaxy profile, galsim.DeVaucouleurs . |
bulgeDisk | Combined exponential (disk) and deVaucouleurs (bulge) profile. |
Galaxy Parameters¶
Name | Type | Description |
---|---|---|
Program Parameters | ||
project | string | Pixel offset of left edge of bounding box relative to left edge of survey image. |
gal | int32 | Pixel offset of right edge of bounding box relative to left edge of survey image. |
ymin | int32 | Pixel offset of bottom edge of bounding box relative to bottom edge of survey image. |
ymax | int32 | Pixel offset of top edge of bounding box relative to bottom edge of survey image. |
Source Properties | ||
f_disk | float32 | Fraction of total galaxy flux to due a Sersic n=1 disk component. |
f_bulge | float32 | Fraction of total galaxy flux to due a Sersic n=4 bulge component. |
flux | float32 | Total detected flux in electrons. |
x | float32 | Source centroid in x relative to image center in arcseconds. |
y | float32 | Source centroid in y relative to image center in arcseconds. |
e1 | float32 | Real part (+) of galaxy ellipticity spinor (Q11-Q22+2*Q12)/(Q11+Q22). |
e2 | float32 | Imaginary part (x) of galaxy ellipticity spinor 2*Q12/(Q11+Q22). |
g1 | float32 | Real part (+) of galaxy ellipticity spinor (Q11-Q22)/(Q11+Q22+2|Q|**0.5). |
g2 | float32 | Imaginary part (x) of galaxy ellipticity spinor (2*Q12)/(Q11+Q22+2|Q|**0.5). |
eta1 | float32 | Conformal shear with a/b = exp(eta). |
eta2 | float32 | Conformal shear with a/b = exp(eta). |
sigma | float32 | Galaxy size arcseconds calculated as |Q|**0.25. |
sigma_p | float32 | Galaxy size in arcseconds calculated as (0.5*trQ)**0.5. |
hlr | float32 | The half-light-radius of the profile. |
beta | float32 | Position angle of second-moment ellipse in radians, or zero when a = b. |
q | float32 | b/a ratio of semi-minor axis to semi-major axis. |
fwhm | float32 | Full-width-half-max of the profile, defined as radius at half the max peak… |
snr | float32 | Signal to noise ration as defined in… |
hlr_b | float32 | hlr for the bulge component of the analysis.models.bulgeDisk model. |
flux_d | float32 | flux for the disk component of the analysis.models.bulgeDisk model. |
flux_b | float32 | flux for the bulge component of the analysis.models.bulgeDisk model. |
R_r | float32 | hlr for the bulge component of the analysis.models.bulgeDisk model. |
PSF Models¶
Name | Description |
---|---|
psf_gaussian | Gaussian galaxy profile, galsim.Gaussian . |
psf_moffat | Moffat galaxy profile, galsim.Moffat . |
PSF Parameters¶
For the PSF, the parameters have the same significance as in the table of parameters as above.
IPython Notebooks¶
- The following iPython notebooks are included in the notebooks/ directory and can be viewed online:
- Individual_Fisher_Analysis: Tutorial of how to use the package to analyze single galaxies using the Fisher Formalism.
- Group_Fisher_Analysis: Tutorial of how to use the package to analyze 2 (or more) galaxies using the Fisher Formalism.
- Fit_Analysis: Tutorial on how to use the fitting package to produce noise fits and compare results to the Fisher Formalism results.
- how_to_add_models: Tutorial on how to add new galaxy models to be analyzed in the package.
- Ring_Test: Tutorial on the ring test.
You can also edit and run these notebooks locally if you have ipython installed. First start the server in its own window (where it log messages will appear):
ipython notebook
This should open up a notebook browser where you can click on any of the notebooks to launch it.
Information for Developers¶
Build the documentation¶
To build a local copy of the HTML documentation, use:
cd .../docs
make html
To view the most recent build of the HTML documentation, point your browser to …/docs/_build/html/index.html
Add a new package module¶
Create a new file analysis/xxx.py for module analysis.xxx with an initial descriptive docstring.
Add the line import xxx to analysis/__init__.py.
Add the line analysis.xxx to the list of submodules in docs/src/analysis.rst.
Create a new documentation file docs/src/analysis.xxx.rst containing (replace xxx in two places):
analysis.xxx module
=================
.. automodule:: analysis.xxx
:members:
:undoc-members:
:show-inheritance:
Update the version¶
Update the version and release values in the sphinx configuration file docs/conf.py, then commit, tag, and push the new version, e.g.:
git tag -l v0.2
Modules API Reference¶
analysis package¶
Submodules¶
analysis.fisher module¶
This module contains functions necessary to produce statistical results of the fisher formalism from a given galaxy.
-
class
analysis.fisher.
Fisher
(g_parameters, image_renderer, snr, var_noise=None)[source]¶ Bases:
object
Produce fisher object (containing fisher analysis) for a given set of galaxy parameters.
Given a galaxy image and the appropiate parameters that describe it, (in the form of
analysis.galfun.GParameters
object) will produce a fisher object that contains the analysis of it using the Fisher Formalism.NOTE: The matrices are in dictionary form, use the function matrixToNumpyArray() to change them to a matrix that is ordered according to param_names.
- Args:
- g_parameters(
GParameters
): String point to the directory - specified by the user.
image_renderer(
ImageRenderer
): Object used to render image of galaxy. snr(float): Value S/N ratio to use in the analysis.- g_parameters(
- Attributes:
image_renderer_partials(
analysis.galfun.ImageRenderer
): Object used to render images of partial derivatives. image(Galsim.Image
): Dictionary whose keys are the ids of each of thegalaxies specified in galaxies.csv, and that map to another dictionary that can be taken in byanalysis.galfun.getGalaxyModel()
var_noise(float): Variance of noise of given S/N . steps(dict): Dictionary containing the step size used when
calculating partial derivatives.- param_names(list): A list containing the keys of fit_params
- in a desirable order.
num_params(int): Number of parameters specified for the galaxy. num_galaxies(int): Number of galaxies specified. derivatives_images(dict): Dictionary containing np.array(s) that represent the
derivative of the galaxy(ies) with respect to each parameter.- second_derivatives_images(dict): Dictionary containing np.array(s) that represent the
- second derivatives of the galaxy(ies) with respect to its parameters.
- fisher_matrix_images(dict): Dictionary containing np.array(s) that represent the
- fisher matrix images of the galaxy(ies) with respect to its parameters.
fisher_matrix(dict): Dictionary containing fisher matrix elements. covariance_matrix(dict): Dictionary containing covariance matrix elements. correlation_matrix(dict): Dictionary containing correlation matrix elements. bias_matrix_images(dict): Dictionary containing bias matrix image elements. bias_matrix(dict): Dictionary containing bias matrix elements. bias_images(dict): Dictionary containing bias images elements. biases(dict): Dictionary containing biases
-
derivativesImages
()[source]¶ Return images of the partial derivatives of the galaxy.
The partial differentiation includes each of the different parameters that describe the galaxy.
analysis.galfun module¶
Multipurpose module that contains important functions ranging from managing parameters of generated galaxies to extracing information from relevant files.
-
class
analysis.galfun.
GParameters
(project=None, id_params=None, omit={})[source]¶ Bases:
object
Class that manages galaxies parameters obtained from galaxies.csv
This class reads a galaxies.csv file located in the specified project directory and extracts the parameters of each galaxy contained in it.
- Args:
- project(str): String point to the directory specified by the user.
id_params(dict): See Attributes for details. Makes it possible to
create a
GParameters
object without galaxies.csv file. - Attributes:
- omit_fit(dict): Dictionary defined in containing
- the parameters that should not be included in the analysis for a particular galaxy model but could be ecessary for drawing the galaxy.
- id_params(dict): Dictionary whose keys are the ids of each of the
- galaxies specified in galaxies.csv, and that map
to another dictionary that can be taken in by
getGalaxyModel()
- params(dict): Dictionary that encodes the same information as
- id_params but in a different form. Combines each of the dictionaries contained in id_params into a single dictionary that contains all parameters but in the form param_#.
- fit_params(dict): Dictionary similar to the params attribute but
- without the parameters specified in omit_fit.
- nfit_params(dict): Dictionary that contains all the parameters not
- contained in fit_params. Usually used for kwargs in conjunction with fit_params to draw Galaxies.
- ordered_fit_names(list): A list containing the keys of fit_params
- in a desirable order.
num_galaxies(int): Number of galaxies specified.
-
static
convertId_Params
(id_params, omit_fit={})[source]¶ Converts id_params to the format of
params
.Parameters: Returns: A dictionary of params (dict).
-
static
convertParams_Id
(params)[source]¶ Convert a dictionary params in the format of
params
to a dictionary in the formatid_params
-
class
analysis.galfun.
ImageRenderer
(pixel_scale=None, nx=None, ny=None, stamp=None, project=None, bounds=None, mask=None)[source]¶ Bases:
object
Object used to produce the image of a galaxy.
Parameters: - pixel_scale (float) – Pixel_scale to use in the image (ratio of pixels to arcsecs)
- nx (float) – Width of the image in pixels.
- ny (float) – height of the image in pixels.
- stamp (int) – optional, galsim.Image of appropiate dimensions to draw the image. does not actually use whatever was originally in the stamp.
- bounds (tuple) – When drawn, the image will be clipped to these bounds.
- mask (
np.array
) – the pixels selected in this mask will be set to 0.
This object is made so it can be passsed in to a class
analysis.fisher.Fisher
object.
-
analysis.galfun.
addNoise
(image, snr, noise_seed=0)[source]¶ Set gaussian noise to the given galsim.Image.
Parameters: Returns: A
galsim.Image
, variance_noise tuple. The image is the noisy version of the original image and variance_noise is the noise variance on each pixel due to the added noise.
-
analysis.galfun.
getGalaxiesModels
(fit_params=None, id_params=None, g_parameters=None, **kwargs)[source]¶ Return the model of a set of galaxies.
- One of the the following must be specified:
- fit_params (and nfit_params as **kwargs. Used specially in
runfits.py
). id_params g_parameters (from which id_params is extracted.)
This function generates each of the galaxies specified in id_params and then sums them together to get a final galaxy.
Parameters: - fit_params (dict) – Partial form of id_params that only includes the
parameters to be used for the fit.
For details,
GParameters
- id_params (dict) – Dictionary containing each of the galaxies
parameters. For details,
GParameters
- g_parameters (
GParameters
) – An object containing different forms of the galaxy parameters. - image (bool) – If :bool:True returns an galsim.Image otherwise it
- a np.array (returns) –
Returns: A
galsim.GSObject
-
analysis.galfun.
getGalaxyModel
(params)[source]¶ Return the image of a single galaxy optionally drawn with a psf.
Look at
analysis.models
to figure out which galaxy models and psf models are supported as well as their corresponding implemented parameters. You can even add your own galaxies if desired.Parameters: - params (dict) – Dictionary containing the information of a single
- where the keys is the name (galaxy) –
- are the values of the parametes. (values) –
Returns: A
galsim.GSObject
analysis.models module¶
This module contains the models that are used for galaxies and psfs for galsim, more information of how to add your own models for both galaxies and psfs can be found in the corresponding tutorial.
-
class
analysis.models.
bulgeDisk
(params=None, params_omit=None)[source]¶ Bases:
analysis.models.model
-
omit_general
= ['delta_e', 'delta_theta', 'n_d', 'n_b']¶
-
parameters
= ['x0', 'y0', 'flux_b', 'flux_d', 'flux_b/flux_total', 'hlr_d', 'hlr_b', 'R_r', 'e1', 'e2', 'eta1', 'eta2', 'delta_e', 'delta_theta', 'n_d', 'n_b']¶
-
-
class
analysis.models.
bulgeDisk6
(params=None, params_omit=None)[source]¶ Bases:
analysis.models.model
-
omit_general
= ['n_d', 'n_b']¶
-
parameters
= ['x0', 'y0', 'flux', 'hlr', 'e1', 'e2', 'eta1', 'eta2', 'n_d', 'n_b']¶
-
-
class
analysis.models.
deVaucouleurs
(params=None, params_omit=None)[source]¶ Bases:
analysis.models.model
-
omit_general
= []¶
-
parameters
= ['x0', 'y0', 'flux', 'hlr', 'fwhm', 'sigma', 'e1', 'e2', 'eta1', 'eta2']¶
-
-
class
analysis.models.
exponential
(params=None, params_omit=None)[source]¶ Bases:
analysis.models.model
-
omit_general
= []¶
-
parameters
= ['x0', 'y0', 'flux', 'hlr', 'fwhm', 'sigma', 'e1', 'e2', 'g1', 'g2', 'eta1', 'eta2', 'q', 'beta']¶
-
-
class
analysis.models.
gaussian
(params=None, params_omit=None)[source]¶ Bases:
analysis.models.model
-
omit_general
= []¶
-
parameters
= ['flux', 'x0', 'y0', 'hlr', 'fwhm', 'sigma', 'e1', 'e2', 'eta1', 'eta2', 'e', 'q', 'beta', 'g1', 'g2']¶
-
-
analysis.models.
getModelCls
(model)[source]¶ Return the corresponding class to the model specified in params
-
analysis.models.
getPsfModelCls
(model)[source]¶ Return the corresponding psf class specified in params.
-
class
analysis.models.
model
(params=None, params_omit=None)[source]¶ Bases:
object
-
omit_general
= []¶
-
parameters
= []¶
-
-
class
analysis.models.
psf_gaussian
(params=None)[source]¶ Bases:
analysis.models.psf_model
-
parameters
= ['psf_flux', 'psf_fwhm', 'psf_e1', 'psf_e2']¶
-
analysis.draw module¶
analysis.defaults module¶
Some of the defaults that are used in the overall program.
-
analysis.defaults.
getInitialValuesFit
(g_parameters)[source]¶ Return a dictionary containing the initial values to be used in the in the fitting of the parameters.
The dictionary is of the form: ‘parameter_name:initial_value’
Args: g_parameters(
analysis.galfun.GParameters
): An object containing differentforms of the galaxy parameters.Returns: A dict.
-
analysis.defaults.
getMaximums
(g_parameters, gal_image)[source]¶ Return a dictionary containing the minimum values to be used in the in the fitting of the parameters.
The dictionary is of the form: ‘parameter_name:maximum_value’
Args: g_parameters(
galfun.GParameters
): An object containing differentforms of the galaxy parameters.Returns: A dict.
-
analysis.defaults.
getMinimums
(g_parameters, gal_image)[source]¶ Return a dictionary containing the minimum values to be used in the in the fitting of the parameters.
The dictionary is of the form: ‘parameter_name:initial_value’
Args: g_parameters(
analysis.galfun.GParameters
): An object containing differentforms of the galaxy parameters.Returns: A dict.
-
analysis.defaults.
getSteps
(g_parameters, image_renderer)[source]¶ Return a dictionary containing the steps to be used in the derivatives of each parameter.
The dictionary is of the form: ‘parameter_name:value_of_step’
Some parameter variations were copied from David’s code suggestions.
Args: g_parameters(
analysis.galfun.GParameters
): An object containing differentforms of the galaxy parameters.Returns: A dict.