Installation¶

From source:

git clone https://github.com/phe-bioinformatics/PHEnix.git
pip2 install -e PHEnix

Directly from github:

pip install git+https://github.com/phe-bioinformatics/PHEnix.git

Overview¶

This code was designed to allow users to input fastq files and a reference sequence and perform:

• Reference mapping
• VCF generation
• VCF filtering
• FASTA sequence of SNPs

The process is comprised of three steps:

• Reference sequence preparation (prepare_reference.py)
• Mapping and filtered VCF generation (run_snp_pipeline.py)
• FASTA generation from single or multiple VCFs (vcf2fasta.py)

Example:

prepare a bwa and gatk reference using a fasta file (myref.fasta)

phenix.py prepare_reference \
--mapper bwa \
--variant gatk \
--reference myref.fasta

map, call and filter variants on fastq files (my.R1.fastq, my.R2.fastq). Filter SNPs on minimum depth, mapping quality and AD ratio

phenix.py run_snp_pipeline \
-r1 my.R1.fastq \
-r2 my.R2.fastq \
-r myref.fasta \
--sample-name mysample \
--mapper bwa \
--variant gatk \
--filters min_depth:5,mq_score:30,ad_ratio:0.9

generate a FASTA file of SNPs using filtered VCFs in current directory

phenix.py vcf2fastq -d ./ -o output.fasta --regex filtered

Requirements¶

A lot of functionality depends on the presence of existing 3rd party tools:

Mappers:

• BWA - Download from [https://github.com/lh3/bwa]
• Bowtie2 - Download from [https://github.com/BenLangmead/bowtie2]

Variant Caller:

• GATK - Download from [https://www.broadinstitute.org/gatk/download/] * Picard - Download from [http://broadinstitute.github.io/picard/]
• MPileup - Download from [https://github.com/samtools/samtools]

In order for them to function properly, they need to be already in you PATH. For commands that run through Java archives, please set appropriate environment variable (see below).

Python¶

• Python >= 2.7
• argparse
• PyVCF
• PyYAML
• matplotlib (_optional_)
• bintrees (_optional_) - If you are using vcf2fasta.py
• numpy (_optional_)
• matplotlib.venn (_optional_) - psycopg2 (_optional_)

3rd Party Requirements¶

Samtools¶

Samtools Samtools can be downloaded from https://github.com/samtools/samtools. It is used to filter and convert to SAM/BAM files and in mpileup variant caller.

BCFTools¶

BCFtools can be downloaded from https://github.com/samtools/bcftools. It is used for calling variants in mpileup.

BWA Heng Li’s mapper can be downloaded from https://github.com/lh3/bwa.

Bowtie2 Bowtie2 mapper available from https://github.com/BenLangmead/bowtie2.

GATK Set GATK_JAR - full path to the GATK Java archive.

Picard Tools¶

Picard is needed for GATK to create dictionary of reference fasta. Either set PICARD_TOOLS_PATH - path to directory where different Picard jars are or set PICARD_JAR - path to picard.jar. Older Picard distributions have many different jars (use first suggestion), where as newer versions have merged all into one jar file.

Config¶

If you have a pipeline, or want to run the same settings for different samples this process can be simplefied by having a single config file. The file is in YAML format and takes common options that may be used for all samples:

• mapper - string
• mapper-options - string
• variant - string
• variant-options - string
• filters - dictionary
• annotators - list

Below is an example of config file:

mapper: bwa
mapper-options: -t 4
variant: gatk
variant-options: --sample_ploidy 2 --genotype_likelihoods_model SNP -rf BadCigar -out_mode EMIT_ALL_SITES -nt 1
filters:
  ad_ratio: 0.9
  min_depth: 5
  qual_score: 30
  mq_score: 30
annotators:
  - coverage

Filters¶

One of the key aims of this project is to provide a flexible and extendable interface for filtering VCFs. While there are a number of tools that will process a VCF file, we felt that none offered in depth understanding that we wanted. Instead we have based this library on PyVCF and extended with variety of filters Filters.

If you feel that there is a filter that has not been implemented, or you want to try your hand at implementing object oriented classes, then please see Implementing Filter.

Mappers¶

We found there was no single neat way of calling different mappers (there is BioPython, of course) but it wasn’t feasible to integrate it into our software stack in the way that we wanted to. Instead, we wrote a simple interfaces that can be easily extended to add new mappers. See Implementing Mapper.

Variant Callers¶

Just like with mappers, there was no neat way of incorporating other interfaces into this software stack. To add your favourite variant caller see Implementing Variant Caller.

run_snp_pipeline.py --help

run_snp_pipeline.py --help

run_snp_pipeline.py --help

class MyFilter(PHEFilterBase):
    name="CoolQUAL"
    _default_threshold=40
    parameter=cool_qual

    def __init__(self, args):
        # Construct any specific details for the object

    def __call__(self, record):
        if record.QUAL < self.threshold:
            return record.QUAL

    def short_desc(self):
        return "My cool QUAL filter"

phenix¶

There is a single entry point to access all commands we have produced so far. Please refer to this documentation or –help on the cammand line.

usage: phenix [-h] [--debug] [--version]
              {run_snp_pipeline,filter_vcf,prepare_reference,vcf2fasta} ...

Options:

--debug=False	More verbose logging (default: turned off).
--version	show program’s version number and exit

Sub-commands:

run_snp_pipeline

Run SNP pipeline.

Run the snp pipeline with specified mapper, variant caller and some filters. Available mappers: [‘bwa’, ‘bowtie2’] Available variant callers: [‘mpileup’, ‘gatk’] Available filters: [‘gq_score’, ‘dp4_ratio’, ‘ad_ratio’, ‘min_depth’, ‘mq_score’, ‘mq0_ratio’, ‘uncall_gt’, ‘qual_score’, ‘mq0f_ratio’] Available annotators: [‘coverage’]

usage: phenix run_snp_pipeline [-h] [--workflow WORKFLOW] [--input INPUT]
                               [-r1 R1] [-r2 R2] [--reference REFERENCE]
                               [--sample-name SAMPLE_NAME] [--outdir OUTDIR]
                               [--config CONFIG] [--mapper MAPPER]
                               [--mapper-options MAPPER_OPTIONS] [--bam BAM]
                               [--variant VARIANT]
                               [--variant-options VARIANT_OPTIONS] [--vcf VCF]
                               [--filters FILTERS]
                               [--annotators ANNOTATORS [ANNOTATORS ...]]
                               [--keep-temp]

Options:

--workflow, -w	Undocumented
--input, -i	Undocumented
-r1	R1/Forward read in Fastq format.
-r2	R2/Reverse read in Fastq format.
--reference, -r
	Rerefence to use for mapping.
--sample-name=test_sample
	Name of the sample for mapper to include as read groups.
--outdir, -o	Undocumented
--config, -c	Undocumented
--mapper=bwa, -m=bwa
	Available mappers: [‘bwa’, ‘bowtie2’]
--mapper-options
	Custom maper options (advanced)
--bam	Undocumented
--variant=gatk, -v=gatk
	Available variant callers: [‘mpileup’, ‘gatk’]
--variant-options
	Custom variant options (advanced)
--vcf	Undocumented
--filters	Filters to be applied to the VCF in key:value pairs, separated by comma (,). Available_filters: [‘gq_score’, ‘dp4_ratio’, ‘ad_ratio’, ‘min_depth’, ‘mq_score’, ‘mq0_ratio’, ‘uncall_gt’, ‘qual_score’, ‘mq0f_ratio’]
--annotators	List of annotators to run before filters. Available: [‘coverage’]
--keep-temp=False
	Keep intermediate files like BAMs and VCFs (default: False).

filter_vcf

Filter a VCF.

Filter the VCF using provided filters.

usage: phenix filter_vcf [-h] --vcf VCF [--filters FILTERS | --config CONFIG]
                         --output OUTPUT [--reference REFERENCE] [--only-good]

Options:

--vcf, -v	VCF file to (re)filter.
--filters, -f	Filter(s) to apply as key:threshold pairs, separated by comma.
--config, -c	Config with filters in YAML format. E.g.filters:-key:value
--output, -o	Location for filtered VCF to be written.
--reference, -r
	mpileup version <= 1.3 do not output all positions. This is required to fix rfrence base in VCF.
--only-good=False
	Write only variants that PASS all filters (default all variants are written).

prepare_reference

Create aux files for reference.

Prepare reference for SNP pipeline by generating required aux files.

usage: phenix prepare_reference [-h] --reference REFERENCE [--mapper MAPPER]
                                [--variant VARIANT]

Options:

--reference, -r
	Path to reference file to prepare.
--mapper	Available mappers: [‘bwa’, ‘bowtie2’]
--variant	Available variants: [‘mpileup’, ‘gatk’]

vcf2fasta

Convert VCFs to FASTA.

Combine multiple VCFs into a single FASTA file.

usage: phenix vcf2fasta [-h]
                        (--directory DIRECTORY | --input INPUT [INPUT ...])
                        [--regexp REGEXP] --out OUT
                        [--with-mixtures WITH_MIXTURES]
                        [--column-Ns COLUMN_NS] [--column-gaps COLUMN_GAPS]
                        [--sample-Ns SAMPLE_NS] [--sample-gaps SAMPLE_GAPS]
                        [--reference REFERENCE]
                        [--include INCLUDE | --exclude EXCLUDE]
                        [--with-stats WITH_STATS] [--tmp TMP]

Options:

--directory, -d
	Path to the directory with .vcf files.
--input, -i	List of VCF files to process.
--regexp	Regular expression for finding VCFs in a directory.
--out, -o	Path to the output FASTA file.
--with-mixtures
	Specify this option with a threshold to output mixtures above this threshold.
--column-Ns	Keeps columns with fraction of Ns below specified threshold.
--column-gaps	Keeps columns with fraction of Ns below specified threshold.
--sample-Ns	Keeps samples with fraction of Ns below specified threshold.
--sample-gaps	Keeps samples with fraction of gaps below specified threshold.
--reference	If path to reference specified (FASTA), then whole genome will be written.
--include	Only include positions in BED file in the FASTA
--exclude	Exclude any positions specified in the BED file.
--with-stats	If a path is specified, then position of the outputed SNPs is stored in this file. Requires mumpy and matplotlib.
--tmp	Location for writing temp files (default: /tmp).

Primer¶

The Phenix Galaxy tool only enables you to filter VCF files and to convert VCF files into FASTA files. The mapping and the SNP calling component of Phenix need to be provided by other Galaxy tools (also available from the Toolshed.)

How to get your own Galaxy server:¶

If you already have a local Galaxy server on which you are an Administrator and on which you can install tools from the Toolshed you can skip this section and continue reading at “How to install Phenix to your Galaxy server”. If not you might want to get one, which is easy.

Prerequisites:

• A workstation running a Linux operating system
• Min. 4GB of RAM (better 8GB or more)
• Root access to that machine.
• git (optional)
• Python 2.6 or 2.7
• GNU Make, gcc to compile and install tool dependencies

The last three are standard on most contemporary Linux installations.

Get your own Galaxy:

Please go to [https://wiki.galaxyproject.org/Admin/GetGalaxy] and follow the instructions in the sections from “Get the Code” to “Become an Admin” (including).

How to install Phenix to your Galaxy server:¶

This page (https://wiki.galaxyproject.org/Admin/Tools/AddToolFromToolShedTutorial) describes the general procedure of installing a tool from the toolshed. Here is a click-by-click guide to what you need to do to install Phenix.

Prerequisites:

There is a small number of Linux packages that need to be installed on your machine, so that Phenix can be installed and run propery. These are:

• a C++ compiler
• a Fortran compiler
• the zlib development libraries
• the samtools package

The exact name of these packages and the commands to install them depend on yout Linux distribution. For Ubuntu, Mint and Debian Linux the required commands should be:

sudo apt-get install build-essential
sudo apt-get install gfortran
sudo apt-get install zlib-devel
sudo apt-get install samtools

On Fedora the commands should be:

sudo dnf install gcc-c++
sudo dnf install gcc-gfortran
sudo dnf install zlib-devel
sudo dnf install samtools

And on OpenSUSE the commands should be:

sudo zypper install gcc-c++
sudo zypper install gcc-fortran
sudo zypper install zlib-devel
sudo zypper install samtools

For more esoteric Linux distributions please refer to your local IT expert and/or Google. After you have successfully installed these packages, follow the instructions below.

• Make sure that you can access ftp sites from the command line running your Galaxy server. This is normally enabled by default, but sometimes requires an additional proxy setting. Try this command

wget ftp://ftp.gnu.org/gnu/libtool/libtool-2.4.tar.gz

If that does not download a file named ‘libtool-2.4.tar.gz’ to your current folder, speak to your local systems’ administrator.

• In your config/galaxy.ini files, set

tool_dependency_dir = tool_dependencies

• Restart your Galaxy
• In your Galaxy, click on ‘Admin’ in the main menu at the top.
• Select ‘Search Tool Shed’ from the menu on the left hand side.
• Click on the little black triangle (the context menu) next to ‘Galaxy Main Tool Shed’ and select ‘Browse valid repositories’.
• Type ‘phephenix’ [sic] into the search box and press Enter.
• You should see the “package_phephenix_1_0” and the “phephenix” repositories. Select ‘Preview and install’ from the context menu of the latter.
• Click ‘Install to Galaxy’.
• Type ‘PHE TOOLS’ into the ‘Add new tool panel section:’ textbox.
• Click ‘Install’.
• You will be presented with a long list of packages that need to be installed. This will take a while. Wait until everything is green. If nothing happens for a little while, try reloading the page.

In your admin tool panel the menu item “Manage installed tools” was added. You can check the status of your installed packages there.

You need to install two more tools that are part of the Phenix workflow:

• Browse the toolshed as before and look for ‘bwa’. Install the tool ‘bwa’ with synopsis “Wrapper for bwa mem, aln, sampe, and samse” owned by ‘devteam’. Put it into the ‘PHE TOOLS’ tool panel section.
• Browse the toolshed as before and install the tool ‘phe_samtools_mpileup’. Put it into the ‘PHE TOOLS’ tool panel section.

How to use Phenix on Galaxy:¶

• In the admin menu, go the ‘Manage installed tools’
• Find the ‘phephenix’ tool and click on it, not on it’s context menu.
• In the ‘Contents of this repository’ box at the bottom of the page, expand the workflows section using the little blue triange if necessary.
• Click on ‘Phenix workflow’ and behold an image representation of the workflow.
• Click on ‘Repository actions’ -> ‘Import workflow to Galaxy’

The Phenix workflow is now ready to use. You need to upload your data to a Galaxy history to use it. There are multiple options depending on your local Galaxy configuration. If you have followed the instructions above under ‘Get your own Galaxy’ the only available option is uploading from your local harddrive. When doing this, please make sure your fastq files are in the ‘fastqsanger’ format and your reference genome is in ‘fasta’ format. The basename of your R1 fastq files will be used as the name for the resulting bam and vcf files with appropriate file endings and will also appear as the sequence header in your resulting SNP alignment. Once your data is ready to use follow theses instructions to run the workflow.

Remark: Processing a single sample with the Phenix workflow can use up to 1.5GB of RAM. It is recommended you do not run more samples than your total system memory divided by 1.5 (2 for 4GB, 5 for 8GB, ...) or only as may samples as you have processor cores, whichever is lower.

• Click on workflow in the top main menu and select ‘run’ from the ‘Phenix workflow’ context menu.
• Select your reference genome in the ‘Reference fasta’ selection box.
• Click both “paper stack” icons next to ‘R1 fastqs’ and next to ‘R2 fastqs’.
• Select all R1.fastq files in the “R1 fastqs” selection box that appeared. An easy way to do this is to type ‘R1’ into the text box immediately under the selection box. The selection box will automatically be filtered for fastq file names that match what has been typed into the text box.
• Do as above for the “R2 fastq” box.

• If you are happy with the above, click ‘Run workflow’.
• Wait until three new history items per sample processed appear. These will be one bam file, one raw vcf, and one filtered vcf per sample.
• Click on PHE TOOLS in the tool panel on the left hand side and select the “VCFs to fasta” tool. This tool will create a multi-fasta file from your filtered vcfs.
• Select all files ending with filtered.vcf in the top file selection box under ‘Input VCF files(s)’, by holding down the Ctrl key.
• Click ‘Execute’. Once everything is completed the “VCFs to fasta” tool with have produced your SNP alignment that can now be used for further processing.