Welcome to Statistical Wavelet Filtering’s documentation!¶

Instalation¶

For instalation just put in your terminal:

pip install statsWaveletFilt

Getting started¶

First you can import some modules like:

import pywt
import numpy as np

import statsWaveletFilt.filtration as fil
import statsWaveletFilt.signals as sig
import statsWaveletFilt.miscellaneous as misc

First step: Signal modelling

In many test for filtration and/or estimation of error, showed in articles, are supose an known ideal signal, how show the code above:

import matplotlib.pyplot as plt

dimension = 1024

t, idealSignal = sig.dopplerFunction(dimension)
plt.plot(t, idealSignal, 'k-')
plt.show()

This package also gives a support for anothers three ideal signal: the heavsine (sig.heavsineFunction()), bump (sig.bumpFunction()) and block (sig.blockFunction()).

And add to this a kind of noise, generally gaussian or, also called, normal.

Second step: Wavelet transformation

Let’s do this using the pywt.wavedec function. With this you can 1) make a wavelet transform of your noisy signal, 2) choice wich wavelet function is used ('haar', 'db', 'gaus', 'mexh' among others) and 3) choice any levels you want to evaluate the transform.

The first position = noiseCoeff has the scale coefficents of the last nível. The consecutive elements  = noiseCoeff[1:] has the wavelet coefficients of the last to the first level of transform.

Using misc.showWaveletCoeff function you can show all coefficients returned of the transform.

Thirth step: Filtration

With the fil.filtration function you can put yours 1) coefficients for filtration, 2) choice the method ('visu', 'sure', 'bayes', 'spc') and, if necessary, 3) set method own parameters (more details in documentation).

And, again, you can show the wavelet coefficients before and after the filtration, with the limiar used using the misc.showWaveletCoeff function.

Fourth step: Recovery wavelet transformation Using the pywt.waverec function we can revovery the signal.

Fifth step: Figures of merith Using sig.differential_snr_dB we can avaliate the filtration using some figures of merith of different forms of evaluate the snr or cnr (see more in Documentation) of the signal.

sig.differential_snr_dB(noisySignal, filtratedSignal, method=’variances’, idealSignal=idealSignal)

Appendix: Miscelaneous

• The misc.generateData function generate data using the signals showed here with different variances of noise and save in format .npy.
• The misc.normalizeData function normalize a data between a min and max value, this function is used in all signals modeled in sig module.

statsWaveletFilt package¶

cusum module¶

Wavelet Based in CUSUM control chart for filtering signals Project (module statsWaveletFilt.cusum ): Functions to analise data using Control Chart CUSUM. In this package the application of this chart is for filtration of wavelet coefficients.

Created by Tiarles Guterres, 2018

cusum.analysisCusum(data, k=0.5, mean=None, std=None, SjBi_start=0, Sjsi_start=0)[source]

Calculates the Control Limits of CUSUM like in . This is a Control Chart defined in  and this type of tool serves to make a control of data who is called in statistic “process”.

For more details about the parameters see  Chapter 9: Cumulative Sum and Exponentially Weighted Moving Average Control Charts.

Parameters: data (list or array-like) – This is the data of “process” who CUSUM has to analize k (int or float) – Optional, 1/2 (or .5) by default. It’s a parameter of the CUSUM algorithm. Helps to the Control Chart acummulate the Control Limits of each element of data. mean (int or float) – Optional, is None by default, but turns the mean of data. Also an intern parameter of the algorithm for help to acumulate the control limits. std (int or float) – Optional, is None by default, but turns the standard deviation of data. The same function of mean in relation of control limits. SjBi_start (int or float) – Optional, is 0 by default. Is the start value for acumulation of superior control limit. Sjsi_start (int or float) – Optional, is 0 by default. Is the start value for acumulation of inferior control limit. A tuple of control limits. In  the superior limits and in  the inferior limits. tuple

thresholdCusum()
Function used to truncation of data using the control limits obtained in this function and a decision interval, called “H”.

References

  MONTGOMERY, D. C. Introduction to Statistical Quality Control. Sixth edition. United States: John Wiley & Sons, Inc., 2009. 733 p.
  PAGE, E. S. Continous Inspection Schemes. Biometrika, v. 41, p. 100-115, 1954
cusum.thresholdCusum(data, SjB, Sjs, std=None, h=5)[source]

Makes the truncation of data accordyling with control limits SjB and Sjs and the interval of decision [H = h * data.std()]. The threshold method it’s showed in , more about cusum it’s showed in , Chapter 9.

Note

The size of data must be equal to size of SjB and Sjs.

Note

After the test (via pytest) the fuction was changed for better performance.

Parameters: data (list or array-like) – The data who corresponding to control limits. SjB (list or array-like) – The control superior limits who corresponding to data . Sjs (list or array-like) – The control inferior limits who corresponding to data . std (int or float) – Optional, is None by default, but turns the standard deviation of data. It’s an intern parameter of the algorithm for help to acumulate the control limits. h (int or float) – Optional, 5 by default. This variable multiply with standard deviation of data to obtain the interval of decision (H). An array with elements of data truncated or not, depending of the control limits and the interval of decision. numpy.array

analysisCusum()
Make the cusum analysis inthe data, return the control limits corresponding to data input.

References

  GUTERRES, T. D. R. M.; BAYER, F. M; KOZAKEVICIUS, A. D. J. (2018) Análise do gráfico de controle CUSUM para filtragem de coeficientes wavelet, Undergraduation Thesis, Universidade Federal de Santa Maria. In portuguese.
  MONTGOMERY, D. C. Introduction to Statistical Quality Control. Sixth edition. United States: John Wiley & Sons, Inc., 2009. 733 p.

filtration module¶

Wavelet Based in CUSUM control chart for filtering signals Project (module statsWaveletFilt.statisticFilter ): Top level functions to filter wavelet coefficients using consagrated methods (threshold) and using Control Chart CUSUM (cusum) proposed in my Undergraduate Thesis, together with prof. Dr. Fábio Mariano Bayer and prof. Dr. Alice de Jesus Kozakevicius called Análise do gráfico de controle CUSUM para a filtragem de coeficientes wavelet, in portuguese for the Universidade Federal de Santa Maria (2°/2018).

Created by Tiarles Guterres, 2018

filtration.filtration(coefficients, method='visu', p=3, mode='hard', dim_t=1024)[source]

Filters the wavelet coefficients returned by the pywt.wavedec function. All methods are implemented and showed in .

Parameters: coefficients (list of 1-D array-like) – The wavelet coefficients and the scale coefficients of the last level. The scale coefficients isn’t modify by the filtration. method (string) – Optional, is ‘visu’ by default. p (int or float) – Optional, is 3 by default. mode (string) – Optional, is ‘hard’ by default. A tuple with  A list if numpy.array. The wavelet coefficients truncated by the choiced method, with scale coefficients. Ready for pywt.waverec function. (a little ‘tip’) and  a list of float. The lambda value used for each wavelet coefficient level. tuple

cusumFiltration()
Function that use Cumulative Sum Control Chart and some variation for filter wavelet coefficients.

References

  KOZAKEVICIUS, A. D. J.; BAYER, F. M. Filtragem de sinais via limiarização de coeficientes wavelet. Ciência e Natura, v. 36, p. 37–51, 2014. In portuguese.

Filters the wavelet coefficients returned by the pywt.wavedec function using the Cumulative Sum Control Chart (CUSUM) .

Parameters: wavCoeff (list of array-like.) – Wavelet coefficients h (int, float or array-like) – Optional, 5 by default . See “method” parameter. k (int, float or array-like) – Optional, 1/2 (or .5) by default . See “method” parameter. method (string) – Optional, “cusumTrad” by default. If “method” is “cusumTrad” the Control Chart considered to filter the wavalet coefficients is the same considered in . If the control limit (called SjB and Sjs here) is bigger than threshold limit the wavelet coefficient corresponded will be zero. In this “method” “k” and “h” will be constant and equals for all wavelet coefficients in all levels, the  recomend h = 5 and k = 1/2, but you can change or just not alterate. But if “method” is “cusumDecay” occurs the same in “cusumTrad” relative to truncation form but the “k” and “h” will be like is described in . Or if “method” is “cusumAdap” will be the same of the “cusumDecay” but the user can be choice who values of “h” and “k” will be for each wavelet level. A tuple with  A list if numpy.array. The wavelet coefficients truncated by the choiced method, with scale coefficients. Ready for pywt.waverec function. (a little ‘tip’),  a list of float. The “k” values used for each wavelet coefficient level and  a list of float. The “h” values used for each wavelet coefficient level. tuple

filtration()
Function that use this function to filter via wavelet coefficients
pywt.wavedec()
Function that decomposes the signal in wavelet and scale coefficients
pywt.waverec()
Function that recomposes the signal from wavelet and scale coefficients

References

  MONTGOMERY, D. C. Introduction to Statistical Quality Control. Sixth edition. United States: John Wiley & Sons, Inc., 2009. 733 p.
  GUTERRES, T. D. R. M.; BAYER, F. M; KOZAKEVICIUS, A. D. J. (2018) Análise do gráfico de controle CUSUM para filtragem de coeficientes wavelet, Undergraduation Thesis, Universidade Federal de Santa Maria. In portuguese.

miscellaneous module¶

Wavelet Based in CUSUM control chart for filtering signals Project (module statsWaveletFilt.miscellaneous ): A Miscellaneous of functions for work with data and show wavelet coefficients

Created by Tiarles Guterres, 2018

miscellaneous.showWaveletCoeff(coefficients, filename='tmp', format='pdf', threshold_value=0, color='black', color_threshold='black', figsize=(7, 8), title='')[source]

Show and save the wavelet and scale coefficients in a plot.

Parameters: coeff (list of numpy.array's) – With in ‘0’ position the scale coefficients. Equal to the pywt.wavedec() return. filename (string) – Optional, is ‘tmp’ by default. This is the first part of the name of the figure. format (string) – Optional, is ‘pdf’ by default. This is the last part of the name of the figure. Can be ‘png’, ‘ps’, ‘eps’ and ‘svg’ too. threshold_value (int, float or list.) – Optional, is 0 by default, this means that bothing new happens. Otherwise, a line in threshold value will be plotted in all wavelet coefficients plots. This value can be a list too, but they was to be the same size of wavelet coefficients (without the scale coefficient). Nothing is returned, the plots is show and save. void

pywt.wavedec()
Function that decomposes the signal in wavelet and scale coefficients
pywt.waverec()
Function that recomposes the signal from wavelet and scale coefficients
filtration.filtration()
Function that use this function to filter via wavelet coefficients
filtration.filtrationCusum()
Function that use Cumulative Sum Control Chart and some variation for filter wavelet coefficients.
miscellaneous.normalizeData(data, min=0, max=1)[source]

Its almost a map function. This function normalize the data between a min and max values.

Parameters: data (list or array-like) – The values that desire normalize. min (int or float) – Optional, is -1 by default. The min value correspond, in the end, of the min value of data. max (int or float) – Optional, is 1 by default. The max value correspond, in the end, of the max value of data. The data normalized between min and max values. numpy.array
miscellaneous.generateData(functions=['doppler', 'block', 'bump', 'heavsine'], varNoises=[0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, 0.01], dim_signals=1024, n_samples_per_sig_per_noise=10000, folder='tmp')[source]

If you like to generate your dataset before run your test you can use this function to generate the data. With the 1) type of signal and 2) quantity of noise (in variance). Saves in .npy

signals module¶

Wavelet Based in CUSUM control chart for filtering signals Project (module statsWaveletFilt.signals ): Functions to evaluate the dignal fitering process using the module statisticFilter or any kind of filtration.

Created by Tiarles Guterres, 2018

signals.for_dB_scale(x)[source]

Converts x to dB scale using 10*log10(x)

Parameters: x (int or float) – The value for convertion The x value converted in dB scale. float
signals.for_real_scale(x)[source]

Converts x to real scale using 10**(x/10)

Parameters: x (int or float) – The value for convertion The x value converted in eal scale. float
signals.snr_square_mean_error(currentSignal, idealSignal)[source]

Calculate the SNR via the current signal and the ideal using the square mean error approach.

Parameters: currentSignal (1-D array-like) – The signal for compare with ideal. idealSignal (1-D array-like) – The ideal signal, based in the currentSignal. Mean of idealSignal by standard deviation of the noise. float
signals.snr_mean_standardNoise(idealSignal, noiseSignal)[source]

Calculate the SNR via ideal signal mean and standard deviation of the noise.

Parameters: idealSignal (1-D array-like) – The ideal signal, based in the currentSignal. noiseSignal (1-D array-like) – Noise apply to ideal signal, could be a initial or residual noise. Mean of idealSignal by standard deviation of the noise. float
signals.snr_variances(idealSignal, noiseSignal)[source]

Calculate the SNR via ratio of variances of ideal signal and noise.

Parameters: idealSignal (1-D array-like) – The ideal signal, based in the currentSignal. noiseSignal (1-D array-like) – Noise apply to ideal signal, could be a initial or residual noise. Variance ratio value between the ideal and noise signals. float
signals.cnr_amplitude_standardNoise(idealSignal, noiseSignal)[source]

Calculate the CNR (contrast-to-noise ratio ) via the amplitude of idealSignal and standard deviation of the noise.

Parameters: idealSignal (1-D array-like) – The ideal signal, based in the currentSignal. noiseSignal (1-D array-like) – Noise apply to ideal signal, could be a initial or residual noise. Ratio of maximum distance of zero and standard deviation of the noise. float
signals.differential_snr_dB(initialSignal, finalSignal, method='square_mean_error', idealSignal=None)[source]

Calculate the SNR or CNR difference between two signals: after and before filtering. Ideal signal may be used.

Parameters: initialSignal (1-D array-like) – Initial Signal, before the filtering process finalSignal (1-D array-like) – Final Signal, after the filtering process method (string, optional) – Is ‘square_mean_error’ by default, other forms of calculate the SNR differential is ‘mean_StandardNoise’, ‘variances’ and ‘amplitude_standardNoise’. idealSignal (1-D array-like or 0, optional) – Is 0 by default, is necessary in all methods except in ‘square_mean_error’ method. The SNR differential value in dB. float
signals.dopplerFunction(dim=1024, normalize=True, fq=0)[source]

Generate the Doppler function in a range of 0 to 1, with dim points.

Parameters: dim (int) – Dimension of the signal. normalize (bool, optional) – It is True by default. This parameter normalize the data values in a range of 0 to 1 with a function present in statsWaveletFilt.miscellaneous. fq (int or float, optional) – It is 0 by default. With this default value the original doppler, shown by Donoho  will be used.  1-D array-like, coordinates in X axis and  1-D array-like, coordinates in Y axis tuple

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.
signals.heavsineFunction(dim=1024, normalize=True, heavs=0)[source]

Generate the Heavsine function in a range of 0 to 1, with dim points.

Parameters: dim (int) – Dimension of the signal. normalize (bool, optional) – It is True by default. This parameter normalize the data values in a range of 0 to 1 with a function present in statsWaveletFilt.miscellaneous. heavs (int or float, optional) – It is 0 by default. This parameter, called * heavs * is the number of discontinuities in the heavens characteristic signal shown by Donoho  with 0 the signal will be the original, used in .  1-D array-like, coordinates in X axis and  1-D array-like, coordinates in Y axis tuple

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.
signals.blockFunction(dim=1024, normalize=True, ht=0)[source]

Generate the Block function in a range of 0 to 1, with dim points.

Parameters: dim (int) – Dimension of the signal. normalize (bool, optional) – It is True by default. This parameter normalize the data values in a range of 0 to 1 with a function present in statsWaveletFilt.miscellaneous. ht (int, optional) – It is 0 by default. The parameter called ht is the commutation characteristic of block signal. The default parameter will generate the signal shown in .  1-D array-like, coordinates in X axis and  1-D array-like, coordinates in Y axis tuple

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.
signals.bumpFunction(dim=1024, normalize=True, wht=0)[source]

Generate the Bump function in a range of 0 to 1, with dim points. Take care to the representation limits of this function is blows infinity in Y axis.

Parameters: dim (int) – Dimension of the signal. normalize (bool, optional) – It is True by default. This parameter normalize the data values in a range of 0 to 1 with a function present in statsWaveletFilt.miscellaneous. wht (int, optional) – It is 0 by default. The parameter called wht is the number of peaks characteristic of bump signal. The default parameter will generate the signal shown in .  1-D array-like, coordinates in X axis and  1-D array-like, coordinates in Y axis tuple

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.

threshold module¶

Wavelet Based in CUSUM control chart for filtering signals Project (module statsWaveletFilt.threshold ): Statistic functions for obtain threshold values for wavelet coefficients based in some referenced works.

Created by Tiarles Guterres, 2018

threshold.lambdasVisuShrink(wavCoeff)[source]

Computes the threshold value (lambda) by VisuShrink  method.

Parameters: wavCoeff (list of lists or array-like) – Wavelet coefficients The threshold values for each wavelet coefficients vector. list of float

filtration()
Function that use this function for filter via wavelet coefficients
pywt.wavedec()
Function that decomposes the signal in wavelet and scale coefficients
pywt.waverec()
Function that recomposes the signal from wavelet and scale coefficients

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.
threshold.lambdasSureShrink(wavCoeff, dim_t=1024)[source]

Computes the threshold value (lambda) by SureShrink  method. It’s showed also in .

Note

After the test (via pytest) the fuction was changed for better performance.

Parameters: wavCoeff (list of lists or array-like) – Wavelet coefficients dim_t (optional, 1024 by default. t-dimension. Input vector from) – internal function _sure(vector, dim_t). The threshold values for each wavelet coefficients vector. list of float

filtration()
Function that use this function to filter via wavelet coefficients
pywt.wavedec()
Function that decomposes the signal in wavelet and scale coefficients
pywt.waverec()
Function that recomposes the signal from wavelet and scale coefficients

References

  DONOHO, D. L.; JOHNSTONE, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika, v. 81, p. 425–455, 1994.
  KOZAKEVICIUS, A. D. J.; BAYER, F. M. Filtragem de sinais via limiarização de coeficientes wavelet. Ciência e Natura, v. 36, p. 37–51, 2014. In portuguese.
threshold.lambdasBayesShrink(wavCoeff)[source]

Computes the threshold value (lambda) by BayesShrink  method. It’s showed also in .

Parameters: wavCoeff (list of lists or array-like) – Wavelet coefficients The threshold values for each wavelet coefficients vector. list of float

filtration()
Function that use this function to filter via wavelet coefficients
pywt.wavedec()
Function that decomposes the signal in wavelet and scale coefficients
pywt.waverec()
Function that recomposes the signal from wavelet and scale coefficients

References

  CHANG, S. G.; YU, B.; VETTERLI, M. Adaptive wavelet thresholding for image denoising and compression. IEEE Transactions on Image Processing, v. 9, p. 1532–1546, 2000.
  KOZAKEVICIUS, A. D. J.; BAYER, F. M. Filtragem de sinais via limiarização de coeficientes wavelet. Ciência e Natura, v. 36, p. 37–51, 2014. In portuguese.
threshold.lambdasSPC_Threshold(wavCoeff, p=3)[source]

Computes the threshold value (lambda) by SPC-Threshold ,  method

Note

After the test (via pytest) the fuction was changed for better performance.

Parameters: wavCoeff (list of lists or array-like) – Wavelet coefficients p (int or float) – Optional, 3 by default. Parameter for the algorithm , generally is used 2 or 3. The threshold values for each wavelet coefficients vector. list of float