crikit.preprocess package — CRIKit2 0.4.3 documentation

# crikit.preprocess package

## Subpackages

- crikit.preprocess.algorithms package
  - Submodules
  - crikit.preprocess.algorithms.abstract_als module
  - crikit.preprocess.algorithms.als module
  - crikit.preprocess.algorithms.anscombe module
  - crikit.preprocess.algorithms.arpls module
  - Module contents
- crikit.preprocess.tests package
  - Submodules
  - crikit.preprocess.tests.test_als module
  - crikit.preprocess.tests.test_anscombe module
  - crikit.preprocess.tests.test_baseline_sub module
  - crikit.preprocess.tests.test_crop module
  - crikit.preprocess.tests.test_subtract_dark module
  - Module contents

## Submodules

## crikit.preprocess.crop module

Created on Tue Jun 21 14:28:49 2016

@author: chc

class crikit.preprocess.crop.CutEveryNSpectra(offset=0, cut_m=1, every_n=100, action='cut')

  Bases: `object`

  Cut m spectra between every n spectra

  Parameters

    - offset (int, option (default = 0)) – Start the process at the offset-th position

    - cut_m (int, optional (default = 1)) – Cut m spectra at a time

    - every_n (int, optional (default = 100)) – Spacing between cuts

    - action (str) – Whether to ‘cut’, use the ‘mean’ of the remaining, replace with the spectrum ‘before’ or ‘after’

  Return type

    ndarray

  Note

  Currently, this class performs the action on a copy of the data, returning a copy

  calculate(data)
  transform(data)

class crikit.preprocess.crop.ZeroColumn(first_or_last=0, zero_col=None)

  Bases: `object`

  Set first or last column that is not all 0’s to 0.

  Parameters

    - first_or_last (int, optional (default = 0 [first])) – Find first (= 0) or last (= -1)

    - zero_col (int, optional (default = None)) – Zero a specified column

  calculate(data)
  transform(data)

class crikit.preprocess.crop.ZeroRow(first_or_last=0, zero_row=None)

  Bases: `object`

  Set first or last row that is not all 0’s to 0.

  Parameters

    - first_or_last (int, optional (default = 0 [first])) – Find first (= 0) or last (= -1)

    - zero_row (int, optional (default = None)) – Zero a specified row

  calculate(data)
  transform(data)

## crikit.preprocess.denoise module

Denoising

Created on Fri Apr 22 23:55:22 2016

@author: chc

class crikit.preprocess.denoise.SVDDecompose(rng=None)

  Bases: `object`

  Compute the SVD of a signal (just wraps numpy.linalg.svd) i.e., decompose the input into components.

  Parameters

    - data (ndarray (2D or 3D).) – Input array.

    - rng (ndarray (1D), optional) – Range of pixels to perform operation over.

  Returns

    U, s, Vh

  Return type

    ndarray, ndarray, ndarray

  Notes

  Undarray (2D)

    U-component from SVD decomposition (spatial componenet with crikit)

  Vhndarray (2D)

    Vh-component from SVD decomposition (spectral componenet with crikit). NOTE: this is the Hermitial/conjugate transpose of the normal V-component in SVD

  sndarray (1D)

    Diagonal elements of S-matrix describing the relative contributions of each singular value

  Sndarray (2D)

    S-matrix derived from s

  References

  calculate(data)

    Calculate SVD and return U, s, and Vh.

class crikit.preprocess.denoise.SVDRecompose(rng=None)

  Bases: `object`

  Reconstruct the original data using the SVD components. The reconstructed signal shape is 2D (or if provided) or matches data_obj.

  Parameters

    - U (ndarray (2D)) – U-component from SVD decomposition (spatial componenet with crikit)

    - s (ndarray (1D)) – Diagonal elements of S-matrix describing the relative contributions of each singular value

    - Vh (ndarray (2D)) – Vh-component from SVD decomposition (spectral componenet with crikit). NOTE: this is the Hermitial/conjugate transpose of the normal V-component in SVD

    - data (ndarray (2D or 3D)) – Original data (for overwrite if selected).

    - rng (ndarray (1D), optional) – Range of pixels to perform operation over.

    - overwrite (bool, optional (default=True)) – Overwrite the original data in data_obj

  Returns

    - ndarray (2D or 3D) – Recomposed data (U*S*Vh). If data_obj is not None, returned object shape matches data_obj. Else 2D.

    - None – Returns None if overwrite is True and overwrites input data_obj.

  Notes

  Sndarray (2D)

    S-matrix derived from s

  References

  calculate(data, U, s, Vh, svs=None)
  transform(data, U, s, Vh, svs=None)

## crikit.preprocess.standardize module

Standardization

Created on Thu Apr 14 08:53:08 2016

@author: chc

class crikit.preprocess.standardize.Anscombe(gauss_std, gauss_mean=0.0, poisson_multi=1.0, rng=None)

  Bases: `object`

  Implement the generalized forward Anscombe transformation.

  Signal : \(X\)

  Mean of Gaussian noise : \(<g>\)

  Standard deviation of Gaussian noise : \(\sigma_g\)

  Noise of type ‘type’ : \(N_{type}\)

  Poisson noise multiplier : \(\alpha\)

  Model : \(X = \alpha*N_{Poisson}\{X\} + N_{Gauss}\{<g>, \sigma_g\},\)

  Parameters

    - data (ndarray.) – Signal with mixed Gaussian and Poisson noise to transform.

    - gauss_std (float) – Standard deviation of Gaussian noise. \(\sigma_g\) in model.

    - poisson_multi (float, optional (default=1.0)) – A multiplier that scales the effect of the Poisson noise. \(\alpha\) in model.

    - gauss_mean (float, optional (default=0.0)) – Mean Gaussian noise level. \(<g>\) in model.

    - rng (ndarray (1D), optional) – Range of pixels to perform operation over.

    - overwrite (bool, optional (default=True)) –

  Returns

    - ndarray – Altered data if overwrite is False

    - None – Return None if overwrite is True

  See also

  - See the docstring of ./algorithms/anscombe for more information.

  References

  [1] M. Mäkitalo and A. Foi, “Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise”, IEEE Trans. Image Process., doi:10.1109/TIP.2012.2202675

  [2] J.L. Starck, F. Murtagh, and A. Bijaoui, Image Processing and Data Analysis, Cambridge University Press, Cambridge, 1998)

  [3] C. H. Camp Jr, Y. J. Lee, and M. T. Cicerone, “Quantitative, comparable coherent anti-Stokes Raman scattering (CARS) spectroscopy: Correcting errors in phase retrieval,” Journal of Raman Spectroscopy 47, 408-415 (2016). arXiv:1507.06543.

  calculate(data)

    Generalized Anscombe transform (Return copy).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns Anscombe-transformed data (or None if fails)

    Return type

      ndarray

  transform(data)

    Generalized Anscombe transform (Overwrite data).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns the success state (True=success)

    Return type

      bool

class crikit.preprocess.standardize.AnscombeInverse(gauss_std, gauss_mean=0.0, poisson_multi=1.0, rng=None)

  Bases: `object`

  Applies an exact, unbiased inverse of the generalized Anscombe variance-stabilizing transformation assuming a mixed Poisson-Gaussian noise model as:

  Signal : \(X\)

  Mean of Gaussian noise : \(<g>\)

  Standard deviation of Gaussian noise : \(\sigma_g\)

  Noise of type ‘type’ : \(N_{type}\)

  Poisson noise multiplier : \(\alpha\)

  Model : \(X = \alpha*N_{Poisson}\{X\} + N_{Gauss}\{<g>, \sigma_g\},\)

  Parameters

    - data (Spectrum (or subclass) object or ndarray.) – Signal with mixed Gaussian and Poisson noise to transform.

    - gauss_std (float) – Standard deviation of Gaussian noise. \(\sigma_g\) in model.

    - poisson_multi (float, optional (default=1.0)) – A multiplier that scales the effect of the Poisson noise. \(\alpha\) in model.

    - gauss_mean (float, optional (default=0.0)) – Mean Gaussian noise level. \(<g>\) in model.

    - rng (ndarray (1D), optional) – Range of pixels to perform operation over.

    - overwrite (bool, optional (default=True)) –

  See also

  - See the docstring of ./algorithms/anscombe for more information.

  References

  [1] M. Mäkitalo and A. Foi, “Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise”, IEEE Trans. Image Process., doi:10.1109/TIP.2012.2202675

  [2] J.L. Starck, F. Murtagh, and A. Bijaoui, Image Processing and Data Analysis, Cambridge University Press, Cambridge, 1998)

  [3] C. H. Camp Jr, Y. J. Lee, and M. T. Cicerone, “Quantitative, comparable coherent anti-Stokes Raman scattering (CARS) spectroscopy: Correcting errors in phase retrieval,” Journal of Raman Spectroscopy 47, 408-415 (2016). arXiv:1507.06543.

  calculate(data)

    Generalized Inverse Anscombe transform (Return copy).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns Anscombe-transformed data (or None if fails)

    Return type

      ndarray

  transform(data)

    Generalized Inverse Anscombe transform (Overwrite data).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns the success state (True=success)

    Return type

      bool

crikit.preprocess.standardize.calc_anscombe_parameters(dark_array, rep_array, axis=0, rng=None, dark_subtracted=False)

  Calculate Anscombe parameters: Gaussian noise mean and standard deviation, Poisson noise amplifer

  Parameters

    - dark_array (ndarray (n_spectra, n_lambda)) – Dark spectra array

    - rep_array (ndarray (n_spectra, n_lambda)) – Repetitive signal spectra

    - axis (int) – Non-spectral axis (ie replicate axis)

    - rng (ndarray (1d)) – Spectral pixel range to use

    - dark_subtracted (bool) – Was the mean dark spectrum already subtracted from the rep_array

    - Returns (dict) – ‘g_mean’, ‘g_std’, ‘alpha’, ‘weighted_mean_alpha’, which are the mean Gaussian noise (ndarray), Gaussian noise standard deviation (ndarray), Poisson multiplication factor (ndarray), weighted mean of the Poisson multiplication factor (float)

## crikit.preprocess.subtract_baseline module

Subtract baseline

Created on Sat May 28 00:41:41 2016

@author: chc

class crikit.preprocess.subtract_baseline.SubtractBaselineALS(smoothness_param=1, asym_param=0.01, redux=10, order=2, rng=None, fix_end_points=False, fix_rng=None, fix_const=1, max_iter=100, min_diff=1e-05, use_imag=True, **kwargs)

  Bases: `object`

  Subtract baseline using asymmetric least squares algorithm

  Parameters

    - smoothness_param (float, optional (default=1.0)) – Smoothness parameter aka ‘lambda’

    - asym_param (float, optional (default=1e-2)) – Asymmetry parameter aka ‘p’

    - redux_factor (int, optional (default=10)) – Down-sampling factor (more down-sampling leads to faster detrending, but with more chance of non-optimal detrending)

    - rng (ndarray (1D), optional (default=None)) – Range in pixels to perform action over

    - use_imag (bool, optional (default=True)) – If spectrum(a) are complex-values, use the imaginary portion?

  calculate(data, **kwargs)
  transform(data, **kwargs)

## crikit.preprocess.subtract_dark module

Subtract mean value (optionally, over a range from all spectrum/spectra/hsi)

Note: If dark > 1D, averaged – even if data has same shape.

class crikit.preprocess.subtract_dark.SubtractDark(dark)

  Bases: `object`

  calculate(data)

    Subtract dark spectrum (return copy).

    Parameters

      data (ndarray) – Data from which dark is subtracted.

    Returns

      Returns data with dark subtracted (or None if fails)

    Return type

      ndarray

  transform(data)

    Subtract dark spectrum (overwrite original data).

    Parameters

      data (ndarray) – Data from which dark is subtracted.

    Returns

      Returns the success state (True=success)

    Return type

      bool

## crikit.preprocess.subtract_mean module

Subtract mean value (optionally, over a range from all spectrum/spectra/hsi)

Created on Thu May 26 14:31:39 2016

@author: chc

class crikit.preprocess.subtract_mean.SubtractMeanOverRange(rng=None)

  Bases: `object`

  calculate(data)

    Subtract the mean intensity over a pixel range (rng). (Return copy).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns data with mean subtracted (or None if fails)

    Return type

      ndarray

  transform(data)

    Subtract the mean intensity over a pixel range (rng). (Overwrite data).

    Parameters

      data (ndarray) – Input data.

    Returns

      Returns the success state (True=success)

    Return type

      bool

## Module contents