mat_handler

Cov

Bases: Matrix

Diagonal and/or dense Covariance matrices

Parameters:

Name	Type	Description	Default
x	`numpy.ndarray`	numeric values	None
names	[`str`]	list of row and column names	[]
isdigonal	`bool`	flag if the Matrix is diagonal	required
autoalign	`bool`	flag to control the autoalignment of Matrix during linear algebra operations	True

Example::

data = pyemu.en.rng.random((10,10))
names = ["par_{0}".format(i) for i in range(10)]
mat = pyemu.Cov(x=data,names=names)
mat.to_binary("mat.jco")

Note

row_names and col_names args are supported in the constructor so support inheritance. However, users should only pass names

T property

wrapper function for Matrix.transpose() method

Returns:

Type	Description
	Matrix: transpose of Matrix

Note

returns a copy of self

A syntatic-sugar overload of Matrix.transpose()

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

as_2d property

get a 2D numeric representation of Matrix.x. If not isdiagonal, simply return reference to Matrix.x, otherwise, constructs and returns a 2D, diagonal ndarray

Returns:

Type	Description
	numpy.ndarray : numpy.ndarray

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
x2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,x2d.shape)

full_s property

Get the full singular value matrix

Returns:

Type	Description
	Matrix: full singular value matrix. Shape is (max(Matrix.shape),max(Matrix.shape))
	with zeros along the diagonal from min(Matrix.shape) to max(Matrix.shape)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.full_s.to_ascii("full_sing_val_matrix.mat")

identity property

get an identity Cov of the same shape

Returns:

Type	Description
	Cov: new Cov instance with identity matrix

Note

the returned identity matrix has the same row-col names as self

inv property

inversion operation of Matrix

Returns:

Type	Description
	Matrix: inverse of Matrix

Note

uses numpy.linalg.inv for the inversion

Example::

mat = pyemu.Matrix.from_binary("my.jco")
mat_inv = mat.inv
mat_inv.to_binary("my_inv.jco")

names property

wrapper for getting row_names. row_names == col_names for Cov

Returns:

Type	Description
	[str]: list of names

ncol property

length of second dimension

Returns:

Type	Description
	int: number of columns

newx property

return a copy of Matrix.x attribute

Returns:

Type	Description
	numpy.ndarray: a copy Matrix.x

nrow property

length of first dimension

Returns:

Type	Description
	int: number of rows

s property

the singular value (diagonal) Matrix

Returns:

Type	Description
	Matrix: singular value matrix. shape is (min(Matrix.shape),min(Matrix.shape))

Example::

# plot the singular spectrum of the jacobian
import matplotlib.pyplot as plt
jco = pyemu.Jco.from_binary("pest.jcb")
plt.plot(jco.s.x)
plt.show()

shape property

get the implied, 2D shape of Matrix

Returns:

Type	Description
	int: length of 2 tuple

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
shape = jco.shape
nrow,ncol = shape #unpack to ints

sqrt property

element-wise square root operation

Returns:

Type	Description
	Matrix: element-wise square root of Matrix

Note

uses numpy.sqrt

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov.sqrt
sqcov.to_uncfile("sqrt_cov.unc")

transpose property

transpose operation of self

Returns:

Type	Description
	Matrix: transpose of Matrix

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

u property

the left singular vector Matrix

Returns:

Type	Description
	Matrix: left singular vectors. Shape is (Matrix.shape[0], Matrix.shape[0])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.u.to_binary("u.jcb")

v property

the right singular vector Matrix

Returns:

Type	Description
	Matrix: right singular vectors. Shape is (Matrix.shape[1], Matrix.shape[1])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.v.to_binary("v.jcb")

x property

return a reference to Matrix.x

Returns:

Type	Description
	numpy.ndarray: reference to Matrix.x

zero property

get a diagonal instance of Cov with all zeros on the diagonal

Returns:

Type	Description
	Cov: new Cov instance with zeros

zero2d property

get an 2D instance of self with all zeros

Returns:

Type	Description
	Matrix: Matrix of zeros

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
zero_jco = jco.zero2d

__add__(other)

Overload of numpy.ndarray.add() - elementwise addition. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to add	required

Returns:

Type	Description
	Matrix: the result of addition

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised. The addition of a scalar does not expand non-zero elements.

Example::

m1 = pyemu.Cov.from_parameter_data(pst)
m1_plus_on1 = m1 + 1.0 #add 1.0 to all non-zero elements
m2 = m1.copy()
m2_plus_m1 = m1 + m2

__getitem__(item)

a very crude overload of object.getitem().

Parameters:

Name	Type	Description	Default
item	`object`	something that can be used as an index	required

Returns:

Type	Description
	Matrix: an object that is a sub-matrix of Matrix

__mul__(other)

Dot product multiplication overload. Tries to speedup by checking for scalars or diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
cov = pyemu.Cov.from_parmaeter_data(pst)
# get the forecast prior covariance matrix
forecast_cov = jco.get(pst.forecast_names).T * cov * jco.get(pst.forecast_names)

__pow__(power)

overload of numpy.ndarray.pow() operator

Parameters:

Name	Type	Description	Default
power	`float`	interpreted as follows: -1 = inverse of self, -0.5 = sqrt of inverse of self, 0.5 = sqrt of self. All other positive ints = elementwise self raised to power	required

Returns:

Type	Description
	Matrix: a new Matrix object raised to the power power

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov**2

__rmul__(other)

Reverse order Dot product multiplication overload.

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

# multiply by a scalar
jco = pyemu.Jco.from_binary("pest.jcb")
jco_times_10 = 10 * jco

__set_svd()

private method to set SVD components.

Note: this should not be called directly

__str__()

overload of object.str()

Returns:

Type	Description
	str: string representation

__sub__(other)

numpy.ndarray.sub() overload. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to subtract	required

Returns:

Type	Description
	Matrix: the result of subtraction

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised

Example::

jco1 = pyemu.Jco.from_binary("pest.1.jcb")
jco2 = pyemu.Jco.from_binary("pest.2.jcb")
diff = jco1 - jco2

align(names, axis=None)

reorder Matrix by names in place. If axis is None, reorder both indices

Parameters:

Name	Type	Description	Default
names	[str]	names in Matrix.row_names and or Matrix.col_names	required
axis	`int`	the axis to reorder. if None, reorder both axes	None

Note

Works in place. Is called programmatically during linear algebra operations

Example::

# load a jco that has more obs (rows) than non-zero weighted obs
# in the control file
jco = pyemu.Jco.from_binary("pest.jcb")
# get an obs noise cov matrix
obscov = pyemu.Cov.from_observation_data(pst)
jco.align(obscov.row_names,axis=0)

condition_on(conditioning_elements)

get a new Covariance object that is conditional on knowing some elements. uses Schur's complement for conditional Covariance propagation

Parameters:

Name	Type	Description	Default
conditioning_elements	[str]	list of names of elements to condition on	required

Returns:

Type	Description
	Cov: new conditional Cov that assumes conditioning_elements have become known

Example::

prior_cov = pyemu.Cov.from_parameter_data(pst)
now_known_pars = pst.adj_par_names[:5]
post_cov = prior_cov.condition_on(now_known_pars)

copy()

get a copy of Matrix

Returns:

Type	Description
	Matrix: copy of this Matrix

df()

wrapper of Matrix.to_dataframe()

drop(names, axis)

drop elements from Matrix in place

Parameters:

Name	Type	Description	Default
names	[str]	list of names to drop	required
axis	`int`	the axis to drop from. must be in [0,1]	required

element_isaligned(other)

check if matrices are aligned for element-wise operations

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

extend(other)

extend Matrix with the elements of other, returning a new matrix.

Args: other (Matrix): the Matrix to extend self by

Returns:

Type	Description
	Matrix: new, extended Matrix

Note

No row or column names can be shared between self and other

Example::

jco1 = pyemu.Jco.from_binary("pest_history.jco")
jco2 = pyemu.Jco.from_binary("pest_forecast.jco")

jco_ext = jco1.extend(jco2)

extract(row_names=None, col_names=None)

wrapper method that Matrix.gets() then Matrix.drops() elements. one of row_names or col_names must be not None.

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names to extract. If None, all row_names are retained.	None
col_names	[str]	col_names to extract. If None, all col_names are retained.	None

Returns:

Type	Description
	Matrix: the extract sub-matrix defined by row_names and/or col_names

Example::

cov = pyemu.Cov.from_parameter_data(pst)
hk_cov = cov.extract(row_names=["hk1","hk2","hk3"])

find_rowcol_indices(names, row_names, col_names, axis=None) staticmethod

fast(er) look of row and column names indices

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

from_ascii(filename) classmethod

load a PEST-compatible ASCII matrix/vector file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	name of the file to read	required

Returns:

Type	Description
	Matrix: Matrix loaded from ASCII file

Example::

mat = pyemu.Matrix.from_ascii("my.mat")
cov = pyemi.Cov.from_ascii("my.cov")

from_binary(filename, forgive=False) classmethod

class method load from PEST-compatible binary file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	Matrix: Matrix loaded from binary file

Example::

mat = pyemu.Matrix.from_binary("my.jco")
cov = pyemu.Cov.from_binary("large_cov.jcb")

from_dataframe(df) classmethod

class method to create a new Matrix instance from a pandas.DataFrame

Parameters:

Name	Type	Description	Default
df	`pandas.DataFrame`	dataframe	required

Returns:

Type	Description
	Matrix: Matrix instance derived from df.

Example::

df = pd.read_csv("my.csv")
mat = pyemu.Matrix.from_dataframe(df)

from_fortranfile(filename) staticmethod

a binary load method to accommodate one of the many bizarre fortran binary writing formats

Parameters:

Name	Type	Description	Default
filename	`str`	name of the binary matrix file	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

from_names(row_names, col_names, isdiagonal=False, autoalign=True, random=False) classmethod

class method to create a new Matrix instance from row names and column names, filled with trash

Parameters:

Name	Type	Description	Default
row_names	[str]	row names for the new Matrix	required
col_names	[str]	col_names for the new matrix	required
isdiagonal	`bool`	flag for diagonal matrix. Default is False	False
autoalign	`bool`	flag for autoaligning new matrix during linear algebra calcs. Default is True	True
random	`bool`	flag for contents of the trash matrix. If True, fill with random numbers, if False, fill with zeros Default is False	False

Returns:

Type	Description
	Matrix: the new Matrix instance

Example::

row_names = ["row_1","row_2"]
col_names = ["col_1,"col_2"]
m = pyemu.Matrix.from_names(row_names,col_names)

from_observation_data(pst) classmethod

instantiates a Cov from pyemu.Pst.observation_data

Parameters:

Name	Type	Description	Default
pst	`pyemu.Pst`	control file instance	required

Returns:

Type	Description
	Cov: a diagonal observation noise covariance matrix derived from the
	weights in the pest control file. Zero-weighted observations
	are included with a weight of 1.0e-30

Example::

obscov = pyemu.Cov.from_observation_data(pst)

from_obsweights(pst_file) classmethod

instantiates a Cov instance from observation weights in a PEST control file.

Parameters:

Name	Type	Description	Default
pst_file	`str`	pest control file name	required

Returns:

Type	Description
	Cov: a diagonal observation noise covariance matrix derived from the
	weights in the pest control file. Zero-weighted observations
	are included with a weight of 1.0e-30

Note

Calls Cov.from_observation_data()

Example::

obscov = pyemu.Cov.from_obsweights("my.pst")

from_parameter_data(pst, sigma_range=4.0, scale_offset=True, subset=None) classmethod

Instantiates a Cov from a pest control file parameter data section using parameter bounds as a proxy for uncertainty.

Parameters:

Name	Type	Description	Default
pst_file	`str`	pest control file name	required
sigma_range	`float`	defines range of upper bound - lower bound in terms of standard deviation (sigma). For example, if sigma_range = 4, the bounds represent 4 * sigma. Default is 4.0, representing approximately 95% confidence of implied normal distribution	4.0
scale_offset	`bool`	flag to apply scale and offset to parameter upper and lower bounds before calculating variance. In some cases, not applying scale and offset can result in undefined (log) variance. Default is True.	True
subset	`list`-like	Subset of parameters to draw	None

Returns:

Type	Description
	Cov: diagonal parameter Cov matrix created from parameter bounds

Note

Calls Cov.from_parameter_data()

from_parbounds(pst_file, sigma_range=4.0, scale_offset=True) classmethod

Instantiates a Cov from a pest control file parameter data section using parameter bounds as a proxy for uncertainty.

Parameters:

Name	Type	Description	Default
pst_file	`str`	pest control file name	required
sigma_range	`float`	defines range of upper bound - lower bound in terms of standard deviation (sigma). For example, if sigma_range = 4, the bounds represent 4 * sigma. Default is 4.0, representing approximately 95% confidence of implied normal distribution	4.0
scale_offset	`bool`	flag to apply scale and offset to parameter upper and lower bounds before calculating variance. In some cases, not applying scale and offset can result in undefined (log) variance. Default is True.	True

Returns:

Type	Description
	Cov: diagonal parameter Cov matrix created from parameter bounds

Note

Calls Cov.from_parameter_data()

from_uncfile(filename, pst=None) classmethod

instaniates a Cov from a PEST-compatible uncertainty file

Parameters:

Name	Type	Description	Default
filename	`str`	uncertainty file name	required
pst	'pyemu.Pst`	a control file instance. this is needed if "first_parameter" and "last_parameter" keywords. Default is None	None

Returns:

Type	Description
	Cov: Cov instance from uncertainty file

Example::

cov = pyemu.Cov.from_uncfile("my.unc")

get(row_names=None, col_names=None, drop=False)

get a new Matrix instance ordered on row_names or col_names

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names for new Matrix. If None, all row_names are used.	None
col_names	[str]	col_names for new Matrix. If None, all col_names are used.	None
drop	`bool`	flag to remove row_names and/or col_names from this Matrix	False

Returns:

Type	Description
	Matrix: a new Matrix

Example:: # load a jco that has more obs (rows) than non-zero weighted obs # in the control file jco = pyemu.Jco.from_binary("pest.jcb") # get an obs noise cov matrix obscov = pyemu.Cov.from_observation_data(pst) nnz_jco = jco.get(row_names = obscov.row_names)

get_dense_binary_info(filename) staticmethod

read the header and row and offsets for a dense binary file.

Parameters

filename (`str`): dense binary filename

Returns:

Type	Description
	tuple containing
	['str']: list of row names
	['int']: list of row offsets
	[str]: list of col names
	bool: flag indicating successful reading of all records found

get_diagonal_vector(col_name='diag')

Get a new Matrix instance that is the diagonal of self. The shape of the new matrix is (self.shape[0],1). Self must be square

Parameters:

Name	Type	Description	Default
col_name	`str`	the name of the single column in the new Matrix	'diag'

Returns:

Type	Description
	Matrix: vector-shaped Matrix instance of the diagonal of this Matrix

Example::

cov = pyemu.Cov.from_unc_file("my.unc")
cov_diag = cov.get_diagonal_vector()
print(cov_diag.col_names)

get_maxsing(eigthresh=1e-05)

Get the number of singular components with a singular value ratio greater than or equal to eigthresh

Args: eigthresh (float): the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Note

this method calls the static method Matrix.get_maxsing_from_s() with Matrix.s.x

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = jco.get_maxsing(eigthresh=pst.svd_data.eigthresh)

get_maxsing_from_s(s, eigthresh=1e-05) staticmethod

static method to work out the maxsing for a given singular spectrum

Parameters:

Name	Type	Description	Default
s	`numpy.ndarray`	1-D array of singular values. This array should come from calling either numpy.linalg.svd or from the pyemu.Matrix.s.x attribute	required
eigthresh	`float`	the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.	1e-05

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = pyemu.Matrix.get_maxsing_from_s(jco.s,eigthresh=pst.svd_data.eigthresh)

hadamard_product(other)

Overload of numpy.ndarray.mult(): element-wise multiplication. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to multiply	required

Returns:

Type	Description
	Matrix: the result of multiplication

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are shared, an exception is raised

Example::

cov = pyemu.Cov.from_parameter_data(pst)
cov2 = cov * 10
cov3 = cov * cov2

identity_like(other) classmethod

Get an identity matrix Cov instance like other Cov

Parameters:

Name	Type	Description	Default
other	`Matrix`	other matrix - must be square	required

Returns:

Type	Description
	Cov: new identity matrix Cov with shape of other

Note

the returned identity cov matrix is treated as non-diagonal

indices(names, axis=None)

get the row and col indices of names. If axis is None, two ndarrays are returned, corresponding the indices of names for each axis

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

Note

thin wrapper around Matrix.find_rowcol_indices static method

mult_isaligned(other)

check if matrices are aligned for dot product multiplication

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

pseudo_inv(maxsing=None, eigthresh=1e-05)

The pseudo inverse of self. Formed using truncated singular value decomposition and Matrix.pseudo_inv_components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required

Returns:

Type	Description
	Matrix: the truncated-SVD pseudo inverse of Matrix (V_1 * s_1^-1 * U^T)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco_psinv = jco.pseudo_inv(pst.svd_data.maxsing,pst.svd_data.eigthresh)
jco_psinv.to_binary("pseudo_inv.jcb")

pseudo_inv_components(maxsing=None, eigthresh=1e-05, truncate=True)

Get the (optionally) truncated SVD components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required
truncate	`bool`	flag to truncate components. If False, U, s, and V will be zeroed out at locations greater than maxsing instead of truncated. Default is True	True

Returns:

Type	Description
	tuple containing
	Matrix: (optionally truncated) left singular vectors
	Matrix: (optionally truncated) singular value matrix
	Matrix: (optionally truncated) right singular vectors

Example::

mat = pyemu.Matrix.from_binary("my.jco")
u1,s1,v1 = mat.pseudo_inv_components(maxsing=10)
resolution_matrix = v1 * v1.T
resolution_matrix.to_ascii("resol.mat")

read_ascii(filename) staticmethod

read a PEST-compatible ASCII matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	file to read from	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: numeric values
	['str']: list of row names
	[str]: list of column names
	bool: diagonal flag

read_binary(filename, forgive=False) staticmethod

static method to read PEST-format binary files

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

read_binary_header(filename) staticmethod

read the first elements of a PEST(++)-style binary file to get format and dimensioning information.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename of the binary file	required

Returns:

Type	Description
	tuple containing
	int: the itemp1 value
	int: the itemp2 value
	int: the icount value

read_dense(filename, forgive=False, close=True, only_rows=None) staticmethod

read a dense-format binary file.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename or the open filehandle	required
forgive	`bool`	flag to forgive incomplete records. If True and an incomplete record is encountered, only the previously read records are returned. If False, an exception is raised for an incomplete record	False
close	`bool`	flag to close the filehandle. Default is True	True
only_rows	`iterable`	rows to read. If None, all rows are read	None

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

replace(other)

replace elements in the covariance matrix with elements from other. if other is not diagonal, then this Cov becomes non diagonal

Parameters:

Name	Type	Description	Default
`Cov`		the Cov to replace elements in this Cov with	required

Note

operates in place. Other must have the same row-col names as self

reset_x(x, copy=True)

reset self.__x private attribute

Parameters:

Name	Type	Description	Default
x	`numpy.ndarray`	the new numeric data	required
copy	`bool`	flag to make a copy of 'x'. Default is True	True

Returns:

Type	Description
	None

Note

operates in place

to_2d()

get a 2D Matrix representation of Matrix. If not Matrix.isdiagonal, simply return a copy of Matrix, otherwise, constructs and returns a new Matrix instance that is stored as diagonal

Returns:

Type	Description
	Martrix: non-diagonal form of Matrix

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
cov2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,cov2d.shape,cov2d.x.shape)

to_ascii(filename, icode=2)

write a PEST-compatible ASCII Matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	filename to write to	required
icode	`int`	PEST-style info code for matrix style. Default is 2.	2

Note: if icode == -1, a 1-d vector is written that represents a diagonal matrix. An exception is raised if icode == -1 and isdiagonal is False

to_binary(filename, droptol=None, chunk=None)

write a PEST-compatible binary file. The format is the same as the format used to storage a PEST Jacobian matrix

Parameters:

Name	Type	Description	Default
filename	`str`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

to_coo(filename, droptol=None, chunk=None)

write an extended PEST-format binary file. The data format is [int,int,float] for i,j,value. It is autodetected during the read with Matrix.from_binary().

Parameters:

Name	Type	Description	Default
filename	`str` or `Path`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

Note

This method is needed when the number of dimensions times 2 is larger than the max value for a 32-bit integer. happens! This method is used by pyemu.Ensemble.to_binary()

to_dataframe()

return a pandas.DataFrame representation of Matrix

Returns:

Type	Description
	pandas.DataFrame: a dataframe derived from Matrix

Note

if self.isdiagonal is True, the full matrix is used to fill the dataframe - lots of zeros.

to_dense(filename, close=True)

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str`	the filename to save to	required
close	`bool`	flag to close the filehandle after saving	True

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

Note

calls Matrix.write_dense()

to_pearson()

Convert Cov instance to Pearson correlation coefficient matrix

Returns:

Type	Description
	Matrix: A Matrix of correlation coefs. Return type is Matrix
	on purpose so that it is clear the returned instance is not a Cov

Example::

# plot the posterior parameter correlation matrix
import matplotlib.pyplot as plt
cov = pyemu.Cov.from_ascii("pest.post.cov")
cc = cov.to_pearson()
cc.x[cc.x==1.0] = np.nan
plt.imshow(cc)

to_uncfile(unc_file, covmat_file='cov.mat', var_mult=1.0, include_path=False)

write a PEST-compatible uncertainty file

Parameters:

Name	Type	Description	Default
unc_file	`str`	filename of the uncertainty file	required
covmat_file	`str`	covariance matrix filename. Default is "Cov.mat". If None, and Cov.isdiaonal, then a standard deviation form of the uncertainty file is written. Exception raised if covmat_file is None and not Cov.isdiagonal	'cov.mat'
var_mult	`float`	variance multiplier for the covmat_file entry	1.0
include_path	`bool`	flag to include the path of unc_file in the name of covmat_file. Default is False - not sure why you would ever make this True...	False

Example::

cov = pyemu.Cov.from_parameter_data(pst)
cov.to_uncfile("my.unc")

write_dense(filename, row_names, col_names, data, close=False) staticmethod

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str` or `file`	the file to write to	required
row_names	[`str`]	row names of the matrix	required
col_names	[`str`]	col names of the matrix	required
data	`np.ndarray`	matrix elements	required
close	`bool`	flag to close the file after writing	False

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

Jco

Bases: Matrix

a thin wrapper class to get more intuitive attribute names. Functions exactly like Matrix

T property

wrapper function for Matrix.transpose() method

Returns:

Type	Description
	Matrix: transpose of Matrix

Note

returns a copy of self

A syntatic-sugar overload of Matrix.transpose()

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

as_2d property

get a 2D numeric representation of Matrix.x. If not isdiagonal, simply return reference to Matrix.x, otherwise, constructs and returns a 2D, diagonal ndarray

Returns:

Type	Description
	numpy.ndarray : numpy.ndarray

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
x2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,x2d.shape)

full_s property

Get the full singular value matrix

Returns:

Type	Description
	Matrix: full singular value matrix. Shape is (max(Matrix.shape),max(Matrix.shape))
	with zeros along the diagonal from min(Matrix.shape) to max(Matrix.shape)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.full_s.to_ascii("full_sing_val_matrix.mat")

inv property

inversion operation of Matrix

Returns:

Type	Description
	Matrix: inverse of Matrix

Note

uses numpy.linalg.inv for the inversion

Example::

mat = pyemu.Matrix.from_binary("my.jco")
mat_inv = mat.inv
mat_inv.to_binary("my_inv.jco")

ncol property

length of second dimension

Returns:

Type	Description
	int: number of columns

newx property

return a copy of Matrix.x attribute

Returns:

Type	Description
	numpy.ndarray: a copy Matrix.x

nobs property

number of observations in the Jco

Returns:

Type	Description
	int: number of observations (rows)

npar property

number of parameters in the Jco

Returns:

Type	Description
	int: number of parameters (columns)

nrow property

length of first dimension

Returns:

Type	Description
	int: number of rows

obs_names property

thin wrapper around Matrix.row_names

Returns:

Type	Description
	['str']: a list of observation names

par_names property

thin wrapper around Matrix.col_names

Returns:

Type	Description
	[str]: a list of parameter names

s property

the singular value (diagonal) Matrix

Returns:

Type	Description
	Matrix: singular value matrix. shape is (min(Matrix.shape),min(Matrix.shape))

Example::

# plot the singular spectrum of the jacobian
import matplotlib.pyplot as plt
jco = pyemu.Jco.from_binary("pest.jcb")
plt.plot(jco.s.x)
plt.show()

shape property

get the implied, 2D shape of Matrix

Returns:

Type	Description
	int: length of 2 tuple

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
shape = jco.shape
nrow,ncol = shape #unpack to ints

sqrt property

element-wise square root operation

Returns:

Type	Description
	Matrix: element-wise square root of Matrix

Note

uses numpy.sqrt

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov.sqrt
sqcov.to_uncfile("sqrt_cov.unc")

transpose property

transpose operation of self

Returns:

Type	Description
	Matrix: transpose of Matrix

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

u property

the left singular vector Matrix

Returns:

Type	Description
	Matrix: left singular vectors. Shape is (Matrix.shape[0], Matrix.shape[0])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.u.to_binary("u.jcb")

v property

the right singular vector Matrix

Returns:

Type	Description
	Matrix: right singular vectors. Shape is (Matrix.shape[1], Matrix.shape[1])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.v.to_binary("v.jcb")

x property

return a reference to Matrix.x

Returns:

Type	Description
	numpy.ndarray: reference to Matrix.x

zero2d property

get an 2D instance of self with all zeros

Returns:

Type	Description
	Matrix: Matrix of zeros

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
zero_jco = jco.zero2d

__add__(other)

Overload of numpy.ndarray.add() - elementwise addition. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to add	required

Returns:

Type	Description
	Matrix: the result of addition

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised. The addition of a scalar does not expand non-zero elements.

Example::

m1 = pyemu.Cov.from_parameter_data(pst)
m1_plus_on1 = m1 + 1.0 #add 1.0 to all non-zero elements
m2 = m1.copy()
m2_plus_m1 = m1 + m2

__getitem__(item)

a very crude overload of object.getitem().

Parameters:

Name	Type	Description	Default
item	`object`	something that can be used as an index	required

Returns:

Type	Description
	Matrix: an object that is a sub-matrix of Matrix

__init(**kwargs)

Jco constructor takes the same arguments as Matrix.

Parameters:

Name	Type	Description	Default
**kwargs	`dict`	constructor arguments for Matrix	{}

Example:

jco = pyemu.Jco.from_binary("my.jco")

__mul__(other)

Dot product multiplication overload. Tries to speedup by checking for scalars or diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
cov = pyemu.Cov.from_parmaeter_data(pst)
# get the forecast prior covariance matrix
forecast_cov = jco.get(pst.forecast_names).T * cov * jco.get(pst.forecast_names)

__pow__(power)

overload of numpy.ndarray.pow() operator

Parameters:

Name	Type	Description	Default
power	`float`	interpreted as follows: -1 = inverse of self, -0.5 = sqrt of inverse of self, 0.5 = sqrt of self. All other positive ints = elementwise self raised to power	required

Returns:

Type	Description
	Matrix: a new Matrix object raised to the power power

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov**2

__rmul__(other)

Reverse order Dot product multiplication overload.

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

# multiply by a scalar
jco = pyemu.Jco.from_binary("pest.jcb")
jco_times_10 = 10 * jco

__set_svd()

private method to set SVD components.

Note: this should not be called directly

__str__()

overload of object.str()

Returns:

Type	Description
	str: string representation

__sub__(other)

numpy.ndarray.sub() overload. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to subtract	required

Returns:

Type	Description
	Matrix: the result of subtraction

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised

Example::

jco1 = pyemu.Jco.from_binary("pest.1.jcb")
jco2 = pyemu.Jco.from_binary("pest.2.jcb")
diff = jco1 - jco2

align(names, axis=None)

reorder Matrix by names in place. If axis is None, reorder both indices

Parameters:

Name	Type	Description	Default
names	[str]	names in Matrix.row_names and or Matrix.col_names	required
axis	`int`	the axis to reorder. if None, reorder both axes	None

Note

Works in place. Is called programmatically during linear algebra operations

Example::

# load a jco that has more obs (rows) than non-zero weighted obs
# in the control file
jco = pyemu.Jco.from_binary("pest.jcb")
# get an obs noise cov matrix
obscov = pyemu.Cov.from_observation_data(pst)
jco.align(obscov.row_names,axis=0)

copy()

get a copy of Matrix

Returns:

Type	Description
	Matrix: copy of this Matrix

df()

wrapper of Matrix.to_dataframe()

drop(names, axis)

drop elements from Matrix in place

Parameters:

Name	Type	Description	Default
names	[str]	list of names to drop	required
axis	`int`	the axis to drop from. must be in [0,1]	required

element_isaligned(other)

check if matrices are aligned for element-wise operations

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

extend(other)

extend Matrix with the elements of other, returning a new matrix.

Args: other (Matrix): the Matrix to extend self by

Returns:

Type	Description
	Matrix: new, extended Matrix

Note

No row or column names can be shared between self and other

Example::

jco1 = pyemu.Jco.from_binary("pest_history.jco")
jco2 = pyemu.Jco.from_binary("pest_forecast.jco")

jco_ext = jco1.extend(jco2)

extract(row_names=None, col_names=None)

wrapper method that Matrix.gets() then Matrix.drops() elements. one of row_names or col_names must be not None.

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names to extract. If None, all row_names are retained.	None
col_names	[str]	col_names to extract. If None, all col_names are retained.	None

Returns:

Type	Description
	Matrix: the extract sub-matrix defined by row_names and/or col_names

Example::

cov = pyemu.Cov.from_parameter_data(pst)
hk_cov = cov.extract(row_names=["hk1","hk2","hk3"])

find_rowcol_indices(names, row_names, col_names, axis=None) staticmethod

fast(er) look of row and column names indices

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

from_ascii(filename) classmethod

load a PEST-compatible ASCII matrix/vector file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	name of the file to read	required

Returns:

Type	Description
	Matrix: Matrix loaded from ASCII file

Example::

mat = pyemu.Matrix.from_ascii("my.mat")
cov = pyemi.Cov.from_ascii("my.cov")

from_binary(filename, forgive=False) classmethod

class method load from PEST-compatible binary file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	Matrix: Matrix loaded from binary file

Example::

mat = pyemu.Matrix.from_binary("my.jco")
cov = pyemu.Cov.from_binary("large_cov.jcb")

from_dataframe(df) classmethod

class method to create a new Matrix instance from a pandas.DataFrame

Parameters:

Name	Type	Description	Default
df	`pandas.DataFrame`	dataframe	required

Returns:

Type	Description
	Matrix: Matrix instance derived from df.

Example::

df = pd.read_csv("my.csv")
mat = pyemu.Matrix.from_dataframe(df)

from_fortranfile(filename) staticmethod

a binary load method to accommodate one of the many bizarre fortran binary writing formats

Parameters:

Name	Type	Description	Default
filename	`str`	name of the binary matrix file	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

from_names(row_names, col_names, isdiagonal=False, autoalign=True, random=False) classmethod

class method to create a new Matrix instance from row names and column names, filled with trash

Parameters:

Name	Type	Description	Default
row_names	[str]	row names for the new Matrix	required
col_names	[str]	col_names for the new matrix	required
isdiagonal	`bool`	flag for diagonal matrix. Default is False	False
autoalign	`bool`	flag for autoaligning new matrix during linear algebra calcs. Default is True	True
random	`bool`	flag for contents of the trash matrix. If True, fill with random numbers, if False, fill with zeros Default is False	False

Returns:

Type	Description
	Matrix: the new Matrix instance

Example::

row_names = ["row_1","row_2"]
col_names = ["col_1,"col_2"]
m = pyemu.Matrix.from_names(row_names,col_names)

from_pst(pst, random=False) classmethod

construct a new empty Jco from a control file optionally filled with trash

Parameters:

Name	Type	Description	Default
pst	`pyemu.Pst`	a pest control file instance. If type is 'str', pst is loaded from filename	required
random	`bool`	flag for contents of the trash matrix. If True, fill with random numbers, if False, fill with zeros Default is False	False

Returns:

Type	Description
	Jco: the new Jco instance

get(row_names=None, col_names=None, drop=False)

get a new Matrix instance ordered on row_names or col_names

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names for new Matrix. If None, all row_names are used.	None
col_names	[str]	col_names for new Matrix. If None, all col_names are used.	None
drop	`bool`	flag to remove row_names and/or col_names from this Matrix	False

Returns:

Type	Description
	Matrix: a new Matrix

Example:: # load a jco that has more obs (rows) than non-zero weighted obs # in the control file jco = pyemu.Jco.from_binary("pest.jcb") # get an obs noise cov matrix obscov = pyemu.Cov.from_observation_data(pst) nnz_jco = jco.get(row_names = obscov.row_names)

get_dense_binary_info(filename) staticmethod

read the header and row and offsets for a dense binary file.

Parameters

filename (`str`): dense binary filename

Returns:

Type	Description
	tuple containing
	['str']: list of row names
	['int']: list of row offsets
	[str]: list of col names
	bool: flag indicating successful reading of all records found

get_diagonal_vector(col_name='diag')

Get a new Matrix instance that is the diagonal of self. The shape of the new matrix is (self.shape[0],1). Self must be square

Parameters:

Name	Type	Description	Default
col_name	`str`	the name of the single column in the new Matrix	'diag'

Returns:

Type	Description
	Matrix: vector-shaped Matrix instance of the diagonal of this Matrix

Example::

cov = pyemu.Cov.from_unc_file("my.unc")
cov_diag = cov.get_diagonal_vector()
print(cov_diag.col_names)

get_maxsing(eigthresh=1e-05)

Get the number of singular components with a singular value ratio greater than or equal to eigthresh

Args: eigthresh (float): the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Note

this method calls the static method Matrix.get_maxsing_from_s() with Matrix.s.x

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = jco.get_maxsing(eigthresh=pst.svd_data.eigthresh)

get_maxsing_from_s(s, eigthresh=1e-05) staticmethod

static method to work out the maxsing for a given singular spectrum

Parameters:

Name	Type	Description	Default
s	`numpy.ndarray`	1-D array of singular values. This array should come from calling either numpy.linalg.svd or from the pyemu.Matrix.s.x attribute	required
eigthresh	`float`	the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.	1e-05

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = pyemu.Matrix.get_maxsing_from_s(jco.s,eigthresh=pst.svd_data.eigthresh)

hadamard_product(other)

Overload of numpy.ndarray.mult(): element-wise multiplication. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to multiply	required

Returns:

Type	Description
	Matrix: the result of multiplication

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are shared, an exception is raised

Example::

cov = pyemu.Cov.from_parameter_data(pst)
cov2 = cov * 10
cov3 = cov * cov2

indices(names, axis=None)

get the row and col indices of names. If axis is None, two ndarrays are returned, corresponding the indices of names for each axis

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

Note

thin wrapper around Matrix.find_rowcol_indices static method

mult_isaligned(other)

check if matrices are aligned for dot product multiplication

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

pseudo_inv(maxsing=None, eigthresh=1e-05)

The pseudo inverse of self. Formed using truncated singular value decomposition and Matrix.pseudo_inv_components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required

Returns:

Type	Description
	Matrix: the truncated-SVD pseudo inverse of Matrix (V_1 * s_1^-1 * U^T)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco_psinv = jco.pseudo_inv(pst.svd_data.maxsing,pst.svd_data.eigthresh)
jco_psinv.to_binary("pseudo_inv.jcb")

pseudo_inv_components(maxsing=None, eigthresh=1e-05, truncate=True)

Get the (optionally) truncated SVD components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required
truncate	`bool`	flag to truncate components. If False, U, s, and V will be zeroed out at locations greater than maxsing instead of truncated. Default is True	True

Returns:

Type	Description
	tuple containing
	Matrix: (optionally truncated) left singular vectors
	Matrix: (optionally truncated) singular value matrix
	Matrix: (optionally truncated) right singular vectors

Example::

mat = pyemu.Matrix.from_binary("my.jco")
u1,s1,v1 = mat.pseudo_inv_components(maxsing=10)
resolution_matrix = v1 * v1.T
resolution_matrix.to_ascii("resol.mat")

read_ascii(filename) staticmethod

read a PEST-compatible ASCII matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	file to read from	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: numeric values
	['str']: list of row names
	[str]: list of column names
	bool: diagonal flag

read_binary(filename, forgive=False) staticmethod

static method to read PEST-format binary files

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

read_binary_header(filename) staticmethod

read the first elements of a PEST(++)-style binary file to get format and dimensioning information.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename of the binary file	required

Returns:

Type	Description
	tuple containing
	int: the itemp1 value
	int: the itemp2 value
	int: the icount value

read_dense(filename, forgive=False, close=True, only_rows=None) staticmethod

read a dense-format binary file.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename or the open filehandle	required
forgive	`bool`	flag to forgive incomplete records. If True and an incomplete record is encountered, only the previously read records are returned. If False, an exception is raised for an incomplete record	False
close	`bool`	flag to close the filehandle. Default is True	True
only_rows	`iterable`	rows to read. If None, all rows are read	None

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

reset_x(x, copy=True)

reset self.__x private attribute

Parameters:

Name	Type	Description	Default
x	`numpy.ndarray`	the new numeric data	required
copy	`bool`	flag to make a copy of 'x'. Default is True	True

Returns:

Type	Description
	None

Note

operates in place

to_2d()

get a 2D Matrix representation of Matrix. If not Matrix.isdiagonal, simply return a copy of Matrix, otherwise, constructs and returns a new Matrix instance that is stored as diagonal

Returns:

Type	Description
	Martrix: non-diagonal form of Matrix

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
cov2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,cov2d.shape,cov2d.x.shape)

to_ascii(filename, icode=2)

write a PEST-compatible ASCII Matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	filename to write to	required
icode	`int`	PEST-style info code for matrix style. Default is 2.	2

Note: if icode == -1, a 1-d vector is written that represents a diagonal matrix. An exception is raised if icode == -1 and isdiagonal is False

to_binary(filename, droptol=None, chunk=None)

write a PEST-compatible binary file. The format is the same as the format used to storage a PEST Jacobian matrix

Parameters:

Name	Type	Description	Default
filename	`str`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

to_coo(filename, droptol=None, chunk=None)

write an extended PEST-format binary file. The data format is [int,int,float] for i,j,value. It is autodetected during the read with Matrix.from_binary().

Parameters:

Name	Type	Description	Default
filename	`str` or `Path`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

Note

This method is needed when the number of dimensions times 2 is larger than the max value for a 32-bit integer. happens! This method is used by pyemu.Ensemble.to_binary()

to_dataframe()

return a pandas.DataFrame representation of Matrix

Returns:

Type	Description
	pandas.DataFrame: a dataframe derived from Matrix

Note

if self.isdiagonal is True, the full matrix is used to fill the dataframe - lots of zeros.

to_dense(filename, close=True)

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str`	the filename to save to	required
close	`bool`	flag to close the filehandle after saving	True

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

Note

calls Matrix.write_dense()

write_dense(filename, row_names, col_names, data, close=False) staticmethod

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str` or `file`	the file to write to	required
row_names	[`str`]	row names of the matrix	required
col_names	[`str`]	col names of the matrix	required
data	`np.ndarray`	matrix elements	required
close	`bool`	flag to close the file after writing	False

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

Matrix

Bases: object

Easy linear algebra in the PEST(++) realm

Parameters:

Name	Type	Description	Default
x	`numpy.ndarray`	numeric values	None
row_names	[`str`]	list of row names	[]
col_names	[str]	list of column names	[]
isdigonal	`bool`	flag if the Matrix is diagonal	required
autoalign	`bool`	flag to control the autoalignment of Matrix during linear algebra operations	True

Example::

data = pyemu.en.rng.random((10,10))
row_names = ["row_{0}".format(i) for i in range(10)]
col_names = ["col_{0}".format(j) for j in range(10)]
mat = pyemu.Matrix(x=data,row_names=row_names,col_names=col_names)
mat.to_binary("mat.jco")

# load an existing jacobian matrix
jco = pyemu.Jco.from_binary("pest.jco")
# form an observation noise covariance matrix from weights
obscov = pyemu.Cov.from_observation_data(pst)
# form the normal matrix, aligning rows and cols on-the-fly
xtqx = jco * obscov.inv * jco.T

Note

this class makes heavy use of property decorators to encapsulate private attributes

T property

wrapper function for Matrix.transpose() method

Returns:

Type	Description
	Matrix: transpose of Matrix

Note

returns a copy of self

A syntatic-sugar overload of Matrix.transpose()

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

as_2d property

get a 2D numeric representation of Matrix.x. If not isdiagonal, simply return reference to Matrix.x, otherwise, constructs and returns a 2D, diagonal ndarray

Returns:

Type	Description
	numpy.ndarray : numpy.ndarray

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
x2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,x2d.shape)

full_s property

Get the full singular value matrix

Returns:

Type	Description
	Matrix: full singular value matrix. Shape is (max(Matrix.shape),max(Matrix.shape))
	with zeros along the diagonal from min(Matrix.shape) to max(Matrix.shape)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.full_s.to_ascii("full_sing_val_matrix.mat")

inv property

inversion operation of Matrix

Returns:

Type	Description
	Matrix: inverse of Matrix

Note

uses numpy.linalg.inv for the inversion

Example::

mat = pyemu.Matrix.from_binary("my.jco")
mat_inv = mat.inv
mat_inv.to_binary("my_inv.jco")

ncol property

length of second dimension

Returns:

Type	Description
	int: number of columns

newx property

return a copy of Matrix.x attribute

Returns:

Type	Description
	numpy.ndarray: a copy Matrix.x

nrow property

length of first dimension

Returns:

Type	Description
	int: number of rows

s property

the singular value (diagonal) Matrix

Returns:

Type	Description
	Matrix: singular value matrix. shape is (min(Matrix.shape),min(Matrix.shape))

Example::

# plot the singular spectrum of the jacobian
import matplotlib.pyplot as plt
jco = pyemu.Jco.from_binary("pest.jcb")
plt.plot(jco.s.x)
plt.show()

shape property

get the implied, 2D shape of Matrix

Returns:

Type	Description
	int: length of 2 tuple

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
shape = jco.shape
nrow,ncol = shape #unpack to ints

sqrt property

element-wise square root operation

Returns:

Type	Description
	Matrix: element-wise square root of Matrix

Note

uses numpy.sqrt

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov.sqrt
sqcov.to_uncfile("sqrt_cov.unc")

transpose property

transpose operation of self

Returns:

Type	Description
	Matrix: transpose of Matrix

Example::

jcb = pyemu.Jco.from_binary("pest.jcb")
jcbt = jcb.T

u property

the left singular vector Matrix

Returns:

Type	Description
	Matrix: left singular vectors. Shape is (Matrix.shape[0], Matrix.shape[0])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.u.to_binary("u.jcb")

v property

the right singular vector Matrix

Returns:

Type	Description
	Matrix: right singular vectors. Shape is (Matrix.shape[1], Matrix.shape[1])

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco.v.to_binary("v.jcb")

x property

return a reference to Matrix.x

Returns:

Type	Description
	numpy.ndarray: reference to Matrix.x

zero2d property

get an 2D instance of self with all zeros

Returns:

Type	Description
	Matrix: Matrix of zeros

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
zero_jco = jco.zero2d

__add__(other)

Overload of numpy.ndarray.add() - elementwise addition. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to add	required

Returns:

Type	Description
	Matrix: the result of addition

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised. The addition of a scalar does not expand non-zero elements.

Example::

m1 = pyemu.Cov.from_parameter_data(pst)
m1_plus_on1 = m1 + 1.0 #add 1.0 to all non-zero elements
m2 = m1.copy()
m2_plus_m1 = m1 + m2

__getitem__(item)

a very crude overload of object.getitem().

Parameters:

Name	Type	Description	Default
item	`object`	something that can be used as an index	required

Returns:

Type	Description
	Matrix: an object that is a sub-matrix of Matrix

__mul__(other)

Dot product multiplication overload. Tries to speedup by checking for scalars or diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
cov = pyemu.Cov.from_parmaeter_data(pst)
# get the forecast prior covariance matrix
forecast_cov = jco.get(pst.forecast_names).T * cov * jco.get(pst.forecast_names)

__pow__(power)

overload of numpy.ndarray.pow() operator

Parameters:

Name	Type	Description	Default
power	`float`	interpreted as follows: -1 = inverse of self, -0.5 = sqrt of inverse of self, 0.5 = sqrt of self. All other positive ints = elementwise self raised to power	required

Returns:

Type	Description
	Matrix: a new Matrix object raised to the power power

Example::

cov = pyemu.Cov.from_uncfile("my.unc")
sqcov = cov**2

__rmul__(other)

Reverse order Dot product multiplication overload.

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to dot product	required

Returns:

Type	Description
	Matrix: the result of dot product

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are found, an exception is raised

Example::

# multiply by a scalar
jco = pyemu.Jco.from_binary("pest.jcb")
jco_times_10 = 10 * jco

__set_svd()

private method to set SVD components.

Note: this should not be called directly

__str__()

overload of object.str()

Returns:

Type	Description
	str: string representation

__sub__(other)

numpy.ndarray.sub() overload. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to subtract	required

Returns:

Type	Description
	Matrix: the result of subtraction

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If no names are shared, an exception is raised

Example::

jco1 = pyemu.Jco.from_binary("pest.1.jcb")
jco2 = pyemu.Jco.from_binary("pest.2.jcb")
diff = jco1 - jco2

align(names, axis=None)

reorder Matrix by names in place. If axis is None, reorder both indices

Parameters:

Name	Type	Description	Default
names	[str]	names in Matrix.row_names and or Matrix.col_names	required
axis	`int`	the axis to reorder. if None, reorder both axes	None

Note

Works in place. Is called programmatically during linear algebra operations

Example::

# load a jco that has more obs (rows) than non-zero weighted obs
# in the control file
jco = pyemu.Jco.from_binary("pest.jcb")
# get an obs noise cov matrix
obscov = pyemu.Cov.from_observation_data(pst)
jco.align(obscov.row_names,axis=0)

copy()

get a copy of Matrix

Returns:

Type	Description
	Matrix: copy of this Matrix

df()

wrapper of Matrix.to_dataframe()

drop(names, axis)

drop elements from Matrix in place

Parameters:

Name	Type	Description	Default
names	[str]	list of names to drop	required
axis	`int`	the axis to drop from. must be in [0,1]	required

element_isaligned(other)

check if matrices are aligned for element-wise operations

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

extend(other)

extend Matrix with the elements of other, returning a new matrix.

Args: other (Matrix): the Matrix to extend self by

Returns:

Type	Description
	Matrix: new, extended Matrix

Note

No row or column names can be shared between self and other

Example::

jco1 = pyemu.Jco.from_binary("pest_history.jco")
jco2 = pyemu.Jco.from_binary("pest_forecast.jco")

jco_ext = jco1.extend(jco2)

extract(row_names=None, col_names=None)

wrapper method that Matrix.gets() then Matrix.drops() elements. one of row_names or col_names must be not None.

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names to extract. If None, all row_names are retained.	None
col_names	[str]	col_names to extract. If None, all col_names are retained.	None

Returns:

Type	Description
	Matrix: the extract sub-matrix defined by row_names and/or col_names

Example::

cov = pyemu.Cov.from_parameter_data(pst)
hk_cov = cov.extract(row_names=["hk1","hk2","hk3"])

find_rowcol_indices(names, row_names, col_names, axis=None) staticmethod

fast(er) look of row and column names indices

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

from_ascii(filename) classmethod

load a PEST-compatible ASCII matrix/vector file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	name of the file to read	required

Returns:

Type	Description
	Matrix: Matrix loaded from ASCII file

Example::

mat = pyemu.Matrix.from_ascii("my.mat")
cov = pyemi.Cov.from_ascii("my.cov")

from_binary(filename, forgive=False) classmethod

class method load from PEST-compatible binary file into a Matrix instance

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	Matrix: Matrix loaded from binary file

Example::

mat = pyemu.Matrix.from_binary("my.jco")
cov = pyemu.Cov.from_binary("large_cov.jcb")

from_dataframe(df) classmethod

class method to create a new Matrix instance from a pandas.DataFrame

Parameters:

Name	Type	Description	Default
df	`pandas.DataFrame`	dataframe	required

Returns:

Type	Description
	Matrix: Matrix instance derived from df.

Example::

df = pd.read_csv("my.csv")
mat = pyemu.Matrix.from_dataframe(df)

from_fortranfile(filename) staticmethod

a binary load method to accommodate one of the many bizarre fortran binary writing formats

Parameters:

Name	Type	Description	Default
filename	`str`	name of the binary matrix file	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

from_names(row_names, col_names, isdiagonal=False, autoalign=True, random=False) classmethod

class method to create a new Matrix instance from row names and column names, filled with trash

Parameters:

Name	Type	Description	Default
row_names	[str]	row names for the new Matrix	required
col_names	[str]	col_names for the new matrix	required
isdiagonal	`bool`	flag for diagonal matrix. Default is False	False
autoalign	`bool`	flag for autoaligning new matrix during linear algebra calcs. Default is True	True
random	`bool`	flag for contents of the trash matrix. If True, fill with random numbers, if False, fill with zeros Default is False	False

Returns:

Type	Description
	Matrix: the new Matrix instance

Example::

row_names = ["row_1","row_2"]
col_names = ["col_1,"col_2"]
m = pyemu.Matrix.from_names(row_names,col_names)

get(row_names=None, col_names=None, drop=False)

get a new Matrix instance ordered on row_names or col_names

Parameters:

Name	Type	Description	Default
row_names	[str]	row_names for new Matrix. If None, all row_names are used.	None
col_names	[str]	col_names for new Matrix. If None, all col_names are used.	None
drop	`bool`	flag to remove row_names and/or col_names from this Matrix	False

Returns:

Type	Description
	Matrix: a new Matrix

Example:: # load a jco that has more obs (rows) than non-zero weighted obs # in the control file jco = pyemu.Jco.from_binary("pest.jcb") # get an obs noise cov matrix obscov = pyemu.Cov.from_observation_data(pst) nnz_jco = jco.get(row_names = obscov.row_names)

get_dense_binary_info(filename) staticmethod

read the header and row and offsets for a dense binary file.

Parameters

filename (`str`): dense binary filename

Returns:

Type	Description
	tuple containing
	['str']: list of row names
	['int']: list of row offsets
	[str]: list of col names
	bool: flag indicating successful reading of all records found

get_diagonal_vector(col_name='diag')

Get a new Matrix instance that is the diagonal of self. The shape of the new matrix is (self.shape[0],1). Self must be square

Parameters:

Name	Type	Description	Default
col_name	`str`	the name of the single column in the new Matrix	'diag'

Returns:

Type	Description
	Matrix: vector-shaped Matrix instance of the diagonal of this Matrix

Example::

cov = pyemu.Cov.from_unc_file("my.unc")
cov_diag = cov.get_diagonal_vector()
print(cov_diag.col_names)

get_maxsing(eigthresh=1e-05)

Get the number of singular components with a singular value ratio greater than or equal to eigthresh

Args: eigthresh (float): the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Note

this method calls the static method Matrix.get_maxsing_from_s() with Matrix.s.x

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = jco.get_maxsing(eigthresh=pst.svd_data.eigthresh)

get_maxsing_from_s(s, eigthresh=1e-05) staticmethod

static method to work out the maxsing for a given singular spectrum

Parameters:

Name	Type	Description	Default
s	`numpy.ndarray`	1-D array of singular values. This array should come from calling either numpy.linalg.svd or from the pyemu.Matrix.s.x attribute	required
eigthresh	`float`	the ratio of smallest to largest singular value to retain. Since it is assumed that s is sorted from largest to smallest, once a singular value is reached that yields a ratio with the first (largest) singular value, the index of this singular is returned.	1e-05

Returns:

Type	Description
	int: the index of the singular value who's ratio with the
	first singular value is less than or equal to eigthresh

Example::

jco = pyemu.Jco.from_binary("pest.jco")
max_sing = pyemu.Matrix.get_maxsing_from_s(jco.s,eigthresh=pst.svd_data.eigthresh)

hadamard_product(other)

Overload of numpy.ndarray.mult(): element-wise multiplication. Tries to speedup by checking for scalars of diagonal matrices on either side of operator

Parameters:

Name	Type	Description	Default
other		(int,float,numpy.ndarray,Matrix): the thing to multiply	required

Returns:

Type	Description
	Matrix: the result of multiplication

Note

if Matrix and other (if applicable) have autoalign set to True, both Matrix and other are aligned based on row and column names. If names are not common between the two, this may result in a smaller returned Matrix. If not common elements are shared, an exception is raised

Example::

cov = pyemu.Cov.from_parameter_data(pst)
cov2 = cov * 10
cov3 = cov * cov2

indices(names, axis=None)

get the row and col indices of names. If axis is None, two ndarrays are returned, corresponding the indices of names for each axis

Parameters:

Name	Type	Description	Default
names	[`str`]	list of names to look for in row_names and/or col_names names	required
row_names	[`str`]	list of row names	required
col_names	[`str`]	list of column names	required
axis	`int`	axis to search along. If None, search both. Default is None	None

Returns:

Type	Description
	numpy.ndarray: array of (integer) index locations. If axis is
	None, a 2 numpy.ndarrays of both row and column name indices is returned

Note

thin wrapper around Matrix.find_rowcol_indices static method

mult_isaligned(other)

check if matrices are aligned for dot product multiplication

Parameters:

Name	Type	Description	Default
other	`Matrix`	the other matrix to check for alignment with	required

Returns:

Type	Description
	bool: True if aligned, False if not aligned

pseudo_inv(maxsing=None, eigthresh=1e-05)

The pseudo inverse of self. Formed using truncated singular value decomposition and Matrix.pseudo_inv_components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required

Returns:

Type	Description
	Matrix: the truncated-SVD pseudo inverse of Matrix (V_1 * s_1^-1 * U^T)

Example::

jco = pyemu.Jco.from_binary("pest.jcb")
jco_psinv = jco.pseudo_inv(pst.svd_data.maxsing,pst.svd_data.eigthresh)
jco_psinv.to_binary("pseudo_inv.jcb")

pseudo_inv_components(maxsing=None, eigthresh=1e-05, truncate=True)

Get the (optionally) truncated SVD components

Parameters:

Name	Type	Description	Default
maxsing	`int`	the number of singular components to use. If None, maxsing is calculated using Matrix.get_maxsing() and eigthresh	None
`eigthresh`		(float, optional): the ratio of largest to smallest singular components to use for truncation. Ignored if maxsing is not None. Default is 1.0e-5	required
truncate	`bool`	flag to truncate components. If False, U, s, and V will be zeroed out at locations greater than maxsing instead of truncated. Default is True	True

Returns:

Type	Description
	tuple containing
	Matrix: (optionally truncated) left singular vectors
	Matrix: (optionally truncated) singular value matrix
	Matrix: (optionally truncated) right singular vectors

Example::

mat = pyemu.Matrix.from_binary("my.jco")
u1,s1,v1 = mat.pseudo_inv_components(maxsing=10)
resolution_matrix = v1 * v1.T
resolution_matrix.to_ascii("resol.mat")

read_ascii(filename) staticmethod

read a PEST-compatible ASCII matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	file to read from	required

Returns:

Type	Description
	tuple containing
	numpy.ndarray: numeric values
	['str']: list of row names
	[str]: list of column names
	bool: diagonal flag

read_binary(filename, forgive=False) staticmethod

static method to read PEST-format binary files

Parameters:

Name	Type	Description	Default
filename	`str`	filename to read	required
forgive	`bool`	flag to forgive incomplete data records. Only applicable to dense binary format. Default is False	False

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

read_binary_header(filename) staticmethod

read the first elements of a PEST(++)-style binary file to get format and dimensioning information.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename of the binary file	required

Returns:

Type	Description
	tuple containing
	int: the itemp1 value
	int: the itemp2 value
	int: the icount value

read_dense(filename, forgive=False, close=True, only_rows=None) staticmethod

read a dense-format binary file.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename or the open filehandle	required
forgive	`bool`	flag to forgive incomplete records. If True and an incomplete record is encountered, only the previously read records are returned. If False, an exception is raised for an incomplete record	False
close	`bool`	flag to close the filehandle. Default is True	True
only_rows	`iterable`	rows to read. If None, all rows are read	None

Returns:

Type	Description
	tuple containing
	numpy.ndarray: the numeric values in the file
	['str']: list of row names
	[str]: list of col_names

reset_x(x, copy=True)

reset self.__x private attribute

Parameters:

Name	Type	Description	Default
x	`numpy.ndarray`	the new numeric data	required
copy	`bool`	flag to make a copy of 'x'. Default is True	True

Returns:

Type	Description
	None

Note

operates in place

to_2d()

get a 2D Matrix representation of Matrix. If not Matrix.isdiagonal, simply return a copy of Matrix, otherwise, constructs and returns a new Matrix instance that is stored as diagonal

Returns:

Type	Description
	Martrix: non-diagonal form of Matrix

Example::

# A diagonal cov
cov = pyemu.Cov.from_parameter_data
cov2d = cov.as_2d # a numpy 2d array
print(cov.shape,cov.x.shape,cov2d.shape,cov2d.x.shape)

to_ascii(filename, icode=2)

write a PEST-compatible ASCII Matrix/vector file

Parameters:

Name	Type	Description	Default
filename	`str`	filename to write to	required
icode	`int`	PEST-style info code for matrix style. Default is 2.	2

Note: if icode == -1, a 1-d vector is written that represents a diagonal matrix. An exception is raised if icode == -1 and isdiagonal is False

to_binary(filename, droptol=None, chunk=None)

write a PEST-compatible binary file. The format is the same as the format used to storage a PEST Jacobian matrix

Parameters:

Name	Type	Description	Default
filename	`str`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

to_coo(filename, droptol=None, chunk=None)

write an extended PEST-format binary file. The data format is [int,int,float] for i,j,value. It is autodetected during the read with Matrix.from_binary().

Parameters:

Name	Type	Description	Default
filename	`str` or `Path`	filename to save binary file	required
droptol	`float`	absolute value tolerance to make values smaller droptol than zero. Default is None (no dropping)	None
chunk	`int`	number of elements to write in a single pass. Default is None, which writes the entire numeric part of the Matrix at once. This is faster but requires more memory.	None

Note

This method is needed when the number of dimensions times 2 is larger than the max value for a 32-bit integer. happens! This method is used by pyemu.Ensemble.to_binary()

to_dataframe()

return a pandas.DataFrame representation of Matrix

Returns:

Type	Description
	pandas.DataFrame: a dataframe derived from Matrix

Note

if self.isdiagonal is True, the full matrix is used to fill the dataframe - lots of zeros.

to_dense(filename, close=True)

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str`	the filename to save to	required
close	`bool`	flag to close the filehandle after saving	True

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

Note

calls Matrix.write_dense()

write_dense(filename, row_names, col_names, data, close=False) staticmethod

experimental new dense matrix storage format to support faster I/O with ensembles

Parameters:

Name	Type	Description	Default
filename	`str` or `file`	the file to write to	required
row_names	[`str`]	row names of the matrix	required
col_names	[`str`]	col names of the matrix	required
data	`np.ndarray`	matrix elements	required
close	`bool`	flag to close the file after writing	False

Returns:

Name	Type	Description
f	`file`	the file handle. Only returned if close is False

concat(mats)

Concatenate Matrix objects. Tries either axis.

Parameters:

Name	Type	Description	Default
mats	[`Matrix`]	list of Matrix objects	required

Returns:

Type	Description
	pyemu.Matrix: a concatenated Matrix instance

get_common_elements(list1, list2)

find the common elements in two lists. used to support auto align might be faster with sets

Parameters:

Name	Type	Description	Default
list1	[`str`]	a list of strings (could be either row or col names, depending on calling function)	required
list2	[`str`]	a list of strings (could be either row or col names, depending on calling function)	required

Returns:

Type	Description
	[str]: list of common strings shared by list1 and list2

Note

result is not ordered WRT list1 or list2

save_coo(x, row_names, col_names, filename, chunk=None)

write a PEST-compatible binary file. The data format is [int,int,float] for i,j,value. It is autodetected during the read with Matrix.from_binary().

Parameters:

Name	Type	Description	Default
x	`numpy.sparse`	coo sparse matrix	required
row_names	[`str`]	list of row names	required
col_names	['str]	list of col_names	required
filename	`str`	filename	required
droptol	`float`	absolute value tolerance to make values smaller than droptol zero. Default is None (no dropping)	required
chunk	`int`	number of elements to write in a single pass. Default is None	None