ev

`ErrVar`

Bases: LinearAnalysis

FOSM-based error variance analysis

Parameters:

Name	Type	Description	Default
`jco`	`varies`	something that can be cast or loaded into a `pyemu.Jco`. Can be a str for a filename or `pyemu.Matrix`/`pyemu.Jco` object.	required
`pst`	`varies`	something that can be cast into a `pyemu.Pst`. Can be an `str` for a filename or an existing `pyemu.Pst`. If `None`, a pst filename is sought with the same base name as the jco argument (if passed)	required
`parcov`	`varies`	prior parameter covariance matrix. If `str`, a filename is assumed and the prior parameter covariance matrix is loaded from a file using the file extension (".jcb"/".jco" for binary, ".cov"/".mat" for PEST-style ASCII matrix, or ".unc" for uncertainty files). If `None`, the prior parameter covariance matrix is constructed from the parameter bounds in `LinearAnalysis.pst`. Can also be a `pyemu.Cov` instance	required
`obscov`	`varies`	observation noise covariance matrix. If `str`, a filename is assumed and the noise covariance matrix is loaded from a file using the file extension (".jcb"/".jco" for binary, ".cov"/".mat" for PEST-style ASCII matrix, or ".unc" for uncertainty files). If `None`, the noise covariance matrix is constructed from the observation weights in `LinearAnalysis.pst`. Can also be a `pyemu.Cov` instance	required
`forecasts`	`varies`	forecast sensitivity vectors. If `str`, first an observation name is assumed (a row in `LinearAnalysis.jco`). If that is not found, a filename is assumed and predictions are loaded from a file using the file extension. If [`str`], a list of observation names is assumed. Can also be a `pyemu.Matrix` instance, a `numpy.ndarray` or a collection. Note if the PEST++ option "++forecasts()" is set in the pest control file (under the `pyemu.Pst.pestpp_options` dictionary), then there is no need to pass this argument (unless you want to analyze different forecasts) of `pyemu.Matrix` or `numpy.ndarray`.	required
`ref_var`	`float`	reference variance. Default is 1.0	required
`verbose`	`bool`	controls screen output. If `str`, a filename is assumed and and log file is written.	required
`sigma_range`	`float`	defines range of upper bound - lower bound in terms of standard deviation (sigma). For example, if sigma_range = 4, the bounds represent 4 * sigma. Default is 4.0, representing approximately 95% confidence of implied normal distribution. This arg is only used if constructing parcov from parameter bounds.	required
`scale_offset`	`bool`	flag to apply parameter scale and offset to parameter bounds when calculating prior parameter covariance matrix from bounds. This arg is onlyused if constructing parcov from parameter bounds.Default is True.	required
`omitted_parameters`	[`str`]	list of parameters to treat as "omitted". Passing this argument activates 3-term error variance analysis.	required
`omitted_parcov`	`varies`	an argument that can be cast to a parcov for the omitted parameters. If None, omitted_parcov will be formed by extracting a sub-matrix from the `LinearAnalsis.parcov` attribute.	required
`omitted_predictions`	`varies`	an argument that can be cast to a "predictions" (e.g. "forecasts") attribute to form prediction sensitivity vectors with respec to the omitted parameters. If None, these vectors will be extracted from the `pyemu.LinearAnalysis.predictions` attribute	required
`kl`	`bool`	flag to perform Karhunen-Loeve scaling on the jacobian before error variance calculations. If `True`, the `pyemu.ErrVar.jco` and `pyemu.ErrVar.parcov` are altered in place. Default is `False`.	required

Example::

ev = pyemu.ErrVar(jco="my.jco",omitted_parameters=["wel1","wel2"])
df = ev.get_errvar_dataframe()

`adj_par_names` `property`

adjustable parameter names

Returns:

Type	Description
	['str`]: list of adjustable parameter names

Note

if LinearAnalysis.pst is None, returns LinearAnalysis.jco.col_names

`fehalf` `property`

Karhunen-Loeve scaling matrix attribute.

Returns:

Type	Description
	`pyemu.Matrix`: the Karhunen-Loeve scaling matrix based on the prior
	parameter covariance matrix

`forecast_names` `property`

get the forecast (aka prediction) names

Returns:

Type	Description
[`str`]	list of forecast names

`forecasts` `property`

the forecast sentivity vectors attribute

Returns:

Type	Description
	`pyemu.Matrix`: a matrix of forecast (prediction) sensitivity vectors (column wise)

`forecasts_iter` `property`

forecast (e.g. prediction) sensitivity vectors iterator

Returns:

Type	Description
	`iterator`: iterator on forecasts (e.g. predictions) sensitivity vectors (matrix)

Note

This is used for processing huge numbers of predictions

This is a synonym for LinearAnalysis.predictions_iter()

`jco` `property`

the jacobian matrix attribute

Returns:

Type	Description
	`pyemu.Jco`: the jacobian matrix attribute

`mle_covariance` `property`

maximum likelihood parameter covariance matrix.

Returns:

Type	Description
	`pyemu.Matrix`: maximum likelihood parameter covariance matrix

`mle_parameter_estimate` `property`

maximum likelihood parameter estimate.

Returns:

Type	Description
	`pandas.Series`: the maximum likelihood parameter estimates

`nnz_obs_names` `property`

non-zero-weighted observation names

Returns:

Type	Description
	['str`]: list of non-zero-weighted observation names

Note

if LinearAnalysis.pst is None, returns LinearAnalysis.jco.row_names

`obscov` `property`

get the observation noise covariance matrix attribute

Returns:

Type	Description
	`pyemu.Cov`: a reference to the `LinearAnalysis.obscov` attribute

`omitted_jco` `property`

the omitted-parameters jacobian matrix

Returns:

Type	Description
	`pyemu.Jco`: the jacobian matrix instance of non-zero-weighted observations and
	omitted parameters

`omitted_parcov` `property`

the prior omitted-parameter covariance matrix

Returns:

Type	Description
	`pyemu.Cov`: the prior parameter covariance matrix of the
	omitted parameters

`omitted_predictions` `property`

omitted prediction sensitivity vectors

Returns:

Type	Description
	`pyemu.Matrix`: a matrix of prediction sensitivity vectors (column wise) to
	omitted parameters

`parcov` `property`

get the prior parameter covariance matrix attribute

Returns:

Type	Description
	`pyemu.Cov`: a reference to the `LinearAnalysis.parcov` attribute

`predictions` `property`

the prediction (aka forecast) sentivity vectors attribute

Returns:

Type	Description
	`pyemu.Matrix`: a matrix of prediction sensitivity vectors (column wise)

`predictions_iter` `property`

prediction sensitivity vectors iterator

Returns:

Type	Description
	`iterator`: iterator on prediction sensitivity vectors (matrix)

Note

this is used for processing huge numbers of predictions

`prior_forecast` `property`

prior forecast (e.g. prediction) variances

Returns:

Type	Description
	`dict`: a dictionary of forecast name, prior variance pairs

`prior_parameter` `property`

prior parameter covariance matrix

Returns:

Type	Description
	`pyemu.Cov`: prior parameter covariance matrix

`prior_prediction` `property`

prior prediction (e.g. forecast) variances

Returns:

Type	Description
	`dict`: a dictionary of prediction name, prior variance pairs

`pst` `property`

the pst attribute

Returns:

Type	Description
	`pyemu.Pst`: the pst attribute

`qhalf` `property`

square root of the cofactor matrix attribute. Create the attribute if it has not yet been created

Returns:

Type	Description
	`pyemu.Matrix`: square root of the cofactor matrix

`qhalfx` `property`

half normal matrix attribute.

Returns:

Type	Description
	`pyemu.Matrix`: half normal matrix attribute

`xtqx` `property`

normal matrix attribute.

Returns:

Type	Description
	`pyemu.Matrix`: normal matrix attribute

`G(singular_value)`

get the parameter solution Matrix at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc G at	required

Returns:

Type	Description
	`pyemu.Matrix`: parameter solution matrix (V_1 * S_1^(_1) * U_1^T) at `singular_value`

`I_minus_R(singular_value)`

get I - R at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calculate I - R at	required

Returns:

Type	Description
	`pyemu.Matrix`: identity matrix minus resolution matrix at `singular_value`

`R(singular_value)`

get resolution Matrix (V_1 * V_1^T) at a given singular value

Args: singular_value (int): singular value to calculate R at

Returns:

Type	Description
	`pyemu.Matrix`: resolution matrix at `singular_value`

`__fromfile(filename, astype=None)`

a private method to deduce and load a filename into a matrix object. Uses extension: 'jco' or 'jcb': binary, 'mat','vec' or 'cov': ASCII, 'unc': pest uncertainty file.

`__load_jco()`

private method to set the jco attribute from a file or a matrix object

`__load_obscov()`

private method to set the obscov attribute from: a pest control file (observation weights) a pst object a matrix object an uncert file an ascii matrix file

`__load_omitted_jco()`

private: set the omitted jco attribute

`__load_omitted_parcov()`

private: set the omitted_parcov attribute

`__load_omitted_predictions()`

private: set the omitted_predictions attribute

`__load_parcov()`

private method to set the parcov attribute from: a pest control file (parameter bounds) a pst object a matrix object an uncert file an ascii matrix file

`__load_predictions()`

private method set the predictions attribute from

mixed list of row names, matrix files and ndarrays a single row name an ascii file

can be none if only interested in parameters.

`__load_pst()`

private method set the pst attribute

`adjust_obscov_resfile(resfile=None)`

reset the elements of obscov by scaling the implied weights based on the phi components in res_file so that the total phi is equal to the number of non-zero weights.

Parameters:

Name	Type	Description	Default
`resfile`	`str`	residual file to use. If None, residual file with case name is sought. default is None	`None`

Note

calls pyemu.Pst.adjust_weights_resfile()

`apply_karhunen_loeve_scaling()`

apply karhuene-loeve scaling to the jacobian matrix.

Note:

This scaling is not necessary for analyses using Schur's
complement, but can be very important for error variance
analyses.  This operation effectively transfers prior knowledge
specified in the parcov to the jacobian and reset parcov to the
identity matrix.

`clean()`

drop regularization and prior information observation from the jco

`drop_prior_information()`

drop the prior information from the jco and pst attributes

`first_forecast(singular_value)`

get the null space term (first term) contribution to forecast (e.g. prediction) error variance at a given singular value.

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc first term at	required

Note

This method is used to construct the error variance dataframe

Just a wrapper around ErrVar.first_forecast

Returns:

Type	Description
	`dict`: dictionary of ("first",prediction_names),error variance pairs at `singular_value`

`first_parameter(singular_value)`

get the null space term (first term) contribution to parameter error variance at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc first term at	required

Returns:

Type	Description
	`pyemu.Cov`: first term contribution to parameter error variance

`first_prediction(singular_value)`

get the null space term (first term) contribution to prediction error variance at a given singular value.

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc first term at	required

Note

This method is used to construct the error variance dataframe

Returns:

Type	Description
	`dict`: dictionary of ("first",prediction_names),error variance pairs at `singular_value`

`get(par_names=None, obs_names=None, astype=None)`

method to get a new LinearAnalysis class using a subset of parameters and/or observations

Parameters:

Name	Type	Description	Default
`par_names`	[`'str`]	par names for new object	`None`
`obs_names`	[`'str`]	obs names for new object	`None`
`astype`	`pyemu.Schur` or `pyemu.ErrVar`	type to cast the new object. If None, return type is same as self	`None`

Returns:

Type	Description
	`LinearAnalysis`: new instance

`get_cso_dataframe()`

get a dataframe of composite observation sensitivity, as returned by PEST in the seo file.

Returns:

Type	Description
	`pandas.DataFrame`: dataframe of observation names and composite observation
	sensitivity

Note

That this formulation deviates slightly from the PEST documentation in that the values are divided by (npar-1) rather than by (npar).

The equation is cso_j = ((Q^1/2JJ^T*Q^1/2)^1/2)_jj/(NPAR-1)

`get_errvar_dataframe(singular_values=None)`

primary entry point for error variance analysis.

Parameters:

Name	Type	Description	Default
`singular_values`	[`int`]	a list singular values to test. If `None`, defaults to `range(0,min(nnz_obs,nadj_par) + 1)`.	`None`

Returns:

Type	Description
	`pandas.DataFrame`: a multi-indexed pandas dataframe summarizing each of the
	error variance terms for each nominated forecast. Rows are the singular values
	tested, columns are a multi-index of forecast name and error variance term number
	(e.g. 1,2 or (optionally) 3).

Example::

ev = pyemu.ErrVar(jco="my.jco",omitted_parameters=["wel1","wel2"])
df = ev.get_errvar_dataframe()

`get_identifiability_dataframe(singular_value=None, precondition=False)`

primary entry point for identifiability analysis

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	the singular spectrum truncation point. Defaults to minimum of non-zero-weighted observations and adjustable parameters	`None`
`precondition`	`bool`	flag to use the preconditioned hessian with the prior parameter covariance matrix (xtqt + sigma_theta^-1). This should be used KL scaling. Default is `False`.	`False`

Returns:

Type	Description
	`pandas.DataFrame`: A pandas dataframe of the right solution-space singular
	vectors and identifiability (identifiabiity is in the column labeled "ident")

Examples::

ev = pyemu.ErrVar(jco="my.jco")
df = ev.get_identifiability_dataframe(singular_value=20)
df.ident.plot(kind="bar")

`get_null_proj(maxsing=None, eigthresh=1e-06)`

get a null-space projection matrix of XTQX

Parameters:

Name	Type	Description	Default
`maxsing`	`int`	number of singular components to use (the truncation point). If None, `pyemu.Matrx.get_maxsing() is used to determine the truncation point with`eigthresh`. Default is None	`None`
`eigthresh`	`float`	the ratio of smallest to largest singular value to keep in the range (solution) space of XtQX. Not used if `maxsing` is not `None`. Default is 1.0e-6	`1e-06`

Note

used for null-space monte carlo operations.

Returns:

Type	Description
	`pyemu.Matrix` the null-space projection matrix (V2V2^T)

`get_obs_competition_dataframe()`

get the observation competition stat a la PEST utility

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of observation names by
	observation names with values equal to the PEST
	competition statistic

`get_par_css_dataframe()`

get a dataframe of composite scaled sensitivities. Includes both PEST-style and Hill-style.

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of parameter names, PEST-style and
	Hill-style composite scaled sensitivity

`reset_obscov(arg=None)`

reset the obscov attribute to None

Parameters:

Name	Type	Description	Default
`arg`	`str` or `pyemu.Matrix`	the value to assign to the obscov attribute. If None, the private __obscov attribute is cleared but not reset	`None`

`reset_parcov(arg=None)`

reset the parcov attribute to None

Parameters:

Name	Type	Description	Default
`arg`	`str` or `pyemu.Matrix`	the value to assign to the parcov attribute. If None, the private __parcov attribute is cleared but not reset	`None`

`reset_pst(arg)`

reset the LinearAnalysis.pst attribute

Parameters:

Name	Type	Description	Default
`arg`	`str` or `pyemu.Pst`	the value to assign to the pst attribute	required

`second_forecast(singular_value)`

get the solution space contribution to forecast (e.g. "prediction") error variance at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc second term at	required

Note

This method is used to construct error variance dataframe

Just a thin wrapper around ErrVar.second_prediction

Returns:

Type	Description
	`dict`: dictionary of ("second",prediction_names), error variance
	arising from the solution space contribution (y^t * G * obscov * G^T * y)

`second_parameter(singular_value)`

get the solution space contribution to parameter error variance at a given singular value (G * obscov * G^T)

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc second term at	required

Returns:

Type	Description
	`pyemu.Cov`: the second term contribution to parameter error variance
	(G * obscov * G^T)

`second_prediction(singular_value)`

get the solution space contribution to predictive error variance at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc second term at	required

Note

This method is used to construct error variance dataframe

`dict`: dictionary of ("second",prediction_names), error variance

Type	Description
	arising from the solution space contribution (y^t * G * obscov * G^T * y)

`third_forecast(singular_value)`

get the omitted parameter contribution to forecast (prediction) error variance at a given singular value.

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc third term at	required

Note

used to construct error variance dataframe just a thin wrapper around ErrVar.third_prediction()

Returns:

Type	Description
	`dict`: a dictionary of ("third",prediction_names),error variance

`third_parameter(singular_value)`

get the omitted parameter contribution to parameter error variance at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc third term at	required

Returns:

Type	Description
	`pyemu.Cov`: the third term contribution to parameter error variance
	calculated at `singular_value` (G * omitted_jco * Sigma_(omitted_pars) *
	omitted_jco^T * G^T). Returns 0.0 if third term calculations are not
	being used.

`third_prediction(singular_value)`

get the omitted parameter contribution to prediction error variance at a given singular value.

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to calc third term at	required

Note

used to construct error variance dataframe

Returns:

Type	Description
	`dict`: a dictionary of ("third",prediction_names),error variance

`variance_at(singular_value)`

get the error variance of all three error variance terms at a given singular value

Parameters:

Name	Type	Description	Default
`singular_value`	`int`	singular value to test	required

Returns:

Type	Description
	`dict`: dictionary of (err var term,prediction_name), variance pairs

ev

ErrVar

adj_par_names property

fehalf property

forecast_names property

forecasts property

forecasts_iter property

jco property

mle_covariance property

mle_parameter_estimate property

nnz_obs_names property

obscov property

omitted_jco property

omitted_parcov property

omitted_predictions property

parcov property

predictions property

predictions_iter property

prior_forecast property

prior_parameter property

prior_prediction property

pst property

qhalf property

qhalfx property

xtqx property

G(singular_value)

I_minus_R(singular_value)

R(singular_value)

__fromfile(filename, astype=None)

__load_jco()

__load_obscov()

__load_omitted_jco()

__load_omitted_parcov()

__load_omitted_predictions()

__load_parcov()

__load_predictions()

__load_pst()

adjust_obscov_resfile(resfile=None)

apply_karhunen_loeve_scaling()

clean()

drop_prior_information()

first_forecast(singular_value)

first_parameter(singular_value)

first_prediction(singular_value)

get(par_names=None, obs_names=None, astype=None)

get_cso_dataframe()

get_errvar_dataframe(singular_values=None)

get_identifiability_dataframe(singular_value=None, precondition=False)

get_null_proj(maxsing=None, eigthresh=1e-06)

get_obs_competition_dataframe()

get_par_css_dataframe()

reset_obscov(arg=None)

reset_parcov(arg=None)

reset_pst(arg)

second_forecast(singular_value)

second_parameter(singular_value)

second_prediction(singular_value)

third_forecast(singular_value)

third_parameter(singular_value)

third_prediction(singular_value)

variance_at(singular_value)

`ErrVar`

`adj_par_names` `property`

`fehalf` `property`

`forecast_names` `property`

`forecasts` `property`

`forecasts_iter` `property`

`jco` `property`

`mle_covariance` `property`

`mle_parameter_estimate` `property`

`nnz_obs_names` `property`

`obscov` `property`

`omitted_jco` `property`

`omitted_parcov` `property`

`omitted_predictions` `property`

`parcov` `property`

`predictions` `property`

`predictions_iter` `property`

`prior_forecast` `property`

`prior_parameter` `property`

`prior_prediction` `property`

`pst` `property`

`qhalf` `property`

`qhalfx` `property`

`xtqx` `property`

`G(singular_value)`

`I_minus_R(singular_value)`

`R(singular_value)`

`__fromfile(filename, astype=None)`

`__load_jco()`

`__load_obscov()`

`__load_omitted_jco()`

`__load_omitted_parcov()`

`__load_omitted_predictions()`

`__load_parcov()`

`__load_predictions()`

`__load_pst()`

`adjust_obscov_resfile(resfile=None)`

`apply_karhunen_loeve_scaling()`

`clean()`

`drop_prior_information()`

`first_forecast(singular_value)`

`first_parameter(singular_value)`

`first_prediction(singular_value)`

`get(par_names=None, obs_names=None, astype=None)`

`get_cso_dataframe()`

`get_errvar_dataframe(singular_values=None)`

`get_identifiability_dataframe(singular_value=None, precondition=False)`

`get_null_proj(maxsing=None, eigthresh=1e-06)`

`get_obs_competition_dataframe()`

`get_par_css_dataframe()`

`reset_obscov(arg=None)`

`reset_parcov(arg=None)`

`reset_pst(arg)`

`second_forecast(singular_value)`

`second_parameter(singular_value)`

`second_prediction(singular_value)`

`third_forecast(singular_value)`

`third_parameter(singular_value)`

`third_prediction(singular_value)`

`variance_at(singular_value)`