pyemu.eds

Module Contents

Classes

EnDS

Ensemble Data Space Analysis using the approach of He et al (2018) (https://doi.org/10.2118/182609-PA)
class pyemu.eds.EnDS(pst=None, sim_ensemble=None, noise_ensemble=None, obscov=None, predictions=None, verbose=False)

Bases: object

Ensemble Data Space Analysis using the approach of He et al (2018) (https://doi.org/10.2118/182609-PA)

Parameters:

  • pst (varies) – something that can be cast into a pyemu.Pst. Can be an str for a filename or an existing pyemu.Pst.

  • sim_ensemble (varies) – something that can be cast into a pyemu.ObservationEnsemble. Can be an str for a filename or pd.DataFrame or an existing pyemu.ObservationEnsemble.

  • noise_ensemble (varies) – something that can be cast into a pyemu.ObservationEnsemble that is the obs+noise realizations. If not passed, a noise ensemble is generated using either obs_cov or the information in pst (i.e. weights or standard deviations)

  • obscov (varies, optional) – observation noise covariance matrix. If str, a filename is assumed and the noise covariance matrix is loaded from a file using the file extension (“.jcb”/”.jco” for binary, “.cov”/”.mat” for PEST-style ASCII matrix, or “.unc” for uncertainty files). If None, the noise covariance matrix is constructed from the observation weights (and optionally “standard_deviation”) . Can also be a pyemu.Cov instance

  • forecasts (enumerable of str) – the names of the entries in pst.osbervation_data (and in ensemble).

  • verbose (bool) – controls screen output. If str, a filename is assumed and and log file is written.

Example:

#assumes "my.pst" exists
ends = pyemu.EnDS(ensemble="my.0.obs.jcb",forecasts=["fore1","fore2"])
ends.get_posterior_prediction_moments() #similar to Schur-style data worth
ends.prep_for_dsi() #setup a new pest interface() based on the DSI approach
property sim_ensemble
property obscov

get the observation noise covariance matrix attribute

Returns:

a reference to the LinearAnalysis.obscov attribute

Return type:

pyemu.Cov

property pst

the pst attribute

Returns:

the pst attribute

Return type:

pyemu.Pst

__fromfile(filename, astype=None)

a private method to deduce and load a filename into a matrix object. Uses extension: ‘jco’ or ‘jcb’: binary, ‘mat’,’vec’ or ‘cov’: ASCII, ‘unc’: pest uncertainty file.

__load_pst()

private method set the pst attribute

__load_ensemble(arg)

private method to set the ensemble attribute from a file or a dataframe object

__load_obscov()

private method to set the obscov attribute from: a pest control file (observation weights) a pst object a matrix object an uncert file an ascii matrix file

reset_pst(arg)

reset the EnDS.pst attribute

Parameters:

arg (str or pyemu.Pst) – the value to assign to the pst attribute

reset_obscov(arg=None)

reset the obscov attribute to None

Parameters:

arg (str or pyemu.Matrix) – the value to assign to the obscov attribute. If None, the private __obscov attribute is cleared but not reset

get_posterior_prediction_convergence_summary(num_realization_sequence, num_replicate_sequence, obslist_dict=None)

repeatedly run `EnDS.get_predictive_posterior_moments() with less than all the possible realizations to evaluate whether the uncertainty estimates have converged

Parameters:

  • num_realization_sequence (`[int’]) – the sequence of realizations to test.

  • num_replicate_sequence ([int]) – The number of replicates of randomly selected realizations to test for each num_realization_sequence value. For example, if num_realization_sequence is [10,100,1000] and num_replicated_sequence is [4,5,6], then EnDS.get_predictive posterior_moments() is called 4 times using 10 randomly selected realizations (new realizations selected 4 times), 5 times using 100 randomly selected realizations, and then 6 times using 1000 randomly selected realizations.

  • obslist_dict (dict, optional) – a nested dictionary-list of groups of observations to pass to EnDS.get_predictive_posterior_moments().

Returns:

a dictionary of num_reals: pd.DataFrame pairs, where the dataframe is the mean

predictive standard deviation results from calling EnDS.get_predictive_posterior_moments() for the desired number of replicates.

Return type:

dict

Example:

ends = pyemu.EnDS(pst="my.pst",sim_ensemble="my.0.obs.csv",predictions=["predhead","predflux"])
obslist_dict = {"hds":["head1","head2"],"flux":["flux1","flux2"]}
num_reals_seq = [10,20,30,100,1000] # assuming there are 1000 reals in "my.0.obs.csv"]
num_reps_seq = [5,5,5,5,5]
mean_dfs = sc.get_posterior_prediction_convergence_summary(num_reals_seq,num_reps_seq,
    obslist_dict=obslist_dict)

get_posterior_prediction_moments(obslist_dict=None, sim_ensemble=None, include_first_moment=True)
A dataworth method to analyze the posterior (expected) mean and uncertainty as a result of conditioning with

some additional observations not used in the conditioning of the current ensemble results.

Parameters:

  • obslist_dict (dict, optional) – a nested dictionary-list of groups of observations that are to be treated as gained/collected. key values become row labels in returned dataframe. If None, then every zero-weighted observation is tested sequentially. Default is None

  • sim_ensemble (pyemu.ObservationEnsemble) – the simulation results ensemble to use. If None, self.sim_ensemble is used. Default is None

  • include_first_moment (bool) – flag to include calculations of the predictive first moments. This can slow things down,so if not needed, better to skip. Default is True

Returns:

tuple containing

  • dict: dictionary of first-moment dataframes. Keys are obslist_dict keys. If include_first_moment

    is None, this is an empty dict.

  • pd.DataFrame: prediction standard deviation summary

  • pd.DataFrame: precent prediction standard deviation reduction summary

Example:

ends = pyemu.EnDS(pst="my.pst",sim_ensemble="my.0.obs.csv",predictions=["predhead","predflux"])
obslist_dict = {"hds":["head1","head2"],"flux":["flux1","flux2"]}
mean_dfs,dfstd,dfpercent = sc.get_posterior_prediction_moments(obslist_dict=obslist_dict)
prep_for_dsi(sim_ensemble=None, t_d='dsi_template')

setup a new PEST interface for the data-space inversion process

Parameters:

  • sim_ensemble (pyemu.ObservationEnsemble) – observation ensemble to use for DSI latent space variables. If None, use self.sim_ensemble. Default is None

  • t_d (str) – template directory to setup the DSI model + pest files in. Default is dsi_template

Example:

#assumes "my.pst" exists

ends = pyemu.EnDS(ensemble=”my.0.obs.jcb”,forecasts=[“fore1”,”fore2”]) ends.prep_for_dsi() #setup a new pest interface() based on the DSI approach pyemu.os_utils.start_workers(“pestpp-ies”,”my.pst”,”dsi_template”,num_workers=20,

master_dir=”dsi_master”)