pyemu.pst package

Submodules

pyemu.pst.pst_controldata module

This module contains several class definitions for obseure parts of the PEST control file: ControlData (’* control data’), RegData (’* regularization’) and SvdData (’* singular value decomposition’). These classes are automatically created and appended to Pst instances; users shouldn’t need to deal with these classes explicitly

class pyemu.pst.pst_controldata.ControlData

Bases: object

an object that encapsulates the control data section: of the PEST control file

Notes

This class works hard to protect the variables in the control data section. It type checks attempts to change values to make sure the type being passed matches the expected type of the attribute.

copy()

property formatted_values

list the entries and current values in the control data section

Returns:: formatted_values for the control data entries
Return type:: pandas.Series

static get_dataframe()

get a generic (default) control section as a dataframe

Returns:: a dataframe of control data information
Return type:: pandas.DataFrame

parse_values_from_lines(lines, iskeyword=False)

cast the string lines for a pest control file into actual inputs

Parameters:: lines ([str]) – raw ASCII lines from pest control file

write(f)

write control data section to a file

Parameters:: f (file handle) – open file handle to write to

write_keyword(f)

write the control data entries to an open file handle using keyword-style format.

Parameters:: f (file handle) – open file handle to write to

Notes

only writes values that have been accessed since instantiation

pyemu.pst.pst_controldata.FFMT(x): lambda function for string formatting float types into 15-char widths

pyemu.pst.pst_controldata.IFMT(x): lambda function for string formatting int types into 10-char widths

class pyemu.pst.pst_controldata.RegData

Bases: object

an object that encapsulates the regularization section of the PEST control file

write(f)

write the regularization section to an open file handle

Parameters:: f (file handle) – open file handle for writing

write_keyword(f)

write the regularization section to an open file handle using the keyword-style format

Parameters:: f (file handle) – open file handle for writing

pyemu.pst.pst_controldata.SFMT(x): lambda function for string formatting str types into 20-char widths

pyemu.pst.pst_controldata.SFMT_LONG(x): lambda function for string formatting str types into 50-char widths

class pyemu.pst.pst_controldata.SvdData(**kwargs)

Bases: object

encapsulates the singular value decomposition section of the PEST control file

Parameters:: kwargs (dict) – optional keyword arguments

parse_values_from_lines(lines)

parse values from lines of the SVD section of a PEST control file

Parameters:: lines ([strs]) – the raw ASCII lines from the control file

write(f)

write an SVD section to a file handle

Parameters:: f (file handle) – open file handle for writing

write_keyword(f)

write an SVD section to a file handle using keyword-style format

Args:
f (file handle): open file handle for writing

pyemu.pst.pst_handler module

class pyemu.pst.pst_handler.Pst(filename, load=True, resfile=None)

Bases: object

All things PEST(++) control file

Parameters:

filename (str) – the name of the control file
load (bool, optional) – flag to load the control file. Default is True
resfile (str, optional) – corresponding residual file. If None, a residual file with the control file base name is sought. Default is None

Note

This class is the primary mechanism for dealing with PEST control files. Support is provided for constructing new control files as well as manipulating existing control files.

Example:

pst = pyemu.Pst("my.pst")
pst.control_data.noptmax = -1
pst.write("my_new.pst")

add_observations(ins_file, out_file=None, pst_path=None, inschek=True)

add new observations to a control file

Parameters:

ins_file (str) – instruction file with exclusively new observation names
out_file (str) – model output file. If None, then ins_file.replace(“.ins”,””) is used. Default is None
pst_path (str) – the path to append to the instruction file and out file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None
inschek (bool) – flag to try to process the existing output file using the pyemu.InstructionFile class. If successful, processed outputs are used as obsvals

Returns:

the data for the new observations that were added

Return type:

pandas.DataFrame

Note

populates the new observation information with default values

Example:

pst = pyemu.Pst(os.path.join("template", "my.pst"))
pst.add_observations(os.path.join("template","new_obs.dat.ins"), pst_path=".")
pst.write(os.path.join("template", "my_new.pst")

add_parameters(template_file, in_file=None, pst_path=None)

add new parameters to an existing control file

Parameters:

template_file (str) – template file with (possibly) some new parameters
in_file (str) – model input file. If None, template_file.replace(‘.tpl’,’’) is used. Default is None.
pst_path (str) – the path to append to the template_file and in_file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None

Returns:

the data for the new parameters that were added. If no new parameters are in the new template file, returns None

Return type:

pandas.DataFrame

Note

populates the new parameter information with default values

Example:

pst = pyemu.Pst(os.path.join("template","my.pst"))
pst.add_parameters(os.path.join("template","new_pars.dat.tpl",pst_path=".")
pst.write(os.path.join("template","my_new.pst")

add_pi_equation(par_names, pilbl=None, rhs=0.0, weight=1.0, obs_group='pi_obgnme', coef_dict={})

a helper to construct a new prior information equation.

Parameters:

par_names ([str]) – parameter names in the equation
pilbl (str) – name to assign the prior information equation. If None, a generic equation name is formed. Default is None
rhs (float) – the right-hand side of the pi equation
weight (float) – the weight of the equation
obs_group (str) – the observation group for the equation. Default is ‘pi_obgnme’
coef_dict (dict) – a dictionary of parameter name, coefficient pairs to assign leading coefficients for one or more parameters in the equation. If a parameter is not listed, 1.0 is used for its coefficients. Default is {}

Example:

pst = pyemu.Pst("pest.pst")
# add a pi equation for the first adjustable parameter
pst.add_pi_equation(pst.adj_par_names[0],pilbl="pi1",rhs=1.0)
# add a pi equation for 1.5 times the 2nd and 3 times the 3rd adj pars to sum together to 2.0
names = pst.adj_par_names[[1,2]]
pst.add_pi_equation(names,coef_dict={names[0]:1.5,names[1]:3})

add_transform_columns()

add transformed values to the Pst.parameter_data attribute

Note

adds parval1_trans, parlbnd_trans and parubnd_trans to Pst.parameter_data

Example:

pst = pyemu.Pst("pest.pst")
pst.add_transform_columns()
print(pst.parameter_data.parval1_trans

property adj_par_groups

get the parameter groups with atleast one adjustable parameter

Returns:: a list of parameter groups with at least one adjustable parameter
Return type:: [str]

property adj_par_names

get the adjustable (not fixed or tied) parameter names

Returns:: list of adjustable (not fixed or tied) parameter names
Return type:: [str]

adjust_weights(obs_dict=None, obsgrp_dict=None)

reset the weights of observations or observation groups to contribute a specified amount to the composite objective function

Parameters:

obs_dict (dict, optional) – dictionary of observation name,new contribution pairs
obsgrp_dict (dict, optional) – dictionary of obs group name,contribution pairs

Notes

If a group is assigned a contribution of 0, all observations in that group will be assigned zero weight.

If a group is assigned a nonzero contribution AND all observations in that group start with zero weight, the observations will be assigned weight of 1.0 to allow balancing.

If groups obsgrp_dict is not passed, all nonzero

Example:

pst = pyemu.Pst("my.pst")

# adjust a single observation
pst.adjust_weights(obs_dict={"obs1":10})

# adjust a single observation group
pst.adjust_weights(obsgrp_dict={"group1":100.0})

# make all non-zero weighted groups have a contribution of 100.0
balanced_groups = {grp:100 for grp in pst.nnz_obs_groups}
pst.adjust_weights(obsgrp_dict=balanced_groups)

adjust_weights_discrepancy(resfile=None, original_ceiling=True, bygroups=False)

adjusts the weights of each non-zero weight observation based on the residual in the pest residual file so each observations contribution to phi is 1.0 (e.g. Mozorov’s discrepancy principal)

Parameters:

resfile (str) – residual file name. If None, try to use a residual file with the Pst case name. Default is None
original_ceiling (bool) – flag to keep weights from increasing - this is generally a good idea. Default is True
bygroups (bool) – flag to adjust weights by groups. If False, the weight of each non-zero weighted observation is adjusted individually. If True, intergroup weighting is preserved (the contribution to each group is used) but this may result in some strangeness if some observations in a group have a really low phi already.

Example:

pst = pyemu.Pst("my.pst")
print(pst.phi) #assumes "my.res" is colocated with "my.pst"
pst.adjust_weights_discrepancy()
print(pst.phi) # phi should equal number of non-zero observations

bounds_report(iterations=None)

report how many parameters are at bounds. If ensemble, the base enbsemble member is evaluated

Parameters:

iterations ([int]) – a list of iterations for which a bounds report is requested If None, all iterations for which par files are located are reported. Default is None

Returns:

a pandas DataFrame object with rows being parameter groups and columns: <iter>_num_at_ub, <iter>_num_at_lb, and <iter>_total_at_bounds row 0 is total at bounds, subsequent rows correspond with groups

Return type:

df

Example

pst = pyemu.Pst(“my.pst”) df = pst.bound_report(iterations=[0,2,3])

build_increments(): experimental method to calculate parameter increments for use in the finite difference pertubation calculations

Note

user beware!

calculate_pertubations(): experimental method to calculate finite difference parameter pertubations.

Note

The pertubation values are added to the Pst.parameter_data attribute - user beware!

control_data

‘* control data’ information. Access with standard PEST variable names

Example:

pst.control_data.noptmax = 2
pst.control_data.pestmode = "estimation"

Type:: pyemu.pst.pst_controldata.ControlData

drop_observations(ins_file, pst_path=None)

remove observations in an instruction file from the control file

Parameters:

ins_file (str) – instruction file to remove
pst_path (str) – the path to append to the instruction file in the control file. If not None, then any existing path in front of the instruction is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None

Returns:

the observation data for the observations that were removed.

Return type:

pandas.DataFrame

Example:

pst = pyemu.Pst(os.path.join("template", "my.pst"))
pst.remove_observations(os.path.join("template","some_obs.dat.ins"), pst_path=".")
pst.write(os.path.join("template", "my_new_with_less_obs.pst")

drop_parameters(tpl_file, pst_path=None)

remove parameters in a template file from the control file

Parameters:

tpl_file (str) – template file to remove
pst_path (str) – the path to append to the template file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None

Returns:

the parameter data for the parameters that were removed.

Return type:

pandas.DataFrame

Note

This method does not check for multiple occurences of the same parameter name(s) in across template files so if you have the same parameter in multiple template files, this is not the method you are looking for

Example:

pst = pyemu.Pst(os.path.join("template", "my.pst"))
pst.remove_parameters(os.path.join("template","boring_zone_pars.dat.tpl"), pst_path=".")
pst.write(os.path.join("template", "my_new_with_less_pars.pst")

enforce_bounds()

enforce bounds violation

Note

cheap enforcement of simply bringing violators back in bounds

Example:

pst = pyemu.Pst("pest.pst")
pst.parrep("random.par")
pst.enforce_bounds()
pst.write("pest_rando.pst")

property estimation

check if the control_data.pestmode is set to estimation

Returns:: True if control_data.pestmode is estmation, False otherwise
Return type:: bool

property forecast_names

get the forecast names from the pestpp options (if any). Returns None if no forecasts are named

Returns:: a list of forecast names.
Return type:: [str]

classmethod from_io_files(tpl_files, in_files, ins_files, out_files, pst_filename=None, pst_path=None)

create a Pst instance from model interface files.

Parameters:

tpl_files ([str]) – list of template file names
in_files ([str]) – list of model input file names (pairs with template files)
ins_files ([str]) – list of instruction file names
out_files ([str]) – list of model output file names (pairs with instruction files)
pst_filename (str) – name of control file to write. If None, no file is written. Default is None
pst_path ('str') – the path from the control file to the IO files. For example, if the control will be in the same directory as the IO files, then pst_path should be ‘.’. Default is None, which doesnt do any path manipulation on the I/O file names

Returns:

new control file instance with parameter and observation names found in tpl_files and ins_files, repsectively.

Return type:

Pst

Note

calls pyemu.helpers.pst_from_io_files()

Assigns generic values for parameter info. Tries to use INSCHEK to set somewhat meaningful observation values

all file paths are relatively to where python is running.

Example:

tpl_files = ["my.tpl"]
in_files = ["my.in"]
ins_files = ["my.ins"]
out_files = ["my.out"]
pst = pyemu.Pst.from_io_files(tpl_files,in_files,ins_files,out_files)
pst.control_data.noptmax = 0
pst.write("my.pst)

classmethod from_par_obs_names(par_names=['par1'], obs_names=['obs1'])

construct a shell Pst instance from parameter and observation names

Parameters:

par_names ([str]) – list of parameter names. Default is [par1]
obs_names ([str]) – list of observation names. Default is [obs1]

Note

While this method works, it does not make template or instruction files. Users are encouraged to use Pst.from_io_files() for more usefulness

Example:

par_names = ["par1","par2"]
obs_names = ["obs1","obs2"]
pst = pyemu.Pst.from_par_obs_names(par_names,obs_names)

get(par_names=None, obs_names=None)

get a new pst object with subset of parameters and/or observations

Parameters:

par_names ([str]) – a list of parameter names to have in the new Pst instance. If None, all parameters are in the new Pst instance. Default is None
obs_names ([str]) – a list of observation names to have in the new Pst instance. If None, all observations are in teh new Pst instance. Default is None

Returns:

a new Pst instance

Return type:

Pst

Note

passing par_names as None and obs_names as None effectively generates a copy of the current Pst

Does not modify model i/o files - this is just a method for performing pyemu operations

Example:

pst = pyemu.Pst("pest.pst")
new_pst = pst.get(pst.adj_par_names[0],pst.obs_names[:10])

get_adj_pars_at_bounds(frac_tol=0.01)

get list of adjustable parameter at/near bounds

Parameters:

frac_tol ('float`) – fractional tolerance of distance to bound. For upper bound, the value parubnd * (1-frac_tol) is used, lower bound uses parlbnd * (1.0 + frac_tol)

Returns:

[`str`]: list of parameters at/near lower bound
[`str`]: list of parameters at/near upper bound

Return type:

tuple containing

Example:

pst = pyemu.Pst("pest.pst")
at_lb,at_ub = pst.get_adj_pars_at_bounds()
print("pars at lower bound",at_lb)

get_par_change_limits()

calculate the various parameter change limits used in pest.

Returns:: a copy of Pst.parameter_data with columns for relative and factor change limits
Return type:: pandas.DataFrame

Note

does not yet support absolute parameter change limits!

Works in control file values space (not log transformed space). Also adds columns for effective upper and lower which account for par bounds and the value of parchglim

example:

pst = pyemu.Pst("pest.pst")
df = pst.get_par_change_limits()
print(df.chg_lower)

static get_phi_components(ogroups, res, obs, pi_ogroups=None, pi_df=None)

get_res_stats(nonzero=True)

get some common residual stats by observation group.

Parameters:: nonzero (bool) – calculate stats using only nonzero-weighted observations. This may seem obsvious to most users, but you never know….
Returns:: a dataframe with columns for groups names and indices of statistic name.
Return type:: pd.DataFrame

Note

Stats are derived from the current obsvals, weights and grouping in Pst.observation_data and the modelled values in Pst.res. The key here is ‘current’ because if obsval, weights and/or groupings have changed in Pst.observation_data since the residuals file was generated then the current values for obsval, weight and group are used

the normalized RMSE is normalized against the obsval range (max - min)

Example:

pst = pyemu.Pst("pest.pst")
stats_df = pst.get_res_stats()
print(stats_df.loc["mae",:])

property greater_than_obs_constraints

get the names of the observations that are listed as active (non-zero weight) greater than inequality constraints.

Returns:: names obseravtions that are non-zero weighted greater than constraints (obgnme startsiwth “g_” or “greater”)
Return type:: pandas.Series

Note

Zero-weighted obs are skipped

property greater_than_pi_constraints

get the names of the prior information eqs that are listed as active (non-zero weight) greater than inequality constraints.

Returns:: pandas.Series names of prior information that are non-zero weighted greater than constraints (obgnme startsiwth “g_” or “greater”)

Note

Zero-weighted pi are skipped

property input_files

list of model input file names

Returns:

a list of model input file names, extracted from: Pst.model_input_data.model_file. Returns None if this attribute is None

Return type:

[str]

Note

Use Pst.model_input_data to access the template-input file information for writing/modification

property instruction_files

list of instruction file names :returns:

a list of instruction file names, extracted from

Pst.model_output_data.pest_file. Returns None if this
attribute is None

Return type:: [str]

Note

Use Pst.model_output_data to access the instruction-output file information for writing/modification

property less_than_obs_constraints

get the names of the observations that are listed as active (non-zero weight) less than inequality constraints.

Returns:: names of observations that are non-zero weighted less than constraints (obgnme starts with ‘l_’ or “less”)
Return type:: pandas.Series

Note

Zero-weighted obs are skipped

property less_than_pi_constraints

get the names of the prior information eqs that are listed as active (non-zero weight) less than inequality constraints.

Returns:: names of prior information that are non-zero weighted less than constraints (obgnme starts with “l_” or “less”)
Return type:: pandas.Series

Note

Zero-weighted pi are skipped

load(filename)

entry point load the pest control file.

Parameters:: filename (str) – pst filename

Note

This method is called from the Pst construtor unless the load arg is False.

property nnz_obs

get the number of non-zero weighted observations

Returns:: the number of non-zeros weighted observations
Return type:: int

property nnz_obs_groups

get the observation groups that contain at least one non-zero weighted: observation

Returns:: a list of observation groups that contain at least one non-zero weighted observation
Return type:: [str]

property nnz_obs_names

get the non-zero weight observation names

Returns:: a list of non-zero weighted observation names
Return type:: [str]

property nobs

get the number of observations

Returns:: the number of observations
Return type:: int

property npar

get number of parameters

Returns:: the number of parameters
Return type:: int

property npar_adj

get the number of adjustable parameters (not fixed or tied)

Returns:: the number of adjustable parameters
Return type:: int

property nprior

number of prior information equations

Returns:: the number of prior info equations
Return type:: int

property obs_groups

get the observation groups

Returns:: a list of unique observation groups
Return type:: [str]

property obs_names

get the observation names

Returns:: a list of observation names
Return type:: [str]

observation_data

‘* observation data’ information. Columns are standard PEST variable names

Example:

pst.observation_data.loc[:,"weight"] = 1.0
pst.observation_data.loc[:,"obgnme"] = "obs_group"

Type:: pandas.DataFrame

property output_files

list of model output file names :returns:

a list of model output file names, extracted from
Pst.model_output_data.model_file. Returns None if this attribute is None

Return type:: [str]

Note

Use Pst.model_output_data to access the instruction-output file information for writing/modification

property par_groups

get the parameter groups

Returns:: a list of parameter groups
Return type:: [str]

property par_names

get the parameter names

Returns:: a list of parameter names
Return type:: [str]

parameter_data

‘* parameter data’ information. Columns are standard PEST variable names

Example:

pst.parameter_data.loc[:,"partrans"] = "log"
pst.parameter_data.loc[:,"parubnd"] = 10.0

Type:: pandas.DataFrame

parrep(parfile=None, enforce_bounds=True, real_name=None, noptmax=0, binary_ens_file=False)

replicates the pest parrep util. replaces the parval1 field in the: parameter data section dataframe with values in a PEST parameter file or a single realization from an ensemble parameter csv file

Parameters:

parfile (str, optional) – parameter file to use. If None, try to find and use a parameter file that corresponds to the case name. If parfile has extension ‘.par’ a single realization parameter file is used If parfile has extention ‘.csv’ an ensemble parameter file is used which invokes real_name If parfile has extention ‘.jcb’ a binary ensemble parameter file is used which invokes real_name Default is None
enforce_bounds (bool, optional) – flag to enforce parameter bounds after parameter values are updated. This is useful because PEST and PEST++ round the parameter values in the par file, which may cause slight bound violations. Default is True
real_name (str or int, optional) – name of the ensemble realization to use for updating the parval1 value in the parameter data section dataframe. If None, try using “base”. If “base” not present, use the real_name with smallest index number. Ignored if parfile is of the PEST parameter file format (e.g. not en ensemble)
noptmax (int, optional) – Value with which to update the pst.control_data.noptmax value Default is 0.
binary_ens_file (bool) – If True, use binary format to load ensemble file, else assume it’s a CSV file

Example:

pst = pyemu.Pst("pest.pst")
pst.parrep("pest.1.base.par")
pst.control_data.noptmax = 0
pst.write("pest_1.pst")

property phi

get the weighted total objective function.

Returns:: sum of squared residuals
Return type:: float

Note

Requires Pst.res (the residuals file) to be available

property phi_components

get the individual components of the total objective function

Returns:: dictionary of observation group, contribution to total phi
Return type:: dict

Note

Requires Pst.res (the residuals file) to be available

property phi_components_normalized

get the individual components of the total objective function: normalized to the total PHI being 1.0

Returns:: dictionary of observation group, normalized contribution to total phi
Return type:: dict

Note

Requires Pst.res (the residuals file) to be available

plot(kind=None, **kwargs)

method to plot various parts of the control. This is sweet as!

Parameters:

kind (str) – options are ‘prior’ (prior parameter histograms, ‘1to1’ (line of equality and sim vs res), ‘obs_v_sim’ (time series using datetime suffix), ‘phi_pie’ (pie chart of phi components)
kwargs (dict) – optional args for plots that are passed to pyemu plot helpers and ultimately to matplotlib

Note

Depending on ‘kind’ argument, a multipage pdf is written

Example:

pst = pyemu.Pst("my.pst")
pst.plot(kind="1to1") # requires Pst.res
pst.plot(kind="prior")
pst.plot(kind="phi_pie")

property prior_groups

get the prior info groups

Returns:: a list of prior information groups
Return type:: [str]

prior_information

‘* prior information’ data. Columns are standard PEST variable names

Type:: pandas.DataFrame

property prior_names

get the prior information names

Returns:: a list of prior information names
Return type:: [str]

process_output_files(pst_path='.')

processing the model output files using the instruction files and existing model output files.

Parameters:: pst_path (str) – relative path from where python is running to where the control file, instruction files and model output files are located. Default is “.” (current python directory)
Returns:: model output values
Return type:: pandas.Series

Note

requires a complete set of model input files at relative path from where python is running to pst_path

Example:

pst = pyemu.Pst("pest.pst")
obsvals = pst.process_output_files()
print(obsvals)

proportional_weights(fraction_stdev=1.0, wmax=100.0, leave_zero=True)

setup weights inversely proportional to the observation value

Parameters:

fraction_stdev (float, optional) – the fraction portion of the observation val to treat as the standard deviation. set to 1.0 for inversely proportional. Default is 1.0
wmax (float, optional) – maximum weight to allow. Default is 100.0
leave_zero (bool, optional) – flag to leave existing zero weights. Default is True

Example:

pst = pyemu.Pst("pest.pst")
# set the weights of the observations to 20% of the observed value
pst.proportional_weights(fraction_stdev=0.2,wmax=10)
pst.write("pest_propo.pst")

rectify_pgroups()

synchronize parameter groups section with the parameter data section

Note

This method is called during Pst.write() to make sure all parameter groups named in * parameter data are included. This is so users don’t have to manually keep this section up. This method can also be called during control file modifications to see what parameter groups are present and prepare for modifying the default values in the * parameter group section

Example:

pst = pyemu.Pst("my.pst")
pst.parameter_data.loc["par1","pargp"] = "new_group"
pst.rectify_groups()
pst.parameter_groups.loc["new_group","derinc"] = 1.0

rectify_pi(): rectify the prior information equation with the current state of the parameter_data dataframe.

Note

Equations that list fixed, tied or missing parameters are removed completely even if adjustable parameters are also listed in the equation. This method is called during Pst.write()

reg_data

‘* regularization’ section information. Access with standard PEST variable names.

Example:

pst.reg_data.phimlim = 1.00 #yeah right!

Type:: pyemu.pst.pst_controldata.RegData

rename_observations(name_dict, pst_path='.', insmap=None)

rename observations in the control and instruction files

Parameters:

name_dict (dict) – mapping of current to new names.
pst_path (str) – the path to the control file from where python is running. Default is “.” (python is running in the same directory as the control file)

Note

This does a lot of string compare, so its gonna be slow as…

Example:

pst = pyemu.Pst(os.path.join("template","pest.pst"))
name_dict = {"obs1":"obs1_better_name"}
pst.rename_observations(name_dict,pst_path="template")

rename_parameters(name_dict, pst_path='.', tplmap=None)

rename parameters in the control and template files

Parameters:

name_dict (dict) – mapping of current to new names.
pst_path (str) – the path to the control file from where python is running. Default is “.” (python is running in the same directory as the control file)

Note

no attempt is made to maintain the length of the marker strings in the template files, so if your model is sensitive to changes in spacing in the template file(s), this is not a method for you

This does a lot of string compare, so its gonna be slow as…

Example:

pst = pyemu.Pst(os.path.join("template","pest.pst"))
name_dict = {"par1":"par1_better_name"}
pst.rename_parameters(name_dict,pst_path="template")

property res

get the residuals dataframe attribute

Returns:: a dataframe containing the residuals information.
Return type:: pandas.DataFrame

Note

if the Pst.__res attribute has not been loaded, this call loads the res dataframe from a file

Example:

# print the observed and simulated values for non-zero weighted obs
print(pst.res.loc[pst.nnz_obs_names,["modelled","measured"]])

sanity_checks(forgive=False)

some basic check for strangeness

Parameters:: forgive (bool) – flag to forgive (warn) for issues. Default is False

Note

checks for duplicate names, atleast 1 adjustable parameter and at least 1 non-zero-weighted observation

Not nearly as comprehensive as pestchek

Example:

pst = pyemu.Pst("pest.pst")
pst.sanity_checks()

set_res(res)

reset the private Pst.res attribute.

Parameters:: res – (pandas.DataFrame or str): something to use as Pst.res attribute. If res is str, a dataframe is read from file res

svd_data

‘* singular value decomposition’ section information. Access with standard PEST variable names

Example:

pst.svd_data.maxsing = 100

Type:: pyemu.pst.pst_controldata.SvdData

property template_files

list of template file names

Returns:

a list of template file names, extracted from: Pst.model_input_data.pest_file. Returns None if this attribute is None

Return type:

[str]

Note

Use Pst.model_input_data to access the template-input file information for writing/modification

property tied

list of tied parameter names

Returns:: a dataframe of tied parameter information. Columns of tied are parnme and partied. Returns None if no tied parameters are found.
Return type:: pandas.DataFrame

try_parse_name_metadata(): try to add meta data columns to parameter and observation data based on item names. Used with the PstFrom process.

Note

metadata is identified in key-value pairs that are separated by a colon. each key-value pair is separated from others by underscore

This works with PstFrom style long names

This method is called programmtically during Pst.load()

write(new_filename, version=None, check_interface=False)

main entry point to write a pest control file.

Parameters:

new_filename (str) – name of the new pest control file
version (int) – flag for which version of control file to write (must be 1 or 2). if None, uses the number of pars to decide: if number of pars iis greater than 10,000, version 2 is used.
check_interface (bool) – flag to check the control file par and obs names against the names found in the template and instruction files. Default is False

Example:

pst = pyemu.Pst("my.pst")
pst.parrep("my.par")
pst.write(my_new.pst")
#write a version 2 control file
pst.write("my_new_v2.pst",version=2)

write_input_files(pst_path='.')

writes model input files using template files and current parval1 values.

Parameters:: pst_path (str) – the path to where control file and template files reside. Default is ‘.’

Note

adds “parval1_trans” column to Pst.parameter_data that includes the effect of scale and offset

Example:

pst = pyemu.Pst("my.pst")

# load final parameter values
pst.parrep("my.par")

# write new model input files with final parameter values
pst.write_input_files()

write_obs_summary_table(filename=None, group_names=None)

write a stand alone observation summary latex table or Excel shet

filename (str): filename. If None, use <case>.par.tex to write as LaTeX. If filename extention is ‘.xls’ or ‘.xlsx’,: tries to write as an Excel file. If filename is “none”, no table is written Default is None

Parameters:

filename (str) – filename. If filename is “none”, no table is written. If None, use <case>.obs.tex. If filename extention is ‘.xls’ or ‘.xlsx’, tries to write as an Excel file. Default is None
group_names (dict) – obs group names : table names. For example {“hds”:”simulated groundwater level”}. Default is None

Returns:

the summary observation group dataframe

Return type:

pandas.DataFrame

Example:

pst = pyemu.Pst("my.pst")
pst.write_obs_summary_table(filename="obs.tex")

write_par_summary_table(filename=None, group_names=None, sigma_range=4.0, report_in_linear_space=False)

write a stand alone parameter summary latex table or Excel sheet

Parameters:

filename (str) – filename. If None, use <case>.par.tex to write as LaTeX. If filename extention is ‘.xls’ or ‘.xlsx’, tries to write as an Excel file. If filename is “none”, no table is written Default is None
group_names (dict) – par group names : table names. For example {“w0”:”well stress period 1”}. Default is None
sigma_range (float) – number of standard deviations represented by parameter bounds. Default is 4.0, implying 95% confidence bounds
report_in_linear_space (bool) – flag, if True, that reports all logtransformed values in linear space. This renders standard deviation meaningless, so that column is skipped

Returns:

the summary parameter group dataframe

Return type:

pandas.DataFrame

Example:

pst = pyemu.Pst("my.pst")
pst.write_par_summary_table(filename="par.tex")

property zero_weight_obs_names

get the zero-weighted observation names

Returns:: a list of zero-weighted observation names
Return type:: [str]

pyemu.pst.pst_handler.get_constraint_tags(ltgt='lt')

pyemu.pst.pst_utils module

Various PEST(++) control file peripheral operations

pyemu.pst.pst_utils.FFMT(x)

pyemu.pst.pst_utils.IFMT(x)

class pyemu.pst.pst_utils.InstructionFile(ins_filename, pst=None)

Bases: object

class for handling instruction files.

Parameters:

ins_filename (str) – path and name of an existing instruction file
pst (pyemu.Pst, optional) – Pst instance - used for checking that instruction file is compatible with the control file (e.g. no duplicates)

Example:

i = InstructionFile("my.ins")
df = i.read_output_file("my.output")

property obs_name_set

read_ins_file(): read the instruction and do some minimal error checking.

Note

This is called by the constructor

read_output_file(output_file)

process a model output file using InstructionFile.instruction_set

Parameters:: output_file (str) – path and name of existing output file
Returns:: a dataframe with observation names and simulated values extracted from output_file
Return type:: pd.DataFrame

throw_ins_error(message, lcount=None)

throw a verbose instruction file error

Parameters:

message (str) – the error message
lcount (int, optional) – error line number. If None, self._ins_linecount is used

throw_ins_warning(message, lcount=None)

throw a verbose PyemuWarning

Parameters:

message (str) – the warning message
lcount (int, optional) – warning line number. If None, self._ins_linecount is used

throw_out_error(message, lcount=None)

throw a verbose output file error

Parameters:

message (str) – the error message
lcount (int, optional) – error line number. If None, self._ins_linecount is used

pyemu.pst.pst_utils.SFMT(item)

pyemu.pst.pst_utils.SFMT_LONG(x)

pyemu.pst.pst_utils.check_interface(pst, pst_path='.', warn=False)

check that the tpl and ins file entries are in sync with the control file entries

Parameters:

pst (pyemu.Pst) – control file instance
pst_path (str) – the path from where python is running to the control file
warn (bool) – flag to treat errors as warnings

pyemu.pst.pst_utils.clean_missing_exponent(pst_filename, clean_filename='clean.pst')

fixes the issue where some terrible fortran program may have written a floating point format without the ‘e’ - like 1.0-3, really?!

Parameters:

pst_filename (str) – the pest control file
clean_filename (str, optional) – the new pest control file to write. Default is “clean.pst”

pyemu.pst.pst_utils.csv_to_ins_file(csv_filename, ins_filename=None, only_cols=None, only_rows=None, marker='~', includes_header=True, includes_index=True, prefix='', head_lines_len=0, sep=',', gpname=False)

write a PEST-style instruction file from an existing CSV file

Parameters:

csv_filename (str) – path and name of existing CSV file
ins_filename (str, optional) – path and name of the instruction file to create. If None, then csv_filename`+”.ins” is used. Default is `None.
only_cols ([str]) – list of columns to add observations for in the resulting instruction file. If None, all columns are used.
only_rows ([str]) – list of rows to add observations for in the resulting instruction file. If None, all rows are used.
marker (str) – the PEST instruction marker to use. Default is “~”
includes_header (bool) – flag to indicate csv_filename includes a header row as the first row. Default is True.
includes_index (bool) – lag to indicate csv_filename includes a index column as the first column. Default is True.
prefix (str, optional) – a prefix to prepend to observation names. Default is “”
gpname (str or [str]) – Optional PEST group name for columns

Returns:

a dataframe of observation names and values found in csv_filename

Return type:

pandas.DataFrame

Note

resulting observation names in ins_filename are a combiation of index and header values.

pyemu.pst.pst_utils.generic_pst(par_names=['par1'], obs_names=['obs1'], addreg=False)

generate a generic pst instance.

Parameters:

par_names ([str], optional) – parameter names to include in the new pyemu.Pst. Default is [“par2”].
obs_names ([str], optional) – observation names to include in the new pyemu.Pst. Default is [“obs1”].
addreg (bool) – flag to add zero-order Tikhonov prior information equations to the new control file

Returns:

a new control file instance. This instance does not have all the info needed to run, but is a placeholder that can then be filled in later.

Return type:

pyemu.Pst

Example:

par_names = ["par1","par2"]
obs_names = ["obs1","obs2"]
pst = pyemu.pst_utils.generic_pst(par_names,obs_names]

pyemu.pst.pst_utils.get_phi_comps_from_recfile(recfile)

read the phi components from a record file by iteration

Parameters:: recfile (str) – pest record file name
Returns:: nested dictionary of iteration number, {group,contribution}
Return type:: dict

Note

It is really poor form to use the record file in this way. Please only use this as a last resort!

pyemu.pst.pst_utils.parse_ins_file(ins_file)

parse a PEST-style instruction file to get observation names

Parameters:: ins_file (str) – path and name of an existing instruction file
Returns:: a list of observation names found in ins_file
Return type:: [str]

Note

This is a basic function for parsing instruction files to look for observation names.

Example:

obs_names = pyemu.pst_utils.parse_ins_file("my.ins")

pyemu.pst.pst_utils.parse_tpl_file(tpl_file)

parse a PEST-style template file to get the parameter names

Args: tpl_file (str): path and name of a template file

Returns:: list of parameter names found in tpl_file
Return type:: [str]

Example:

par_names = pyemu.pst_utils.parse_tpl_file("my.tpl")

pyemu.pst.pst_utils.process_output_files(pst, pst_path='.')

helper function to process output files using the: InstructionFile class

Parameters:

pst (pyemu.Pst) – control file instance
pst_path – path to instruction and output files to append to the front of the names in the Pst instance

pyemu.pst.pst_utils.read_parfile(parfile)

load a PEST-style parameter value file into a pandas.DataFrame

Parameters:: parfile (str) – path and name of existing parameter file
Returns:: a dataframe with columns of “parnme”, “parval1”, “scale” and “offset”
Return type:: pandas.DataFrame

Example:

df = pyemu.pst_utils.read_parfile("my.par1")

pyemu.pst.pst_utils.read_resfile(resfile)

load a PEST-style residual file into a pandas.DataFrame

Parameters:: resfile – path and name of an existing residual file

pyemu.pst.pst_utils.res_from_en(pst, enfile)

load ensemble results from PESTPP-IES into a PEST-style residuals pandas.DataFrame

Parameters:: enfile (str) – CSV-format ensemble file name
Returns:: a dataframe with the same columns as a residual dataframe (a la pst_utils.read_resfile())
Return type:: pandas.DataFrame

Note

If a “base” realization is found in the ensemble, it is used as the “modelled” column in the residuals dataframe. Otherwise, the mean of the ensemble is used as “modelled”

Example:

df = pyemu.pst_utils.res_from_en("my.0.obs.csv")
df.residual.plot(kind="hist")

pyemu.pst.pst_utils.res_from_obseravtion_data(observation_data)

create a PEST-style residual dataframe filled with np.NaN for missing information

Parameters:: observation_data (pandas.DataFrame) – the “* observation data” pandas.DataFrame from pyemu.Pst.observation_data
Returns:: a dataframe with the same columns as the residual dataframe (“name”,”group”,”measured”,”modelled”, “residual”,”weight”).
Return type:: pandas.DataFrame

pyemu.pst.pst_utils.str_con(item)

pyemu.pst.pst_utils.try_process_output_file(ins_file, output_file=None)

attempt to process a model output file using a PEST-style instruction file

Parameters:

ins_file (str) – path and name of an instruction file
output_file (str,optional) – path and name of existing model output file to process. If None, ins_file.replace(“.ins”,””) is used. Default is None.

Returns:

a dataframe of observation name and simulated outputs extracted from output_file.

Return type:

pandas.DataFrame

Note

If an exception is raised when processing the output file, the exception is echoed to the screen and None is returned.

Example:

df = pyemu.pst_utils.try_process_output_file("my.ins","my.output")

pyemu.pst.pst_utils.try_process_output_pst(pst)

attempt to process each instruction file, model output file pair in a pyemu.Pst.

Parameters:: pst (pyemu.Pst) – a control file instance
Returns:: a dataframe of observation names and simulated outputs extracted from model output files.
Return type:: pandas.DataFrame

Note

This function first tries to process the output files using the InstructionFile class, If that failes, then it tries to run INSCHEK. If an instructionfile is processed successfully, the extract simulated values are used to populate the pst.observation_data.obsval attribute.

pyemu.pst.pst_utils.try_read_input_file_with_tpl(tpl_file, input_file=None)

attempt to read parameter values from an input file using a template file :param tpl_file: path and name of a template file :type tpl_file: str :param input_file: path and name of existing model

input file to process. If None, tpl_file.replace(“.tpl”,””) is used. Default is None.

Returns:: a dataframe of parameter name and values extracted from input_file.
Return type:: pandas.DataFrame

Note

If an exception is raised when reading the input file, the exception is echoed to the screen and None is returned.

Example:

df = pyemu.pst_utils.try_process_output_file("my.tpl","my.input")

pyemu.pst.pst_utils.write_input_files(pst, pst_path='.')

write parameter values to model input files

Parameters:

pst (pyemu.Pst) – a Pst instance
pst_path (str) – the path to where the control file and template files reside. Default is ‘.’.

Note

This function uses template files with the current parameter values (stored in pst.parameter_data.parval1).

This function uses multiprocessing - one process per template file

This is a simple implementation of what PEST does. It does not handle all the special cases, just a basic function…user beware

pyemu.pst.pst_utils.write_parfile(df, parfile)

write a PEST-style parameter file from a dataframe

Parameters:

df (pandas.DataFrame) – a dataframe with column names that correspond to the entries in the parameter data section of the pest control file
parfile (str) – name of the parameter file to write

Example:

pyemu.pst_utils.write_parfile(pst.parameter_data,"my.par")

pyemu.pst.pst_utils.write_to_template(parvals, tpl_file, in_file)

write parameter values to a model input file using the corresponding template file

Parameters:

parvals (dict) – a container of parameter names and values. Can also be a pandas.Series
tpl_file (str) – path and name of a template file
in_file (str) – path and name of model input file to write

Examples:

pyemu.pst_utils.write_to_template(par.parameter_data.parval1,
                                  "my.tpl","my.input")

Module contents

This is the pyemu module for handling PEST control files. It provides functionality for reading and writing control files, creating new control files, and manipulating all sections of the control file. The primary object is the Pst…start there.