pyemu.pst
This is the pyemu module for handling PEST control files. It provides functionality for reading and writing control files, creating new control files, and manipulating all sections of the control file. The primary object is the Pst…start there.
Submodules
Package Contents
Classes
an object that encapsulates the control data section |
|
All things PEST(++) control file |
- class pyemu.pst.ControlData
Bases:
object
- an object that encapsulates the control data section
of the PEST control file
Notes
This class works hard to protect the variables in the control data section. It type checks attempts to change values to make sure the type being passed matches the expected type of the attribute.
- property formatted_values
list the entries and current values in the control data section
- Returns:
formatted_values for the control data entries
- Return type:
pandas.Series
- __setattr__(key, value)
Implement setattr(self, name, value).
- __getattr__(item)
- static get_dataframe()
get a generic (default) control section as a dataframe
- Returns:
a dataframe of control data information
- Return type:
pandas.DataFrame
- static _parse_value(value)
- parse_values_from_lines(lines, iskeyword=False)
cast the string lines for a pest control file into actual inputs
- Parameters:
lines ([str]) – raw ASCII lines from pest control file
- copy()
- write_keyword(f)
write the control data entries to an open file handle using keyword-style format.
- Parameters:
f (file handle) – open file handle to write to
Notes
only writes values that have been accessed since instantiation
- write(f)
write control data section to a file
- Parameters:
f (file handle) – open file handle to write to
- class pyemu.pst.Pst(filename, load=True, resfile=None)
Bases:
object
All things PEST(++) control file
- Parameters:
filename (str) – the name of the control file
load (bool, optional) – flag to load the control file. Default is True
resfile (str, optional) – corresponding residual file. If None, a residual file with the control file base name is sought. Default is None
Note
This class is the primary mechanism for dealing with PEST control files. Support is provided for constructing new control files as well as manipulating existing control files.
Example:
pst = pyemu.Pst("my.pst") pst.control_data.noptmax = -1 pst.write("my_new.pst")
- property phi
get the weighted total objective function.
- Returns:
sum of squared residuals
- Return type:
float
Note
Requires Pst.res (the residuals file) to be available
- property phi_components
get the individual components of the total objective function
- Returns:
dictionary of observation group, contribution to total phi
- Return type:
dict
Note
Requires Pst.res (the residuals file) to be available
- property phi_components_normalized
- get the individual components of the total objective function
normalized to the total PHI being 1.0
- Returns:
dictionary of observation group, normalized contribution to total phi
- Return type:
dict
Note
Requires Pst.res (the residuals file) to be available
- property res
get the residuals dataframe attribute
- Returns:
a dataframe containing the residuals information.
- Return type:
pandas.DataFrame
Note
if the Pst.__res attribute has not been loaded, this call loads the res dataframe from a file
Example:
# print the observed and simulated values for non-zero weighted obs print(pst.res.loc[pst.nnz_obs_names,["modelled","measured"]])
- property nprior
number of prior information equations
- Returns:
the number of prior info equations
- Return type:
int
- property nnz_obs
get the number of non-zero weighted observations
- Returns:
the number of non-zeros weighted observations
- Return type:
int
- property nobs
get the number of observations
- Returns:
the number of observations
- Return type:
int
- property npar_adj
get the number of adjustable parameters (not fixed or tied)
- Returns:
the number of adjustable parameters
- Return type:
int
- property npar
get number of parameters
- Returns:
the number of parameters
- Return type:
int
- property forecast_names
get the forecast names from the pestpp options (if any). Returns None if no forecasts are named
- Returns:
a list of forecast names.
- Return type:
[str]
- property obs_groups
get the observation groups
- Returns:
a list of unique observation groups
- Return type:
[str]
- property nnz_obs_groups
- get the observation groups that contain at least one non-zero weighted
observation
- Returns:
a list of observation groups that contain at least one non-zero weighted observation
- Return type:
[str]
- property adj_par_groups
get the parameter groups with atleast one adjustable parameter
- Returns:
a list of parameter groups with at least one adjustable parameter
- Return type:
[str]
- property par_groups
get the parameter groups
- Returns:
a list of parameter groups
- Return type:
[str]
- property prior_groups
get the prior info groups
- Returns:
a list of prior information groups
- Return type:
[str]
- property prior_names
get the prior information names
- Returns:
a list of prior information names
- Return type:
[str]
- property par_names
get the parameter names
- Returns:
a list of parameter names
- Return type:
[str]
- property adj_par_names
get the adjustable (not fixed or tied) parameter names
- Returns:
list of adjustable (not fixed or tied) parameter names
- Return type:
[str]
- property obs_names
get the observation names
- Returns:
a list of observation names
- Return type:
[str]
- property nnz_obs_names
get the non-zero weight observation names
- Returns:
a list of non-zero weighted observation names
- Return type:
[str]
- property zero_weight_obs_names
get the zero-weighted observation names
- Returns:
a list of zero-weighted observation names
- Return type:
[str]
- property estimation
check if the control_data.pestmode is set to estimation
- Returns:
True if control_data.pestmode is estmation, False otherwise
- Return type:
bool
- property tied
list of tied parameter names
- Returns:
a dataframe of tied parameter information. Columns of tied are parnme and partied. Returns None if no tied parameters are found.
- Return type:
pandas.DataFrame
- property template_files
list of template file names
- Returns:
- a list of template file names, extracted from
Pst.model_input_data.pest_file. Returns None if this attribute is None
- Return type:
[str]
Note
Use Pst.model_input_data to access the template-input file information for writing/modification
- property input_files
list of model input file names
- Returns:
- a list of model input file names, extracted from
Pst.model_input_data.model_file. Returns None if this attribute is None
- Return type:
[str]
Note
Use Pst.model_input_data to access the template-input file information for writing/modification
- property instruction_files
list of instruction file names :returns:
- a list of instruction file names, extracted from
- Pst.model_output_data.pest_file. Returns None if this
attribute is None
- Return type:
[str]
Note
Use Pst.model_output_data to access the instruction-output file information for writing/modification
- property output_files
list of model output file names :returns:
- a list of model output file names, extracted from
Pst.model_output_data.model_file. Returns None if this attribute is None
- Return type:
[str]
Note
Use Pst.model_output_data to access the instruction-output file information for writing/modification
- property less_than_obs_constraints
get the names of the observations that are listed as active (non-zero weight) less than inequality constraints.
- Returns:
names of observations that are non-zero weighted less than constraints (obgnme starts with ‘l_’ or “less”)
- Return type:
pandas.Series
Note
Zero-weighted obs are skipped
- property less_than_pi_constraints
get the names of the prior information eqs that are listed as active (non-zero weight) less than inequality constraints.
- Returns:
names of prior information that are non-zero weighted less than constraints (obgnme starts with “l_” or “less”)
- Return type:
pandas.Series
Note
Zero-weighted pi are skipped
- property greater_than_obs_constraints
get the names of the observations that are listed as active (non-zero weight) greater than inequality constraints.
- Returns:
names obseravtions that are non-zero weighted greater than constraints (obgnme startsiwth “g_” or “greater”)
- Return type:
pandas.Series
Note
Zero-weighted obs are skipped
- property greater_than_pi_constraints
get the names of the prior information eqs that are listed as active (non-zero weight) greater than inequality constraints.
- Returns:
pandas.Series names of prior information that are non-zero weighted greater than constraints (obgnme startsiwth “g_” or “greater”)
Note
Zero-weighted pi are skipped
- parameter_data
‘* parameter data’ information. Columns are standard PEST variable names
Example:
pst.parameter_data.loc[:,"partrans"] = "log" pst.parameter_data.loc[:,"parubnd"] = 10.0
- Type:
pandas.DataFrame
- observation_data
‘* observation data’ information. Columns are standard PEST variable names
Example:
pst.observation_data.loc[:,"weight"] = 1.0 pst.observation_data.loc[:,"obgnme"] = "obs_group"
- Type:
pandas.DataFrame
- prior_information
‘* prior information’ data. Columns are standard PEST variable names
- Type:
pandas.DataFrame
- control_data
‘* control data’ information. Access with standard PEST variable names
Example:
pst.control_data.noptmax = 2 pst.control_data.pestmode = "estimation"
- svd_data
‘* singular value decomposition’ section information. Access with standard PEST variable names
Example:
pst.svd_data.maxsing = 100
- reg_data
‘* regularization’ section information. Access with standard PEST variable names.
Example:
pst.reg_data.phimlim = 1.00 #yeah right!
- __setattr__(key, value)
Implement setattr(self, name, value).
- classmethod from_par_obs_names(par_names=['par1'], obs_names=['obs1'])
construct a shell Pst instance from parameter and observation names
- Parameters:
par_names ([str]) – list of parameter names. Default is [par1]
obs_names ([str]) – list of observation names. Default is [obs1]
Note
While this method works, it does not make template or instruction files. Users are encouraged to use Pst.from_io_files() for more usefulness
Example:
par_names = ["par1","par2"] obs_names = ["obs1","obs2"] pst = pyemu.Pst.from_par_obs_names(par_names,obs_names)
- static get_phi_components(ogroups, res, obs, pi_ogroups=None, pi_df=None)
- set_res(res)
reset the private Pst.res attribute.
- Parameters:
res – (pandas.DataFrame or str): something to use as Pst.res attribute. If res is str, a dataframe is read from file res
- static _read_df(f, nrows, names, converters, defaults=None)
a private method to read part of an open file into a pandas.DataFrame.
- Parameters:
f (file) – open file handle
nrows (int) – number of rows to read
names ([str]) – names to set the columns of the dataframe with
converters (dict) – dictionary of lambda functions to convert strings to numerical format
defaults (dict) – dictionary of default values to assign columns. Default is None
- Returns:
dataframe of control file section info
- Return type:
pandas.DataFrame
- _read_line_comments(f, forgive)
private method to read comment lines from a control file
- _read_section_comments(f, forgive)
private method to read comments from a section of the control file
- static _parse_external_line(line, pst_path='.')
private method to parse a file for external file info
- static _parse_path_agnostic(filename)
private method to parse a file path for any os sep
- static _cast_df_from_lines(section, lines, fieldnames, converters, defaults, alias_map={}, pst_path='.')
private method to cast a pandas dataframe from raw control file lines
- _cast_prior_df_from_lines(section, lines, pst_path='.')
private method to cast prior information lines to a dataframe
- _load_version2(filename)
private method to load a version 2 control file
- load(filename)
entry point load the pest control file.
- Parameters:
filename (str) – pst filename
Note
This method is called from the Pst construtor unless the load arg is False.
- _reset_file_paths_os()
- _try_load_longnames()
- _parse_pestpp_line(line)
- _update_control_section()
private method to synchronize the control section counters with the various parts of the control file. This is usually called during the Pst.write() method.
- rectify_pgroups()
synchronize parameter groups section with the parameter data section
Note
This method is called during Pst.write() to make sure all parameter groups named in * parameter data are included. This is so users don’t have to manually keep this section up. This method can also be called during control file modifications to see what parameter groups are present and prepare for modifying the default values in the * parameter group section
Example:
pst = pyemu.Pst("my.pst") pst.parameter_data.loc["par1","pargp"] = "new_group" pst.rectify_groups() pst.parameter_groups.loc["new_group","derinc"] = 1.0
- _parse_pi_par_names()
private method to get the parameter names from prior information equations. Sets a ‘names’ column in Pst.prior_information that is a list of parameter names
- add_pi_equation(par_names, pilbl=None, rhs=0.0, weight=1.0, obs_group='pi_obgnme', coef_dict={})
a helper to construct a new prior information equation.
- Parameters:
par_names ([str]) – parameter names in the equation
pilbl (str) – name to assign the prior information equation. If None, a generic equation name is formed. Default is None
rhs (float) – the right-hand side of the pi equation
weight (float) – the weight of the equation
obs_group (str) – the observation group for the equation. Default is ‘pi_obgnme’
coef_dict (dict) – a dictionary of parameter name, coefficient pairs to assign leading coefficients for one or more parameters in the equation. If a parameter is not listed, 1.0 is used for its coefficients. Default is {}
Example:
pst = pyemu.Pst("pest.pst") # add a pi equation for the first adjustable parameter pst.add_pi_equation(pst.adj_par_names[0],pilbl="pi1",rhs=1.0) # add a pi equation for 1.5 times the 2nd and 3 times the 3rd adj pars to sum together to 2.0 names = pst.adj_par_names[[1,2]] pst.add_pi_equation(names,coef_dict={names[0]:1.5,names[1]:3})
- rectify_pi()
rectify the prior information equation with the current state of the parameter_data dataframe.
Note
Equations that list fixed, tied or missing parameters are removed completely even if adjustable parameters are also listed in the equation. This method is called during Pst.write()
- _write_df(name, f, df, formatters, columns)
private method to write a dataframe to a control file
- sanity_checks(forgive=False)
some basic check for strangeness
- Parameters:
forgive (bool) – flag to forgive (warn) for issues. Default is False
Note
checks for duplicate names, atleast 1 adjustable parameter and at least 1 non-zero-weighted observation
Not nearly as comprehensive as pestchek
Example:
pst = pyemu.Pst("pest.pst") pst.sanity_checks()
- _write_version2(new_filename, use_pst_path=True, pst_rel_path='.')
private method to write a version 2 control file
- write(new_filename, version=None, check_interface=False)
main entry point to write a pest control file.
- Parameters:
new_filename (str) – name of the new pest control file
version (int) – flag for which version of control file to write (must be 1 or 2). if None, uses the number of pars to decide: if number of pars iis greater than 10,000, version 2 is used.
check_interface (bool) – flag to check the control file par and obs names against the names found in the template and instruction files. Default is False
Example:
pst = pyemu.Pst("my.pst") pst.parrep("my.par") pst.write(my_new.pst") #write a version 2 control file pst.write("my_new_v2.pst",version=2)
- _rectify_parchglim()
private method to just fix the parchglim vs cross zero issue
- _write_version1(new_filename)
private method to write a version 1 pest control file
- bounds_report(iterations=None)
report how many parameters are at bounds. If ensemble, the base enbsemble member is evaluated
- Parameters:
iterations ([int]) – a list of iterations for which a bounds report is requested If None, all iterations for which par files are located are reported. Default is None
- Returns:
- a pandas DataFrame object with rows being parameter groups and columns
<iter>_num_at_ub, <iter>_num_at_lb, and <iter>_total_at_bounds row 0 is total at bounds, subsequent rows correspond with groups
- Return type:
df
Example
pst = pyemu.Pst(“my.pst”) df = pst.bound_report(iterations=[0,2,3])
- get(par_names=None, obs_names=None)
get a new pst object with subset of parameters and/or observations
- Parameters:
par_names ([str]) – a list of parameter names to have in the new Pst instance. If None, all parameters are in the new Pst instance. Default is None
obs_names ([str]) – a list of observation names to have in the new Pst instance. If None, all observations are in teh new Pst instance. Default is None
- Returns:
a new Pst instance
- Return type:
Pst
Note
passing par_names as None and obs_names as None effectively generates a copy of the current Pst
Does not modify model i/o files - this is just a method for performing pyemu operations
Example:
pst = pyemu.Pst("pest.pst") new_pst = pst.get(pst.adj_par_names[0],pst.obs_names[:10])
- parrep(parfile=None, enforce_bounds=True, real_name=None, noptmax=0, binary_ens_file=False)
- replicates the pest parrep util. replaces the parval1 field in the
parameter data section dataframe with values in a PEST parameter file or a single realization from an ensemble parameter csv file
- Parameters:
parfile (str, optional) – parameter file to use. If None, try to find and use a parameter file that corresponds to the case name. If parfile has extension ‘.par’ a single realization parameter file is used If parfile has extention ‘.csv’ an ensemble parameter file is used which invokes real_name If parfile has extention ‘.jcb’ a binary ensemble parameter file is used which invokes real_name Default is None
enforce_bounds (bool, optional) – flag to enforce parameter bounds after parameter values are updated. This is useful because PEST and PEST++ round the parameter values in the par file, which may cause slight bound violations. Default is True
real_name (str or int, optional) – name of the ensemble realization to use for updating the parval1 value in the parameter data section dataframe. If None, try using “base”. If “base” not present, use the real_name with smallest index number. Ignored if parfile is of the PEST parameter file format (e.g. not en ensemble)
noptmax (int, optional) – Value with which to update the pst.control_data.noptmax value Default is 0.
binary_ens_file (bool) – If True, use binary format to load ensemble file, else assume it’s a CSV file
Example:
pst = pyemu.Pst("pest.pst") pst.parrep("pest.1.base.par") pst.control_data.noptmax = 0 pst.write("pest_1.pst")
- adjust_weights_discrepancy(resfile=None, original_ceiling=True, bygroups=False)
adjusts the weights of each non-zero weight observation based on the residual in the pest residual file so each observations contribution to phi is 1.0 (e.g. Mozorov’s discrepancy principal)
- Parameters:
resfile (str) – residual file name. If None, try to use a residual file with the Pst case name. Default is None
original_ceiling (bool) – flag to keep weights from increasing - this is generally a good idea. Default is True
bygroups (bool) – flag to adjust weights by groups. If False, the weight of each non-zero weighted observation is adjusted individually. If True, intergroup weighting is preserved (the contribution to each group is used) but this may result in some strangeness if some observations in a group have a really low phi already.
Example:
pst = pyemu.Pst("my.pst") print(pst.phi) #assumes "my.res" is colocated with "my.pst" pst.adjust_weights_discrepancy() print(pst.phi) # phi should equal number of non-zero observations
- _adjust_weights_by_phi_components(components, original_ceiling)
private method that resets the weights of observations by group to account for residual phi components.
- Parameters:
components (dict) – a dictionary of obs group:phi contribution pairs
original_ceiling (bool) – flag to keep weights from increasing.
- __reset_weights(target_phis, res_idxs, obs_idxs)
private method to reset weights based on target phi values for each group. This method should not be called directly
- Parameters:
target_phis (dict) – target phi contribution for groups to reweight
res_idxs (dict) – the index positions of each group of interest in the res dataframe
obs_idxs (dict) – the index positions of each group of interest in the observation data dataframe
- _adjust_weights_by_list(obslist, weight)
a private method to reset the weight for a list of observation names. Supports the data worth analyses in pyemu.Schur class. This method only adjusts observation weights in the current weight is nonzero. User beware!
- Parameters:
obslist ([str]) – list of observation names
weight (float) – new weight to assign
- adjust_weights(obs_dict=None, obsgrp_dict=None)
reset the weights of observations or observation groups to contribute a specified amount to the composite objective function
- Parameters:
obs_dict (dict, optional) – dictionary of observation name,new contribution pairs
obsgrp_dict (dict, optional) – dictionary of obs group name,contribution pairs
Notes
If a group is assigned a contribution of 0, all observations in that group will be assigned zero weight.
If a group is assigned a nonzero contribution AND all observations in that group start with zero weight, the observations will be assigned weight of 1.0 to allow balancing.
If groups obsgrp_dict is not passed, all nonzero
Example:
pst = pyemu.Pst("my.pst") # adjust a single observation pst.adjust_weights(obs_dict={"obs1":10}) # adjust a single observation group pst.adjust_weights(obsgrp_dict={"group1":100.0}) # make all non-zero weighted groups have a contribution of 100.0 balanced_groups = {grp:100 for grp in pst.nnz_obs_groups} pst.adjust_weights(obsgrp_dict=balanced_groups)
- proportional_weights(fraction_stdev=1.0, wmax=100.0, leave_zero=True)
setup weights inversely proportional to the observation value
- Parameters:
fraction_stdev (float, optional) – the fraction portion of the observation val to treat as the standard deviation. set to 1.0 for inversely proportional. Default is 1.0
wmax (float, optional) – maximum weight to allow. Default is 100.0
leave_zero (bool, optional) – flag to leave existing zero weights. Default is True
Example:
pst = pyemu.Pst("pest.pst") # set the weights of the observations to 20% of the observed value pst.proportional_weights(fraction_stdev=0.2,wmax=10) pst.write("pest_propo.pst")
- calculate_pertubations()
experimental method to calculate finite difference parameter pertubations.
Note
The pertubation values are added to the Pst.parameter_data attribute - user beware!
- build_increments()
experimental method to calculate parameter increments for use in the finite difference pertubation calculations
Note
user beware!
- add_transform_columns()
add transformed values to the Pst.parameter_data attribute
Note
adds parval1_trans, parlbnd_trans and parubnd_trans to Pst.parameter_data
Example:
pst = pyemu.Pst("pest.pst") pst.add_transform_columns() print(pst.parameter_data.parval1_trans
- enforce_bounds()
enforce bounds violation
Note
cheap enforcement of simply bringing violators back in bounds
Example:
pst = pyemu.Pst("pest.pst") pst.parrep("random.par") pst.enforce_bounds() pst.write("pest_rando.pst")
- classmethod from_io_files(tpl_files, in_files, ins_files, out_files, pst_filename=None, pst_path=None)
create a Pst instance from model interface files.
- Parameters:
tpl_files ([str]) – list of template file names
in_files ([str]) – list of model input file names (pairs with template files)
ins_files ([str]) – list of instruction file names
out_files ([str]) – list of model output file names (pairs with instruction files)
pst_filename (str) – name of control file to write. If None, no file is written. Default is None
pst_path ('str') – the path from the control file to the IO files. For example, if the control will be in the same directory as the IO files, then pst_path should be ‘.’. Default is None, which doesnt do any path manipulation on the I/O file names
- Returns:
new control file instance with parameter and observation names found in tpl_files and ins_files, repsectively.
- Return type:
Pst
Note
calls pyemu.helpers.pst_from_io_files()
Assigns generic values for parameter info. Tries to use INSCHEK to set somewhat meaningful observation values
all file paths are relatively to where python is running.
Example:
tpl_files = ["my.tpl"] in_files = ["my.in"] ins_files = ["my.ins"] out_files = ["my.out"] pst = pyemu.Pst.from_io_files(tpl_files,in_files,ins_files,out_files) pst.control_data.noptmax = 0 pst.write("my.pst)
- add_parameters(template_file, in_file=None, pst_path=None)
add new parameters to an existing control file
- Parameters:
template_file (str) – template file with (possibly) some new parameters
in_file (str) – model input file. If None, template_file.replace(‘.tpl’,’’) is used. Default is None.
pst_path (str) – the path to append to the template_file and in_file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None
- Returns:
the data for the new parameters that were added. If no new parameters are in the new template file, returns None
- Return type:
pandas.DataFrame
Note
populates the new parameter information with default values
Example:
pst = pyemu.Pst(os.path.join("template","my.pst")) pst.add_parameters(os.path.join("template","new_pars.dat.tpl",pst_path=".") pst.write(os.path.join("template","my_new.pst")
- drop_observations(ins_file, pst_path=None)
remove observations in an instruction file from the control file
- Parameters:
ins_file (str) – instruction file to remove
pst_path (str) – the path to append to the instruction file in the control file. If not None, then any existing path in front of the instruction is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None
- Returns:
the observation data for the observations that were removed.
- Return type:
pandas.DataFrame
Example:
pst = pyemu.Pst(os.path.join("template", "my.pst")) pst.remove_observations(os.path.join("template","some_obs.dat.ins"), pst_path=".") pst.write(os.path.join("template", "my_new_with_less_obs.pst")
- drop_parameters(tpl_file, pst_path=None)
remove parameters in a template file from the control file
- Parameters:
tpl_file (str) – template file to remove
pst_path (str) – the path to append to the template file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None
- Returns:
the parameter data for the parameters that were removed.
- Return type:
pandas.DataFrame
Note
This method does not check for multiple occurences of the same parameter name(s) in across template files so if you have the same parameter in multiple template files, this is not the method you are looking for
Example:
pst = pyemu.Pst(os.path.join("template", "my.pst")) pst.remove_parameters(os.path.join("template","boring_zone_pars.dat.tpl"), pst_path=".") pst.write(os.path.join("template", "my_new_with_less_pars.pst")
- add_observations(ins_file, out_file=None, pst_path=None, inschek=True)
add new observations to a control file
- Parameters:
ins_file (str) – instruction file with exclusively new observation names
out_file (str) – model output file. If None, then ins_file.replace(“.ins”,””) is used. Default is None
pst_path (str) – the path to append to the instruction file and out file in the control file. If not None, then any existing path in front of the template or in file is split off and pst_path is prepended. If python is being run in a directory other than where the control file will reside, it is useful to pass pst_path as .. Default is None
inschek (bool) – flag to try to process the existing output file using the pyemu.InstructionFile class. If successful, processed outputs are used as obsvals
- Returns:
the data for the new observations that were added
- Return type:
pandas.DataFrame
Note
populates the new observation information with default values
Example:
pst = pyemu.Pst(os.path.join("template", "my.pst")) pst.add_observations(os.path.join("template","new_obs.dat.ins"), pst_path=".") pst.write(os.path.join("template", "my_new.pst")
- write_input_files(pst_path='.')
writes model input files using template files and current parval1 values.
- Parameters:
pst_path (str) – the path to where control file and template files reside. Default is ‘.’
Note
adds “parval1_trans” column to Pst.parameter_data that includes the effect of scale and offset
Example:
pst = pyemu.Pst("my.pst") # load final parameter values pst.parrep("my.par") # write new model input files with final parameter values pst.write_input_files()
- process_output_files(pst_path='.')
processing the model output files using the instruction files and existing model output files.
- Parameters:
pst_path (str) – relative path from where python is running to where the control file, instruction files and model output files are located. Default is “.” (current python directory)
- Returns:
model output values
- Return type:
pandas.Series
Note
requires a complete set of model input files at relative path from where python is running to pst_path
Example:
pst = pyemu.Pst("pest.pst") obsvals = pst.process_output_files() print(obsvals)
- get_res_stats(nonzero=True)
get some common residual stats by observation group.
- Parameters:
nonzero (bool) – calculate stats using only nonzero-weighted observations. This may seem obsvious to most users, but you never know….
- Returns:
a dataframe with columns for groups names and indices of statistic name.
- Return type:
pd.DataFrame
Note
Stats are derived from the current obsvals, weights and grouping in Pst.observation_data and the modelled values in Pst.res. The key here is ‘current’ because if obsval, weights and/or groupings have changed in Pst.observation_data since the residuals file was generated then the current values for obsval, weight and group are used
the normalized RMSE is normalized against the obsval range (max - min)
Example:
pst = pyemu.Pst("pest.pst") stats_df = pst.get_res_stats() print(stats_df.loc["mae",:])
- static _stats_rss(df)
- static _stats_mean(df)
- static _stats_mae(df)
- static _stats_rmse(df)
- static _stats_nrmse(df)
- plot(kind=None, **kwargs)
method to plot various parts of the control. This is sweet as!
- Parameters:
kind (str) – options are ‘prior’ (prior parameter histograms, ‘1to1’ (line of equality and sim vs res), ‘obs_v_sim’ (time series using datetime suffix), ‘phi_pie’ (pie chart of phi components)
kwargs (dict) – optional args for plots that are passed to pyemu plot helpers and ultimately to matplotlib
Note
Depending on ‘kind’ argument, a multipage pdf is written
Example:
pst = pyemu.Pst("my.pst") pst.plot(kind="1to1") # requires Pst.res pst.plot(kind="prior") pst.plot(kind="phi_pie")
- write_par_summary_table(filename=None, group_names=None, sigma_range=4.0, report_in_linear_space=False)
write a stand alone parameter summary latex table or Excel sheet
- Parameters:
filename (str) – filename. If None, use <case>.par.tex to write as LaTeX. If filename extention is ‘.xls’ or ‘.xlsx’, tries to write as an Excel file. If filename is “none”, no table is written Default is None
group_names (dict) – par group names : table names. For example {“w0”:”well stress period 1”}. Default is None
sigma_range (float) – number of standard deviations represented by parameter bounds. Default is 4.0, implying 95% confidence bounds
report_in_linear_space (bool) – flag, if True, that reports all logtransformed values in linear space. This renders standard deviation meaningless, so that column is skipped
- Returns:
the summary parameter group dataframe
- Return type:
pandas.DataFrame
Example:
pst = pyemu.Pst("my.pst") pst.write_par_summary_table(filename="par.tex")
- write_obs_summary_table(filename=None, group_names=None)
- write a stand alone observation summary latex table or Excel shet
- filename (str): filename. If None, use <case>.par.tex to write as LaTeX. If filename extention is ‘.xls’ or ‘.xlsx’,
tries to write as an Excel file. If filename is “none”, no table is written Default is None
- Parameters:
filename (str) – filename. If filename is “none”, no table is written. If None, use <case>.obs.tex. If filename extention is ‘.xls’ or ‘.xlsx’, tries to write as an Excel file. Default is None
group_names (dict) – obs group names : table names. For example {“hds”:”simulated groundwater level”}. Default is None
- Returns:
the summary observation group dataframe
- Return type:
pandas.DataFrame
Example:
pst = pyemu.Pst("my.pst") pst.write_obs_summary_table(filename="obs.tex")
- get_par_change_limits()
calculate the various parameter change limits used in pest.
- Returns:
a copy of Pst.parameter_data with columns for relative and factor change limits
- Return type:
pandas.DataFrame
Note
does not yet support absolute parameter change limits!
Works in control file values space (not log transformed space). Also adds columns for effective upper and lower which account for par bounds and the value of parchglim
example:
pst = pyemu.Pst("pest.pst") df = pst.get_par_change_limits() print(df.chg_lower)
- get_adj_pars_at_bounds(frac_tol=0.01)
get list of adjustable parameter at/near bounds
- Parameters:
frac_tol ('float`) – fractional tolerance of distance to bound. For upper bound, the value parubnd * (1-frac_tol) is used, lower bound uses parlbnd * (1.0 + frac_tol)
- Returns:
[`str`]: list of parameters at/near lower bound
[`str`]: list of parameters at/near upper bound
- Return type:
tuple containing
Example:
pst = pyemu.Pst("pest.pst") at_lb,at_ub = pst.get_adj_pars_at_bounds() print("pars at lower bound",at_lb)
- try_parse_name_metadata()
try to add meta data columns to parameter and observation data based on item names. Used with the PstFrom process.
Note
metadata is identified in key-value pairs that are separated by a colon. each key-value pair is separated from others by underscore
This works with PstFrom style long names
This method is called programmtically during Pst.load()
- rename_parameters(name_dict, pst_path='.', tplmap=None)
rename parameters in the control and template files
- Parameters:
name_dict (dict) – mapping of current to new names.
pst_path (str) – the path to the control file from where python is running. Default is “.” (python is running in the same directory as the control file)
Note
no attempt is made to maintain the length of the marker strings in the template files, so if your model is sensitive to changes in spacing in the template file(s), this is not a method for you
This does a lot of string compare, so its gonna be slow as…
Example:
pst = pyemu.Pst(os.path.join("template","pest.pst")) name_dict = {"par1":"par1_better_name"} pst.rename_parameters(name_dict,pst_path="template")
- rename_observations(name_dict, pst_path='.', insmap=None)
rename observations in the control and instruction files
- Parameters:
name_dict (dict) – mapping of current to new names.
pst_path (str) – the path to the control file from where python is running. Default is “.” (python is running in the same directory as the control file)
Note
This does a lot of string compare, so its gonna be slow as…
Example:
pst = pyemu.Pst(os.path.join("template","pest.pst")) name_dict = {"obs1":"obs1_better_name"} pst.rename_observations(name_dict,pst_path="template")