`pyemu.pst.pst_utils`

Various PEST(++) control file peripheral operations

Module Contents

Classes

InstructionFile

class for handling instruction files.

Functions

`SFMT`(item)
`str_con`(item)
`read_resfile`(resfile)	load a PEST-style residual file into a pandas.DataFrame
`res_from_en`(pst, enfile)	load ensemble results from PESTPP-IES into a PEST-style
`read_parfile`(parfile)	load a PEST-style parameter value file into a pandas.DataFrame
`write_parfile`(df, parfile)	write a PEST-style parameter file from a dataframe
`parse_tpl_file`(tpl_file)	parse a PEST-style template file to get the parameter names
`write_input_files`(pst[, pst_path])	write parameter values to model input files
`_write_chunk_to_template`(chunk, parvals, pst_path)
`write_to_template`(parvals, tpl_file, in_file)	write parameter values to a model input file using
`_get_marker_indices`(marker, line)	method to find the start and end parameter markers
`parse_ins_file`(ins_file)	parse a PEST-style instruction file to get observation names
`_parse_ins_string`(string)	split up an instruction file line to get the observation names
`_populate_dataframe`(index, columns, default_dict, dtype)	helper function to populate a generic Pst dataframe attribute.
`generic_pst`([par_names, obs_names, addreg])	generate a generic pst instance.
`try_read_input_file_with_tpl`(tpl_file[, input_file])	attempt to read parameter values from an input file using a template file
`_read_infile_with_tplfile`(tpl_file, input_file)	attempt to read parameter values from an input file using a template file,
`try_process_output_file`(ins_file[, output_file])	attempt to process a model output file using a PEST-style instruction file
`try_process_output_pst`(pst)	attempt to process each instruction file, model output
`_try_run_inschek`(ins_file, out_file[, cwd])	try to run inschek and load the resulting obf file
`get_phi_comps_from_recfile`(recfile)	read the phi components from a record file by iteration
`res_from_obseravtion_data`(observation_data)	create a PEST-style residual dataframe filled with np.NaN for
`clean_missing_exponent`(pst_filename[, clean_filename])	fixes the issue where some terrible fortran program may have
`csv_to_ins_file`(csv_filename[, ins_filename, ...])	write a PEST-style instruction file from an existing CSV file
`process_output_files`(pst[, pst_path])	helper function to process output files using the
`check_interface`(pst[, pst_path, warn])	check that the tpl and ins file entries are in

Attributes

`SFMT_LONG`
`IFMT`
`FFMT`
`pst_config`

pyemu.pst.pst_utils.SFMT(item)

pyemu.pst.pst_utils.SFMT_LONG

pyemu.pst.pst_utils.IFMT

pyemu.pst.pst_utils.FFMT

pyemu.pst.pst_utils.str_con(item)

pyemu.pst.pst_utils.pst_config

pyemu.pst.pst_utils.read_resfile(resfile)

load a PEST-style residual file into a pandas.DataFrame

Parameters:: resfile – path and name of an existing residual file

pyemu.pst.pst_utils.res_from_en(pst, enfile)

load ensemble results from PESTPP-IES into a PEST-style residuals pandas.DataFrame

Parameters:: enfile (str) – CSV-format ensemble file name
Returns:: a dataframe with the same columns as a residual dataframe (a la pst_utils.read_resfile())
Return type:: pandas.DataFrame

Note

If a “base” realization is found in the ensemble, it is used as the “modelled” column in the residuals dataframe. Otherwise, the mean of the ensemble is used as “modelled”

Example:

df = pyemu.pst_utils.res_from_en("my.0.obs.csv")
df.residual.plot(kind="hist")

pyemu.pst.pst_utils.read_parfile(parfile)

load a PEST-style parameter value file into a pandas.DataFrame

Parameters:: parfile (str) – path and name of existing parameter file
Returns:: a dataframe with columns of “parnme”, “parval1”, “scale” and “offset”
Return type:: pandas.DataFrame

Example:

df = pyemu.pst_utils.read_parfile("my.par1")

pyemu.pst.pst_utils.write_parfile(df, parfile)

write a PEST-style parameter file from a dataframe

Parameters:

df (pandas.DataFrame) – a dataframe with column names that correspond to the entries in the parameter data section of the pest control file
parfile (str) – name of the parameter file to write

Example:

pyemu.pst_utils.write_parfile(pst.parameter_data,"my.par")

pyemu.pst.pst_utils.parse_tpl_file(tpl_file)

parse a PEST-style template file to get the parameter names

Args: tpl_file (str): path and name of a template file

Returns:: list of parameter names found in tpl_file
Return type:: [str]

Example:

par_names = pyemu.pst_utils.parse_tpl_file("my.tpl")

pyemu.pst.pst_utils.write_input_files(pst, pst_path='.')

write parameter values to model input files

Parameters:

pst (pyemu.Pst) – a Pst instance
pst_path (str) – the path to where the control file and template files reside. Default is ‘.’.

Note

This function uses template files with the current parameter values (stored in pst.parameter_data.parval1).

This function uses multiprocessing - one process per template file

This is a simple implementation of what PEST does. It does not handle all the special cases, just a basic function…user beware

pyemu.pst.pst_utils._write_chunk_to_template(chunk, parvals, pst_path)

pyemu.pst.pst_utils.write_to_template(parvals, tpl_file, in_file)

write parameter values to a model input file using the corresponding template file

Parameters:

parvals (dict) – a container of parameter names and values. Can also be a pandas.Series
tpl_file (str) – path and name of a template file
in_file (str) – path and name of model input file to write

Examples:

pyemu.pst_utils.write_to_template(par.parameter_data.parval1,
                                  "my.tpl","my.input")

pyemu.pst.pst_utils._get_marker_indices(marker, line): method to find the start and end parameter markers on a template file line. Used by write_to_template()

pyemu.pst.pst_utils.parse_ins_file(ins_file)

parse a PEST-style instruction file to get observation names

Parameters:: ins_file (str) – path and name of an existing instruction file
Returns:: a list of observation names found in ins_file
Return type:: [str]

Note

This is a basic function for parsing instruction files to look for observation names.

Example:

obs_names = pyemu.pst_utils.parse_ins_file("my.ins")

pyemu.pst.pst_utils._parse_ins_string(string): split up an instruction file line to get the observation names

pyemu.pst.pst_utils._populate_dataframe(index, columns, default_dict, dtype): helper function to populate a generic Pst dataframe attribute.

Note

This function is called as part of constructing a generic Pst instance

pyemu.pst.pst_utils.generic_pst(par_names=['par1'], obs_names=['obs1'], addreg=False)

generate a generic pst instance.

Parameters:

par_names ([str], optional) – parameter names to include in the new pyemu.Pst. Default is [“par2”].
obs_names ([str], optional) – observation names to include in the new pyemu.Pst. Default is [“obs1”].
addreg (bool) – flag to add zero-order Tikhonov prior information equations to the new control file

Returns:

a new control file instance. This instance does not have all the info needed to run, but is a placeholder that can then be filled in later.

Return type:

pyemu.Pst

Example:

par_names = ["par1","par2"]
obs_names = ["obs1","obs2"]
pst = pyemu.pst_utils.generic_pst(par_names,obs_names]

pyemu.pst.pst_utils.try_read_input_file_with_tpl(tpl_file, input_file=None)

attempt to read parameter values from an input file using a template file :param tpl_file: path and name of a template file :type tpl_file: str :param input_file: path and name of existing model

input file to process. If None, tpl_file.replace(“.tpl”,””) is used. Default is None.

Returns:: a dataframe of parameter name and values extracted from input_file.
Return type:: pandas.DataFrame

Note

If an exception is raised when reading the input file, the exception is echoed to the screen and None is returned.

Example:

df = pyemu.pst_utils.try_process_output_file("my.tpl","my.input")

pyemu.pst.pst_utils._read_infile_with_tplfile(tpl_file, input_file)

attempt to read parameter values from an input file using a template file, raising heaps of exceptions.

Args:
tpl_file (str): path and name of a template file input_file (str): path and name of existing model

Returns:
pandas.DataFrame: a dataframe of parameter name and values extracted from input_file.

Note:
use try_read_inputfile_with_tpl instead of this one.

pyemu.pst.pst_utils.try_process_output_file(ins_file, output_file=None)

attempt to process a model output file using a PEST-style instruction file

Parameters:

ins_file (str) – path and name of an instruction file
output_file (str,optional) – path and name of existing model output file to process. If None, ins_file.replace(“.ins”,””) is used. Default is None.

Returns:

a dataframe of observation name and simulated outputs extracted from output_file.

Return type:

pandas.DataFrame

Note

If an exception is raised when processing the output file, the exception is echoed to the screen and None is returned.

Example:

df = pyemu.pst_utils.try_process_output_file("my.ins","my.output")

pyemu.pst.pst_utils.try_process_output_pst(pst)

attempt to process each instruction file, model output file pair in a pyemu.Pst.

Parameters:: pst (pyemu.Pst) – a control file instance
Returns:: a dataframe of observation names and simulated outputs extracted from model output files.
Return type:: pandas.DataFrame

Note

This function first tries to process the output files using the InstructionFile class, If that failes, then it tries to run INSCHEK. If an instructionfile is processed successfully, the extract simulated values are used to populate the pst.observation_data.obsval attribute.

pyemu.pst.pst_utils._try_run_inschek(ins_file, out_file, cwd='.'): try to run inschek and load the resulting obf file

pyemu.pst.pst_utils.get_phi_comps_from_recfile(recfile)

read the phi components from a record file by iteration

Parameters:: recfile (str) – pest record file name
Returns:: nested dictionary of iteration number, {group,contribution}
Return type:: dict

Note

It is really poor form to use the record file in this way. Please only use this as a last resort!

pyemu.pst.pst_utils.res_from_obseravtion_data(observation_data)

create a PEST-style residual dataframe filled with np.NaN for missing information

Parameters:: observation_data (pandas.DataFrame) – the “* observation data” pandas.DataFrame from pyemu.Pst.observation_data
Returns:: a dataframe with the same columns as the residual dataframe (“name”,”group”,”measured”,”modelled”, “residual”,”weight”).
Return type:: pandas.DataFrame

pyemu.pst.pst_utils.clean_missing_exponent(pst_filename, clean_filename='clean.pst')

fixes the issue where some terrible fortran program may have written a floating point format without the ‘e’ - like 1.0-3, really?!

Parameters:

pst_filename (str) – the pest control file
clean_filename (str, optional) – the new pest control file to write. Default is “clean.pst”

pyemu.pst.pst_utils.csv_to_ins_file(csv_filename, ins_filename=None, only_cols=None, only_rows=None, marker='~', includes_header=True, includes_index=True, prefix='', head_lines_len=0, sep=',', gpname=False)

write a PEST-style instruction file from an existing CSV file

Parameters:

csv_filename (str) – path and name of existing CSV file
ins_filename (str, optional) – path and name of the instruction file to create. If None, then csv_filename`+”.ins” is used. Default is `None.
only_cols ([str]) – list of columns to add observations for in the resulting instruction file. If None, all columns are used.
only_rows ([str]) – list of rows to add observations for in the resulting instruction file. If None, all rows are used.
marker (str) – the PEST instruction marker to use. Default is “~”
includes_header (bool) – flag to indicate csv_filename includes a header row as the first row. Default is True.
includes_index (bool) – lag to indicate csv_filename includes a index column as the first column. Default is True.
prefix (str, optional) – a prefix to prepend to observation names. Default is “”
gpname (str or [str]) – Optional PEST group name for columns

Returns:

a dataframe of observation names and values found in csv_filename

Return type:

pandas.DataFrame

Note

resulting observation names in ins_filename are a combiation of index and header values.

class pyemu.pst.pst_utils.InstructionFile(ins_filename, pst=None)

Bases: object

class for handling instruction files.

Parameters:

ins_filename (str) – path and name of an existing instruction file
pst (pyemu.Pst, optional) – Pst instance - used for checking that instruction file is compatible with the control file (e.g. no duplicates)

Example:

i = InstructionFile("my.ins")
df = i.read_output_file("my.output")

property obs_name_set

read_ins_file(): read the instruction and do some minimal error checking.

Note

This is called by the constructor

throw_ins_warning(message, lcount=None)

throw a verbose PyemuWarning

Parameters:

message (str) – the warning message
lcount (int, optional) – warning line number. If None, self._ins_linecount is used

throw_ins_error(message, lcount=None)

throw a verbose instruction file error

Parameters:

message (str) – the error message
lcount (int, optional) – error line number. If None, self._ins_linecount is used

throw_out_error(message, lcount=None)

throw a verbose output file error

Parameters:

message (str) – the error message
lcount (int, optional) – error line number. If None, self._ins_linecount is used

read_output_file(output_file)

process a model output file using InstructionFile.instruction_set

Parameters:: output_file (str) – path and name of existing output file
Returns:: a dataframe with observation names and simulated values extracted from output_file
Return type:: pd.DataFrame

_execute_ins_line(ins_line, ins_lcount): private method to process output file lines with an instruction line

_readline_ins(): consolidate private method to read the next instruction file line. Casts to lower and splits on whitespace

_readline_output(): consolidate private method to read the next output file line. Casts to lower

pyemu.pst.pst_utils.process_output_files(pst, pst_path='.')

helper function to process output files using the: InstructionFile class

Parameters:

pst (pyemu.Pst) – control file instance
pst_path – path to instruction and output files to append to the front of the names in the Pst instance

pyemu.pst.pst_utils.check_interface(pst, pst_path='.', warn=False)

check that the tpl and ins file entries are in sync with the control file entries

Parameters:

pst (pyemu.Pst) – control file instance
pst_path (str) – the path from where python is running to the control file
warn (bool) – flag to treat errors as warnings

pyemu.pst.pst_utils

Module Contents

Classes

Functions

Attributes

`pyemu.pst.pst_utils`