pst_utils

Various PEST(++) control file peripheral operations

`InstructionFile`

Bases: object

class for handling instruction files.

Parameters:

Name	Type	Description	Default
`ins_filename`	`str`	path and name of an existing instruction file	required
`pst`	`pyemu.Pst`	Pst instance - used for checking that instruction file is compatible with the control file (e.g. no duplicates)	`None`

Example::

i = InstructionFile("my.ins")
df = i.read_output_file("my.output")

`read_ins_file()`

read the instruction and do some minimal error checking.

Note:

This is called by the constructor

`read_output_file(output_file)`

process a model output file using InstructionFile.instruction_set

Parameters:

Name	Type	Description	Default
`output_file`	`str`	path and name of existing output file	required

Returns:

`pd.DataFrame`: a dataframe with observation names and simulated values
extracted from `output_file`

`throw_ins_error(message, lcount=None)`

throw a verbose instruction file error

Parameters:

Name	Type	Description	Default
`message`	`str`	the error message	required
`lcount`	`int`	error line number. If None, self._ins_linecount is used	`None`

`throw_ins_warning(message, lcount=None)`

throw a verbose PyemuWarning

Parameters:

Name	Type	Description	Default
`message`	`str`	the warning message	required
`lcount`	`int`	warning line number. If None, self._ins_linecount is used	`None`

`throw_out_error(message, lcount=None)`

throw a verbose output file error

Parameters:

Name	Type	Description	Default
`message`	`str`	the error message	required
`lcount`	`int`	error line number. If None, self._ins_linecount is used	`None`

`check_interface(pst, pst_path='.', warn=False)`

check that the tpl and ins file entries are in sync with the control file entries

Parameters:

Name	Type	Description	Default
`pst`	`pyemu.Pst`	control file instance	required
`pst_path`	`str`	the path from where python is running to the control file	`'.'`
`warn`	`bool`	flag to treat errors as warnings	`False`

`clean_missing_exponent(pst_filename, clean_filename='clean.pst')`

fixes the issue where some terrible fortran program may have written a floating point format without the 'e' - like 1.0-3, really?!

Parameters:

Name	Type	Description	Default
`pst_filename`	`str`	the pest control file	required
`clean_filename`	`str`	the new pest control file to write. Default is "clean.pst"	`'clean.pst'`

`csv_to_ins_file(csv_filename, ins_filename=None, only_cols=None, only_rows=None, marker='~', includes_header=True, includes_index=True, prefix='', head_lines_len=0, sep=',', gpname=False)`

write a PEST-style instruction file from an existing CSV file

Parameters:

Name	Type	Description	Default
`csv_filename`	`str`	path and name of existing CSV file	required
`ins_filename`	`str`	path and name of the instruction file to create. If `None`, then `csv_filename`+".ins" is used. Default is `None`.	`None`
`only_cols`	[`str`]	list of columns to add observations for in the resulting instruction file. If `None`, all columns are used.	`None`
`only_rows`	[`str`]	list of rows to add observations for in the resulting instruction file. If `None`, all rows are used.	`None`
`marker`	`str`	the PEST instruction marker to use. Default is "~"	`'~'`
`includes_header`	`bool`	flag to indicate `csv_filename` includes a header row as the first row. Default is True.	`True`
`includes_index`	`bool`	lag to indicate `csv_filename` includes a index column as the first column. Default is True.	`True`
`prefix`	`str`	a prefix to prepend to observation names. Default is ""	`''`
`gpname`	`str` or [`str`]	Optional PEST group name for columns	`False`

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of observation names and values found in
	`csv_filename`

Note

resulting observation names in ins_filename are a combination of index and header values.

`generic_pst(par_names=['par1'], obs_names=['obs1'], addreg=False)`

generate a generic pst instance.

Parameters:

Name	Type	Description	Default
`par_names`	[`str`]	parameter names to include in the new `pyemu.Pst`. Default is ["par2"].	`['par1']`
`obs_names`	[`str`]	observation names to include in the new `pyemu.Pst`. Default is ["obs1"].	`['obs1']`
`addreg`	`bool`	flag to add zero-order Tikhonov prior information equations to the new control file	`False`

Returns:

Type	Description
	`pyemu.Pst`: a new control file instance. This instance does not have
	all the info needed to run, but is a placeholder that can then be
	filled in later.

Example::

par_names = ["par1","par2"]
obs_names = ["obs1","obs2"]
pst = pyemu.pst_utils.generic_pst(par_names,obs_names]

`get_phi_comps_from_recfile(recfile)`

read the phi components from a record file by iteration

Parameters:

Name	Type	Description	Default
`recfile`	`str`	pest record file name	required

Returns:

Type	Description
	`dict`: nested dictionary of iteration number, {group,contribution}

Note

It is really poor form to use the record file in this way. Please only use this as a last resort!

`parse_ins_file(ins_file)`

parse a PEST-style instruction file to get observation names

Parameters:

Name	Type	Description	Default
`ins_file`	`str`	path and name of an existing instruction file	required

Returns:

Type	Description
	[`str`]: a list of observation names found in `ins_file`

Note

This is a basic function for parsing instruction files to look for observation names.

Example::

obs_names = pyemu.pst_utils.parse_ins_file("my.ins")

`parse_tpl_file(tpl_file)`

parse a PEST-style template file to get the parameter names

Args: tpl_file (str): path and name of a template file

Returns:

Type	Description
	[`str`] : list of parameter names found in `tpl_file`

Example::

par_names = pyemu.pst_utils.parse_tpl_file("my.tpl")

`process_output_files(pst, pst_path='.')`

helper function to process output files using the InstructionFile class

Parameters:

Name	Type	Description	Default
`pst`	`pyemu.Pst`	control file instance	required
`pst_path`	`str`	path to instruction and output files to append to the front of the names in the Pst instance	`'.'`

Returns: pd.DataFrame: dataframe of observation names and simulated values extracted from the model output files listed in pst

Example::

 pst = pyemu.Pst("my.pst")
 df = pyemu.pst_utils.process_output_files(pst)

`read_parfile(parfile)`

load a PEST-style parameter value file into a pandas.DataFrame

Parameters:

Name	Type	Description	Default
`parfile`	`str`	path and name of existing parameter file	required

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe with columns of "parnme", "parval1",
	"scale" and "offset"

Example::

df = pyemu.pst_utils.read_parfile("my.par1")

`read_resfile(resfile)`

load a PEST-style residual file into a pandas.DataFrame

Parameters:

Name	Type	Description	Default
`resfile`	`str`	path and name of an existing residual file	required

Returns: pandas.DataFrame: a dataframe of info from the residuals file. Column names are the names from the residuals file: "name", "group", "measured", "modelled" (with two "L"s), "residual", "weight".

Example::

 df = pyemu.pst_utils.read_resfile("my.res")
 df.residual.plot(kind="hist")

`res_from_en(pst, enfile)`

load ensemble results from PESTPP-IES into a PEST-style residuals pandas.DataFrame

Parameters:

Name	Type	Description	Default
`enfile`	`str`	CSV-format ensemble file name	required

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe with the same columns as a
	residual dataframe (a la `pst_utils.read_resfile()`)

Note

If a "base" realization is found in the ensemble, it is used as the "modelled" column in the residuals dataframe. Otherwise, the mean of the ensemble is used as "modelled"

Example::

df = pyemu.pst_utils.res_from_en("my.0.obs.csv")
df.residual.plot(kind="hist")

`res_from_obseravtion_data(observation_data)`

create a PEST-style residual dataframe filled with np.nan for missing information

Parameters:

Name	Type	Description	Default
`observation_data`	`pandas.DataFrame`	the "* observation data" `pandas.DataFrame` from `pyemu.Pst.observation_data`	required

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe with the same columns as the
	residual dataframe ("name","group","measured","modelled",
	"residual","weight").

`try_process_output_file(ins_file, output_file=None, logger=None)`

attempt to process a model output file using a PEST-style instruction file

Parameters:

Name	Type	Description	Default
`ins_file`	`str`	path and name of an instruction file	required
`output_file`	`str`,optional	path and name of existing model output file to process. If `None`, `ins_file.replace(".ins","")` is used. Default is None.	`None`

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of observation name and simulated outputs
	extracted from `output_file`.

Note

If an exception is raised when processing the output file, the exception is echoed to the screen and None is returned.

Example::

df = pyemu.pst_utils.try_process_output_file("my.ins","my.output")

`try_process_output_pst(pst)`

attempt to process each instruction file, model output file pair in a pyemu.Pst.

Parameters:

Name	Type	Description	Default
`pst`	`pyemu.Pst`	a control file instance	required

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of observation names and simulated outputs
	extracted from model output files.

Note

This function first tries to process the output files using the InstructionFile class, If that fails, then it tries to run INSCHEK. If an instructionfile is processed successfully, the extract simulated values are used to populate the pst.observation_data.obsval attribute.

`try_read_input_file_with_tpl(tpl_file, input_file=None)`

attempt to read parameter values from an input file using a template file Args: tpl_file (str): path and name of a template file input_file (str,optional): path and name of existing model input file to process. If None, tpl_file.replace(".tpl","") is used. Default is None.

Returns:

Type	Description
	`pandas.DataFrame`: a dataframe of parameter name and values
	extracted from `input_file`.

Note

If an exception is raised when reading the input file, the exception is echoed to the screen and None is returned.

Example::

df = pyemu.pst_utils.try_process_output_file("my.tpl","my.input")

`write_input_files(pst, pst_path='.')`

write parameter values to model input files

Parameters:

Name	Type	Description	Default
`pst`	`pyemu.Pst`	a Pst instance	required
`pst_path`	`str`	the path to where the control file and template files reside. Default is '.'.	`'.'`

Note:

This function uses template files with the current parameter         values (stored in `pst.parameter_data.parval1`).

This function uses multiprocessing - one process per template file

This is a simple implementation of what PEST does.  It does not
handle all the special cases, just a basic function...user beware

`write_parfile(df, parfile)`

write a PEST-style parameter file from a dataframe

Parameters:

Name	Type	Description	Default
`df`	`pandas.DataFrame`	a dataframe with column names that correspond to the entries in the parameter data section of the pest control file	required
`parfile`	`str`	name of the parameter file to write	required

Example::

pyemu.pst_utils.write_parfile(pst.parameter_data,"my.par")

`write_to_template(parvals, tpl_file, in_file)`

write parameter values to a model input file using the corresponding template file

Parameters:

Name	Type	Description	Default
`parvals`	`dict`	a container of parameter names and values. Can also be a `pandas.Series`	required
`tpl_file`	`str`	path and name of a template file	required
`in_file`	`str`	path and name of model input file to write	required

Examples::

pyemu.pst_utils.write_to_template(par.parameter_data.parval1,
                                  "my.tpl","my.input")