pyemu.legacy.pstfromflopy

Module Contents

Classes

PstFromFlopyModel

a monster helper class to setup a complex PEST interface around

Functions

write_const_tpl(name, tpl_file, suffix[, zn_array, ...])

write a constant (uniform) template file for a 2-D array

write_grid_tpl(name, tpl_file, suffix[, zn_array, ...])

write a grid-based template file for a 2-D array

write_zone_tpl(name, tpl_file[, suffix, zn_array, ...])

write a zone-based template file for a 2-D array

apply_list_pars()

a function to apply boundary condition multiplier parameters.

setup_temporal_diff_obs(pst, ins_file[, out_file, ...])

a helper function to setup difference-in-time observations based on an existing

apply_temporal_diff_obs(config_file)

process an instruction-output file pair and formulate difference observations.

Attributes

wildass_guess_par_bounds_dict

pyemu.legacy.pstfromflopy.wildass_guess_par_bounds_dict
pyemu.legacy.pstfromflopy.write_const_tpl(name, tpl_file, suffix, zn_array=None, shape=None, longnames=False)

write a constant (uniform) template file for a 2-D array

Parameters:

  • name (str) – the base parameter name

  • tpl_file (str) – the template file to write

  • zn_array (numpy.ndarray, optional) – an array used to skip inactive cells, and optionally get shape info.

  • shape (tuple) – tuple nrow and ncol. Either zn_array or shape must be passed

  • longnames (bool) – flag to use longer names that exceed 12 chars in length. Default is False.

Returns:

a dataframe with parameter information

Return type:

pandas.DataFrame

Note

This function is used during the PstFrom setup process

pyemu.legacy.pstfromflopy.write_grid_tpl(name, tpl_file, suffix, zn_array=None, shape=None, spatial_reference=None, longnames=False)

write a grid-based template file for a 2-D array

Parameters:

  • name (str) – the base parameter name

  • tpl_file (str) – the template file to write - include path

  • zn_array (numpy.ndarray, optional) – zone array to identify inactive cells. Default is None

  • shape (tuple, optional) – a length-two tuple of nrow and ncol. Either zn_array or shape must be passed.

  • spatial_reference (flopy.utils.SpatialReference) – a spatial reference instance. If longnames is True, then spatial_reference is used to add spatial info to the parameter names.

  • longnames (bool) – flag to use longer names that exceed 12 chars in length. Default is False.

Returns:

a dataframe with parameter information

Return type:

pandas.DataFrame

Note

This function is used during the PstFrom setup process

Example:

pyemu.helpers.write_grid_tpl("hk_layer1","hk_Layer_1.ref.tpl","gr",
                             zn_array=ib_layer_1,shape=(500,500))
pyemu.legacy.pstfromflopy.write_zone_tpl(name, tpl_file, suffix='', zn_array=None, shape=None, longnames=False, fill_value='1.0')

write a zone-based template file for a 2-D array

Parameters:

  • name (str) – the base parameter name

  • tpl_file (str) – the template file to write

  • suffix (str) – suffix to add to parameter names. Only used if longnames=True

  • zn_array (numpy.ndarray, optional) – an array used to skip inactive cells, and optionally get shape info. zn_array values less than 1 are given fill_value

  • shape (tuple) – tuple nrow and ncol. Either zn_array or shape must be passed

  • longnames (bool) – flag to use longer names that exceed 12 chars in length. Default is False.

  • fill_value (str) – value to fill locations where zn_array is zero or less. Default is “1.0”.

Returns:

a dataframe with parameter information

Return type:

pandas.DataFrame

Note

This function is used during the PstFrom setup process

pyemu.legacy.pstfromflopy.apply_list_pars()

a function to apply boundary condition multiplier parameters.

Note

Used to implement the parameterization constructed by PstFromFlopyModel during a forward run

Requires either “temporal_list_pars.csv” or “spatial_list_pars.csv”

Should be added to the forward_run.py script (called programmaticlly by the PstFrom forward run script)

pyemu.legacy.pstfromflopy.setup_temporal_diff_obs(pst, ins_file, out_file=None, include_zero_weight=False, include_path=False, sort_by_name=True, long_names=True, prefix='dif')

a helper function to setup difference-in-time observations based on an existing set of observations in an instruction file using the observation grouping in the control file

Parameters:

  • pst (pyemu.Pst) – existing control file

  • ins_file (str) – an existing instruction file

  • out_file (str, optional) – an existing model output file that corresponds to the instruction file. If None, ins_file.replace(“.ins”,””) is used

  • include_zero_weight (bool, optional) – flag to include zero-weighted observations in the difference observation process. Default is False so that only non-zero weighted observations are used.

  • include_path (bool, optional) – flag to setup the binary file processing in directory where the hds_file is located (if different from where python is running). This is useful for setting up the process in separate directory for where python is running.

  • sort_by_name (bool,optional) – flag to sort observation names in each group prior to setting up the differencing. The order of the observations matters for the differencing. If False, then the control file order is used. If observation names have a datetime suffix, make sure the format is year-month-day to use this sorting. Default is True

  • long_names (bool, optional) – flag to use long, descriptive names by concating the two observation names that are being differenced. This will produce names that are too long for tradtional PEST(_HP). Default is True.

  • prefix (str, optional) – prefix to prepend to observation names and group names. Default is “dif”.

Returns:

tuple containing

  • str: the forward run command to execute the binary file process during model runs.

  • pandas.DataFrame: a dataframe of observation information for use in the pest control file

Note

This is the companion function of helpers.apply_temporal_diff_obs().

pyemu.legacy.pstfromflopy.apply_temporal_diff_obs(config_file)

process an instruction-output file pair and formulate difference observations.

Parameters:

config_file (str) – configuration file written by pyemu.helpers.setup_temporal_diff_obs.

Returns:

processed difference observations

Return type:

diff_df (pandas.DataFrame)

Note

Writes config_file.replace(“.config”,”.processed”) output file that can be read with the instruction file that is created by pyemu.helpers.setup_temporal_diff_obs().

This is the companion function of helpers.setup_setup_temporal_diff_obs().

class pyemu.legacy.pstfromflopy.PstFromFlopyModel(model, new_model_ws, org_model_ws=None, pp_props=[], const_props=[], temporal_bc_props=[], temporal_list_props=[], grid_props=[], grid_geostruct=None, pp_space=None, zone_props=[], pp_geostruct=None, par_bounds_dict=None, sfr_pars=False, temporal_sfr_pars=False, temporal_list_geostruct=None, remove_existing=False, k_zone_dict=None, mflist_waterbudget=True, mfhyd=True, hds_kperk=[], use_pp_zones=False, obssim_smp_pairs=None, external_tpl_in_pairs=None, external_ins_out_pairs=None, extra_pre_cmds=None, extra_model_cmds=None, extra_post_cmds=None, redirect_forward_output=True, tmp_files=None, model_exe_name=None, build_prior=True, sfr_obs=False, spatial_bc_props=[], spatial_list_props=[], spatial_list_geostruct=None, hfb_pars=False, kl_props=None, kl_num_eig=100, kl_geostruct=None)

Bases: object

a monster helper class to setup a complex PEST interface around an existing MODFLOW-2005-family model.

Parameters:

  • model (flopy.mbase) – a loaded flopy model instance. If model is an str, it is treated as a MODFLOW nam file (requires org_model_ws)

  • new_model_ws (str) – a directory where the new version of MODFLOW input files and PEST(++) files will be written

  • org_model_ws (str) – directory to existing MODFLOW model files. Required if model argument is an str. Default is None

  • pp_props ([[str,[int]]]) – pilot point multiplier parameters for grid-based properties. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices. For example, [“lpf.hk”,[0,1,2,]] would setup pilot point multiplier parameters for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, [“rch.rech”,[0,4,10,15]] would setup pilot point multiplier parameters for recharge for stress period 1,5,11,and 16.

  • const_props ([[str,[int]]]) – constant (uniform) multiplier parameters for grid-based properties. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices. For example, [“lpf.hk”,[0,1,2,]] would setup constant (uniform) multiplier parameters for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, [“rch.rech”,[0,4,10,15]] would setup constant (uniform) multiplier parameters for recharge for stress period 1,5,11,and 16.

  • temporal_list_props ([[str,[int]]]) – list-type input stress-period level multiplier parameters. A nested list of list-type input elements to parameterize using name, iterable pairs. The iterable is zero-based stress-period indices. For example, to setup multipliers for WEL flux and for RIV conductance, temporal_list_props = [[“wel.flux”,[0,1,2]],[“riv.cond”,None]] would setup multiplier parameters for well flux for stress periods 1,2 and 3 and would setup one single river conductance multiplier parameter that is applied to all stress periods

  • spatial_list_props ([[str,[int]]]) – list-type input for spatial multiplier parameters. A nested list of list-type elements to parameterize using names (e.g. [[“riv.cond”,0],[“wel.flux”,1] to setup up cell-based parameters for each list-type element listed. These multiplier parameters are applied across all stress periods. For this to work, there must be the same number of entries for all stress periods. If more than one list element of the same type is in a single cell, only one parameter is used to multiply all lists in the same cell.

  • grid_props ([[str,[int]]]) – grid-based (every active model cell) multiplier parameters. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., [“lpf.hk”,[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 in every active model cell). For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, [“rch.rech”,[0,4,10,15]] would setup grid-based multiplier parameters in every active model cell for recharge for stress period 1,5,11,and 16.

  • sfr_pars (bool) – setup parameters for the stream flow routing modflow package. If list is passed it defines the parameters to set up.

  • sfr_temporal_pars (bool) – flag to include stress-period level spatially-global multipler parameters in addition to the spatially-discrete sfr_pars. Requires sfr_pars to be passed. Default is False

  • grid_geostruct (pyemu.geostats.GeoStruct) – the geostatistical structure to build the prior parameter covariance matrix elements for grid-based parameters. If None, a generic GeoStruct is created using an “a” parameter that is 10 times the max cell size. Default is None

  • pp_space (int) – number of grid cells between pilot points. If None, use the default in pyemu.pp_utils.setup_pilot_points_grid. Default is None

  • zone_props ([[str,[int]]]) – zone-based multiplier parameters. A nested list of zone-based model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., [“lpf.hk”,[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 for unique zone values in the ibound array. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, [“rch.rech”,[0,4,10,15]] would setup zone-based multiplier parameters for recharge for stress period 1,5,11,and 16.

  • pp_geostruct (pyemu.geostats.GeoStruct) – the geostatistical structure to use for building the prior parameter covariance matrix for pilot point parameters. If None, a generic GeoStruct is created using pp_space and grid-spacing information. Default is None

  • par_bounds_dict (dict) – a dictionary of model property/boundary condition name, upper-lower bound pairs. For example, par_bounds_dict = {“hk”:[0.01,100.0],”flux”:[0.5,2.0]} would set the bounds for horizontal hydraulic conductivity to 0.001 and 100.0 and set the bounds for flux parameters to 0.5 and 2.0. For parameters not found in par_bounds_dict, pyemu.helpers.wildass_guess_par_bounds_dict is used to set somewhat meaningful bounds. Default is None

  • temporal_list_geostruct (pyemu.geostats.GeoStruct) – the geostastical struture to build the prior parameter covariance matrix for time-varying list-type multiplier parameters. This GeoStruct express the time correlation so that the ‘a’ parameter is the length of time that boundary condition multiplier parameters are correlated across. If None, then a generic GeoStruct is created that uses an ‘a’ parameter of 3 stress periods. Default is None

  • spatial_list_geostruct (pyemu.geostats.GeoStruct) – the geostastical struture to build the prior parameter covariance matrix for spatially-varying list-type multiplier parameters. If None, a generic GeoStruct is created using an “a” parameter that is 10 times the max cell size. Default is None.

  • remove_existing (bool) – a flag to remove an existing new_model_ws directory. If False and new_model_ws exists, an exception is raised. If True and new_model_ws exists, the directory is destroyed - user beware! Default is False.

  • k_zone_dict (dict) – a dictionary of zero-based layer index, zone array pairs. e.g. {lay: np.2darray} Used to override using ibound zones for zone-based parameterization. If None, use ibound values greater than zero as zones. Alternatively a dictionary of dictionaries can be passed to allow different zones to be defined for different parameters. e.g. {“upw.hk” {lay: np.2darray}, “extra.rc11” {lay: np.2darray}} or {“hk” {lay: np.2darray}, “rc11” {lay: np.2darray}}

  • use_pp_zones (bool) – a flag to use ibound zones (or k_zone_dict, see above) as pilot point zones. If False, ibound values greater than zero are treated as a single zone for pilot points. Default is False

  • ([[str (external_ins_out_pairs) – a list of observed-simulated PEST-type SMP file pairs to get observations from and include in the control file. Default is []

  • str]] – a list of observed-simulated PEST-type SMP file pairs to get observations from and include in the control file. Default is []

  • ([[str – a list of existing template file, model input file pairs to parse parameters from and include in the control file. Default is []

  • str]] – a list of existing template file, model input file pairs to parse parameters from and include in the control file. Default is []

  • ([[str – a list of existing instruction file, model output file pairs to parse observations from and include in the control file. Default is []

  • str]] – a list of existing instruction file, model output file pairs to parse observations from and include in the control file. Default is []

  • extra_pre_cmds ([str]) – a list of preprocessing commands to add to the forward_run.py script commands are executed with os.system() within forward_run.py. Default is None.

  • redirect_forward_output (bool) – flag for whether to redirect forward model output to text files (True) or allow model output to be directed to the screen (False). Default is True

  • extra_post_cmds ([str]) – a list of post-processing commands to add to the forward_run.py script. Commands are executed with os.system() within forward_run.py. Default is None.

  • tmp_files ([str]) – a list of temporary files that should be removed at the start of the forward run script. Default is [].

  • model_exe_name (str) – binary name to run modflow. If None, a default from flopy is used, which is dangerous because of the non-standard binary names (e.g. MODFLOW-NWT_x64, MODFLOWNWT, mfnwt, etc). Default is None.

  • build_prior (bool) – flag to build prior covariance matrix. Default is True

  • sfr_obs (bool) – flag to include observations of flow and aquifer exchange from the sfr ASCII output file

  • hfb_pars (bool) – add HFB parameters. uses pyemu.gw_utils.write_hfb_template(). the resulting HFB pars have parval1 equal to the values in the original file and use the spatial_list_geostruct to build geostatistical covariates between parameters

  • kl_props ([[str,[int]]]) – karhunen-loeve based multiplier parameters. A nested list of KL-based model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., [“lpf.hk”,[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 for unique zone values in the ibound array. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, [“rch.rech”,[0,4,10,15]] would setup zone-based multiplier parameters for recharge for stress period 1,5,11,and 16.

  • kl_num_eig (int) – the number of KL-based eigenvector multiplier parameters to use for each KL parameter set. default is 100

  • kl_geostruct (pyemu.geostats.Geostruct) – the geostatistical structure to build the prior parameter covariance matrix elements for KL-based parameters. If None, a generic GeoStruct is created using an “a” parameter that is 10 times the max cell size. Default is None

Note

Setup up multiplier parameters for an existing MODFLOW model.

Does all kinds of coolness like building a meaningful prior, assigning somewhat meaningful parameter groups and bounds, writes a forward_run.py script with all the calls need to implement multiplier parameters, run MODFLOW and post-process.

While this class does work, the newer PstFrom class is a more pythonic implementation

_setup_sfr_obs()

setup sfr ASCII observations

_setup_sfr_pars(par_cols=None, include_temporal_pars=None)

setup multiplier parameters for sfr segment data Adding support for reachinput (and isfropt = 1)

_setup_hfb_pars()

setup non-mult parameters for hfb (yuck!)

_setup_mult_dirs()

setup the directories to use for multiplier parameterization. Directories are make within the PstFromFlopyModel.m.model_ws directory

_setup_model(model, org_model_ws, new_model_ws)

setup the flopy.mbase instance for use with multipler parameters. Changes model_ws, sets external_path and writes new MODFLOW input files

_get_count(name)

get the latest counter for a certain parameter type.

_prep_mlt_arrays()

prepare multipler arrays. Copies existing model input arrays and writes generic (ones) multiplier arrays

_write_u2d(u2d)

write a flopy.utils.Util2D instance to an ASCII text file using the Util2D filename

_write_const_tpl(name, tpl_file, zn_array)

write a template file a for a constant (uniform) multiplier parameter

_write_grid_tpl(name, tpl_file, zn_array)

write a template file a for grid-based multiplier parameters

_grid_prep()

prepare grid-based parameterizations

_pp_prep(mlt_df)

prepare pilot point based parameterization

_kl_prep(mlt_df)

prepare KL based parameterizations

_setup_array_pars()

main entry point for setting up array multipler parameters

_setup_observations()

main entry point for setting up observations

draw(num_reals=100, sigma_range=6, use_specsim=False, scale_offset=True)

draw from the geostatistically-implied parameter covariance matrix

Parameters:

  • num_reals (int) – number of realizations to generate. Default is 100

  • sigma_range (float) – number of standard deviations represented by the parameter bounds. Default is 6.

  • use_specsim (bool) – flag to use spectral simulation for grid-based parameters. Requires a regular grid but is wicked fast. Default is False

  • scale_offset (bool, optional) – flag to apply scale and offset to parameter bounds when calculating variances - this is passed through to pyemu.Cov.from_parameter_data. Default is True.

Note

operates on parameters by groups to avoid having to construct a very large covariance matrix for problems with more the 30K parameters.

uses helpers.geostatitical_draw()

Returns:

The realized parameter ensemble

Return type:

pyemu.ParameterEnsemble

build_prior(fmt='ascii', filename=None, droptol=None, chunk=None, sigma_range=6)

build and optionally save the prior parameter covariance matrix.

Parameters:

  • fmt (str, optional) – the format to save the cov matrix. Options are “ascii”,”binary”,”uncfile”, “coo”. Default is “ascii”. If “none” (lower case string, not None), then no file is created.

  • filename (str, optional) – the filename to save the prior cov matrix to. If None, the name is formed using model nam_file name. Default is None.

  • droptol (float, optional) – tolerance for dropping near-zero values when writing compressed binary. Default is None.

  • chunk (int, optional) – chunk size to write in a single pass - for binary only. Default is None (no chunking).

  • sigma_range (float) – number of standard deviations represented by the parameter bounds. Default is 6.

Returns:

the full prior parameter covariance matrix, generated by processing parameters by groups

Return type:

pyemu.Cov

build_pst(filename=None)

build the pest control file using the parameters and observations.

Parameters:

filename (str) – the filename to save the contorl file to. If None, the name if formed from the model namfile name. Default is None. The control is saved in the PstFromFlopy.m.model_ws directory.

Note

calls pyemu.Pst.from_io_files

calls PESTCHEK

_add_external()

add external (existing) template files and/or instruction files to the Pst instance

write_forward_run()

write the forward run script forward_run.py

Note

This method can be called repeatedly, especially after any changed to the pre- and/or post-processing routines.

_parse_k(k, vals)

parse the iterable from a property or boundary condition argument

_parse_pakattr(pakattr)

parse package-iterable pairs from a property or boundary condition argument

_setup_list_pars()

main entry point for setting up list multiplier parameters

_setup_temporal_list_pars()
_setup_spatial_list_pars()
_list_helper(k, pak, attr, col)

helper to setup list multiplier parameters for a given k, pak, attr set.

_setup_hds()

setup modflow head save file observations for given kper (zero-based stress period index) and k (zero-based layer index) pairs using the kperk argument.

_setup_smp()

setup observations from PEST-style SMP file pairs

_setup_hob()

setup observations from the MODFLOW HOB package

_setup_hyd()

setup observations from the MODFLOW HYDMOD package

_setup_water_budget_obs()

setup observations from the MODFLOW list file for volume and flux water buget information