legacy

This module contains legacy, deprecated classes and methods, including PstFromFlopyModel -- superseded by pyemu.utils.PstFrom()

PstFromFlopyModel

Bases: object

a monster helper class to setup a complex PEST interface around an existing MODFLOW-2005-family model.

Parameters:

Name	Type	Description	Default
model	`flopy.mbase`	a loaded flopy model instance. If model is an str, it is treated as a MODFLOW nam file (requires org_model_ws)	required
new_model_ws	`str`	a directory where the new version of MODFLOW input files and PEST(++) files will be written	required
org_model_ws	`str`	directory to existing MODFLOW model files. Required if model argument is an str. Default is None	None
pp_props	[[`str`,[`int`]]]	pilot point multiplier parameters for grid-based properties. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices. For example, ["lpf.hk",[0,1,2,]] would setup pilot point multiplier parameters for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, ["rch.rech",[0,4,10,15]] would setup pilot point multiplier parameters for recharge for stress period 1,5,11,and 16.	[]
const_props	[[`str`,[`int`]]]	constant (uniform) multiplier parameters for grid-based properties. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices. For example, ["lpf.hk",[0,1,2,]] would setup constant (uniform) multiplier parameters for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, ["rch.rech",[0,4,10,15]] would setup constant (uniform) multiplier parameters for recharge for stress period 1,5,11,and 16.	[]
temporal_list_props	[[`str`,[`int`]]]	list-type input stress-period level multiplier parameters. A nested list of list-type input elements to parameterize using name, iterable pairs. The iterable is zero-based stress-period indices. For example, to setup multipliers for WEL flux and for RIV conductance, temporal_list_props = [["wel.flux",[0,1,2]],["riv.cond",None]] would setup multiplier parameters for well flux for stress periods 1,2 and 3 and would setup one single river conductance multiplier parameter that is applied to all stress periods	[]
spatial_list_props	[[`str`,[`int`]]]	list-type input for spatial multiplier parameters. A nested list of list-type elements to parameterize using names (e.g. [["riv.cond",0],["wel.flux",1] to setup up cell-based parameters for each list-type element listed. These multiplier parameters are applied across all stress periods. For this to work, there must be the same number of entries for all stress periods. If more than one list element of the same type is in a single cell, only one parameter is used to multiply all lists in the same cell.	[]
grid_props	[[`str`,[`int`]]]	grid-based (every active model cell) multiplier parameters. A nested list of grid-scale model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., ["lpf.hk",[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 in every active model cell). For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, ["rch.rech",[0,4,10,15]] would setup grid-based multiplier parameters in every active model cell for recharge for stress period 1,5,11,and 16.	[]
sfr_pars	`bool`	setup parameters for the stream flow routing modflow package. If list is passed it defines the parameters to set up.	False
grid_geostruct	`pyemu.geostats.GeoStruct`	the geostatistical structure to build the prior parameter covariance matrix elements for grid-based parameters. If None, a generic GeoStruct is created using an "a" parameter that is 10 times the max cell size. Default is None	None
pp_space	`int`	number of grid cells between pilot points. If None, use the default in pyemu.pp_utils.setup_pilot_points_grid. Default is None	None
zone_props	[[`str`,[`int`]]]	zone-based multiplier parameters. A nested list of zone-based model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., ["lpf.hk",[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 for unique zone values in the ibound array. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, ["rch.rech",[0,4,10,15]] would setup zone-based multiplier parameters for recharge for stress period 1,5,11,and 16.	[]
pp_geostruct	`pyemu.geostats.GeoStruct`	the geostatistical structure to use for building the prior parameter covariance matrix for pilot point parameters. If None, a generic GeoStruct is created using pp_space and grid-spacing information. Default is None	None
par_bounds_dict	`dict`	a dictionary of model property/boundary condition name, upper-lower bound pairs. For example, par_bounds_dict = {"hk":[0.01,100.0],"flux":[0.5,2.0]} would set the bounds for horizontal hydraulic conductivity to 0.001 and 100.0 and set the bounds for flux parameters to 0.5 and 2.0. For parameters not found in par_bounds_dict, pyemu.helpers.wildass_guess_par_bounds_dict is used to set somewhat meaningful bounds. Default is None	None
temporal_list_geostruct	`pyemu.geostats.GeoStruct`	the geostastical structure to build the prior parameter covariance matrix for time-varying list-type multiplier parameters. This GeoStruct express the time correlation so that the 'a' parameter is the length of time that boundary condition multiplier parameters are correlated across. If None, then a generic GeoStruct is created that uses an 'a' parameter of 3 stress periods. Default is None	None
spatial_list_geostruct	`pyemu.geostats.GeoStruct`	the geostastical structure to build the prior parameter covariance matrix for spatially-varying list-type multiplier parameters. If None, a generic GeoStruct is created using an "a" parameter that is 10 times the max cell size. Default is None.	None
remove_existing	`bool`	a flag to remove an existing new_model_ws directory. If False and new_model_ws exists, an exception is raised. If True and new_model_ws exists, the directory is destroyed - user beware! Default is False.	False
k_zone_dict	`dict`	a dictionary of zero-based layer index, zone array pairs. e.g. {lay: np.2darray} Used to override using ibound zones for zone-based parameterization. If None, use ibound values greater than zero as zones. Alternatively a dictionary of dictionaries can be passed to allow different zones to be defined for different parameters. e.g. {"upw.hk" {lay: np.2darray}, "extra.rc11" {lay: np.2darray}} or {"hk" {lay: np.2darray}, "rc11" {lay: np.2darray}}	None
use_pp_zones	`bool`	a flag to use ibound zones (or k_zone_dict, see above) as pilot point zones. If False, ibound values greater than zero are treated as a single zone for pilot points. Default is False	False
obssim_smp_pairs ([[`str`,`str`]]		a list of observed-simulated PEST-type SMP file pairs to get observations from and include in the control file. Default is []	required
external_tpl_in_pairs ([[`str`,`str`]]		a list of existing template file, model input file pairs to parse parameters from and include in the control file. Default is []	required
external_ins_out_pairs ([[`str`,`str`]]		a list of existing instruction file, model output file pairs to parse observations from and include in the control file. Default is []	required
extra_pre_cmds	[`str`]	a list of preprocessing commands to add to the forward_run.py script commands are executed with os.system() within forward_run.py. Default is None.	None
redirect_forward_output	`bool`	flag for whether to redirect forward model output to text files (True) or allow model output to be directed to the screen (False). Default is True	True
extra_post_cmds	[`str`]	a list of post-processing commands to add to the forward_run.py script. Commands are executed with os.system() within forward_run.py. Default is None.	None
tmp_files	[`str`]	a list of temporary files that should be removed at the start of the forward run script. Default is [].	None
model_exe_name	`str`	binary name to run modflow. If None, a default from flopy is used, which is dangerous because of the non-standard binary names (e.g. MODFLOW-NWT_x64, MODFLOWNWT, mfnwt, etc). Default is None.	None
build_prior	`bool`	flag to build prior covariance matrix. Default is True	True
sfr_obs	`bool`	flag to include observations of flow and aquifer exchange from the sfr ASCII output file	False
hfb_pars	`bool`	add HFB parameters. uses pyemu.gw_utils.write_hfb_template(). the resulting HFB pars have parval1 equal to the values in the original file and use the spatial_list_geostruct to build geostatistical covariates between parameters	False
kl_props	[[`str`,[`int`]]]	karhunen-loeve based multiplier parameters. A nested list of KL-based model properties to parameterize using name, iterable pairs. For 3D properties, the iterable is zero-based layer indices (e.g., ["lpf.hk",[0,1,2,]] would setup a multiplier parameter for layer property file horizontal hydraulic conductivity for model layers 1,2, and 3 for unique zone values in the ibound array. For time-varying properties (e.g. recharge), the iterable is for zero-based stress period indices. For example, ["rch.rech",[0,4,10,15]] would setup zone-based multiplier parameters for recharge for stress period 1,5,11,and 16.	None
kl_num_eig	`int`	the number of KL-based eigenvector multiplier parameters to use for each KL parameter set. default is 100	100
kl_geostruct	`pyemu.geostats.Geostruct`	the geostatistical structure to build the prior parameter covariance matrix elements for KL-based parameters. If None, a generic GeoStruct is created using an "a" parameter that is 10 times the max cell size. Default is None	None

Note:

Setup up multiplier parameters for an existing MODFLOW model.

Does all kinds of coolness like building a
meaningful prior, assigning somewhat meaningful parameter groups and
bounds, writes a forward_run.py script with all the calls need to
implement multiplier parameters, run MODFLOW and post-process.

While this class does work, the newer `PstFrom` class is a more pythonic
implementation

build_prior(fmt='ascii', filename=None, droptol=None, chunk=None, sigma_range=6)

build and optionally save the prior parameter covariance matrix.

Parameters:

Name	Type	Description	Default
fmt	`str`	the format to save the cov matrix. Options are "ascii","binary","uncfile", "coo". Default is "ascii". If "none" (lower case string, not None), then no file is created.	'ascii'
filename	`str`	the filename to save the prior cov matrix to. If None, the name is formed using model nam_file name. Default is None.	None
droptol	`float`	tolerance for dropping near-zero values when writing compressed binary. Default is None.	None
chunk	`int`	chunk size to write in a single pass - for binary only. Default is None (no chunking).	None
sigma_range	`float`	number of standard deviations represented by the parameter bounds. Default is 6.	6

Returns:

Type	Description
	pyemu.Cov: the full prior parameter covariance matrix, generated by processing parameters by
	groups

build_pst(filename=None)

build the pest control file using the parameters and observations.

Parameters:

Name	Type	Description	Default
filename	`str`	the filename to save the control file to. If None, the name if formed from the model namfile name. Default is None. The control is saved in the PstFromFlopy.m.model_ws directory.	None

Note:

calls pyemu.Pst.from_io_files

calls PESTCHEK

draw(num_reals=100, sigma_range=6, use_specsim=False, scale_offset=True)

draw from the geostatistically-implied parameter covariance matrix

Parameters:

Name	Type	Description	Default
num_reals	`int`	number of realizations to generate. Default is 100	100
sigma_range	`float`	number of standard deviations represented by the parameter bounds. Default is 6.	6
use_specsim	`bool`	flag to use spectral simulation for grid-based parameters. Requires a regular grid but is wicked fast. Default is False	False
scale_offset	`bool`	flag to apply scale and offset to parameter bounds when calculating variances - this is passed through to pyemu.Cov.from_parameter_data. Default is True.	True

Note

operates on parameters by groups to avoid having to construct a very large covariance matrix for problems with more the 30K parameters.

uses helpers.geostatitical_draw()

Returns:

Type	Description
	pyemu.ParameterEnsemble: The realized parameter ensemble

write_forward_run()

write the forward run script forward_run.py

Note

This method can be called repeatedly, especially after any changed to the pre- and/or post-processing routines.

SpatialReference

Bases: object

a class to locate a structured model grid in x-y space. Lifted wholesale from Flopy, and preserved here... ...maybe slightly over-engineered for here

Args:

delr (`numpy ndarray`): the model discretization delr vector (An array of spacings along a row)
delc (`numpy ndarray`): the model discretization delc vector (An array of spacings along a column)
lenuni (`int`): the length units flag from the discretization package. Default is 2.
xul (`float`): The x coordinate of the upper left corner of the grid. Enter either xul and yul or xll and yll.
yul (`float`): The y coordinate of the upper left corner of the grid. Enter either xul and yul or xll and yll.
xll (`float`): The x coordinate of the lower left corner of the grid. Enter either xul and yul or xll and yll.
yll (`float`): The y coordinate of the lower left corner of the grid. Enter either xul and yul or xll and yll.
rotation (`float`): The counter-clockwise rotation (in degrees) of the grid
proj4_str (`str`): a PROJ4 string that identifies the grid in space. warning: case sensitive!
units (`string`): Units for the grid.  Must be either "feet" or "meters"
epsg (`int`): EPSG code that identifies the grid in space. Can be used in lieu of
    proj4. PROJ4 attribute will auto-populate if there is an internet
    connection(via get_proj4 method).
    See https://www.epsg-registry.org/ or spatialreference.org
length_multiplier (`float`): multiplier to convert model units to spatial reference units.
    delr and delc above will be multiplied by this value. (default=1.)

bounds property

Return bounding box in shapely order.

length_multiplier property

Attempt to identify multiplier for converting from model units to sr units, defaulting to 1.

vertices property

Returns a list of vertices for

get_extent()

Get the extent of the rotated and offset grid

get_grid_lines()

Get the grid lines as a list

get_ij(x, y)

Return the row and column of a point or sequence of points in real-world coordinates.

Parameters:

Name	Type	Description	Default
x	`float`	scalar or sequence of x coordinates	required
y	`float`	scalar or sequence of y coordinates	required

Returns:

Type	Description
	tuple of
	int : row or sequence of rows (zero-based)
	int : column or sequence of columns (zero-based)

get_vertices(i, j)

Get vertices for a single cell or sequence if i, j locations.

get_xcenter_array()

Return a numpy one-dimensional float array that has the cell center x coordinate for every column in the grid in model space - not offset or rotated.

get_xedge_array()

Return a numpy one-dimensional float array that has the cell edge x coordinates for every column in the grid in model space - not offset or rotated. Array is of size (ncol + 1)

get_ycenter_array()

Return a numpy one-dimensional float array that has the cell center x coordinate for every row in the grid in model space - not offset of rotated.

get_yedge_array()

Return a numpy one-dimensional float array that has the cell edge y coordinates for every row in the grid in model space - not offset or rotated. Array is of size (nrow + 1)

load(namefile=None, reffile='usgs.model.reference') staticmethod

Attempts to load spatial reference information from the following files (in order): 1) usgs.model.reference 2) NAM file (header comment) 3) SpatialReference.default dictionary

read_usgs_model_reference_file(reffile='usgs.model.reference') staticmethod

read spatial reference info from the usgs.model.reference file https://water.usgs.gov/ogw/policy/gw-model/modelers-setup.html

rotate(x, y, theta, xorigin=0.0, yorigin=0.0) staticmethod

Given x and y array-like values calculate the rotation about an arbitrary origin and then return the rotated coordinates. theta is in degrees.

set_spatialreference(xul=None, yul=None, xll=None, yll=None, rotation=0.0)

set spatial reference - can be called from model instance

transform(x, y, inverse=False)

Given x and y array-like values, apply rotation, scale and offset, to convert them from model coordinates to real-world coordinates.

write_gridspec(filename)

write a PEST-style grid specification file

apply_array_pars(arr_par='arr_pars.csv', arr_par_file=None, chunk_len=50)

a function to apply array-based multiplier parameters.

Parameters:

Name	Type	Description	Default
arr_par	`str` or `pandas.DataFrame`	if type str,	'arr_pars.csv'
chunk_len (`int`)		the number of files to process per chunk with multiprocessing - applies to both fac2real and process_ input_files. Default is 50.	required

Note

Used to implement the parameterization constructed by PstFromFlopyModel during a forward run

This function should be added to the forward_run.py script but can be called on any correctly formatted csv

This function using multiprocessing, spawning one process for each model input array (and optionally pp files). This speeds up execution time considerably but means you need to make sure your forward run script uses the proper multiprocessing idioms for freeze support and main thread handling (PstFrom does this for you).

apply_list_pars()

a function to apply boundary condition multiplier parameters.

Note

Used to implement the parameterization constructed by PstFromFlopyModel during a forward run

Requires either "temporal_list_pars.csv" or "spatial_list_pars.csv"

Should be added to the forward_run.py script (called programmaticlly by the PstFrom forward run script)

apply_temporal_diff_obs(config_file)

process an instruction-output file pair and formulate difference observations.

Parameters:

Name	Type	Description	Default
config_file	`str`	configuration file written by pyemu.helpers.setup_temporal_diff_obs.	required

Returns:

Type	Description
	diff_df (pandas.DataFrame) : processed difference observations

Note

Writes config_file.replace(".config",".processed") output file that can be read with the instruction file that is created by pyemu.helpers.setup_temporal_diff_obs().

This is the companion function of helpers.setup_setup_temporal_diff_obs().

geostatistical_draws(pst, struct_dict, num_reals=100, sigma_range=4, verbose=True, scale_offset=True, subset=None, rng=None)

construct a parameter ensemble from a prior covariance matrix implied by geostatistical structure(s) and parameter bounds.

Parameters:

Name	Type	Description	Default
pst	`pyemu.Pst`	a control file (or the name of control file). The parameter bounds in pst are used to define the variance of each parameter group.	required
struct_dict	`dict`	a dict of GeoStruct (or structure file), and list of pilot point template files pairs. If the values in the dict are pd.DataFrames, then they must have an 'x','y', and 'parnme' column. If the filename ends in '.csv', then a pd.DataFrame is loaded, otherwise a pilot points file is loaded.	required
num_reals	`int`	number of realizations to draw. Default is 100	100
sigma_range	`float`	a float representing the number of standard deviations implied by parameter bounds. Default is 4.0, which implies 95% confidence parameter bounds.	4
verbose	`bool`	flag to control output to stdout. Default is True. flag for stdout.	True
scale_offset	`bool`,optional	flag to apply scale and offset to parameter bounds when calculating variances - this is passed through to pyemu.Cov.from_parameter_data(). Default is True.	True
subset	`array-like`	list, array, set or pandas index defining subset of parameters for draw.	None
rng	`numpy.random.RandomState`	random number generator if not using default from pyemu.en	None

Returns pyemu.ParameterEnsemble: the realized parameter ensemble.

Note

Parameters are realized by parameter group.

The variance of each parameter is used to scale the resulting geostatistical covariance matrix Therefore, the sill of the geostatistical structures in struct_dict should be 1.0

Example::

pst = pyemu.Pst("my.pst")
sd = {"struct.dat":["hkpp.dat.tpl","vka.dat.tpl"]}
pe = pyemu.helpers.geostatistical_draws(pst,struct_dict=sd}
pe.to_csv("my_pe.csv")

kl_setup(num_eig, sr, struct, prefixes, factors_file='kl_factors.dat', islog=True, basis_file=None, tpl_dir='.')

setup a karhuenen-Loeve based parameterization for a given geostatistical structure.

Parameters:

Name	Type	Description	Default
num_eig	`int`	the number of basis vectors to retain in the reduced basis	required
sr	`flopy.reference.SpatialReference`	a spatial reference instance	required
struct	`str`	a PEST-style structure file. Can also be a pyemu.geostats.Geostruct instance.	required
prefixes	[`str`]	a list of parameter prefixes to generate KL parameterization for.	required
factors_file	`str`	name of the PEST-style interpolation factors file to write (can be processed with FAC2REAL). Default is "kl_factors.dat".	'kl_factors.dat'
islog	`bool`	flag to indicate if the parameters are log transformed. Default is True	True
basis_file	`str`	the name of the PEST-style binary (e.g. jco) file to write the reduced basis vectors to. Default is None (not saved).	None
tpl_dir	`str`	the directory to write the resulting template files to. Default is "." (current directory).	'.'

Returns:

Type	Description
	pandas.DataFrame: a dataframe of parameter information.

Note

This is the companion function to helpers.apply_kl()

Example::

m = flopy.modflow.Modflow.load("mymodel.nam")
prefixes = ["hk","vka","ss"]
df = pyemu.helpers.kl_setup(10,m.sr,"struct.dat",prefixes)

setup_temporal_diff_obs(pst, ins_file, out_file=None, include_zero_weight=False, include_path=False, sort_by_name=True, long_names=True, prefix='dif')

a helper function to setup difference-in-time observations based on an existing set of observations in an instruction file using the observation grouping in the control file

Parameters:

Name	Type	Description	Default
pst	`pyemu.Pst`	existing control file	required
ins_file	`str`	an existing instruction file	required
out_file	`str`	an existing model output file that corresponds to the instruction file. If None, ins_file.replace(".ins","") is used	None
include_zero_weight	`bool`	flag to include zero-weighted observations in the difference observation process. Default is False so that only non-zero weighted observations are used.	False
include_path	`bool`	flag to setup the binary file processing in directory where the hds_file is located (if different from where python is running). This is useful for setting up the process in separate directory for where python is running.	False
sort_by_name	`bool`,optional	flag to sort observation names in each group prior to setting up the differencing. The order of the observations matters for the differencing. If False, then the control file order is used. If observation names have a datetime suffix, make sure the format is year-month-day to use this sorting. Default is True	True
long_names	`bool`	flag to use long, descriptive names by concatenating the two observation names that are being differenced. This will produce names that are too long for traditional PEST(_HP). Default is True.	True
prefix	`str`	prefix to prepend to observation names and group names. Default is "dif".	'dif'

Returns:

Type	Description
	tuple containing
	str: the forward run command to execute the binary file process during model runs.
	pandas.DataFrame: a dataframe of observation information for use in the pest control file

Note

This is the companion function of helpers.apply_temporal_diff_obs().

write_const_tpl(name, tpl_file, suffix, zn_array=None, shape=None, longnames=False)

write a constant (uniform) template file for a 2-D array

Parameters:

Name	Type	Description	Default
name	`str`	the base parameter name	required
tpl_file	`str`	the template file to write	required
zn_array	`numpy.ndarray`	an array used to skip inactive cells, and optionally get shape info.	None
shape	`tuple`	tuple nrow and ncol. Either zn_array or shape must be passed	None
longnames	`bool`	flag to use longer names that exceed 12 chars in length. Default is False.	False

Returns:

Type	Description
	pandas.DataFrame: a dataframe with parameter information

Note

This function is used during the PstFrom setup process

write_grid_tpl(name, tpl_file, suffix, zn_array=None, shape=None, spatial_reference=None, longnames=False)

write a grid-based template file for a 2-D array

Parameters:

Name	Type	Description	Default
name	`str`	the base parameter name	required
tpl_file	`str`	the template file to write - include path	required
zn_array	`numpy.ndarray`	zone array to identify inactive cells. Default is None	None
shape	`tuple`	a length-two tuple of nrow and ncol. Either zn_array or shape must be passed.	None
spatial_reference	`flopy.utils.SpatialReference`	a spatial reference instance. If longnames is True, then spatial_reference is used to add spatial info to the parameter names.	None
longnames	`bool`	flag to use longer names that exceed 12 chars in length. Default is False.	False

Returns:

Type	Description
	pandas.DataFrame: a dataframe with parameter information

Note

This function is used during the PstFrom setup process

Example::

pyemu.helpers.write_grid_tpl("hk_layer1","hk_Layer_1.ref.tpl","gr",
                             zn_array=ib_layer_1,shape=(500,500))

write_zone_tpl(name, tpl_file, suffix='', zn_array=None, shape=None, longnames=False, fill_value='1.0')

write a zone-based template file for a 2-D array

Parameters:

Name	Type	Description	Default
name	`str`	the base parameter name	required
tpl_file	`str`	the template file to write	required
suffix	`str`	suffix to add to parameter names. Only used if longnames=True	''
zn_array	`numpy.ndarray`	an array used to skip inactive cells, and optionally get shape info. zn_array values less than 1 are given fill_value	None
shape	`tuple`	tuple nrow and ncol. Either zn_array or shape must be passed	None
longnames	`bool`	flag to use longer names that exceed 12 chars in length. Default is False.	False
fill_value	`str`	value to fill locations where zn_array is zero or less. Default is "1.0".	'1.0'

Returns:

Type	Description
	pandas.DataFrame: a dataframe with parameter information

Note

This function is used during the PstFrom setup process