os_utils

Operating system utilities in the PEST(++) realm

PortManager

Bases: object

Cross-platform port manager for parallel processes.

__init__(port_range=(4004, 4999), lock_dir=None, max_retries=50, lock_timeout=5, log_level=logging.INFO)

Initialize the port manager. Args: port_range: Tuple of (min_port, max_port) to search within lock_dir: Directory to store lock files (default: system temp dir) max_retries: Maximum attempts to find an available port lock_timeout: Time in seconds after which a lock is considered stale

get_available_port()

Find and reserve an available port. Returns: An available port number. Raises: RuntimeError: If no available port can be found after max_retries.

reserved_port()

Context manager that reserves a port and releases it after use.

PyPestWorker

Bases: object

a pure python worker for pest++. the pest++ master doesnt even know...

Parameters:

Name	Type	Description	Default
pst	str or Pst	something about a control file	required
host	str	master hostname or IPv4 address	required
port	int	port number that the master is listening on	required
timeout	float	number of seconds to sleep at different points in the process. if you have lots of pars and/obs, a longer sleep can be helpful, but if you make this smaller, the worker responds faster...'it depends'	0.25
verbose	bool	flag to echo what's going on to stdout	True
socket_timeout	float	number of seconds that the socket should wait before giving up. generally, this can be a big number...	None

run(cmd_str, cwd='.', verbose=False, use_sp=False, **kwargs)

main run function so both run_sp and run_ossystem can coexist

Parameters:

Name	Type	Description	Default
cmd_str	`str`	the str to execute with os.system()	required
cwd	`str`	the directory to execute the command in. Default is ".".	'.'
verbose	`bool`	flag to echo to stdout the cmd_str. Default is False.	False

Notes: by default calls run_ossystem which is the OG function from pyemu that uses os.system() if use_sp is True, then run_sp is called which uses subprocess.Popen instead of os.system

Example:: pyemu.os_utils.run("pestpp-ies my.pst",cwd="template")

run_ossystem(cmd_str, cwd='.', verbose=False)

an OS agnostic function to execute a command line

Parameters:

Name	Type	Description	Default
cmd_str	`str`	the str to execute with os.system()	required
cwd	`str`	the directory to execute the command in. Default is ".".	'.'
verbose	`bool`	flag to echo to stdout the cmd_str. Default is False.	False

Notes

uses platform to detect OS and adds .exe suffix or ./ prefix as appropriate if os.system returns non-zero, an exception is raised

Example::

pyemu.os_utils.run("pestpp-ies my.pst",cwd="template")

run_sp(cmd_str, cwd='.', verbose=True, logfile=False, **kwargs)

an OS agnostic function to execute a command line with subprocess

Parameters:

Name	Type	Description	Default
cmd_str	`str`	the str to execute with sp.Popen()	required
cwd	`str`	the directory to execute the command in. Default is ".".	'.'
verbose	`bool`	flag to echo to stdout the cmd_str. Default is False.	True
shell	`bool`	flag to use shell=True in the subprocess.Popen call. Not recommended	required

Notes

uses sp Popen to execute the command line. By default does not run in shell mode (ie. does not look for the exe in env variables)

start_workers(worker_dir, exe_rel_path, pst_rel_path, num_workers=None, worker_root='..', port=None, rel_path=None, local=True, cleanup=True, master_dir=None, verbose=False, silent_master=False, reuse_master=False, restart=False, ppw_function=None, ppw_kwargs={})

start a group of pest(++) workers on the local machine

Parameters:

Name	Type	Description	Default
worker_dir	`str`	the path to a complete set of input files need by PEST(++). This directory will be copied to make worker (and optionally the master) directories	required
exe_rel_path	`str`	the relative path to and name of the pest(++) executable from within the worker_dir. For example, if the executable is up one directory from worker_dir, the exe_rel_path would be os.path.join("..","pestpp-ies")	required
pst_rel_path	`str`	the relative path to and name of the pest control file from within worker_dir.	required
num_workers	`int`	number of workers to start. defaults to number of cores	None
worker_root	`str`	the root directory to make the new worker directories in. Default is ".." (up one directory from where python is running).	'..'
rel_path	`str`	the relative path to where pest(++) should be run from within the worker_dir, defaults to the uppermost level of the worker dir. This option is usually not needed unless you are one of those crazy people who spreads files across countless subdirectories.	None
local	`bool`	flag for using "localhost" instead of actual hostname/IP address on worker command line. Default is True. local can also be passed as an str, in which case local is used as the hostname (for example local="192.168.10.1")	True
cleanup	`bool`	flag to remove worker directories once processes exit. Default is True. Set to False for debugging issues	True
master_dir	`str`	name of directory for master instance. If master_dir exists, then it will be REMOVED!!! If master_dir, is None, no master instance will be started. If not None, a copy of worker_dir will be made into master_dir and the PEST(++) executable will be started in master mode in this directory. Default is None	None
verbose	`bool`	flag to echo useful information to stdout. Default is False	False
silent_master	`bool`	flag to pipe master output to devnull and instead print a simple message to stdout every few seconds. This is only for pestpp Travis testing so that log file sizes dont explode. Default is False	False
reuse_master	`bool`	flag to use an existing master_dir as is - this is an advanced user option for cases where you want to construct your own master_dir then have an async process started in it by this function.	False
restart	`bool`	flag to add a restart flag to the master start. If True, this will include /r in the master call string.	False
ppw_function	function	a function pointer that uses PyPestWorker to execute model runs. This is to help speed up pure-python and really fast running forward models and the way its implemented in start_workers assumes each PyPestWorker instance does not need a separate set of model+files, that is, these workers are assumed to execute entirely in memory. The first three arguments to this function must be pst_rel_path, host, and port. Default is None.	None
ppw_kwargs	dict	keyword arguments to pass to ppw_function. Default is empty dict.	{}

Notes

If all workers (and optionally master) exit gracefully, then the worker dirs will be removed unless cleanup is False

Example::

# start 10 workers using the directory "template" as the base case and
# also start a master instance in a directory "master".
pyemu.helpers.start_workers("template","pestpp-ies","pest.pst",10,master_dir="master",
                            worker_root=".")