Model Drivers

class yggdrasil.drivers.ExecutableModelDriver.ExecutableModelDriver(*args, **kwargs)[source]

Class for running executable based models.

static before_registration(cls)[source]

Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

classmethod executable_command(args, unused_kwargs=None, **kwargs)[source]

Compose a command for running a program using the exectuable for this language (compiler/interpreter) with the provided arguments.

Parameters
  • args (list) – The program that returned command should run and any arguments that should be provided to it.

  • unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to None and is ignored.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Arguments composing the command required to run the program

from the command line using the executable for this language.

Return type

list

classmethod is_configured()[source]

Determine if the appropriate configuration has been performed (e.g. installation of supporting libraries etc.)

Returns

True if the language has been configured.

Return type

bool

classmethod is_language_installed()[source]

Determine if the interpreter/compiler for the associated programming language is installed.

Returns

True if the language interpreter/compiler is installed.

Return type

bool

classmethod language_version(version_flags=None, **kwargs)[source]

Determine the version of this language.

Parameters

**kwargs – Keyword arguments are passed to cls.run_executable.

Returns

Version of compiler/interpreter for this language.

Return type

str

class yggdrasil.drivers.PythonModelDriver.PythonModelDriver(*args, **kwargs)[source]

Class for running Python models.

static finalize_registration(cls)[source]

Operations that should be performed after a class has been fully initialized and registered.

classmethod format_function_param(key, default=None, **kwargs)[source]

Return the formatted version of the specified key.

Parameters
  • key (str) – Key in cls.function_param mapping that should be formatted.

  • default (str, optional) – Format that should be returned if key is not in cls.function_param. Defaults to None.

  • **kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns

Formatted string.

Return type

str

Raises

NotImplementedError – If key is not in cls.function_param and default is not set.

classmethod get_testing_options(**kwargs)[source]

Method to return a dictionary of testing options for this class.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class.

Returns

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type

dict

classmethod install_dependency(package=None, package_manager=None, **kwargs)[source]

Install a dependency.

Parameters
  • package (str) – Name of the package that should be installed. If the package manager supports it, this can include version requirements.

  • package_manager (str, optional) – Package manager that should be used to install the package.

  • **kwargs – Additional keyword arguments are passed to the parent class.

classmethod is_language_installed()[source]

Determine if this model driver is installed on the current machine.

Returns

Truth of if this model driver can be run on the current

machine.

Return type

bool

classmethod is_library_installed(lib, **kwargs)[source]

Determine if a dependency is installed.

Parameters
  • lib (str) – Name of the library that should be checked.

  • **kwargs – Additional keyword arguments are ignored.

Returns

True if the library is installed, False otherwise.

Return type

bool

run_validation()[source]

Run the validation script for the model.

set_env(**kwargs)[source]

Get environment variables that should be set for the model process.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Environment variables for the model process.

Return type

dict

classmethod write_finalize_oiter(var)[source]

Get the lines necessary to finalize an array after iteration.

Parameters

var (dict, str) – Name or information dictionary for the variable being initialized.

Returns

The lines finalizing the variable.

Return type

list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]

Get the lines necessary to initialize an array for iteration output.

Parameters
  • var (dict, str) – Name or information dictionary for the variable being initialized.

  • value (str, optional) – Value that should be assigned to the variable.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

The lines initializing the variable.

Return type

list

class yggdrasil.drivers.MatlabModelDriver.MatlabModelDriver(*args, **kwargs)[source]

Base class for running Matlab models.

Parameters
  • name (str) – Driver name.

  • args (str or list) – Argument(s) for running the model in matlab. Generally, this should be the full path to a Matlab script.

  • use_symunit (bool, optional) – If True, input/output variables with units will be represented in Matlab using symunit. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to parent class’s __init__ method.

started_matlab

True if the driver had to start a new matlab engine. False otherwise.

Type

bool

screen_session

Screen session that Matlab was started in.

Type

str

mlengine

Matlab engine used to run script.

Type

object

mlsession

Name of the Matlab session that was started.

Type

str

Raises

RuntimeError – If Matlab is not installed.

Note

Matlab models that call exit will shut down the shared engine.

after_loop()[source]

Actions to perform after run_loop has finished. Mainly checking if there was an error and then handling it.

static after_registration(cls, **kwargs)[source]

Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

static before_registration(cls)[source]

Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

before_start()[source]

Actions to perform before the run loop.

check_exits()[source]

Check to make sure the program dosn’t contain any exits as exits will shut down the Matlab engine as well as the program.

Raises

RuntimeError – If there are any exit calls in the file.

cleanup()[source]

Close the Matlab session and engine.

classmethod comm_atexit(comm)[source]

Operations performed on comm at exit including draining receive.

Parameters

comm (CommBase) – Communication object.

classmethod configure(cfg, disable_engine=None, hide_matlab_libiomp=None)[source]

Add configuration options for this language. This includes locating any required external libraries and setting option defaults.

Parameters
  • cfg (YggConfigParser) – Config class that options should be set for.

  • disable_engine (bool, optional) – Disable the use of the Python engine for Matlab. If not provided, this argument will be ignored.

  • hide_matlab_libiomp (bool, optional) – Hide the Matlab version of libiomp by moving it and recording the name of the file. If False, the file will be restored if it exists. If not provided, this argument will be ignored.

Returns

Section, option, description tuples for options that could not

be set.

Return type

list

classmethod decode_format(format_str)[source]

Method for decoding format strings created in this language.

Parameters

format_str (str) – Encoded format string.

Returns

Decoded format string.

Return type

str

classmethod executable_command(args, **kwargs)[source]

Compose a command for running a program in this language with the provied arguments. If not already present, the interpreter command and interpreter flags are prepended to the provided arguments.

Parameters
  • args (list) – The program that returned command should run and any arguments that should be provided to it.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Arguments composing the command required to run the program

from the command line using the interpreter for this language.

Return type

list

classmethod get_matlab_info()[source]

Determine the root directory where Matlab is installed and the version that is installed (if Matlab is installed at all). This will fail if Matlab is not installed, cannot be started, or does not operate as expected.

Returns

Matlab root directory and Matlab version string.

Return type

tuple

Raises

RuntimeError – If Matlab cannot be started or the root directory or release cannot be determiend.

classmethod get_testing_options(**kwargs)[source]

Method to return a dictionary of testing options for this class.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class.

Returns

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type

dict

classmethod language_version(skip_config=False)[source]

Determine the version of this language.

Parameters

skip_config (bool, optional) – If True, the config option for the version (if it exists) will be ignored and the version will be determined fresh.

Returns

Version of compiler/interpreter for this language.

Return type

str

parse_arguments(args)[source]

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters

args (list) – List of arguments provided.

classmethod run_code(lines, **kwargs)[source]

Run code by first writing it as an executable and then calling the driver.

Parameters
  • lines (list) – Lines of code to be wrapped as an executable.

  • **kwargs – Additional keyword arguments are passed to the write_executable method.

classmethod run_executable(args, dont_wrap_error=False, fname_wrapper=None, matlab_engine=None, **kwargs)[source]

Run a program using the executable for this language and the provided arguments.

Parameters
  • args (list) – The program that should be run and any arguments that should be provided to it.

  • dont_wrap_error (bool, optional) – If False, the executable will be wrapped in a try/catch block to prevent errors from stopping Matlab shutdown. If True, the command will be executed as is with the Matlab interpreter. Defaults to False.

  • fname_wrapper (str, optional) – File where wrapper should be saved. If not provided, one is created. Defaults to None.

  • matlab_engine (MatlabEngine, optional) – Matlab engine that should be used to run the command. If not provided, the Matlab interpreter is used instead. Defaults to None.

  • **kwargs – Additional keyword arguments are passed to cls.executable_command and tools.popen_nobuffer.

Returns

Output to stdout from the run command.

Return type

str

Raises
run_loop()[source]

Loop to check if model is still running and forward output.

set_env()[source]

Get environment variables that should be set for the model process.

Returns

Environment variables for the model process.

Return type

dict

start_matlab_engine()[source]

Start matlab session and connect to it.

classmethod write_error_wrapper(fname, try_lines, env=None, matlab_engine=None)[source]

Write a wrapper for the model that encloses it in a try except so that the error can be propagated appropriately.

Parameters
  • fname (str) – File where the wrapper should be written.

  • try_lines (list) – List of lines to go in the try block.

  • model_file (str) – Path to model that should be wrapped.

  • env (dict, optional) – Dictionary of environment variables that should be set before calling the model. Defaults to None and is ignored.

  • matlab_engine (MatlabEngine, optional) – Matlab engine that will be used to call the wrapper. If not provided, it is assumed the error will be called using the Matlab interpreter on the command line. Defautls to None.

Raises:

classmethod write_finalize_iiter(var, **kwargs)[source]

Get the lines necessary to initialize an array for iteration output.

Parameters
  • var (dict, str) – Name or information dictionary for the variable being initialized.

  • value (str, optional) – Value that should be assigned to the variable.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

The lines initializing the variable.

Return type

list

classmethod write_finalize_oiter(var)[source]

Get the lines necessary to finalize an array after iteration.

Parameters

var (dict, str) – Name or information dictionary for the variable being initialized.

Returns

The lines finalizing the variable.

Return type

list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]

Get the lines necessary to initialize an array for iteration output.

Parameters
  • var (dict, str) – Name or information dictionary for the variable being initialized.

  • value (str, optional) – Value that should be assigned to the variable.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

The lines initializing the variable.

Return type

list

class yggdrasil.drivers.MakeModelDriver.MakeModelDriver(*args, **kwargs)[source]

Class for running make file compiled drivers. Before running the make command, the necessary compiler & linker flags for the interface’s C/C++ library are stored the environment variables CFLAGS and LDFLAGS respectively. These should be used in the make file to correctly compile with the interface’s C/C++ libraries.

Parameters
  • name (str) – Driver name.

  • args (str, list) – Executable that should be created (make target) and any arguments for the executable.

  • makefile (str, optional) – Path to make file either absolute, relative to makedir (if provided), or relative to working_dir. Defaults to Makefile.

  • makedir (str, optional) – Directory where make should be invoked from if it is not the same as the directory containing the makefile. Defaults to directory containing makefile if provided, otherwise working_dir.

  • **kwargs – Additional keyword arguments are passed to parent class.

makefile

Path to make file either relative to makedir or absolute.

Type

str

makedir

Directory where make should be invoked from.

Type

str

Raises

RuntimeError – If neither the IPC or ZMQ C libraries are available.

static before_registration(cls)[source]

Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

compile_model(target=None, **kwargs)[source]

Compile model executable(s).

Parameters
  • target (str, optional) – Target to build.

  • **kwargs – Keyword arguments are passed on to the parent class’s method.

classmethod fix_path(path, for_env=False, **kwargs)[source]

Update a path.

Parameters
  • path (str) – Path that should be formatted.

  • for_env (bool, optional) – If True, the path is formatted for use in and environment variable. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Updated path.

Return type

str

classmethod get_buildfile_lock(**kwargs)[source]

Get a lock for a buildfile to prevent simultaneous access, creating one as necessary.

classmethod get_language_for_buildfile(buildfile, target=None)[source]

Determine the target language based on the contents of a build file.

Parameters
  • buildfile (str) – Full path to the build configuration file.

  • target (str, optional) – Target that will be built. Defaults to None and the default target in the build file will be used.

classmethod get_target_language_info(*args, **kwargs)[source]

Get a dictionary of information about language compilation tools.

Parameters
  • *args – Arguments are passed to the parent class’s method.

  • **kwargs – Keyword arguments are passed to the parent class’s method.

Returns

Information about language compilers and linkers.

Return type

dict

parse_arguments(args, **kwargs)[source]

Sort arguments based on their syntax to determine if an argument is a source file, compilation flag, or runtime option/flag that should be passed to the model executable.

Parameters
  • args (list) – List of arguments provided.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

class yggdrasil.drivers.CMakeModelDriver.CMakeModelDriver(*args, **kwargs)[source]

Class for running cmake compiled drivers. Before running the cmake command, the cmake commands for setting the necessary compiler & linker flags for the interface’s C/C++ library are written to a file called ‘ygg_cmake.txt’ that should be included in the CMakeLists.txt file (after the target executable has been added).

Parameters
  • name (str) – Driver name.

  • args (str, list) – Executable that should be created (cmake target) and any arguments for the executable.

  • sourcedir (str, optional) – Source directory to call cmake on. If not provided it is set to working_dir. This should be the directory containing the CMakeLists.txt file. It can be relative to working_dir or absolute.

  • builddir (str, optional) – Directory where the build should be saved. Defaults to <sourcedir>/build. It can be relative to working_dir or absolute.

  • configuration (str, optional) – Build type/configuration that should be built. Defaults to ‘Release’.

  • **kwargs – Additional keyword arguments are passed to parent class.

sourcedir

Source directory to call cmake on.

Type

str

add_libraries

If True, interface libraries and dependency libraries are added using CMake’s ADD_LIBRARY directive. If False, interface libraries are found using FIND_LIBRARY.

Type

bool

configuration

Build type/configuration that should be built. This is only used on Windows.

Type

str

Raises

RuntimeError – If neither the IPC or ZMQ C libraries are available.

property buildfile_orig

Full path to where the original CMakeLists.txt file will be stored during compilation of the modified file.

Type

str

property buildfile_ygg

Full path to the verison of the CMakeLists.txt that has been updated w/ yggdrasil compilation flags.

Type

str

compile_model(target=None, **kwargs)[source]

Compile model executable(s) and appends any products produced by the compilation that should be removed after the run is complete.

Parameters
  • target (str, optional) – Target to build.

  • **kwargs – Keyword arguments are passed on to the call_compiler method.

classmethod fix_path(path, for_env=False, **kwargs)[source]

Update a path.

Parameters
  • path (str) – Path that should be formatted.

  • for_env (bool, optional) – If True, the path is formatted for use in and environment variable. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Updated path.

Return type

str

classmethod get_language_for_buildfile(buildfile, target=None)[source]

Determine the target language based on the contents of a build file.

Parameters
  • buildfile (str) – Full path to the build configuration file.

  • target (str, optional) – Target that will be built. Defaults to None and the default target in the build file will be used.

classmethod get_target_language_info(target_compiler_flags=None, target_linker_flags=None, compiler_flag_kwargs=None, linker_flag_kwargs=None, without_wrapper=False, **kwargs)[source]

Get a dictionary of information about language compilation tools.

Parameters
  • target_compiler_flags (list, optional) – Compilation flags that should be passed to the target language compiler. Defaults to [].

  • target_linker_flags (list, optional) – Linking flags that should be passed to the target language linker. Defaults to [].

  • compiler_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_compiler_flags method. Defaults to None.

  • linker_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_linker_flags method. Defaults to None.

  • **kwargs – Keyword arguments are passed to the parent class’s method.

Returns

Information about language compilers and linkers.

Return type

dict

parse_arguments(args, **kwargs)[source]

Sort arguments based on their syntax to determine if an argument is a source file, compilation flag, or runtime option/flag that should be passed to the model executable.

Parameters
  • args (list) – List of arguments provided.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

classmethod prune_sh_gcc(path, gcc)[source]

Remove instances of sh.exe from the path that are not associated with the selected gcc compiler. This can happen on windows when rtools or git install a version of sh.exe that is added to the path before the compiler.

Parameters
  • path (str) – Contents of the path variable.

  • gcc (str) – Full path to the gcc executable.

Returns

Modified path that removes the extra instances

of sh.exe.

Return type

str

classmethod update_compiler_kwargs(**kwargs)[source]

Update keyword arguments supplied to the compiler get_flags method for various options.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Keyword arguments for a get_flags method providing compiler

flags.

Return type

dict

write_wrappers(**kwargs)[source]

Write any wrappers needed to compile and/or run a model.

Parameters

**kwargs – Keyword arguments are passed to the parent class’s method.

Returns

Full paths to any created wrappers.

Return type

list

class yggdrasil.drivers.LPyModelDriver.LPyModelDriver(*args, **kwargs)[source]

Class for running LPy models.

function_param = None
classmethod get_testing_options(**kwargs)[source]

Method to return a dictionary of testing options for this class.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class.

Returns

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type

dict

classmethod language_version(**kwargs)[source]

Determine the version of this language.

Parameters

**kwargs – Keyword arguments are passed to cls.run_executable.

Returns

Version of compiler/interpreter for this language.

Return type

str

Model Base Classes

class yggdrasil.drivers.InterpretedModelDriver.InterpretedModelDriver(*args, **kwargs)[source]

Base class for models written in interpreted languages.

Parameters
  • name (str) – Driver name.

  • args (str or list) – Argument(s) for running the model on the command line. This driver will check to see if there is an interpreter included in the args provided. If not, one will be added unless skip_interpreter is True.

  • interpreter (str, optional) – Name or path of interpreter executable that should be used to run the model. If not provided, the interpreter will be determined based on configuration options for the language (if present) and the default_interpreter class attribute.

  • interpreter_flags (list, optional) – Flags that should be passed to the interpreter when running the model. If not provided, the flags are determined based on configuration options for the language (if present) and the default_interpreter_flags class attribute.

  • skip_interpreter (bool, optional) – If True, no interpreter will be added to the arguments. This should only be used for subclasses that will not be invoking the model via the command line. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to the parent class.

Class Attributes:
default_interpreter (str): Name of interpreter that will be used if not

set explicitly by instance or config file. Defaults to the language name if not set.

default_interpreter_flags (list): Flags that will be passed to the

interpreter when running the model by default if not set explicitly by instance or config file.

interpreter

Name or path to the interpreter that will be used.

Type

str

interpreter_flags

Flags that will be passed to the interpreter when running a model.

Type

list

path_env_variable

Name of the environment variable containing path information for the interpreter for this language.

Type

str

paths_to_add

Paths that should be added to the path_env_variable for this language on the process the model is run in.

Type

list

comm_atexit

Function taking a comm instance as input that performs any necessary operations during exit. If None, no additional actions are taken.

Type

function

comm_linger

If True, interface comms will linger during close. This should only be required if the language will disreguard Python threads at exit (e.g. when using a Matlab engine).

Type

bool

decode_format (function

Function decoding format string created in this language. If None, no additional actions are taken.

recv_converters

Mapping between the names of message types (e.g. ‘array’, ‘pandas’) and functions that should be used to prepare such objects for return when they are received.

Type

dict

send_converters

Mapping between the names of message types (e.g. ‘array’, ‘pandas’) and functions that should be used to prepare such objects for sending.

Type

dict

static after_registration(cls, **kwargs)[source]

Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

classmethod executable_command(args, exec_type='interpreter', unused_kwargs=None, skip_interpreter_flags=False, interpreter=None, interpreter_flags=None, **kwargs)[source]

Compose a command for running a program in this language with the provied arguments. If not already present, the interpreter command and interpreter flags are prepended to the provided arguments.

Parameters
  • args (list) – The program that returned command should run and any arguments that should be provided to it.

  • exec_type (str, optional) – Type of executable command that will be returned. If ‘interpreter’, a command using the interpreter is returned and if ‘direct’, the raw args being provided are returned. Defaults to ‘interpreter’.

  • skip_interpreter_flags (bool, optional) – If True, interpreter flags will not be added to the command after the interpreter. Defaults to False. Interpreter flags will not be added, reguardless of this keyword, if the first element of args is already an interpreter.

  • unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to {}.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Arguments composing the command required to run the program

from the command line using the interpreter for this language.

Return type

list

Raises

ValueError – If exec_type is not ‘interpreter’ or ‘direct’.

classmethod get_interpreter()[source]

Command required to run a model written in this language from the command line.

Returns

Name of (or path to) interpreter executable.

Return type

str

classmethod get_interpreter_flags()[source]

Get the flags that should be passed to the interpreter when using it to run a model on the command line.

Returns

Flags that should be passed to the interpreter on the command

line.

Return type

list

classmethod is_interpreter(cmd)[source]

Determine if a command line argument is an interpreter.

Parameters

cmd (str) – Command that should be checked.

Returns

True if the command is an interpreter, False otherwise.

Return type

bool

classmethod language2python(pyobj)[source]

Prepare an object from the target language for receipt in Python.

Parameters

pyobj (object) – Python object transformed from the target language.

Returns

Python object in a form that conforms with the

expected Python type.

Return type

object

classmethod language_executable()[source]

Command required to compile/run a model written in this language from the command line.

Returns

Name of (or path to) compiler/interpreter executable required

to run the compiler/interpreter from the command line.

Return type

str

parse_arguments(*args, **kwargs)[source]

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters
  • *args – Arguments are passed to the parent class’s method.

  • **kwargs – Keyword arguments are passed to the parent class’s method.

classmethod python2language(pyobj)[source]

Prepare a python object for transformation in target language.

Parameters

pyobj (object) – Python object.

Returns

Python object in a form that is friendly to the

target language.

Return type

object

run_model(*args, **kwargs)[source]

Run the model. Unless overridden, the model will be run using run_executable.

Parameters
  • *args – Arguments are passed to the parent class’s method.

  • **kwargs – Keyword arguments are passed to the parent class’s method.

set_env(**kwargs)[source]

Get environment variables that should be set for the model process.

Returns

Environment variables for the model process.

Return type

dict

class yggdrasil.drivers.CompiledModelDriver.CompiledModelDriver(*args, **kwargs)[source]

Base class for models written in compiled languages.

Parameters
  • name (str) – Driver name.

  • args (str or list) – The model executable and any arguments that should be passed to the model executable.

  • source_files (list, optional) – Source files that should be compiled into an executable. Defaults to an empty list and the driver will search for a source file based on the model executable (the first model argument).

  • compiler (str, optional) – Command or path to executable that should be used to compile the model. If not provided, the compiler will be determined based on configuration options for the language (if present) and the registered compilers that are available on the current operating system.

  • compiler_flags (list, optional) – Flags that should be passed to the compiler during compilation. If nto provided, the compiler flags will be determined based on configuration options for the language (if present), the compiler defaults, and the default_compiler_flags class attribute.

  • linker (str, optional) – Command or path to executable that should be used to link the model. If not provided, the linker will be determined based on configuration options for the language (if present) and the registered linkers that are available on the current operating system

  • linker_flags (list, optional) – Flags that should be passed to the linker during compilation. If nto provided, the linker flags will be determined based on configuration options for the language (if present), the linker defaults, and the default_linker_flags class attribute.

  • **kwargs – Additional keyword arguments are passed to parent class.

Class Attributes:
default_compiler (str): Name of compiler that will be used if not

set explictly by instance or config file.

default_compiler_flags (list): Flags that should be passed to the

compiler by default for this language.

default_linker (str): Name of linker that will be used if not

set explictly by instance or config file

default_linker_flags (list): Flags that should be passed to the

linker by default for this language.

allow_parallel_build (bool): If True, a file can be compiled by

two processes simultaneously. If False, it cannot and an MPI barrier will be used to prevent simultaneous compilation. Defaults to False.

source_files

Source files.

Type

list

compiler

Name or path to the compiler that should be used.

Type

str

compiler_flags

Compiler flags.

Type

list

linker

Name or path to the linker that should be used.

Type

str

linker_flags

Linker flags.

Type

list

compiler_tool

Compiler tool that will be used.

Type

CompilerBase

linker_tool

Linker tool that will be used.

Type

LinkerBase

static after_registration(cls, **kwargs)[source]

Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

buildfile_locked(dry_run=False)[source]

Context manager for locked build file.

classmethod call_compiler(src, language=None, toolname=None, dont_build=None, **kwargs)[source]

Compile a source file into an executable or linkable object file, checking for errors.

Parameters
  • src (str) – Full path to source file.

  • out (str, optional) – Full path to the output object file that should be created. Defaults to None and is created from the provided source file.

  • flags (list, optional) – Compilation flags. Defaults to []. If compiler_flags is present, flags is replaced by compiler_flags.

  • compiler_flags (list, optional) – Alternative to flags. Ignored if not provided.

  • dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable. Defaults to False.

  • dont_build (bool, optional) – If True, cmake configuration/generation will be run, but the project will not be built. Defaults to None. If provided, this overrides dont_link.

  • overwrite (bool, optional) – If True, the existing compile file will be overwritten. Otherwise, it will be kept and this function will return without recompiling the source file.

  • language (str, optional) – Language that should be used to compile the files. Defaults to None and the language of the current driver is used.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • products (list, optional) – Existing Python list that additional products produced by the compilation should be appended to. Defaults to None and is ignored.

  • **kwargs – Additional keyword arguments are passed to run_executable. and call_linker if dont_link is False.

Returns

Full path to compiled source.

Return type

str

Raises
  • RuntimeError – If there is an error in calling the compiler.

  • RuntimeError – If the compilation command dosn’t yield the specified output file.

classmethod call_linker(obj, language=None, toolname=None, **kwargs)[source]

Link several object files to create an executable or library (shared or static), checking for errors.

Parameters
  • obj (list) – Object files that should be linked.

  • language (str, optional) – Language that should be used to link the files. Defaults to None and the language of the current driver is used.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • **kwargs – Additional keyword arguments are passed to run_executable.

Returns

Full path to compiled source.

Return type

str

classmethod cleanup_dependencies(products=None, verbose=False, **kwargs)[source]

Cleanup dependencies.

classmethod compile_dependencies(toolname=None, dep=None, **kwargs)[source]

Compile any required internal libraries, including the interface.

compile_dependencies_instance(*args, **kwargs)[source]

Compile dependencies specifically for this instance.

compile_model(source_files=None, skip_interface_flags=False, **kwargs)[source]

Compile model executable(s).

Parameters
  • source_files (list, optional) – Source files that will be compiled. Defaults to None and is set to the source_files attribute.

  • skip_interface_flags (bool, optional) – If True, interface flags will not be added. This includes the logger flag specifying the current logging level. Defaults to False.

  • **kwargs – Keyword arguments are passed on to the call_compiler method.

Returns

Compiled model file path.

Return type

str

classmethod configure(cfg, **kwargs)[source]

Add configuration options for this language.

Parameters
  • cfg (CisConfigParser) – Config class that options should be set for.

  • **kwargs – Additional keyword arguments are used to set tool configuration options (e.g. ‘compiler’).

Returns

Section, option, description tuples for options that could not

be set.

Return type

list

classmethod configure_executable_type(cfg)[source]

Add configuration options specific in the executable type before the libraries are configured.

Parameters

cfg (CisConfigParser) – Config class that options should be set for.

Returns

Section, option, description tuples for options that could not

be set.

Return type

list

classmethod configure_libraries(cfg)[source]

Add configuration options for external libraries in this language.

Parameters

cfg (CisConfigParser) – Config class that options should be set for.

Returns

Section, option, description tuples for options that could not

be set.

Return type

list

classmethod configure_library(cfg, k)[source]

Add configuration options for an external library.

Parameters

cfg (YggConfigParser) – Config class that options should be set for.

Returns

Section, option, description tuples for options that could not

be set.

Return type

list

classmethod executable_command(args, exec_type='compiler', toolname=None, **kwargs)[source]

Compose a command for running a program using the compiler for this language and the provied arguments. If not already present, the compiler command and compiler flags are prepended to the provided arguments.

Parameters
  • args (list) – The program that returned command should run and any arguments that should be provided to it. For the compiler, this means the source files, for the linker, this means the object files.

  • exec_type (str, optional) – Type of executable command that will be returned. If ‘compiler’, a command using the compiler is returned, if ‘linker’, a command using the linker is returned, and if ‘direct’, the raw args being provided are returned. Defaults to ‘compiler’.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • **kwargs – Additional keyword arguments are passed to either get_linker_flags or get_compiler_flags.

Returns

Arguments composing the command required to run the program

from the command line using the compiler for this language.

Return type

list

Raises

ValueError – If exec_type is not ‘compiler’, ‘linker’, or ‘direct’.

classmethod get_available_tools(tooltype)[source]

Return the registry of compilation tools for this language.

Parameters

tooltype (str) – Name of tool type that should be returned. Values include ‘compiler’, ‘linker’, or ‘archiver’.

Returns

Registry of tools for this language.

Return type

dict

classmethod get_buildfile_lock(fname=None, context=None, instance=None, **kwargs)[source]

Get a lock for a buildfile to prevent simultaneous access, creating one as necessary.

Parameters
  • name (str) – Build file.

  • context (threading.Context) – Threading context.

  • instance (ModelDriver) – Driver instance that should be used.

  • **kwargs – Additional keyword arguments are passed to the FileLock initialization.

Returns

Lock for the buildfile.

Return type

FileLock

classmethod get_compiler_flags(toolname=None, compiler=None, **kwargs)[source]

Determine the flags required by the current compiler.

Parameters
  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • compiler (CompilerBase, optional) – Compilation tool class for the compiler that should be used. Defaults to None and is set based on toolname.

  • **kwargs – Keyword arguments are passed to cls.update_compiler_kwargs first and then the compiler’s get_flags method.

Returns

Flags for the compiler.

Return type

list

classmethod get_dependency_include_dirs(dep, toolname=None, default=None)[source]

Get the include directories for a dependency.

Parameters
  • dep (str) – Name of internal or external dependency or full path to the library.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

Returns

Full paths to the directories containing the dependency’s

header(s).

Return type

list

Raises

ValueError – If the include directory cannot be determined for the specified dependency and default is None.

classmethod get_dependency_info(dep, toolname=None, default=None)[source]

Get the dictionary of information associated with a dependency.

Parameters
  • dep (str) – Name of internal or external dependency.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • default (dict, optional) – Information dictionary that should be returned if dep cannot be located. Defaults to None and an error will be raised if dep cannot be found.

Returns

Dependency info.

Return type

dict

classmethod get_dependency_library(dep, default=None, libtype=None, commtype=None, toolname=None)[source]

Get the library location for a dependency.

Parameters
  • dep (str) – Name of internal or external dependency or full path to the library.

  • default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

  • libtype (str, optional) – Library type that should be returned. Valid values are ‘static’ and ‘shared’. Defaults to None and will be set on the dependency’s libtype (if it has one) or the _default_libtype parameter (if it doesn’t).

  • commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used in the names of internal libraries.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns

Full path to the library file. For header only libraries,

an empty string will be returned.

Return type

str

Raises
  • ValueError – If libtype is not ‘static’ or ‘shared’.

  • ValueError – If the path to the library cannot be determined for the specified dependency and default is None.

classmethod get_dependency_object(dep, default=None, commtype=None, toolname=None)[source]

Get the location of an object file for a dependency.

Parameters
  • dep (str) – Name of internal or external dependency or full path to the object file.

  • default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

  • commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used in the names of internal libraries.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns

Full path to the object file.

Return type

str

classmethod get_dependency_order(deps, toolname=None)[source]

Get the correct dependency order, including any dependencies for the direct dependencies.

Parameters
  • deps (list) – Dependencies in order.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns

Dependency order.

Return type

list

classmethod get_dependency_source(dep, toolname=None, default=None)[source]

Get the path to the library source files (or header files) for a dependency.

Parameters
  • dep (str) – Name of internal or external dependency or full path to the library.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

Returns

Full path to the library source file. For header only libraries

this will be the header location.

Return type

str

classmethod get_external_libraries(no_comm_libs=False)[source]

Determine the external libraries that are required based on the default comm.

Parameters

no_comm_libs (bool, optional) – If True, libraries for the installed comms are not included in the returned list. Defaults to False.

Returns

The names of external libraries required by the interface

library, including the dependency libraries for the installed comm libraries.

Return type

list

classmethod get_internal_suffix(commtype=None)[source]

Determine the suffix that should be used for internal libraries.

Parameters

commtype (str, optional) – If provided, this is the communication type that should be used for the model. If None, the default comm is used.

Returns

Suffix that should be added to internal libraries to

differentiate between different dependencies.

Return type

str

classmethod get_linker_flags(toolname=None, **kwargs)[source]

Determine the flags required by the current linker.

Parameters
  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • **kwargs – Keyword arguments are passed to cls.update_linker_kwargs first and then the linker’s get_flags method.

Returns

Flags for the linker.

Return type

list

classmethod get_testing_options(**kwargs)[source]

Method to return a dictionary of testing options for this class.

Parameters

**kwargs – Additional keyword arguments are passed to the parent class.

Returns

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type

dict

classmethod get_tool(*args, **kwargs)[source]

Get tool from a driver class.

Parameters
  • *args – Arguments are passed to the get_tool_static method.

  • **kwargs – Keyword arguments are passed to the get_tool_static method.

Returns

Class providing an interface to the specified

compilation tool.

Return type

CompilationToolBase

get_tool_instance(*args, **kwargs)[source]

Get tool from a driver instance.

Parameters
  • *args – Arguments are passed to the get_tool_static method.

  • **kwargs – Keyword arguments are passed to the get_tool_static method.

Returns

Class providing an interface to the specified

compilation tool.

Return type

CompilationToolBase

static get_tool_static(cls, tooltype, toolname=None, return_prop='tool', default=False, language=None)[source]

Get the class associated with the specified compilation tool for this language.

Parameters
  • cls (class, instance) – Compiled driver class or instance of compiled driver class to get tool for.

  • tooltype (str) – Type of compilation tool that should be returned.

  • toolname (str, optional) – Name of the tool that should be returned. Defaults to None and the tool name associated with the provided class/instance will be used.

  • return_prop (str, optional) – Value that should be returned. If ‘tool’, the tool is returned. If ‘name’, the tool name is returned. If ‘flags’, the tool flags are returned. Defaults to ‘tool’.

  • default (object, optiona) – Tool that should be returned if one cannot be identified. If False, an error will be raised when a tool cannot be located. Defaults to False.

  • language (str, optional) – Language of tools that should be returned. Defaults to None if not provided.

Returns

Class providing an interface to the specified

compilation tool.

Return type

CompilationToolBase

Raises
classmethod identify_source_files(args=None, working_dir=None, source_files=None, **kwargs)[source]

Determine the source file based on model arguments.

Parameters
  • args (list, optional) – Arguments provided.

  • working_dir (str, optional) – Working directory.

  • source_files (list, optional) – Source files in the model.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Source files.

Return type

list

classmethod is_configured()[source]

Determine if the appropriate configuration has been performed (e.g. installation of supporting libraries etc.)

Returns

True if the language has been configured.

Return type

bool

classmethod is_language_installed()[source]

Determine if the interpreter/compiler for the associated programming language is installed.

Returns

True if the language interpreter/compiler is installed.

Return type

bool

classmethod is_library_installed(lib, cfg=None)[source]

Determine if a dependency is installed by check for the appropriate config options setting the path to the library files.

Parameters
  • lib (str) – Name of the library that should be checked.

  • cfg (CisConfigParser, optional) – Config class that should be checked. Defaults to yggdrasil.config.ygg_cfg if not provided.

Returns

True if the library is installed, False otherwise.

Return type

bool

classmethod is_tool_installed(tooltype)[source]

Determine if a compilation tool of a certain is installed for this language.

Parameters

tooltype (str) – Type of tool to check for. Supported values include ‘compiler’, ‘linker’, & ‘archiver’.

Returns

True if a tool of the specified type is installed.

Return type

bool

classmethod language_executable(toolname=None)[source]

Command required to compile/run a model written in this language from the command line.

Parameters

toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns

Name of (or path to) compiler/interpreter executable required

to run the compiler/interpreter from the command line.

Return type

str

classmethod language_version(toolname=None, **kwargs)[source]

Determine the version of this language.

Parameters
  • **kwargs – Keyword arguments are passed to cls.run_executable.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns

Version of compiler/interpreter for this language.

Return type

str

model_command()[source]

Return the command that should be used to run the model.

Returns

Any commands/arguments needed to run the model from the

command line.

Return type

list

classmethod mpi_partner_cleanup(self)[source]

Actions cleaning up an MPIPartnerModel.

classmethod mpi_partner_init(self)[source]

Actions initializing an MPIPartnerModel.

parse_arguments(args, **kwargs)[source]

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters
  • args (list) – List of arguments provided.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

classmethod partner_buildfile_lock(self)[source]

Actions completing buildfile lock on MPIPartnerModels.

run_model(**kwargs)[source]

Run the model. Unless overridden, the model will be run using run_executable.

Parameters

**kwargs – Keyword arguments are passed to run_executable.

set_env(for_compile=False, compile_kwargs=None, toolname=None, **kwargs)[source]

Get environment variables that should be set for the model process.

Parameters
  • for_compile (bool, optional) – If True, environment variables are set that are necessary for compiling. Defaults to False.

  • compile_kwargs (dict, optional) – Keyword arguments that should be passed to the compiler’s set_env method. Defaults to empty dict.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Environment variables for the model process.

Return type

dict

classmethod set_env_compiler(compiler=None, toolname=None, **kwargs)[source]

Get environment variables that should be set for the compilation process.

Parameters
  • compiler (CompilerBase, optional) – Compiler that set_env shoudl be called for. If not provided, the default compiler for this language will be used.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • **kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns

Environment variables for the model process.

Return type

dict

set_target_language()[source]

Set the language of the target being compiled (usually the same as the language associated with this driver.

Returns

Name of language.

Return type

str

classmethod update_compiler_kwargs(for_api=False, for_model=False, commtype=None, toolname=None, directory=None, include_dirs=None, definitions=None, skip_interface_flags=False, **kwargs)[source]

Update keyword arguments supplied to the compiler get_flags method for various options.

Parameters
  • dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable. Defaults to False.

  • for_api (bool, optional) – If True, flags are added that are required for compiling internal api libraries in this language. This includes external communication libraries. Defaults to False.

  • for_model (bool, optional) – If True, flags are added that are required for including the interface library. Defaults to False.

  • commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used if for_model is True.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • include_dirs (list, optional) – If provided, each list element will be added as an included directory flag. Defaults to None and is initialized as an empty list.

  • definitions (list, optional) – If provided, each list element will be added as a defintion. Defaults to None and is initialized to an empty list.

  • skip_interface_flags (bool, optional) – If True, interface flags will not be added. Defaults to False.

  • internal_dependencies (list, optional) – If provided, a list of names of internal libraries that are required or linkable object files for dependencies. Defaults to an empty list.

  • external_dependencies (list, optional) – If provided, a list of names of external libraries that are required or linkable object files for dependencies. Defaults to an empty list.

  • **kwargs – Additional keyword arguments are passed to the compiler class’s ‘get_flags’ method and get_linker_flags if dont_link is False.

Returns

Keyword arguments for a get_flags method providing compiler

flags.

Return type

dict

classmethod update_linker_kwargs(for_api=False, for_model=False, commtype=None, toolname=None, libtype='object', skip_interface_flags=False, use_library_path_internal=False, **kwargs)[source]

Update keyword arguments supplied to the linker/archiver get_flags method for various options.

Parameters
  • for_api (bool, optional) – If True, flags are added that are required for linking internal api libraries in this language. This includes external communication libraries. Defaults to False.

  • for_model (bool, optional) – If True, flags are added that are required for including the interface library. Defaults to False.

  • commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used if for_model is True.

  • toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

  • libtype (str, optional) – Library type that should be created by the linker/archiver. Valid values are ‘static’, ‘shared’, or ‘object’. Defaults to ‘object’.

  • skip_interface_flags (bool, optional) – If True, interface flags will not be added. Defaults to False.

  • libraries (list, optional) – Full paths to libraries that should be linked against. Defaults to an empty list.

  • internal_dependencies (list, optional) – If provided, a list of names of internal libraries that are required or linkable object files for dependencies. Defaults to an empty list.

  • external_dependencies (list, optional) – If provided, a list of names of external libraries that are required or linkable object files for dependencies. Defaults to an empty list.

  • use_library_path_internal (bool, optional) – If True, internal dependencies are included as full paths. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to the linker (or archiver if static is True) ‘get_flags’ method.

Returns

Keyword arguments for a get_flags method providing linker/

archiver flags.

Return type

dict

write_wrappers(**kwargs)[source]

Write any wrappers needed to compile and/or run a model.

Parameters

**kwargs – Keyword arguments are passed to the parent class’s method.

Returns

Full paths to any created wrappers.

Return type

list