Python Code

Submodules

yggdrasil.command_line module

class yggdrasil.command_line.ArgumentBase(arguments=None, conditions=None, **kwargs)[source]

Bases: object

class yggdrasil.command_line.ArgumentGroup(exclusive=False, **kwargs)[source]

Bases: yggdrasil.command_line.ArgumentBase

class yggdrasil.command_line.ArgumentParser(arguments=None, conditions=None, **kwargs)[source]

Bases: yggdrasil.command_line.ArgumentBase

class yggdrasil.command_line.ArgumentSubparser(parsers=None, **kwargs)[source]

Bases: yggdrasil.command_line.ArgumentBase

class yggdrasil.command_line.ArgumentTuple(args, kwargs)[source]

Bases: tuple

class yggdrasil.command_line.ConditionalArgumentTuple(args, kwargs, conditions=None)[source]

Bases: yggdrasil.command_line.ArgumentTuple

yggdrasil.command_line.ReplacementWarning(old, new)[source]
class yggdrasil.command_line.SubCommand(*args, **kwargs)[source]

Bases: object

Class for handling subcommands so that they can be run as subcommands or individually.

classmethod add_argument_to_parser(parser, x)[source]
classmethod add_arguments(parser, args=None)[source]
classmethod add_subparser(subparsers, args=None)[source]
allow_unknown = False
arguments = []
classmethod call(args=None, namespace=None, **kwargs)[source]
classmethod func(args)[source]
classmethod get_parser(args=None)[source]
help = None
name = None
classmethod parse_args(parser, args=None, allow_unknown=False, namespace=None)[source]
class yggdrasil.command_line.SubCommandMeta[source]

Bases: type

Meta class for subcommands.

class yggdrasil.command_line.cc_flags(*args, **kwargs)[source]

Bases: yggdrasil.command_line.cc_toolname

Get the compiler flags necessary for including the interface library in a C or C++ program.

arguments = [(('--cpp',), {'action': 'store_true', 'help': 'Get the compilation flags used for C++ programs.'}), (('--toolname',), {'default': None, 'help': 'Name of the tool that associated flags be returned for.'})]
help = 'Get the compilation flags necessary for a C/C++ program.'
name = 'compiler-flags'
classmethod parse_args(*args, **kwargs)[source]
class yggdrasil.command_line.cc_toolname(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Output the name of the compiler used to compile C or C++ programs.

arguments = [(('--cpp',), {'action': 'store_true', 'help': 'Get the compiler used for C++ programs.'}), (('--fullpath',), {'action': 'store_true', 'help': 'Get the full path to the tool exectuable.'})]
classmethod func(args, **kwargs)[source]
help = 'Get the compiler used for C/C++ programs.'
name = 'compiler-tool'
classmethod parse_args(*args, **kwargs)[source]
class yggdrasil.command_line.generate_gha_workflow(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Re-generate the Github actions workflow yaml.

arguments = [(('--base', '--base-file'), {'help': 'Version of GHA workflow yaml that contains anchors.'}), (('--dest',), {'help': 'Name of target GHA workflow yaml file.'}), (('--verbose',), {'action': 'store_true', 'help': 'Print yaml contents.'})]
classmethod func(args, gitdir=None)[source]
help = 'Generate a Github Actions (GHA) workflow yaml file from a version of the file that uses anchors (not supported by GHA as of 2021-01-14).'
name = 'gha'
yggdrasil.command_line.githook()[source]

Git hook to determine if the Github workflow need to be re-generated.

class yggdrasil.command_line.integration_service_manager(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Start or manage the yggdrasil service manager.

arguments = [(('--manager-name',), {'help': 'Name that will be used to identify the service manager.'}), (('--service-type',), {'choices': ['flask', 'rmq'], 'help': "Type of service that should be started. If not provided, the default will be the ('services', 'default_type') configuration option, if set, and will otherwise be 'flask'."}), (('--commtype',), {'type': <class 'str'>, 'help': "Type of communicator that should be used for connections to services. If not provided, the default will be determined by the ('services', 'default_comm') configuration option, if set."}), (('--address',), {'type': <class 'str'>, 'help': "URL for requests to the service manager. If not provided, the default will be determined by the ('services', 'default_type') configuration option, if set, and based on the selected 'service-type', if not. For a service-type of 'flask', this should be the http address with the port that should be used and the default will be 'http://localhost:{port}'. For a service-type of 'rmq', this should be an amqp broker address and the default will be 'amqp://guest:guest@localhost:{port}/%%2f'."}), (('--port',), {'type': <class 'int'>, 'help': 'Port that should be used to access the app. Defaults to 5000 for a flask service manager and 5672 for a RabbitMQ service manager if not set via the PORT environment variable.'}), <yggdrasil.command_line.ArgumentSubparser object>]
classmethod func(args)[source]
help = 'Start or manage a integration service manager.'
name = 'integration-service-manager'
class yggdrasil.command_line.ld_flags(*args, **kwargs)[source]

Bases: yggdrasil.command_line.cc_toolname

Get the linker flags necessary for including the interface library in a C or C++ program.

arguments = [(('--cpp',), {'action': 'store_true', 'help': 'Get the compilation flags used for C++ programs.'}), (('--toolname',), {'default': None, 'help': 'Name of the tool that associated flags be returned for.'})]
help = 'Get the compilation flags necessary for a C/C++ program.'
name = 'linker-flags'
classmethod parse_args(*args, **kwargs)[source]
class yggdrasil.command_line.ld_toolname(*args, **kwargs)[source]

Bases: yggdrasil.command_line.cc_toolname

Output the name of the linker used to compile C or C++ programs.

arguments = [(('--cpp',), {'action': 'store_true', 'help': 'Get the linker used for C++ programs.'}), (('--fullpath',), {'action': 'store_true', 'help': 'Get the full path to the tool exectuable.'})]
help = 'Get the linker used for C/C++ programs.'
name = 'linker-tool'
classmethod parse_args(*args, **kwargs)[source]
class yggdrasil.command_line.main(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Runner for yggdrasil CLI.

arguments = []
classmethod func(args)[source]
classmethod get_parser(**kwargs)[source]
help = 'Command line interface for the yggdrasil package.'
name = 'yggdrasil'
classmethod parse_args(parser, args=None, **kwargs)[source]
subcommands = [<class 'yggdrasil.command_line.yggrun'>, <class 'yggdrasil.command_line.ygginfo'>, <class 'yggdrasil.command_line.validate_yaml'>, <class 'yggdrasil.command_line.yggcc'>, <class 'yggdrasil.command_line.yggcompile'>, <class 'yggdrasil.command_line.yggclean'>, <class 'yggdrasil.command_line.ygginstall'>, <class 'yggdrasil.command_line.update_config'>, <class 'yggdrasil.command_line.regen_metaschema'>, <class 'yggdrasil.command_line.regen_schema'>, <class 'yggdrasil.command_line.yggmodelform'>, <class 'yggdrasil.command_line.yggdevup'>, <class 'yggdrasil.command_line.timing_plots'>, <class 'yggdrasil.command_line.generate_gha_workflow'>, <class 'yggdrasil.command_line.integration_service_manager'>]
yggdrasil.command_line.rebuild_c_api()[source]

Rebuild the C/C++ API.

class yggdrasil.command_line.regen_metaschema(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Regenerate the yggdrasil metaschema.

classmethod func(args)[source]
help = 'Regenerate the yggdrasil metaschema.'
name = 'metaschema'
class yggdrasil.command_line.regen_schema(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Regenerate the yggdrasil schema.

arguments = [(('--only-constants',), {'action': 'store_true', 'help': 'Only update the constants.py file without updating the schema.'})]
classmethod func(args)[source]
help = 'Regenerate the yggdrasil schema.'
name = 'schema'
class yggdrasil.command_line.timing_plots(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Create performance plots using timing results.

arguments = [<yggdrasil.command_line.ArgumentSubparser object>]
classmethod func(args)[source]
help = 'Create performance plots using timing results.'
name = 'timing'
class yggdrasil.command_line.update_config(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Update the user config file for yggdrasil.

classmethod add_arguments(parser, **kwargs)[source]
arguments = [(('languages',), {'nargs': '*', 'default': [], 'help': 'One or more languages that should be configured.'}), (('--languages',), {'nargs': '+', 'dest': 'languages_flag', 'default': [], 'help': 'One or more languages that should be configured.'}), (('--show-file',), {'action': 'store_true', 'help': 'Print the path to the config file without updating it.'}), (('--remove-file',), {'action': 'store_true', 'help': 'Remove the existing config file and return.'}), (('--overwrite',), {'action': 'store_true', 'help': 'Overwrite the existing file.'}), (('--disable-languages',), {'nargs': '+', 'default': [], 'choices': ['c', 'c++', 'cpp', 'cxx', 'fortran', 'R', 'matlab', 'python', 'r', 'cmake', 'make', 'lpy', 'osr', 'sbml', 'dummy', 'executable', 'mpi', 'timesync'], 'help': 'One or more languages that should be disabled.'}), (('--enable-languages',), {'nargs': '+', 'default': [], 'choices': ['c', 'c++', 'cpp', 'cxx', 'fortran', 'R', 'matlab', 'python', 'r', 'cmake', 'make', 'lpy', 'osr', 'sbml', 'dummy', 'executable', 'mpi', 'timesync'], 'help': 'One or more languages that should be enabled.'}), (('--quiet', '-q'), {'action': 'store_true', 'help': 'Suppress output.'}), (('--allow-multiple-omp',), {'action': 'store_true', 'default': None, 'help': 'Have yggdrasil set the environment variable KMP_DUPLICATE_LIB_OK to \'True\' during model runs to disable errors resembling "OMP: Error #15: Initializing libomp.dylib..." that result from having multiple versions of OpenMP loaded during runtime.'}), (('--dont-allow-multiple-omp',), {'action': 'store_true', 'default': None, 'help': "Don't set the KMP_DUPLICATE_LIB_OK environment variable when running models (see help for '--allow-multiple-omp' for more information)."}), (('--c-compiler',), {'help': 'Name or path to compiler that should be used to compile models written in c.'}), (('--c++-compiler',), {'help': 'Name or path to compiler that should be used to compile models written in c++.'}), (('--fortran-compiler',), {'help': 'Name or path to compiler that should be used to compile models written in fortran.'}), (('--c-linker',), {'help': 'Name or path to linker that should be used to link models written in c.'}), (('--c++-linker',), {'help': 'Name or path to linker that should be used to link models written in c++.'}), (('--fortran-linker',), {'help': 'Name or path to linker that should be used to link models written in fortran.'}), (('--c-archiver',), {'help': 'Name or path to archiver that should be used to create static libraries for models written in c.'}), (('--c++-archiver',), {'help': 'Name or path to archiver that should be used to create static libraries for models written in c++.'}), (('--fortran-archiver',), {'help': 'Name or path to archiver that should be used to create static libraries for models written in fortran.'})]
classmethod func(args)[source]
help = 'Update the user config file.'
language_arguments = {'c': [(('--vcpkg-dir',), {'help': 'Directory containing the vcpkg installation.'}), (('--macos-sdkroot', '--sdkroot'), {'help': 'The full path to the MacOS SDK that should be used.'})], 'matlab': [(('--disable-matlab-engine-for-python',), {'action': 'store_true', 'default': None, 'dest': 'disable_engine', 'help': 'Disable use of the Matlab engine for Python.'}), (('--enable-matlab-engine-for-python',), {'action': 'store_true', 'default': None, 'dest': 'enable_engine', 'help': 'Enable use of the Matlab engine for Python.'}), (('--hide-matlab-libiomp',), {'action': 'store_true', 'default': None, 'help': 'Hide the version of libiomp installed by Matlab by slightly changing the filename so that the conda version of libomp is used instead. This helps to solve the error "OMP: Error #15: Initializing libomp.dylib..." that can occur when using a conda environment. The hidden file location will be set in the configuration file and can be restored via the \'--restore-matlab-libiomp\' option.'}), (('--restore-matlab-libiomp',), {'action': 'store_true', 'default': None, 'help': "Restore the version of libiomp installed by Matlab. (See help for '--hide-matlab-libiomp')"})], 'osr': [(('--osr-repository-path',), {'dest': 'repository', 'help': 'The full path to the OpenSimRoot repository.'})]}
name = 'config'
opposite_arguments = [('allow_multiple_omp', 'dont_allow_multiple_omp'), ('disable_engine', 'enable_engine'), ('hide_matlab_libiomp', 'restore_matlab_libiomp')]
class yggdrasil.command_line.validate_yaml(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Validate a set of or or more YAMLs defining an integration.

arguments = [(('yamlfile',), {'nargs': '+', 'help': 'One or more YAML specification files.'}), (('--model-only',), {'action': 'store_true', 'help': 'Validate a YAML containing an isolated model without ensuring that it is part of a complete integration.'}), (('--model-submission',), {'action': 'store_true', 'help': 'Validate a YAML against the requirements for submissions to the yggdrasil model repository.'})]
classmethod func(args)[source]
help = 'Validate a set of YAML specification files for an integration.'
name = 'validate'
class yggdrasil.command_line.yggcc(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Compile a program.

arguments = [(('source',), {'nargs': '+', 'help': 'One or more source files or the directory containing an R package.'}), (('--language',), {'default': None, 'choices': [None, 'c', 'c++', 'cpp', 'cxx', 'fortran', 'R', 'r'], 'help': 'Language of the source code. If not provided, the language will be determined from the source extension.'}), (('--toolname',), {'help': 'Name of compilation tool that should be used'}), (('--flags',), {'nargs': '*', 'help': 'Additional flags that should be added to the compilation command'}), (('--use-ccache',), {'action': 'store_true', 'help': 'Run compilation with ccache.'}), (('--Rpkg-language',), {'help': 'Language that R package is written in (only used if the specified language is R).'})]
classmethod func(args)[source]
help = 'Compile a program from source files for use in an yggdrasil integration.'
name = 'compile'
class yggdrasil.command_line.yggclean(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Cleanup dependency files.

arguments = [(('language',), {'nargs': '*', 'default': ['all'], 'help': 'One or more languages to clean up dependencies for.'})]
classmethod func(args, verbose=True)[source]
help = 'Remove dependency libraries compiled by yggdrasil.'
name = 'clean'
class yggdrasil.command_line.yggcompile(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Compile interface library/libraries.

arguments = [(('language',), {'nargs': '*', 'default': ['all'], 'help': 'One or more languages to compile dependencies for.'}), (('--toolname',), {'help': 'Name of compilation tool that should be used'})]
classmethod func(args)[source]
help = 'Compile yggdrasil dependency libraries. Existing libraries are first deleted.'
name = 'compile-deps'
class yggdrasil.command_line.yggdevup(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Cleanup old libraries, re-install languages, and re-compile interface libraries following an update to the code (when doing development).

classmethod func(args)[source]
help = 'Perform cleanup and reinitialization following an update to the code during development.'
name = 'dev-update'
class yggdrasil.command_line.ygginfo(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Print information about yggdrasil installation.

arguments = [(('--no-languages',), {'action': 'store_true', 'dest': 'no_languages', 'help': "Don't print information about individual languages."}), (('--no-comms',), {'action': 'store_true', 'dest': 'no_comms', 'help': "Don't print information about individual comms."}), (('--verbose', '-v'), {'action': 'store_true', 'help': 'Increase the verbosity of the printed information.'}), <yggdrasil.command_line.ArgumentSubparser object>]
classmethod func(args, return_str=False)[source]
help = 'Display information about the current yggdrasil installation.'
name = 'info'
class yggdrasil.command_line.ygginstall(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Call installation script.

classmethod func(args)[source]
classmethod get_parser(**kwargs)[source]
help = 'Complete yggdrasil installation for one or more languages.'
name = 'install'
class yggdrasil.command_line.yggmodelform(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Save/print a JSON schema that can be used for generating a form for composing a model specification files.

arguments = [(('--file',), {'help': 'Path to file where the schema should be saved.'})]
classmethod func(args)[source]
help = 'Save/print the JSON schema for generating the model specification form.'
name = 'model-form-schema'
class yggdrasil.command_line.yggrun(*args, **kwargs)[source]

Bases: yggdrasil.command_line.SubCommand

Start a run.

classmethod add_arguments(parser, **kwargs)[source]
arguments = [(('yamlfile',), {'nargs': '+', 'help': 'One or more yaml specification files.'}), (('--with-mpi', '--mpi-nproc'), {'type': <class 'int'>, 'default': 1, 'help': 'Number of MPI processes to run on.'}), (('--mpi-tag-start',), {'type': <class 'int'>, 'default': 0, 'help': 'Tag that MPI communications should start at.'}), (('--validate',), {'action': 'store_true', 'help': 'Validate the run via model validation commands on completion.'})]
classmethod func(args)[source]
help = 'Run an integration.'
name = 'run'
yggdrasil.command_line.yggtime_comm()[source]

Plot timing statistics comparing the different communication mechanisms.

yggdrasil.command_line.yggtime_lang()[source]

Plot timing statistics comparing the different languages.

yggdrasil.command_line.yggtime_os()[source]

Plot timing statistics comparing the different operating systems.

yggdrasil.command_line.yggtime_paper()[source]

Create plots for timing.

yggdrasil.command_line.yggtime_py()[source]

Plot timing statistics comparing the different versions of Python.

yggdrasil.components module

class yggdrasil.components.ClassRegistry(*args, import_function=None, **kwargs)[source]

Bases: collections.OrderedDict

Class for registering classes.

get(key, default=None)[source]

Return the value for key if key is in the dictionary, else default.

has_entry(key)[source]
import_classes()[source]

Import all classes in the same directory.

items() a set-like object providing a view on D's items[source]
keys() a set-like object providing a view on D's keys[source]
values() an object providing a view on D's values[source]
class yggdrasil.components.ComponentBase(*args, **kwargs)[source]

Bases: yggdrasil.components.ComponentBaseUnregistered

Base class for schema components.

Parameters
  • skip_component_schema_normalization (bool, optional) – If True, the schema will not be used to normalize/validate input keyword arguments (e.g. in case they were already parsed). Defaults to False.

  • **kwargs – Keyword arguments are added to the class as attributes according to the class attributes _schema_properties and _schema_excluded_from_class. Keyword arguments not added to the class as attributes are assigned to the extra_kwargs dictionary.

extra_kwargs

Keyword arguments that were not parsed.

Type

dict

Class Attributes:

_schema_type (str): Name of the component type the class represents. _schema_subtype_key (str): Attribute that should be used to identify the

subtype associated with each class.

_schema_subtype_description (str): Description for the subtype represented

by the class that should be used in documentation tables.

_schema_required (list): Keys from _schema_properties that are required

to produce a valid component.

_schema_properties (dict): Schemas describing keyword arguments that

can be supplied to the class constructor and used to specify component behavior in YAML/JSON files. At initialization, these keywords are added to the class instance as attributes of the same name unless they are in _schema_excluded_from_class. Unless _schema_inherit is False, these properties will be added in addition to the schema properties defined by the class base.

_schema_excluded_from_class (list): Keywords in _schema_properties that

should not be added to the class as attributes during initialization.

_schema_excluded_from_inherit (list): Keywords in _schema_properties that

should not be inherited either from the base class or by child classes.

_schema_inherit (bool, ComponentBase): If False, the base schema will

not be inherited. If a Component subclass, the schema from that class will be inherited instead of the base class. Defaults to True.

_dont_register (bool): If True, the component class will be be registered

and the before_registration class method will not be called. Defaults to False.

static after_registration(cls)[source]

Operations that should be preformed to modify class attributes after registration. These actions will not be performed if the environment variable YGGDRASIL_REGISTRATION_IN_PROGRESS is set.

static before_registration(cls)[source]

Operations that should be performed to modify class attributes prior to registration. These actions will still be performed if the environment variable YGGDRASIL_REGISTRATION_IN_PROGRESS is set.

static finalize_registration(cls)[source]

Final operations to perform after a class has been fully initialized. These actions will not be performed if the environment variable YGGDRASIL_REGISTRATION_IN_PROGRESS is set.

class yggdrasil.components.ComponentBaseUnregistered[source]

Bases: object

Base class for schema components w/o schema and registration.

disconnect()[source]

Disconnect attributes that are aliases.

exception yggdrasil.components.ComponentError[source]

Bases: BaseException

Error raised when there is a problem import a component.

class yggdrasil.components.ComponentMeta(name, bases, class_dict)[source]

Bases: type

Meta class for registering schema components.

yggdrasil.components.create_component(comptype, subtype=None, **kwargs)[source]

Dynamically create an instance of a component with the specified options as outlined in the component schema. This function requires loading the component schemas and so should not be used at the module or class level to prevent circular dependencies.

Parameters
  • comptype (str) – Component type.

  • subtype (str, optional) – Component subtype. If subtype is not one of the registered subtypes for the specified comptype, subtype is treated as the name of the class. Defaults to None if not provided and the default subtype defined in the schema for the specified component will be used. If the subtype is specified by the component subtype key in the remaining kwargs, that subtype will be used instead.

  • **kwargs – Additional keyword arguments are treated as options for the component as outlined in the component schema.

Returns

Instance of the specified component type/subtype and

options.

Return type

ComponentBase

Raises

ComponentError – If comptype is not a registered component type.

yggdrasil.components.get_component_base_class(comptype, subtype=None, **kwargs)[source]

Determine the base class for a component type.

Parameters
  • comptype (str) – The name of a component to test against.

  • subtype (str, optional) – Subtype to use to determine the component base class. Defaults to None.

  • **kwargs – Additional keyword arguments are used to determine the subtype if it is None.

Returns

Component base class.

Return type

ComponentBase

yggdrasil.components.get_registry(comptype=None)[source]

Get the registry that should be used for looking up components.

Parameters

comptype (str, optional) – The name of a component to get the registry for. Defaults to None and the entire registry will be returned.

yggdrasil.components.import_component(comptype, subtype=None, **kwargs)[source]

Dynamically import a component by name.

Parameters
  • comptype (str) – Component type.

  • subtype (str, optional) – Component subtype. If subtype is not one of the registered subtypes for the specified comptype, subtype is treated as the name of class. Defaults to None if not provided and the default subtype defined in the schema for the specified component will be used.

  • **kwargs – Additional keyword arguments are used to determine the subtype if it is None.

Returns

Component class.

Return type

class

Raises
  • ComponentError – If comptype is not a registered component type.

  • ComponentError – If subtype is not a registered subtype or the name of a registered subtype class for the specified comptype.

yggdrasil.components.inherit_schema(orig, new_properties=None, new_required=None, remove_keys=[], **kwargs)[source]

Create an inherited schema, adding new value to accepted ones for dependencies.

Parameters
  • orig (dict) – Schema that will be inherited.

  • new_properties (dict, optional) – Dictionary of new properties to add. Defaults to None and is ignored.

  • new_requried (list, optional) – Properties that should be required by the new class. Defaults to None and is ignored.

  • remove_keys (list, optional) – Keys that should be removed form orig before adding the new keys. Defaults to empty list.

  • **kwargs – Additional keyword arguments will be added to the schema with dependency on the provided key/value pair.

Returns

New schema properties and a list of requried.

Return type

tuple(dict, list)

yggdrasil.components.init_registry(recurse=False)[source]

Initialize the registries and schema.

yggdrasil.components.isinstance_component(x, comptype, subtype=None, **kwargs)[source]

Determine if an object is an instance of a component type.

Parameters
  • x (object) – Object to test.

  • comptype (str, list) – The name of one or more components to test against.

  • subtype (str, optional) – Subtype to use to determine the component base class. Defaults to None.

  • **kwargs – Additional keyword arguments are used to determine the subtype if it is None.

Returns

True if the object is an instance of the specified component(s).

Return type

bool

yggdrasil.components.registering(recurse=False)[source]

Context for preforming registration.

yggdrasil.components.registration_in_progress()[source]

Determine if a registration is in progress.

yggdrasil.components.restore_registry(reg_dict)[source]

Restore the registry to values in the provided dictionary.

yggdrasil.components.suspend_registry()[source]

Suspend the registry by storing the global registries in a dictionary.

yggdrasil.config module

This module imports the configuration for yggdrasil.

Todo

Remove reference to environment variables for accessing config options.

class yggdrasil.config.ConfigEnv(old_env, new_env)[source]

Bases: object

Container for environment variable modification.

restore()[source]

Restore the old environment variables.

class yggdrasil.config.YggConfigParser(files=None)[source]

Bases: configparser.ConfigParser, object

Config parser that returns None if option not provided on get.

backwards_str2val(val)[source]
property file_to_update

Full path to file that should be updated if update_file is called without an explicit file path.

Type

str

classmethod from_files(files, **kwargs)[source]

Construct a config parser from a set of files.

Parameters
  • files (list) – One or more files that options should be read from in the order they should be loaded.

  • **kwargs – Additional keyword arguments are passed to the class constructor.

Returns

Config parser with information loaded from the

provided files.

Return type

YggConfigParser

get(section, option, default=None, **kwargs)[source]

Return None if the section/option does not exist.

Parameters
  • section (str) – Name of section.

  • option (str) – Name of option in section.

  • default (obj, optional) – Value that should be returned if the section and/or option are not found or are an empty string. Defaults to None.

  • **kwargs – Additional keyword arguments are passed to the parent class’s get.

Returns

String entry if the section & option exist, otherwise default.

Return type

obj

read(*args, **kwargs)[source]

Read and parse a filename or an iterable of filenames.

Files that cannot be opened are silently ignored; this is designed so that you can specify an iterable of potential configuration file locations (e.g. current directory, user’s home directory, systemwide directory), and all existing configuration files in the iterable will be read. A single filename may also be given.

Return list of successfully read files.

reload()[source]

Reload parameters from the original files.

set(section, option, value=None)[source]

Set an option.

update_file(fname=None)[source]

Write out updated contents to a file.

Parameters

fname (str, optional) – Full path to file where contents should be saved. If None, file_to_update is used. Defaults to None.

Raises

RuntimeError – If fname is None and file_to_update is None.

yggdrasil.config.acquire_env(new_env)[source]

Get the existing environment variable values and set the environment based on the provided dictionary.

Parameters

new_env (dict) – Mapping from configuration key to values that environment variables should be set to.

Returns

Mapping of environment variables and values that were overridden

by new_env.

Return type

dict

yggdrasil.config.cfg_environment(env=None, cfg=None)[source]

Set environment variables based on config options.

Parameters
  • env (dict, optional) – Dictionary of environment variables that should be updated. Defaults to os.environ.

  • cfg (yggdrasil.config.YggConfigParser, optional) – Config parser with options that should be used to update the environment. Defaults to yggdrasil.config.ygg_cfg.

yggdrasil.config.cfg_logging(cfg=None)[source]

Set logging levels from config options.

Parameters

cfg (yggdrasil.config.YggConfigParser, optional) – Config parser with options that should be used to update the environment. Defaults to yggdrasil.config.ygg_cfg.

yggdrasil.config.get_config_parser(parser=None, description=None, skip_sections=None)[source]

Create a parser or add to an existing parser arguments based on configuration options.

Parameters
  • parser (argparse.ArgumentParser, optional) – Existing argument parser that arguments should be added to. Defaults to None and a new parser is created.

  • description (str, optional) – The description that should be used if a new parser is created. Defaults to None.

  • skip_sections (list, optional) – Configuration sections that should be skipped when adding arguments to the parser. Defaults to None and arguments for all configuration sections will be added.

Returns

Argument parser with arguments added that

are associated with configuration options.

Return type

argparse.ArgumentParser

yggdrasil.config.get_language_order(drivers)[source]

Get the correct language order, including any base languages.

Parameters

drivers (dict) – Drivers in order.

Returns

Drivers in sorted order.

Return type

dict

yggdrasil.config.get_ygg_loglevel(cfg=None, default='DEBUG')[source]

Get the current log level.

Parameters
  • cfg (yggdrasil.config.YggConfigParser, optional) – Config parser with options that should be used to determine the log level. Defaults to yggdrasil.config.ygg_cfg.

  • default (str, optional) – Log level that should be returned if the log level option is not set in cfg. Defaults to ‘DEBUG’.

Returns

Log level string.

Return type

str

yggdrasil.config.parser_config(args, **kwargs)[source]

Context manager for a run that modifies configuration options using values from an argument parser.

Parameters
  • args (argparse.Namespace) – Argument parsing results.

  • **kwargs – Additional environment variable key/value pairs and/or argument name key/value pairs that should be added to the environment.

yggdrasil.config.resolve_config_parser(args)[source]

Process argument results by setting flags that are set by combination arguments.

Parameters

args (argparse.ArgumentParser) – Argument parser to set combination child flags for.

Returns

Argument parser with child flags for

combination flags set.

Return type

argparse.ArgumentParser

yggdrasil.config.restore_env(old_env)[source]

Restore environment variables to a previous state.

Parameters

old_env (dict) – Mapping from environment variable to value for state that should be restored.

yggdrasil.config.set_ygg_loglevel(level, cfg=None)[source]

Set the current log level.

Parameters
  • level (str) – Level that the log should be set to.

  • cfg (yggdrasil.config.YggConfigParser, optional) – Config parser with options that should be used to update the environment. Defaults to yggdrasil.config.ygg_cfg.

yggdrasil.config.temp_config(**kwargs)[source]

Context manager for a run that modifies configuration options using a dictionary of key/value pairs.

Parameters

**kwargs – Environment variable key/value pairs and/or argument name key/value pairs that should be added to the environment.

yggdrasil.config.update_language_config(languages=None, skip_warnings=False, disable_languages=None, enable_languages=None, allow_multiple_omp=None, lang_kwargs=None, overwrite=False, verbose=False)[source]

Update configuration options for a language driver.

Parameters
  • languages (list, optional) – List of languages to configure. Defaults to None and all supported languages will be configured.

  • skip_warnings (bool, optional) – If True, warnings about missing options will not be raised. Defaults to False.

  • disable_languages (list, optional) – List of languages that should be disabled. Defaults to an empty list.

  • enable_languages (list, optional) – List of languages that should be enabled. Defaults to an empty list.

  • allow_multiple_omp (bool, optional) – Set the allow_multiple_omp config option controlling whether or not the KMP_DUPLICATE_LIB_OK environment variable is set for model environments. Defaults to None and is ignored.

  • overwrite (bool, optional) – If True, the existing file will be overwritten. Defaults to False.

  • verbose (bool, optional) – If True, information about the config file will be displayed. Defaults to False.

  • lang_kwargs (dict, optional) – Dictionary containing language specific keyword arguments. Defaults to {}.

yggdrasil.constants module

Constants used by yggdrasil.

yggdrasil.doctools module

yggdrasil.doctools.component2table(comp, table_type, include_required=None, subtype_ref=None, **kwargs)[source]

Create a table describing a component.

Parameters
  • comp (str) – Name of a component type to create a table for.

  • table_type (str) –

    Type of table that should be created for the component. Supported values include:

    • ’general’: Create a table of properties common to all components of the specified type.

    • ’specific’: Create a table of properties specific to only some components of the specified type.

    • ’subtype’: Create a table describing the subtypes available for the specified component type.

  • include_required (bool, optional) – If True, a required column is included. Defaults to True if table_type is ‘general’ and False otherwise.

  • subtype_ref (str, optional) – Reference for the subtype table for the specified component that should be used in the description for the subtype property. Defaults to None and is ignored.

  • **kwargs – Additional keyword arguments are passed to dict2table.

Returns

Lines comprising the table.

Return type

list

yggdrasil.doctools.dict2table(args, key_column_name='option', val_column_name='description', column_order=None, wrapped_columns=None, list_columns=None, sort_on_key=True, sort_required_first=True, last_column=None, prune_empty_columns=False, style='simple', **kwargs)[source]

Convert a dictionary to a table.

Parameters
  • args (dict) – Dictionary of entries for the table. The keys will go in the first column (the key column) as named by key_column_name. If the values are dictionaries, the elements in each dictionary will be put in different columns. If the values are strings, they will be put in the column identified by val_column_name.

  • key_column_name (str, optional) – Label that the key column should have. Defaults to ‘option’.

  • val_column_name (str, optional) – Lable that the value column should have. Defaults to ‘description’.

  • column_order (list, optional) – List specifying the roder that fields should be added to the table as columns. Defaults to None and is set to [key_column_name, val_column_name]. If a column is missing from an entry in args, an empty value will be added to the table in its place. Columns in args entries that are not in column_order are appended to column_order in sorted order.

  • wrapped_columns (dict, optional) – Dictionary specifying fields that should be wrapped as columns and the widths that the corresponding column should be wrapped to. Defaults to {‘Description’: 80}.

  • list_columns (list, optional) – List of fields with values that should be formatted with one item per line if it is a list. Defaults to None and is ignored.

  • sort_on_key (bool, optional) – If True, the entries in args are added as rows in the order determined by sorting on the keys. If False, the order will be determine by prop (which is not deterministic if a Python 2 dictionary). Defaults to True.

  • sort_required_first (bool, optional) – If True, the required entries are placed first in the table. Defaults to True.

  • last_column (str, optional) – Name of the field that should be placed in the last column position. Defaults to None and is ignored.

  • prune_empty_columns (bool, optional) – If True, empty columns will be removed. If False, they will be included. Defaults to False.

  • style (str, optional) – Style of table that should be used. ‘simple’ dosn’t allow for multi-column or row cells, while ‘complex’ does. Defaults to ‘simple’.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Lines comprising the table.

Return type

list

yggdrasil.doctools.docs2args(docs, keys=['Args:', 'Arguments:'])[source]

Get a dictionary of arguments and argument descriptions from a docstring.

Parameters
  • docs (str) – Docstring that should be parsed.

  • keys (list) – Strings that should be used to identify the target section in the docstring. Defaults to [‘Args:’, ‘Arguments:’].

Returns

Dictionary of arguments/description pairs.

Return type

dict

yggdrasil.doctools.document_cli(fname, headline_char='*')[source]

Get documentation for all yggdrasil command line interfaces.

Parameters
  • fname (str) – Path to file where CLI documentation should be saved.

  • headline_char (str, optional) – Character that should be used to mark the title. Defaults to ‘*’.

Returns

Documentation lines.

Return type

list

yggdrasil.doctools.document_cli_subcommand(prog, sub, alias=None, dummy_file=None, headline_char='*', dummy_def=None)[source]

Get documentation for a command line interface.

Parameters
  • prog (str) – Program that calls the subcommand class.

  • sub (yggdrasil.command_line.SubCommand) – Class for the subcommand.

  • alias (str, optional) – Command that prog is an alias for. Defaults to None and is ignored.

  • dummy_file (str, optional) – Full path to file where an alias function will be stored. Defaults to None and the original module will be used.

  • dummy_def (list, optional) – Existing list that the dummy definition lines should be added to. Must be provided if dummy_file is.

  • headline_char (str, optional) – Character that should be used to mark the title. Defaults to ‘*’.

Returns

Documentation lines.

Return type

list

yggdrasil.doctools.get_docs_section(docs, keys=['Args:', 'Arguments:'], join_lines=False)[source]

Get contents of a particular sections in a docstring.

Parameters
  • docs (str) – Docstring that should be parsed.

  • keys (list) – Strings that should be used to identify the target section in the docstring. Defaults to [‘Args:’, ‘Arguments:’].

  • join_lines (bool, optional) – If True, the returned lines will be stripped and joined into a single string. Defaults to False.

Returns

Lines in the section(s) specified by keys.

Return type

list

yggdrasil.doctools.write_classdocs_table(class_, table_type='all', **kwargs)[source]

Write a table containing entries from a section in a class’s documentation.

Parameters
  • class (type) – Class with docstring that should be parsed.

  • table_type (str, optional) –

    Type of table that should be created for the class. Defaults to ‘all’. Supported values include:

    • ’all’: Create each type of table for the specified class.

    • ’args’: Create a table of the class’s argumetns.

    • ’attr’: Create a table of the class’s attributes.

    • ’classattr’: Create a table of the class’s class attributes.

  • **kwargs – Additional keyword arguments are passed to dict2table and write_table.

Returns

Name of file or files created.

Return type

str, list

Raises

ValueError – If table_type is not one of the supported values.

yggdrasil.doctools.write_comm_devnotes_table(**kwargs)[source]

Write a table containing development notes for the registered comms.

Parameters

**kwargs – Additional keyword arguments are passed to dict2table and write_table.

Returns

Name of file or files created.

Return type

str, list

yggdrasil.doctools.write_component_table(comp='all', table_type='all', **kwargs)[source]

Write a component table to a file.

Parameters
  • comp (str) – Name of a component type to create a table for. Defaults to ‘all’ and tables are created for each of the registered components.

  • table_type (str, optional) –

    Type of table that should be created for the component. Defaults to ‘all’. Supported values include:

    • ’all’: Create each type of table for the specified component.

    • ’general’: Create a table of properties common to all components of the specified type.

    • ’specific’: Create a table of properties specific to only some components of the specified type.

    • ’subtype’: Create a table describing the subtypes available for the specified component type.

  • **kwargs – Additional keyword arguments are passed to component2table and write_table.

Returns

Name of file or files created.

Return type

str, list

yggdrasil.doctools.write_datatype_mapping_table(**kwargs)[source]

Write a table containing mapping of datatypes between different languages.

Parameters

**kwargs – Additional keyword arguments are passed to dict2table and write_table.

Returns

Name of file or files created.

Return type

str, list

yggdrasil.doctools.write_datatype_table(table_type='all', **kwargs)[source]

Write a table containing entries from the descriptions of datatypes.

Parameters
  • table_type (str, optional) –

    Type of table that should be created for the class. Defaults to ‘all’. Supported values include:

    • ’all’: Create each type of table.

    • ’simple’: Create a table of standard JSON datatypes.

    • ’container’: Create a table of container datatypes.

    • ’yggdrasil’: Create a table of yggdrasil specific datatypes.

  • **kwargs – Additional keyword arguments are passed to dict2table and write_table.

Returns

Name of file or files created.

Return type

str, list

Raises

ValueError – If table_type is not one of the supported values.

yggdrasil.doctools.write_interface_mapping_table(split_table_for_notebook=False, **kwargs)[source]

Write a table containing mapping of datatypes between different languages.

Parameters

**kwargs – Additional keyword arguments are passed to dict2table and write_table.

Returns

Name of file or files created.

Return type

str, list

yggdrasil.doctools.write_table(lines, fname=None, fname_base=None, fname_dir=None, verbose=False, **kwargs)[source]

Write a table to a file.

Parameters
  • fname (str, optional) – Name of the file where the table should be written. If not provided, one is constructed from fname_base and fname_dir.

  • fname_base (str, optional) – Base name of the file where the table should be written if fname is not provided. If fname is None, fname_base must be provided.

  • fname_dir (str, optional) – Directory where the table should be written if fname is not provided. If not provided, this defaults to the current working directory.

  • verbose (bool, optional) – If True, the lines are printed to standard out in addition to the file. Defaults to False.

  • **kwargs – Additional keyword arguments are ignored.

Returns

Name of file created.

Return type

str

Raises

ValueError – If neither fname nor fname_base are provided.

yggdrasil.multitasking module

exception yggdrasil.multitasking.AliasDisconnectError[source]

Bases: RuntimeError

class yggdrasil.multitasking.AliasMeta(name, bases, class_dict)[source]

Bases: type

Meta class for adding aliased methods to the class.

class yggdrasil.multitasking.AliasObject(*args, dont_initialize_base=False, **kwargs)[source]

Bases: object

Alias object that calls to attribute.

Parameters

dont_initialize_base (bool, optional) – If True the base object will not be initialized. Defaults to False.

check_for_base(attr)[source]

Raise an error if the aliased object has been disconnected.

disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

property dummy_copy

Dummy copy of base.

classmethod from_base(base, *args, **kwargs)[source]

Create an instance by creating a based from the provided base class.

exception yggdrasil.multitasking.BreakLoopError(*args, **kwargs)[source]

Bases: yggdrasil.multitasking.BreakLoopException

Version of BreakLoopException that sets an error message.

break_stack
exception yggdrasil.multitasking.BreakLoopException(*args, **kwargs)[source]

Bases: BaseException

Special exception that can be raised by the target function for a loop in order to break the loop.

break_stack
class yggdrasil.multitasking.Context(task_method='thread', dont_initialize_base=False)[source]

Bases: yggdrasil.multitasking.MultiObject

Context for managing threads/processes.

Dict(*args, **kwargs)[source]

Get a shared dictionary in this context.

Event(*args, **kwargs)[source]

Get an event in this context.

Queue(*args, **kwargs)[source]

Get a queue in this context.

RLock(*args, **kwargs)[source]

Get a recursive lock in this context.

Task(*args, **kwargs)[source]

Get a task in this context.

current_task()[source]

Current task (process/thread).

main_task()[source]

Main task (process/thread).

parallel
task_method
class yggdrasil.multitasking.ContextObject(*args, task_method='threading', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.MultiObject

Base class for object intialized in a context.

property context

Context used to create this object.

Type

Context

disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

classmethod get_base_class(context)[source]

Get instance of base class that will be represented.

class yggdrasil.multitasking.Dict(*args, task_method='threading', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.ContextObject

Multiprocessing/threading shared dictionary.

clear(*args, **kwargs)
copy(*args, **kwargs)
disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

get(*args, **kwargs)
classmethod get_base_class(context)[source]

Get instance of base class that will be represented.

items(*args, **kwargs)
keys(*args, **kwargs)
pop(*args, **kwargs)
popitem(*args, **kwargs)
setdefault(*args, **kwargs)
update(*args, **kwargs)
values(*args, **kwargs)
class yggdrasil.multitasking.DummyContextObject[source]

Bases: object

property context
disconnect()[source]
class yggdrasil.multitasking.DummyEvent(value=False)[source]

Bases: yggdrasil.multitasking.DummyContextObject

clear()[source]
is_set()[source]
set()[source]
wait(*args, **kwargs)[source]
class yggdrasil.multitasking.DummyQueue[source]

Bases: yggdrasil.multitasking.DummyContextObject

close()[source]
empty()[source]
full()[source]
get(*args, **kwargs)[source]
get_nowait(*args, **kwargs)[source]
join(*args, **kwargs)[source]
join_thread(*args, **kwargs)[source]
put(*args, **kwargs)[source]
put_nowait(*args, **kwargs)[source]
qsize()[source]
class yggdrasil.multitasking.DummyRLock[source]

Bases: yggdrasil.multitasking.DummyContextObject

acquire(*args, **kwargs)[source]
release(*args, **kwargs)[source]
class yggdrasil.multitasking.DummyTask(name='', exitcode=0, daemon=False)[source]

Bases: yggdrasil.multitasking.DummyContextObject

daemon
exitcode
is_alive()[source]
join(*args, **kwargs)[source]
kill()[source]
name
terminate()[source]
class yggdrasil.multitasking.Event(*args, **kwargs)[source]

Bases: yggdrasil.multitasking.ContextObject

Multiprocessing/threading event.

add_callback(callback, args=(), kwargs={}, trigger='set')[source]

Add a callback that will be called when set or clear is invoked.

Parameters
  • callback (callable) – Callable executed when set is called.

  • args (tuple, optional) – Arguments to pass to the callback.

  • kwargs (dict, optional) – Keyword arguments to pass to the callback.

  • trigger (str, optional) – Action triggering the set call. Options are ‘set’ or ‘clear’. Defaults to ‘set’.

clear()[source]

Clear the event.

disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

property dummy_copy

Dummy copy of base.

property is_set
set()[source]

Set the event.

property wait
class yggdrasil.multitasking.LockedDict(*args, task_method='process', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.LockedObject

Dictionary that can be shared between threads.

add_subdict(key)[source]

Add a subdictionary.

clear(*args, **kwargs)
copy(*args, **kwargs)
property dummy_copy

Dummy copy of base.

get(*args, **kwargs)
items(*args, **kwargs)
keys(*args, **kwargs)
pop(*args, **kwargs)
popitem(*args, **kwargs)
setdefault(*args, **kwargs)
update(*args, **kwargs)
values(*args, **kwargs)
class yggdrasil.multitasking.LockedObject(*args, task_method='process', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.AliasObject

Container that provides a lock that is acquired before accessing the object.

disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

class yggdrasil.multitasking.MPIErrorExchange(global_tag=0)[source]

Bases: object

Set of MPI messages to check for errors.

closing_messages = ['ERROR', 'COMPLETE']
finalize(failure)[source]

Finalize an instance by waiting for completions.

Parameters

failure (bool) – True if there was an error.

recv(wait=False)[source]

Check for response to receive request.

reset(global_tag=0)[source]

Rest comms for the next test.

send(msg)[source]

Send a message.

sync(local_tag=None, msg=None, get_tags=False, check_equal=False, dont_raise=False, local_error=False, sync_tag=False, check_complete=False)[source]

Synchronize processes.

Parameters
  • local_tag (int) – Next tag that will be used by the local MPI comm.

  • get_tags (bool, optional) – If True, tags will be exchanged between all processes. Defaults to False.

  • check_equal (bool, optional) – If True, tags will be checked to be equal. Defaults to False.

  • dont_raise (bool, optional) – If True and a MPIPartnerError were to be raised, the sync will abort. Defaults to False.

Raises
  • MPIPartnerError – If there was an error on one of the other MPI processes.

  • AssertionError – If check_equal is True and the tags are not equivalent.

tags = {'ERROR_ON_RANK0': 1, 'ERROR_ON_RANKX': 2}
exception yggdrasil.multitasking.MPIPartnerError[source]

Bases: Exception

Error raised when there is an error on another process.

class yggdrasil.multitasking.MPIRequestWrapper(request, completed=False, **kwargs)[source]

Bases: yggdrasil.multitasking.WaitableFunction

Wrapper for an MPI request.

cancel()[source]

Cancel the request.

canceled
completed
request
property result

The result of the MPI request.

Type

object

test()[source]

Test to see if the request has completed.

wait(timeout=None, on_timeout=False)[source]

Wait for the request to be completed.

Parameters
  • timeout (float, optional) – Time (in seconds) that should be waited for the process to finish. A value of None will wait indefinitely. Defaults to None.

  • on_timeout (callable, bool, str, optional) – Object indicating what action should be taken in the event that the timeout is reached. If a callable is provided, it will be called. A value of False will cause a TimeoutError to be raised. A value of True will cause the function value to be returned. A string will be used as the error message for a raised timeout error. Defaults to False.

Returns

The result of the request.

Return type

object

class yggdrasil.multitasking.MultiObject(*args, task_method='threading', **kwargs)[source]

Bases: yggdrasil.multitasking.AliasObject

Concurrent/parallel processing object using either threads or processes.

parallel
task_method
class yggdrasil.multitasking.ProcessEvent(*args, **kwargs)[source]

Bases: object

Multiprocessing/threading event associated with a process that has a discreet start and end.

has_started()[source]

bool: True if the process has started.

has_stopped()[source]

bool: True if the process has stopped.

is_running()[source]

bool: True if the processes has started, but hasn’t stopped.

start()[source]

Set the started event.

started
stop()[source]

Set the stopped event.

stopped
class yggdrasil.multitasking.Queue(*args, task_method='threading', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.ContextObject

Multiprocessing/threading queue.

disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

property dummy_copy

Dummy copy of base.

empty()[source]
full(*args, **kwargs)
get(*args, **kwargs)
classmethod get_base_class(context)[source]

Get instance of base class that will be represented.

get_nowait(*args, **kwargs)
join(*args, **kwargs)[source]
join_thread(*args, **kwargs)
put(*args, **kwargs)[source]
put_nowait(*args, **kwargs)[source]
qsize(*args, **kwargs)
class yggdrasil.multitasking.RLock(*args, task_method='threading', task_context=None, **kwargs)[source]

Bases: yggdrasil.multitasking.ContextObject

Recursive lock. Acquiring the lock after disconnect is called through use as a context will not raise an error, but will not do anything.

acquire(*args, **kwargs)
property dummy_copy

Dummy copy of base.

release(*args, **kwargs)
class yggdrasil.multitasking.SafeThread(*args, **kwargs)[source]

Bases: threading.Thread

Thread that sets Event on error.

property exitcode

Exit code. 1 if error, 0 otherwise.

Type

int

property pid

Process ID.

run(*args, **kwargs)[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

class yggdrasil.multitasking.Task(target=None, args=(), kwargs={}, **kws)[source]

Bases: yggdrasil.multitasking.ContextObject

Multiprocessing/threading process.

property authkey
property daemon
disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

property dummy_copy

Dummy copy of base.

property exitcode
getName(*args, **kwargs)
classmethod get_base_class(context)[source]

Get instance of base class that will be represented.

property ident

Process ID.

isDaemon(*args, **kwargs)
is_alive()[source]

Determine if the process/thread is alive.

join(*args, **kwargs)
kill(*args, **kwargs)[source]

Kill the task.

property name
property pid
run(*args, **kwargs)
property sentinel
setDaemon(*args, **kwargs)
setName(*args, **kwargs)
start(*args, **kwargs)
target(*args, **kwargs)[source]

Run the target.

terminate(*args, **kwargs)
class yggdrasil.multitasking.TaskLoop(target=None, polling_interval=0.0, **kws)[source]

Bases: yggdrasil.multitasking.Task

Class for looping over a task.

break_flag
break_loop(break_stack=None)[source]

Break the task loop.

break_stack
disconnect()[source]

Disconnect from the aliased object by replacing it with a dummy object.

kill(*args, **kwargs)[source]

Kill the task.

loop_target(*args, **kwargs)[source]

Continue calling the target until the loop is broken.

polling_interval
exception yggdrasil.multitasking.TimeoutError(msg, function_value)[source]

Bases: asyncio.exceptions.TimeoutError

Error to raise when a wait times out.

class yggdrasil.multitasking.ValueEvent(*args, **kwargs)[source]

Bases: yggdrasil.multitasking.Event

Class for handling storing a value that also triggers an event.

clear()[source]

Clear the event.

get()[source]
set(value=None)[source]

Set the event.

class yggdrasil.multitasking.WaitableFunction(function, polling_interval=0.01)[source]

Bases: object

Create an object that can be waited on until a function returns True.

Parameters
  • function (callable) – Callable function that takes no arguments and returns a boolean.

  • polling_interval (float, optional) – Time (in seconds) that should be waited in between function calls. Defaults to 0.1 seconds.

function
polling_interval
wait(timeout=None, on_timeout=False)[source]

Wait for the function to return True.

Parameters
  • timeout (float, optional) – Time (in seconds) that should be waited for the process to finish. A value of None will wait indefinitely. Defaults to None.

  • on_timeout (callable, bool, str, optional) – Object indicating what action should be taken in the event that the timeout is reached. If a callable is provided, it will be called. A value of False will cause a TimeoutError to be raised. A value of True will cause the function value to be returned. A string will be used as the error message for a raised timeout error. Defaults to False.

Returns

The result of the function call.

Return type

object

class yggdrasil.multitasking.YggTask(*args, **kwargs)[source]

Bases: yggdrasil.tools.YggClass

Class for managing Ygg thread/process.

atexit()[source]

Actions performed when python exits.

before_start()[source]

Actions to perform on the main thread/process before starting the thread/process.

call_target()[source]

Call target.

check_flag_attr(attr)[source]

Determine if a flag is set.

cleanup()[source]

Actions to perform to clean up the thread after it has stopped.

clear_flag_attr(attr)[source]

Clear a flag.

create_flag_attr(attr)[source]

Create a flag.

property daemon

Indicates whether the thread/process is daemonic or not.

Type

bool

property exitcode

Exit code.

get_current_task()[source]

Get the current process/thread.

get_flag_attr(attr)[source]

Return the flag attribute.

get_main_proc()[source]

Get the main process/thread.

property ident

Process ID.

is_alive(*args, **kwargs)[source]

Determine if the process/thread is alive.

join(*args, **kwargs)[source]

Join the process/thread.

kill(*args, **kwargs)[source]

Kill the process.

property main_terminated

True if the main thread/process has terminated.

Type

bool

property pid

Process ID.

poll()[source]

Check if the process is finished and return the return code if it is.

printStatus(return_str=False)[source]

Print the class status.

property returncode

Return code.

run(*args, **kwargs)[source]

Continue running until terminate event set.

run_error()[source]

Actions to perform on error in try/except wrapping run.

run_finally()[source]

Actions to perform in finally clause of try/except wrapping run.

run_init()[source]

Actions to perform at beginning of run.

set_flag_attr(attr, value=True)[source]

Set a flag.

set_started_flag(value=True)[source]

Set the started flag for the thread/process to True.

set_terminated_flag(value=True)[source]

Set the terminated flag for the thread/process to True.

start(*args, **kwargs)[source]

Start thread/process and print info.

terminate(no_wait=False)[source]

Set the terminate event and wait for the thread/process to stop.

Parameters

no_wait (bool, optional) – If True, terminate will not block until the thread/process stops. Defaults to False and blocks.

Raises

AssertionError – If no_wait is False and the thread/process has not stopped after the timeout.

wait(timeout=None, key=None)[source]

Wait until thread/process finish to return using sleeps rather than blocking.

Parameters
  • timeout (float, optional) – Maximum time that should be waited for the driver to finish. Defaults to None and is infinite.

  • key (str, optional) – Key that should be used to register the timeout. Defaults to None and is set based on the stack trace.

wait_flag_attr(attr, timeout=None)[source]

Wait until a flag is True.

property was_started

True if the thread/process was started. False otherwise.

Type

bool

property was_terminated

True if the thread/process was terminated. False otherwise.

Type

bool

class yggdrasil.multitasking.YggTaskLoop(*args, **kwargs)[source]

Bases: yggdrasil.multitasking.YggTask

Class to run a loop inside a thread/process.

after_loop()[source]

Actions performed after the loop.

before_loop()[source]

Actions performed before the loop.

call_target()[source]

Call target.

property loop_count

Number of loops performed.

Type

int

on_main_terminated(dont_break=False)[source]

Actions performed when 1st main terminated.

Parameters

dont_break (bool, optional) – If True, the break flag won’t be set. Defaults to False.

pause()[source]

Pause the loop execution.

resume()[source]

Resume the loop execution.

run(*args, **kwargs)[source]

Continue running until terminate event set.

run_error()[source]

Actions to perform on error in try/except wrapping run.

run_loop(*args, **kwargs)[source]

Actions performed on each loop iteration.

set_break_flag(value=True, break_stack=None)[source]

Set the break flag for the thread/process to True.

set_loop_flag(value=True)[source]

Set the loop flag for the thread/process to True.

terminate(*args, **kwargs)[source]

Also set break flag.

wait_for_loop(timeout=None, key=None, nloop=0)[source]

Wait until thread/process enters loop to return using sleeps rather than blocking.

Parameters
  • timeout (float, optional) – Maximum time that should be waited for the thread/process to enter loop. Defaults to None and is infinite.

  • key (str, optional) – Key that should be used to register the timeout. Defaults to None and is set based on the stack trace.

  • nloop (int, optional) – Number of loops that should be performed before breaking. Defaults to 0.

property was_break

True if the break flag was set.

Type

bool

property was_loop

True if the thread/process was loop. False otherwise.

Type

bool

yggdrasil.multitasking.add_aliased_attribute(cls, name, with_lock=False)[source]

Factory to alias an attribute so that it refers to the wrapped object.

Parameters
  • name (str) – Name of attribute to alias.

  • with_lock (bool, optional) – If True, the class’s lock will be acquired before getting the attribute. Defaults to False.

yggdrasil.multitasking.add_aliased_method(cls, name, with_lock=False)[source]

Factory to alias a method so that it refers to the wrapped object.

Parameters
  • name (str) – Name of method to alias.

  • with_lock (bool, optional) – If True, the class’s lock will be acquired before executing the method. Defaults to False.

yggdrasil.multitasking.check_locks()[source]

Check for locks in lock registry that are locked.

yggdrasil.multitasking.check_processes()[source]

Check for processes that are still running.

yggdrasil.multitasking.check_sockets()[source]

Check registered sockets.

yggdrasil.multitasking.check_threads()[source]

Check for threads that are still running.

yggdrasil.multitasking.test_target_error()[source]
yggdrasil.multitasking.test_target_sleep()[source]
yggdrasil.multitasking.wait_on_function(function, timeout=None, on_timeout=False, polling_interval=0.1)[source]

Wait for the function to return True.

Parameters
  • function (callable) – Callable function that takes no arguments and returns a boolean.

  • timeout (float, optional) – Time (in seconds) that should be waited for the process to finish. A value of None will wait indefinitely. Defaults to None.

  • on_timeout (callable, bool, str, optional) – Object indicating what action should be taken in the event that the timeout is reached. If a callable is provided, it will be called. A value of False will cause a TimeoutError to be raised. A value of True will cause the function value to be returned. A string will be used as the error message for a raised timeout error. Defaults to False.

  • polling_interval (float, optional) – Time (in seconds) that should be waited in between function calls. Defaults to 0.1 seconds.

Returns

The result of the function call.

Return type

object

yggdrasil.multitasking.ygg_atexit()[source]

Things to do at exit.

yggdrasil.platform module

This module handle platform compatibility issues.

yggdrasil.runner module

This module provides tools for running models using yggdrasil.

exception yggdrasil.runner.IntegrationError[source]

Bases: BaseException

Error raised when there is an error in an integration.

class yggdrasil.runner.YggFunction(*args, **kwargs)[source]

Bases: yggdrasil.tools.YggClass

This class wraps function-like behavior around a model.

Parameters
  • model_yaml (str, list) – Full path to one or more YAML specification files containing information defining a partial integration. If service_address is set, this should be the name of a service registered with the service manager running at the provided address.

  • service_address (str, optional) – Address for service manager that is capable of running the specified integration. Defaults to None and is ignored.

  • **kwargs – Additional keyword arguments are passed to the YggRunner constructor.

outputs

Input channels providing access to model output.

Type

dict

inputs

Output channels providing access to model input.

Type

dict

runner

Runner for model.

Type

YggRunner

__call__(*args, **kwargs)[source]

Call the model as a function by sending variables.

Parameters
  • *args – Any positional arguments are expected to be input variables in the correct order.

  • **kwargs – Any keyword arguments are expected to be named input variables for the model.

Raises
  • RuntimeError – If an input argument is missing.

  • RuntimeError – If sending an input argument to a model fails.

  • RuntimeError – If receiving an output value from a model fails.

Returns

Returned values for each return variable.

Return type

dict

model_info()[source]

Display information about the wrapped model(s).

stop()[source]

Stop the model(s) from running.

class yggdrasil.runner.YggRunner(*args, **kwargs)[source]

Bases: yggdrasil.tools.YggClass

This class handles the orchestration of starting the model and IO drivers, monitoring their progress, and cleaning up on exit.

Parameters
  • modelYmls (list) – List of paths to yaml files specifying the models that should be run.

  • namespace (str, optional) – Name that should be used to uniquely identify any RMQ exchange. Defaults to the value in the config file.

  • host (str, optional) – Name of the host that the models will be launched from. Defaults to None.

  • rank (int, optional) – Rank of this set of models if run in parallel. Defaults to 0.

  • ygg_debug_level (str, optional) – Level for Ygg debug messages. Defaults to environment variable ‘YGG_DEBUG’.

  • rmq_debug_level (str, optional) – Level for RabbitMQ debug messages. Defaults to environment variable ‘RMQ_DEBUG’.

  • ygg_debug_prefix (str, optional) – Prefix for Ygg debug messages. Defaults to namespace.

  • as_service (bool, optional) – If True, the integration is running as a service. If True, complete_partial is set to True. Defaults to False.

  • complete_partial (bool, optional) – If True, unpaired input/output channels are allowed and reserved for use (e.g. for calling the model as a function). Defaults to False.

  • partial_commtype (dict, optional) – Communicator kwargs that should be be used for the connections to the unpaired channels when complete_partial is True. Defaults to None and will be ignored.

  • yaml_param (dict, optional) – Parameters that should be used in mustache formatting of YAML files. Defaults to None and is ignored.

  • validate (bool, optional) – If True, the validation scripts for each modle (if present), will be run after the integration finishes running. Defaults to False.

namespace

Name that should be used to uniquely identify any RMQ exchange.

Type

str

host

Name of the host that the models will be launched from.

Type

str

rank

Rank of this set of models if run in parallel.

Type

int

modeldrivers

Model drivers associated with this run.

Type

dict

connectiondrivers

Connection drivers for this run.

Type

dict

interrupt_time

Time of last interrupt signal.

Type

float

error_flag

True if one or more models raises an error.

Type

bool

..todo:: namespace, host, and rank do not seem strictly necessary.

property all_drivers

For all drivers.

Type

iterator

atexit(*args, **kwargs)[source]

At exit ensure that the runner has stopped and cleaned up.

bridge_mpi_connections(yml)[source]

Bridge connections over MPI processes.

cleanup()[source]

Perform cleanup operations for all drivers.

closeChannels(force_stop=False)[source]

Stop IO drivers and join the threads.

Parameters

force_stop (bool, optional) – If True, the terminate method is used to stop the drivers. Otherwise, the stop method is used. The stop method will try to exit gracefully while terminate will exit as quickly as possible. Defaults to False.

create_connection_driver(yml)[source]

Create a connection driver instance from the yaml information.

Parameters

yml (yaml) – Yaml object containing driver information.

Returns

An instance of the specified driver.

Return type

object

create_driver(yml, **kwargs)[source]

Create a driver instance from the yaml information.

Parameters

yml (yaml) – Yaml object containing driver information.

Returns

An instance of the specified driver.

Return type

object

distribute_mpi()[source]

Distribute models between MPI processes.

do_client_exits(model)[source]

Perform exits for IO drivers associated with a client model.

Parameters

model (dict) – Dictionary of model parameters including any associated IO drivers.

expand_duplicates()[source]

Expand model copies so they can be split across MPI processes.

get_models(name, rank=None)[source]

Get the set of drivers referenced by a model name.

Parameters
  • name (str, list) – Name of model(s).

  • rank (int, optional) – If provided, only models that will run on MPI processes with this rank will be returned. Defaults to None and is ignored.

Returns

Set of drivers for a model.

Return type

list

io_drivers()[source]

Return the input and output drivers for all models.

Returns

Access to list of I/O drivers.

Return type

iterator

property is_alive

True if all of the models are still running, False otherwise.

Type

bool

loadDrivers()[source]

Load all of the necessary drivers, doing the IO drivers first and adding IO driver environmental variables back tot he models.

pause()[source]

Pause all drivers.

pprint(*args)[source]

Print with color.

printStatus(return_str=False)[source]

Print the status of all drivers, starting with the IO drivers.

reduce_duplicates()[source]

Join model duplicates after they were split between processes.

reset_signal_handler()[source]

Reset signal handlers to old ones.

resume()[source]

Resume all paused drivers.

run(signal_handler=None, timer=None, t0=None)[source]

Run all of the models and wait for them to exit.

Parameters
  • signal_handler (function, optional) – Function that should be used as a signal handler. Defaults to None and is set by set_signal_handler.

  • timer (function, optional) – Function that should be called to get intermediate timing statistics. Defaults to time.time if not provided.

  • t0 (float, optional) – Zero point for timing statistics. Is set using the provided timer if not provided.

Returns

Intermediate times from the run.

Return type

dict

set_signal_handler(signal_handler=None)[source]

Set the signal handler.

Parameters

signal_handler (function, optional) – Function that should handle received SIGINT and SIGTERM signals. Defaults to self.signal_handler.

signal_handler(sig, frame)[source]

Terminate all drivers on interrrupt.

startDrivers()[source]

Start drivers, starting with the IO drivers.

start_server(name)[source]

Start a server driver.

stop_server(name)[source]

Stop a server driver.

terminate()[source]

Immediately stop all drivers, beginning with IO drivers.

waitModels(timeout=False)[source]

Wait for all model drivers to finish. When a model finishes, join the thread and perform exits for associated IO drivers.

yggdrasil.runner.get_runner(models, **kwargs)[source]

Get runner for a set of models, getting run information from the environment.

Parameters
  • models (list) – List of yaml files containing information on the models that should be run.

  • **kwargs – Additonal keyword arguments are passed to YggRunner.

Returns

Runner for the provided models.

Return type

YggRunner

Raises

Exception – If config option ‘namespace’ in ‘rmq’ section not set.

yggdrasil.runner.run(*args, **kwargs)[source]

yggdrasil.scanf module

Small scanf implementation.

Python has powerful regular expressions but sometimes they are totally overkill when you just want to parse a simple-formatted string. C programmers use the scanf-function for these tasks (see link below).

This implementation of scanf translates the simple scanf-format into regular expressions. Unlike C you can be sure that there are no buffer overflows possible.

For more information see
Original code from:

http://code.activestate.com/recipes/502213-simple-scanf-implementation/

Modified original to make the %f more robust, as well as added %* modifier to skip fields.

Version: 1.3.3

Releases:
1.0
2010-10-11
  • Initial release

1.1
2010-10-13
  • Changed regex from ‘match’ (only matches at beginning of line) to ‘search’ (matches anywhere in line)

  • Bugfix - ignore cast for skipped fields

1.2
2013-05-30
  • Added ‘collapseWhitespace’ flag (defaults to True) to take the search string and replace all whitespace with regex string to match repeated whitespace. This enables better matching in log files where the data has been formatted for easier reading. These cases have variable amounts of whitespace between the columns, depending on the number of characters in the data itself.

1.3
2016-01-18
  • Add ‘extractdata’ function.

1.3.1
2016-06-23
  • Release to PyPi, now including README.md

Updates by Meagan Lang:
2018-04-12
  • Added ability to parse components of field specifiers and cope with whitespace padding.

yggdrasil.scanf.scanf(format, s=None, collapseWhitespace=True)[source]
scanf supports the following formats:

%c One character %5c 5 characters %d, %i int value %7d, %7i int value with length 7 %f float value %o octal value %X, %x hex value %s string terminated by whitespace

Examples: >>> scanf(“%s - %d errors, %d warnings”, “/usr/sbin/sendmail - 0 errors, 4 warnings”) (‘/usr/sbin/sendmail’, 0, 4) >>> scanf(“%o %x %d”, “0123 0x123 123”) (66, 291, 123)

If the parameter s is a file-like object, s.readline is called. If s is not specified, stdin is assumed.

The function returns a tuple of found values or None if the format does not match.

yggdrasil.scanf.scanf_compile(format, collapseWhitespace=True)[source]

Translate the format into a regular expression

For example: >>> format_re, casts = _scanf_compile(‘%s - %d errors, %d warnings’) >>> print format_re.pattern (S+) - ([+-]?d+) errors, ([+-]?d+) warnings

Translated formats are cached for faster reuse

yggdrasil.schema module

class yggdrasil.schema.ComponentSchema(schema_type, subtype_key, schema_registry=None, module=None, schema_subtypes=None)[source]

Bases: object

Schema information for one component.

Parameters
  • schema_type (str) – The name of the component.

  • subtype_key (str) – The name of the schema property/class attribute that should be used to differentiate between subtypes of this component.

  • schema_registry (SchemaRegistry) – Registry of schemas that this schema is dependent on.

  • **kwargs – Additional keyword arguments are entries in the component schema.

  • schema_type – The name of the component.

  • schema_registry – Registry of schemas.

  • subtype_key – Schema property that is used to differentiate between subtypes of this component.

  • schema_subtypes (dict) – Mapping between component class names and the associated values of the subtype_key property for this component.

append(comp_cls, verify=False)[source]

Append component class to the schema.

Parameters
  • comp_cls (class) – Component class that should be added.

  • verify (bool, optional) – If True, verify the schema after adding the component class. Defaults to False.

property base_subtype_class

Base class for the subtype.

Type

ComponentClass

property base_subtype_class_name

Name of base class for the subtype.

Type

str

property class2subtype

Mapping from class to list of subtypes.

Type

dict

property classes

All available classes for this schema.

Type

list

property default_subtype

Default subtype.

Type

str

classmethod from_registry(schema_type, registry, **kwargs)[source]

Construct a ComponentSchema from a registry entry.

Parameters
  • schema_type (str) – Name of component type to build.

  • registry (dict) – Registry information for the component.

  • **kwargs – Additional keyword arguments are passed to the class __init__ method.

Returns

Schema with information from classes.

Return type

ComponentSchema

classmethod from_schema(schema, schema_registry=None)[source]

Construct a ComponentSchema from a schema.

Parameters

schema (dict) – Schema.

Returns

Schema with information from schema.

Return type

ComponentSchema

property full_schema

Schema for evaluating YAML input file that fully specifies the properties for each component.

Type

dict

get_schema(relaxed=False, allow_instance=False, for_form=False)[source]

Get the schema defining this component.

Parameters
  • relaxed (bool, optional) – If True, the returned schema (and any definitions it includes) are relaxed to allow for objects with objects with additional properties to pass validation. Defaults to False.

  • allow_instance (bool, optional) – If True, the returned schema will validate instances of this component in addition to documents describing a component. Defaults to False.

  • for_form (bool, optional) – If True, the returned schema will be formatted for easy parsing by form generation tools. Defaults to False. Causes relaxed and allow_instance to be ignored.

Returns

Schema for this component.

Return type

dict

get_subtype_properties(subtype)[source]

Get the valid properties for a specific subtype.

Parameters

subtype (str) – Name of the subtype to get keys for.

Returns

Valid properties for the specified subtype.

Return type

list

get_subtype_schema(subtype, unique=False, relaxed=False, allow_instance=False, for_form=False)[source]

Get the schema for the specified subtype.

Parameters
  • subtype (str) – Component subtype to return schema for. If ‘base’, the schema for evaluating the component base will be returned.

  • unique (bool, optional) – If True, the returned schema will only contain properties that are specific to the specified subtype. If subtype is ‘base’, these will be properties that are valid for all of the registerd subtypes. Defaults to False.

  • relaxed (bool, optional) – If True, the schema will allow additional properties. Defaults to False.

  • allow_instance (bool, optional) – If True, the returned schema will validate instances of this component in addition to documents describing a component. Defaults to False.

  • for_form (bool, optional) – If True, the returned schema will be formatted for easy parsing by form generation tools. Defaults to False. Causes relaxed and allow_instance to be ignored.

Returns

Schema for specified subtype.

Return type

dict

identify_subtype(doc)[source]

Identify the subtype associated with a document by validating it against the schemas for the different subtypes.

Parameters

doc (dict) – JSON object that conforms to one of the component subtypes.

Returns

Name of the subtype that valdiates the provided document.

Return type

str

property properties

Valid properties for this component.

Type

list

property schema

Schema for this component.

Type

dict

property subtype2class

Mapping from subtype to class.

Type

dict

property subtypes

All subtypes for this schema type.

Type

list

class yggdrasil.schema.SchemaDict[source]

Bases: collections.OrderedDict

OrderedDict subclass for ordering schemas on read in Python 2.

class yggdrasil.schema.SchemaRegistry(registry=None, required=None)[source]

Bases: object

Registry of schema’s for different integration components.

Parameters
  • registry (dict, optional) – Dictionary of registered components. Defaults to None and the registry will be empty.

  • required (list, optional) – Components that are required. Defaults to [‘comm’, ‘file’, ‘model’, ‘connection’]. Ignored if registry is None.

Raises

ValueError – If registry is provided and one of the required components is missing.

add(k, v, verify=False)[source]

Add a new component schema to the registry.

property definitions

Schema definitions for different components.

Type

dict

property form_schema

Schema for generating a YAML form.

Type

dict

classmethod from_file(fname)[source]

Create a SchemaRegistry from a file.

Parameters

fname (str) – Full path to the file the schema should be loaded from.

property full_schema

Schema for evaluating YAML input file that fully specifies the properties for each component.

Type

dict

get(k, *args, **kwargs)[source]

Return a component schema from the registry.

get_component_keys(comp_name)[source]

Get the properties associated with a certain component.

Parameters

comp_name (str) – Name of the component to return keys for.

Returns

All of the valid properties for the specified component.

Return type

list

get_component_schema(comp_name, subtype=None, relaxed=False, allow_instance=False, allow_instance_definitions=False, for_form=False)[source]

Get the schema for a certain component.

Parameters
  • comp_name (str) – Name of the component to get the schema for.

  • subtype (str, optional) – Component subtype to get schema for. Defaults to None and the schema for evaluating any subtype of the specified component is returned.

  • relaxed (bool, optional) – If True, the returned schema (and any definitions it includes) are relaxed to allow for objects with objects with additional properties to pass validation. Defaults to False.

  • allow_instance (bool, optional) – If True, the returned schema will validate instances of this component in addition to documents describing a component. Defaults to False.

  • allow_instance_definitions (bool, optional) – If True, the definitions in the returned schema will allow for instances of the components. Defaults to False.

  • for_form (bool, optional) – If True, the returned schema will be formatted for easy parsing by form generation tools. Defaults to False. Causes relaxed and allow_instance to be ignored.

  • **kwargs – Additonal keyword arguments are paseed to get_schema or get_subtype_schema for the selected component type.

Returns

Schema for the specified component.

Return type

dict

get_definitions(relaxed=False, allow_instance=False, for_form=False, dont_copy=False)[source]

Get schema definitions for the registered components.

Parameters
  • relaxed (bool, optional) – If True, the returned schema (and any definitions it includes) are relaxed to allow for objects with objects with additional properties to pass validation. Defaults to False.

  • allow_instance (bool, optional) – If True, the returned definitions will validate instances of the components in addition to documents describing components. Defaults to False.

  • for_form (bool, optional) – If True, the returned schema will be formatted for easy parsing by form generation tools. Defaults to False. Causes relaxed and allow_instance to be ignored.

  • dont_copy (bool, optional) – If True, a the cached definitions are returned without copying. Defaults to False.

Returns

Schema defintiions for each of the registered components.

Return type

dict

get_schema(relaxed=False, allow_instance=False, for_form=False)[source]

Get the schema defining this component.

Parameters
  • relaxed (bool, optional) – If True, the returned schema (and any definitions it includes) are relaxed to allow for objects with objects with additional properties to pass validation. Defaults to False.

  • allow_instance (bool, optional) – If True, the returned schema will validate instances of this component in addition to documents describing a component. Defaults to False.

  • for_form (bool, optional) – If True, the returned schema will be formatted for easy parsing by form generation tools. Defaults to False. Causes relaxed and allow_instance to be ignored.

Returns

Schema for this component.

Return type

dict

is_valid_component(comp_name, obj)[source]

Determine if an object is a valid represenation of a component.

Parameters
  • comp_name (str) – Name of the component to validate against.

  • obj (object) – Object to validate.

Returns

True if the object is valid, False otherwise.

Return type

bool

items()[source]
keys()[source]
load(fname)[source]

Load schema from a file.

Parameters

fname (str) – Full path to the file the schema should be loaded from.

property model_form_schema

Schema for generating a model YAML form.

Type

dict

property model_form_schema_props

Information about how properties should be modified for the model form schema.

Type

dict

normalize(obj, backwards_compat=False, **kwargs)[source]

Normalize an object against this schema.

Parameters
  • obj (object) – Object to normalize.

  • **kwargs – Additional keyword arguments are passed to normalize_instance.

Returns

Normalized object.

Return type

object

classmethod register_normalizer(path)[source]

Register a normalizer that will be applied to elements in the instance at the specified path.

Parameters

path (tuple) – Location in schema where normalizer will be applied.

Returns

Decorator for registering the normalizer function.

Return type

function

save(fname)[source]

Save the schema to a file.

Parameters
  • fname (str) – Full path to the file the schema should be saved to.

  • schema (dict) – yggdrasil YAML options.

property schema

Schema for evaluating YAML input file.

Type

dict

validate(obj, **kwargs)[source]

Validate an object against this schema.

Parameters
  • obj (object) – Object to valdiate.

  • **kwargs – Additional keyword arguments are passed to validate_instance.

validate_component(comp_name, obj, **kwargs)[source]

Validate an object against a specific component.

Parameters
  • comp_name (str) – Name of the component to validate against.

  • obj (object) – Object to validate.

  • **kwargs – Additional keyword arguments are passed to get_component_schema.

validate_model_submission(obj, **kwargs)[source]

Validate an object against the schema for models submitted to the yggdrasil model repository.

Parameters
  • obj (object) – Object to validate.

  • **kwargs – Additional keyword arguments are ignored.

yggdrasil.schema.cdriver2commtype(driver)[source]

Convert a connection driver to a file type.

Parameters

driver (str) – The name of the connection driver.

Returns

The corresponding file type for the driver.

Return type

str

yggdrasil.schema.cdriver2filetype(driver)[source]

Convert a connection driver to a file type.

Parameters

driver (str) – The name of the connection driver.

Returns

The corresponding file type for the driver.

Return type

str

yggdrasil.schema.clear_schema()[source]

Clear global schema.

yggdrasil.schema.convert_extended2base(s)[source]

Covert schema from the extended form to a strictly JSON form.

Parameters

s (object) – Object to updated.

Returns

Updated JSON object.

Return type

object

yggdrasil.schema.create_schema()[source]

Create a new schema from the registry.

yggdrasil.schema.get_json_schema(fname_dst=None, indent=None)[source]

Return the yggdrasil schema as a strictly JSON schema without any of the extended datatypes.

Parameters
  • fname_dst (str, optional) – Full path to file where the JSON schema should be saved. Defaults to None and no file is created.

  • indent (str, optional) – Indentation that should be used when saving the schema to a file.

Returns

Converted structure.

Return type

dict

yggdrasil.schema.get_model_form_schema(fname_dst=None, **kwargs)[source]

Return the yggdrasil schema that can be used to generate a form for creating a model specification file.

Parameters
  • fname_dst (str, optional) – Full path to file where the JSON schema should be saved. Defaults to None and no file is created.

  • **kwargs – Additional keyword arguments are passed to the json.dump call if fname_dst is provided and ignored otherwise.

Returns

Schema structure.

Return type

dict

yggdrasil.schema.get_schema(fname=None)[source]

Return the yggdrasil schema for YAML options.

Parameters

fname (str, optional) – Full path to the file that the schema should be loaded from. If the file dosn’t exist, it is created. Defaults to _schema_fname.

Returns

yggdrasil YAML options.

Return type

dict

yggdrasil.schema.init_schema(fname=None)[source]

Initialize global schema.

yggdrasil.schema.load_schema(fname=None)[source]

Return the yggdrasil schema for YAML options.

Parameters

fname (str, optional) – Full path to the file that the schema should be loaded from. If the file dosn’t exist, it is created. Defaults to _schema_fname.

Returns

yggdrasil YAML options.

Return type

dict

yggdrasil.schema.migrate_keys(from_dict, to_dict, exclude_key_list=None, include_key_list=None)[source]

Migrate keys from one component to another that are not in a list of predefined keys.

Parameters
  • from_dict (dict) – Component dictionary to migrate keys from.

  • to_dict (list) – List of component dictionaries to migrate keys to. If this is an empty list, keys will not be migrated.

  • exclude_key_list (list, optional) – List of keys in from_dict that should not be migrated to to_dict. All keys in include_key_list that are not in this list are moved. Defaults to None and no keys are excluded.

  • include_key_list (list, optional) – List of keys that should be migrated from from_dict to to_dict dictionaries. If not provided, all keys in from_dict that are not in exclude_key_list are moved. Defaults to None and all keys in from_dict are included.

yggdrasil.schema.ordered_dump(data, **kwargs)[source]

Dump object as a YAML document, representing SchemaDict objects as mapping type.

Parameters
  • data (object) – Python object that should be dumped.

  • **kwargs – Additional keyword arguments are passed to encode_yaml.

Returns

YAML document representating data.

Return type

str

yggdrasil.schema.ordered_load(stream, object_pairs_hook=<class 'yggdrasil.schema.SchemaDict'>, **kwargs)[source]

Load YAML document from a file using a specified class to represent mapping types that allows for ordering.

Parameters
  • stream (file) – File stream to load the schema YAML from.

  • object_pairs_hook (type, optional) – Class that should be used to represent loaded maps. Defaults to SchemaDict.

  • **kwargs – Additional keyword arguments are passed to decode_yaml.

Returns

Result of ordered load.

Return type

object

yggdrasil.schema.rwmeth2filetype(rw_meth)[source]

Get the alternate properties that corresponding to the old read_meth/write_meth keywords.

Parameters

rw_meth (str) – Read/write method name.

Returns

Property values equivalent to provided read/write method.

Return type

dict

yggdrasil.schema.standardize(instance, keys, is_singular=False, suffixes=None, altkeys=None)[source]

Standardize a component such that each key contains a list of dictionaries.

Parameters
  • instance (dict) – Component to standardize.

  • keys (list) – Keys to standardize in the instance.

  • is_singular (bool, optional) – If False, the keys are assumed to be plural and singular alternatives are also checked. If True, the keys are assumed to be singular and plural alternatives are also checked. Defaults to False.

  • suffixes (list, optional) – Suffixes to add to the keys to get a set of alternate keys that should also be checked. Defaults to None as is ignored.

  • altkeys (list, optional) – List of lists of alternate keys. Defaults to None.

yggdrasil.schema.update_constants(schema=None)[source]

Update constants.py with info from the schema.

yggdrasil.services module

exception yggdrasil.services.ClientError[source]

Bases: BaseException

Error raised by errors when calling the server from the client.

class yggdrasil.services.FlaskService(*args, **kwargs)[source]

Bases: yggdrasil.services.ServiceBase

Flask based service.

call(request, **kwargs)[source]

Send a request.

Parameters
  • request (object) – JSON serializable request.

  • **kwargs – Keyword arguments are ignored.

Returns

Response.

Return type

object

default_address = 'http://localhost:{port}'
default_commtype = 'rest'
default_port = 5000
deserialize(msg)[source]

Deserialize a message.

Parameters

msg (str) – Message to deserialize.

Returns

Deserialized message.

Return type

object

classmethod is_installed()[source]

bool: True if the class is fully installed, False otherwise.

run_server()[source]

Begin listening for requests.

serialize(msg)[source]

Serialize a message.

Parameters

msg (object) – Message to serialize.

Returns

Serialized message.

Return type

str

serialize_response(response)[source]

Serialize a response message.

Parameters

request (object) – Request to serialize.

Returns

Serialized request.

Return type

str

service_type = 'flask'
set_log_level(log_level)[source]

Set the logging level.

Parameters

log_level (int) – Logging level.

setup_client(*args, **kwargs)[source]

Set up the machinery for sending requests.

setup_server(*args, **kwargs)[source]

Set up the machinery for receiving requests.

Parameters
  • *args – Arguments are ignored.

  • **kwargs – Keyword arguments are ignored.

shutdown()[source]

Shutdown the process from the server.

yggdrasil.services.IntegrationServiceManager(service_type=None, **kwargs)[source]

Start a management service to track running integrations.

Parameters
  • service_type (ServiceBase, str, optional) – Base class that should be used. Defaults to (‘services’, ‘default_type’) configuration options, if set, and ‘flask’ if not. If there is an address provided, the service type will be determined by parsing the address.

  • **kwargs – Additional keyword arguments are used to intialized the manager class instance.

class yggdrasil.services.IntegrationServiceRegistry(filename='~/.yggdrasil_services.yml')[source]

Bases: object

Class for managing integration services.

Parameters

filename (str, optional) – File where the registry will be/is stored. Defaults to ‘~/.yggdrasil_services.yml’.

add(name, yamls=None, **kwargs)[source]

Add an integration service to the registry.

Parameters
  • name (str) – Name that will be used to access the integration service when starting or stopping it.

  • yamls (str, list) – Set of one or more YAML specification files defining the integration.

  • **kwargs – Additional keyword arguments are added to the new entry.

Raises

ValueError – If there is already an integration with the specified name.

add_from_repository(model_repository, directory=None)[source]

Add integration services to the registry from a repository of model YAMLs.

Parameters
  • model_repository (str) – URL of directory in a Git repository containing YAMLs that should be added to the model registry.

  • directory (str, optional) – Directory where services from the model_repository should be cloned. Defaults to ‘~/.yggdrasil_service’.

Returns

The directory where the repositories were cloned.

Return type

str

load()[source]

Load the dictionary of existing integrations that have been registered.

Returns

Existing registry of integrations.

Return type

dict

load_collection(name)[source]

Read a collection of integration registry entries from an YAML.

Parameters

name (str) – Full path to a YAML file containing one or more registry entries mapping between integration name and YAML specification files.

Returns

Loaded registry entries.

Return type

dict

property registry

Existing registry of integrations.

Type

dict

remove(name)[source]

Remove an integration service from the registry.

Parameters

name (str) – Name associated with the integration service that should be removed from the registry.

Raises

KeyError – If there is not an integration service associated with the specified name.

save(registry)[source]

Save the registry to self.filename.

Parameters

registry (dict) – Dictionary of integrations to save.

class yggdrasil.services.RMQService(*args, **kwargs)[source]

Bases: yggdrasil.services.ServiceBase

RabbitMQ based service.

call(request, timeout=10.0, **kwargs)[source]

Send a request.

Parameters
  • request (str) – Serialized request.

  • timeout (float, optional) – Time (in seconds) that should be waited for a response to be returned. Defaults to 10.

  • **kwargs – Keyword arguments are ignored.

Returns

Serialized response.

Return type

str

default_commtype = 'rmq'
default_port = 5672
classmethod is_installed()[source]

bool: True if the class is fully installed, False otherwise.

property is_running

True if the server is running.

Type

bool

run_server()[source]

Listen for requests.

service_type = 'rmq'
setup_client(*args, **kwargs)[source]

Set up the machinery for sending requests.

Parameters
  • *args – Arguments are used to initialize the RabbitMQ connections via _init_rmq.

  • **kwargs – Keyword arguments are used to initialize the RabbitMQ connections via _init_rmq.

setup_server(*args, **kwargs)[source]

Set up the machinery for receiving requests.

Parameters
  • *args – Arguments are used to initialize the RabbitMQ connections via _init_rmq.

  • **kwargs – Keyword arguments are used to initialize the RabbitMQ connections via _init_rmq.

shutdown(in_callback=False)[source]

Shutdown the process from the server.

exception yggdrasil.services.ServerError[source]

Bases: BaseException

Error raised when there was an error on the server.

class yggdrasil.services.ServiceBase(*args, **kwargs)[source]

Bases: yggdrasil.tools.YggClass

Base class for sending/responding to service requests.

Parameters
  • name (str) – Name that should be used to initialize an address for the service.

  • for_request (bool, optional) – If True, a client-side connection is initialized. If False a server-side connection is initialized. Defaults to False.

  • address (str, optional) – The address that the service can be accessed from. Defaults to (‘services’, ‘address’) configuration option, if set, and None if not.

  • *args – Additional arguments are used to initialize the client/server connection.

  • **kwargs – Additional keyword arguments are used to initialize the client/server connection.

call(*args, **kwargs)[source]

Send a request.

default_address = None
default_port = None
deserialize(msg)[source]

Deserialize a message.

Parameters

msg (str) – Message to deserialize.

Returns

Deserialized message.

Return type

object

deserialize_request(request)[source]

Deserialize a request message.

Parameters

request (str) – Serialized request.

Returns

Deserialized request.

Return type

object

classmethod is_installed()[source]

bool: True if the class is fully installed, False otherwise.

property is_running

True if the server is running.

Type

bool

process_request(request, **kwargs)[source]

Process a request and return a response.

Parameters
  • request (str) – Serialized request that should be processed.

  • **kwargs – Additional keyword arguments are passed to the respond method.

Returns

Serialized response.

Return type

str

process_response(response)[source]

Process a response.

Parameters

response (str) – Serialized response that should be processed.

Returns

The deserialized, processed response.

Return type

object

respond(request, **kwargs)[source]

Create a response to the request.

run_server()[source]

Begin listening for requests.

send_request(request, **kwargs)[source]

Send a request.

Parameters
  • request (object) – Request to send.

  • **kwargs – Additional keyword arguments are passed to the call method.

Returns

Response.

Return type

object

serialize(msg)[source]

Serialize a message.

Parameters

msg (object) – Message to serialize.

Returns

Serialized message.

Return type

str

serialize_response(response)[source]

Serialize a response message.

Parameters

request (object) – Request to serialize.

Returns

Serialized request.

Return type

str

set_log_level(log_level)[source]

Set the logging level.

Parameters

log_level (int) – Logging level.

setup_client(*args, **kwargs)[source]

Set up the machinery for sending requests.

setup_server(*args, **kwargs)[source]

Set up the machinery for receiving requests.

shutdown(*args, **kwargs)[source]

Shutdown the process from the server.

start_server(remote_url=None, with_coverage=False, log_level=None, model_repository=None)[source]

Start the server.

Parameters
  • remote_url (str optional) – Address for the URL that remote integrations will use to connect to this server. Defaults to None and is set based on the YGGDRASIL_SERVICE_HOST_URL environment variable if it is set and is the local address otherwise.

  • with_coverage (bool, optional) – If True, the server is started with coverage. Defaults to False.

  • log_level (int, optional) – Level of log messages that should be printed. Defaults to None and is ignored.

  • model_repository (str, optional) – URL of directory in a Git repository containing YAMLs that should be added to the model registry. Defaults to None and is ignored.

wait_for_server(timeout=15.0)[source]

Wait for a service to start running.

Parameters

timeout (float, optional) – Time (in seconds) that should be waited for the server to start. Defaults to 15.

Raises

RuntimeError – If the time limit is reached and the server still hasn’t started.

yggdrasil.services.create_service_manager_class(service_type=None)[source]

Create an integration manager service with the specified base.

Parameters

service_type (ServiceBase, str, optional) – Base class that should be used. Defaults to (‘services’, ‘default_type’) configuration options, if set, and ‘flask’ if not.

Returns

Subclass of ServiceBase to handle starting/stopping integrations

running as services.

Return type

type

yggdrasil.services.validate_model_submission(fname)[source]

Validate a YAML file according to the standards for submission to the yggdrasil model repository.

Parameters

fname (str) – YAML file to validate or directory in which to check each of the YAML files.

yggdrasil.timing module

class yggdrasil.timing.TimedRun(*args, **kwargs)[source]

Bases: yggdrasil.tools.YggClass

Class to time sending messages from one language to another.

Parameters
  • lang_src (str) – Language that messages should be sent from.

  • lang_dst (str) – Language that messages should be sent to.

  • test_name (str, optional) – Name of the example. Defaults to ‘timed_pipe’.

  • filename (str, optional) – Full path to the file where timing statistics will be saved. This can be a pyperf json or a Python pickle of run data. Defaults to ‘scalings_{test_name}_{comm_type}.json’ if dont_use_pyperf is False, otherwise the extension is ‘.dat’.

  • comm_type (str, optional) – Name of communication class that should be used for tests. Defaults to the current default comm class.

  • platform (str, optional) – Platform that the test should be run on. If the data doesn’t already exist and this doesn’t match the current platform, an error will be raised. Defaults to the current platform.

  • python_ver (str, optional) – Version of Python that the test should be run with. If the data doesn’t already exist and this doesn’t match the current version of python, an error will be raised. Defaults to the current version of python.

  • max_errors (int, optional) – Maximum number of errors that should be retried. Defaults to 5.

  • matlab_running (bool, optional) – If True, the test will assert that there is an existing Matlab engine before starting, otherwise the test will assert that there is not an existing Matlab engine. Defaults to False.

  • dont_use_pyperf (bool, optional) – If True, the timings will be run without using the pyperf package. Defaults to False.

lang_src

Language that messages should be sent from.

Type

str

lang_dst

Language that messages should be sent to.

Type

str

platform

Platform that the test is being run on.

Type

str

python_ver

Version of Python that the test should be run with.

Type

str

filename

Full path to the file where timing statistics will be saved. This can be a pyperf json or a Python pickle of run data.

Type

str

comm_type

Name of communication class that should be used for tests.

Type

str

max_errors

Maximum number of errors that should be retried.

Type

int

matlab_running

True if there was a Matlab engine running when the test was created. False otherwise.

Type

bool

dont_use_pyperf

If True, the timings will be run without using the pyperf package.

Type

bool

after_run(run_uuid, result)[source]

Actions that should be performed after a run.

Parameters
  • nmsg (int) – Number of messages that were sent.

  • msg_size (int) – Size of each message that were sent.

  • result (float) – Time required (in seconds) to execute the program.

property base_msg_count

Message count to use for tests varrying message size.

Type

int

property base_msg_size

Message size to use for tests varrying message count.

Type

int

before_run(nmsg, msg_size)[source]

Actions that should be performed before a run.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • msg_size (int) – Size of each message that should be sent.

Returns

Unique identifier for the run.

Return type

str

can_run(raise_error=False)[source]

Determine if the test can be run from the current platform and python version.

Parameters

raise_error (bool, optional) – If True, an error will be raised if the test cannot be completed from the current platform. Defaults to False.

Returns

True if the test can be run, False otherwise.

Return type

bool

check_output(fout, nmsg, msg_size)[source]

Assert that the output file contains the expected result.

Parameters
  • fout (str) – The file that should be checked.

  • nmsg (int) – The number of messages that will be sent.

  • msg_size (int) – The size of the the messages that will be sent.

classmethod class_plot(lang_src='python', lang_dst='python', **kwargs)[source]

Create the class for a given combo of languages and comm types, then call plot_scaling_joint.

Parameters
  • lang_src (str, optional) – Language that messages should be sent from. Defaults to ‘python’.

  • lang_dst (str, optional) – Language that messages should be sent to. Defaults to ‘python’.

  • **kwargs – Additional keywords are passed to either the class constructor or plot_scaling_joint as appropriate.

Returns

Pair of axes containing the

plotted scalings and the fit.

Return type

tuple(matplotlib.Axes, matplotlib.Axes)

cleanup_output(fout)[source]

Cleanup the output file.

Parameters

fout (str) – The file to be cleaned up.

property data

Timing statistics data.

Type

dict or pyperf.BenchmarkSuite

property default_msg_count

Default message count for scaling tests.

Type

list

property default_msg_size

Default message sizes for scaling tests. This will vary depending on the comm type so that the maximum size is not more than 10x larger than the maximum message size.

Type

list

entry_name(nmsg, msg_size)[source]

Get a unique identifier for a run.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • msg_size (int) – Size of each message that should be sent.

fit_scaling_count(msg_size=None, counts=None, **kwargs)[source]

Do a linear fit to the scaling of execution time with message count.

Parameters
  • msg_size (int, optional) – Size of each message that should be sent. Defaults to self.base_msg_size.

  • counts (list, optional) – List of counts to test. Defaults to self.default_msg_count if not provided.

  • **kwargs – Additional keyword arguments are passed to scaling_count.

Returns

The slope and intercept of the linear fit.

Return type

tuple

fit_scaling_size(msg_count=None, sizes=None, **kwargs)[source]

Do a linear fit to the scaling of execution time with message count.

Parameters
  • msg_count (int, optional) – Number of messages that should be sent for each size. Defaults to self.base_msg_count.

  • sizes (list, optional) – List of sizes to test. Defaults to self.default_msg_size if not provided.

  • **kwargs – Additional keyword arguments are passed to scaling_size.

Returns

The slope and intercept of the linear fit.

Return type

tuple

get_entry(name)[source]

Get values for an entry.

Parameters

name (str) – Name of the entry to return.

get_new_uuid()[source]

Get a new unique ID.

Returns

Unique identifier.

Return type

str

get_yaml_dst(lang)[source]

Get the yaml entry for the destination model.

Parameters

lang (str) – Language for the destination model.

get_yaml_src(lang)[source]

Get the yaml entry for the source model.

Parameters

lang (str) – Language for the source model.

has_entry(name)[source]

Check to see if there is an entry with the provided name.

Parameters

name (str) – Name of the entry to check for.

load(as_json=False)[source]

Load scalings data from a pyperf BenchmarkSuite json file or a Python pickle.

Parameters

as_json (bool, optional) – If True and self.dont_use_pyperf is False, the pyperf BenchmarkSuite data will be loaded as a dictionary from the json. Defaults to False.

Returns

Loaded scalings data. None is returned

if the file does not exit.

Return type

pyperf.BenchmarkSuite or dict

make_yamlfile(path)[source]

Create a YAML file for running the test.

Parameters

path (str) – Full path to file where the YAML should be saved.

property max_msg_size

Largest size of message that can be sent without being split.

Type

int

output_content(nmsg, msg_size)[source]

Get the result that should be output to file during the run.

Parameters
  • nmsg – The number of messages that will be sent.

  • msg_sizze – The size of the the messages that will be sent.

Returns

The contents expected in the file.

Return type

str

property output_file_format

Full path to the output file created by the run.

Type

str

plot_scaling(msg_size, msg_count, axs=None, label=None, xscale=None, yscale='linear', plot_kws={}, time_method='average', per_message=False, **kwargs)[source]

Plot scaling of run time with a variable.

Parameters
  • msg_size (int, list, np.ndarray) – List of message sizes to use as x variable, or message size to use when plotting dependent on message count.

  • msg_count (int, list, np.ndarray) – List of message counts to use as x variable, or message count to use when plotting dependent on message size.

  • axs (matplotlib.Axes, optional) – Axes object that line should be added to. If not provided, one is created.

  • label (str, optional) – Label that should be used for the line. Defaults to None.

  • xscale (str, optional) – ‘log’ or ‘linear’ to indicate what scale the x axis should use. Defaults to ‘linear’.

  • yscale (str, optional) – ‘log’ or ‘linear’ to indicate what scale the y axis should use. Defaults to ‘linear’.

  • plot_kws (dict, optional) – Ploting keywords that should be passed. Defaults to {}.

  • time_method (str, optional) – Timing method that should be used. Valid values include ‘bestof’ and ‘average’. Defaults to ‘average’.

  • per_message (bool, optional) – If True, the time per message is returned rather than the total time. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to scaling_size or scaling_count.

Returns

Axes containing the plotted scaling.

Return type

matplotlib.Axes

plot_scaling_joint(msg_size0=None, msg_count0=None, msg_size=None, msg_count=None, axs=None, **kwargs)[source]

Plot scaling of run time with both count and size, side by side. Anywhere data is exchanged as a tuple for each plot, the plot of scaling with count is first and the scaling with size is second.

Parameters
  • msg_size0 (int, optional) – Size of messages to use for count scaling. Defaults to self.base_msg_size.

  • msg_count0 (int, optional) – Number of messages to use for size scaling. Defaults to self.base_msg_count.

  • msg_size (list, np.ndarray, optional) – List of message sizes to use as x variable on the size scaling plot. Defaults to self.default_msg_size.

  • msg_count (list, np.ndarray, optional)) – List of message counts to use as x variable on the count scaling plot. Defaults to self.default_msg_count.

  • axs (tuple, optional) – Pair of axes objects that lines should be added to. If not provided, they are created.

  • **kwargs – Additional keyword arguments are passed to plot_scaling.

Returns

Pair of axes containing the

plotted scalings and the fit.

Return type

tuple(matplotlib.Axes, matplotlib.Axes)

property pyperfscript

Format string for creating a pyperf script.

Type

str

reload()[source]

Reload scalings data and store it in the data attribute.

remove_entry(name)[source]

Remove all runs associated with an entry.

Parameters

name (str) – Name of the entry to be removed.

run(run_uuid, timer=<built-in function time>, t0=None)[source]

Run test sending a set of messages between the designated models.

Parameters
  • run_uuid (str) – Unique ID for the run.

  • timer (function, optional) – Function that should be called to get intermediate timing statistics. Defaults to time.time if not provided.

  • t0 (float, optional) – Zero point for timing statistics. Is set using the provided timer if not provided.

Returns

Intermediate times from the run.

Return type

dict

save(data, overwrite=False)[source]

Save scalings data to a new pyperf BenchmarkSuite json file or a Python pickle. If the file exists and overwrite is not set, an error will be raised. No file is written if data is None.

Parameters
  • data (pyperf.BenchmarkSuite or dict) – Data to be saved.

  • overwrite (bool, optional) – If True, any existing file will be overwritten. Defaults to False.

Raises

RuntimeError – If the file already exists and overwrite is False.

scaling_count(msg_size, counts=None, min_count=1, max_count=100, nsamples=10, scaling='linear', per_message=False, **kwargs)[source]

Get scaling of run time with message count.

Parameters
  • msg_size (int) – Size of each message that should be sent.

  • counts (list, optional) – List of counts to test. Defaults to None and a list is created based on the other keyword arguments.

  • min_count (int, optional) – Minimum message count that should be timed. Defaults to 1. This is ignored if ‘counts’ is provided.

  • max_count (int, optional) – Maximum message count that should be timed. Defaults to 100. This is ignored if ‘counts’ is provided.

  • nsamples (int, optional) – Number of samples that should be done between ‘min_count’ and ‘max_count’. Defaults to 10. This is ignored if ‘counts’ is provided.

  • scaling (str, optional) – Scaling for sampling of message counts between ‘min_count’ and ‘max_count’. Defaults to ‘linear’. This is ignored if ‘counts’ is provided.

  • per_message (bool, optional) – If True, the time per message is returned rather than the total time. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to time_run.

Returns

Lists of counts timed, minimum execution time, average

execution times, and standard deviations.

Return type

tuple

scaling_size(nmsg, sizes=None, min_size=1, max_size=10000000.0, nsamples=10, scaling='log', per_message=False, **kwargs)[source]

Get scaling of run time with message size.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • sizes (list, optional) – List of sizes to test. Defaults to None and a list is created based on the other keyword arguments.

  • min_size (int, optional) – Minimum message size that should be timed. Defaults to 1. This is ignored if ‘sizes’ is provided.

  • max_size (int, optional) – Maximum message size that should be timed. Defaults to 1e7. This is ignored if ‘sizes’ is provided.

  • nsamples (int, optional) – Number of samples that should be done between ‘min_size’ and ‘max_size’. Defaults to 10. This is ignored if ‘sizes’ is provided.

  • scaling (str, optional) – Scaling for sampling of message sizes between ‘min_size’ and ‘max_size’. Defaults to ‘linear’. This is ignored if ‘sizes’ is provided.

  • per_message (bool, optional) – If True, the time per message is returned rather than the total time. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to time_run.

Returns

Lists of sizes timed, minimum execution times, average

execution times, and standard deviations.

Return type

tuple

property source_dst

Source file for language messages will be sent to.

Type

str

property source_src

Source file for language messages will be sent from.

Type

str

property startup_time

Time required to set up communications and start models.

Type

float

property tempdir

Temporary directory.

Type

str

property time_per_byte
property time_per_message

Time required to send a single message of 1000 bytes.

Type

float

time_run(nmsg, msg_size, nrep=10, overwrite=False)[source]

Time sending a set of messages between the designated models.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • msg_size (int) – Size of each message that should be sent.

  • nrep (int, optional) – Number of times the test should be repeated to get an average execution time and standard deviation. Defaults to 10.

  • overwrite (bool, optional) – If True, any existing entry for this run will be overwritten. Defaults to False.

Returns

Best of, average and standard deviation in the time (in seconds)

required to execute the program.

Return type

tuple

time_run_mine(nmsg, msg_size, nrep)[source]

Time sending a set of messages between the designated models without using the pyperf package.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • msg_size (int) – Size of each message that should be sent.

  • nrep (int) – Number of times the test should be repeated. These reps will be appended to the existing reps for this entry.

time_run_pyperf(nmsg, msg_size, nrep)[source]

Time sending a set of messages between the designated models using the pyperf package.

Parameters
  • nmsg (int) – Number of messages that should be sent.

  • msg_size (int) – Size of each message that should be sent.

  • nrep (int) – Number of times the test should be repeated. These reps will be appended to the existing reps for this entry.

property yamlfile_format

Format string for creating a yaml file.

Type

str

yggdrasil.timing.change_default_comm(default_comm)[source]
yggdrasil.timing.debug_log()[source]

Set the log level to debug.

yggdrasil.timing.get_comm_list()[source]

Get the list of testable communication methods on the current platform.

Returns

Names of testable communication methods using timed_pipe.

Return type

list

yggdrasil.timing.get_lang_list()[source]

Get the list of testable languages on the current platform.

Returns

Names of testable languages using timed_pipe.

Return type

list

yggdrasil.timing.get_source(lang, direction, test_name='timed_pipe')[source]

Get the path to the source file.

Parameters
  • lang (str) – Language that should be returned.

  • direction (str) – ‘src’ or ‘dst’.

  • test_name (str, optional) – Name of the example. Defaults to ‘timed_pipe’.

Returns

Full path to the source file.

Return type

str

yggdrasil.timing.plot_scalings(compare='comm_type', compare_values=None, plotfile=None, test_name='timed_pipe', cleanup_plot=False, use_paper_values=False, **kwargs)[source]

Plot comparison of scaling for chosen variable.

Parameters
  • compare (str, optional) – Name of variable that should be compared. Valid values are ‘language’, ‘comm_type’, ‘platform’, ‘python_ver’. Defaults to ‘comm_type’.

  • compare_values (list, optional) – Values that should be plotted. If not provided, the values will be determined based on the current platform.

  • plotfile (str, optional) – Full path to the file where the plot will be saved. If not provided, one is created based on the test parameters in the current working directory.

  • test_name (str, optional) – Name of the test that should be used. Defaults to ‘timed_pipe’.

  • cleanup_plot (bool, optional) – If True, the create plotfile will be removed before the function returns. This is generally only useful for testing. Defaults to False.

  • use_paper_values (bool, optional) – If True, use the values from the paper. Defaults to False.

  • **kwargs – Additional keyword arguments are passed to plot_scaling_joint.

Returns

Path where the figure was saved.

Return type

str

yggdrasil.timing.pyperf_func(loops, timer, nmsg, msg_size, max_errors)[source]

Function to do pyperf loops over function.

Parameters
  • loops (int) – Number of loops to perform.

  • timer (TimedRun) – Class with information about the run and methods required for setup/teardown.

  • nmsg (int) – Number of messages that should be sent in the test.

  • msg_size (int) – Size of messages that should be sent in the test.

  • max_errors (int) – Maximum number of errors that should be retried.

Returns

Time (in seconds) required to perform the test the required

number of times.

Return type

float

yggdrasil.timing.pyperfjson_to_pandas(json_file)[source]

Convert pyperf benchmarks json file to a Pandas data frame.

Parameters

json_file (str) – Full path to the JSON benchmarks file that should be added.

Returns

Data frame version of pyperf benchmarks.

Return type

pandas.DataFrame

yggdrasil.timing.write_pyperf_script(script_file, nmsg, msg_size, lang_src, lang_dst, comm_type, nrep=10, max_errors=5, matlab_running=False)[source]

Write a script to run pyperf.

Parameters
  • script_file (str) – Full path to the file where the script should be saved.

  • nmsg (int) – The number of messages that should be sent during the run.

  • msg_size (int) – The size (in bytes) of the test messages that should be sent during the run.

  • lang_src (str) – The language that the source program should be in.

  • lang_dst (str) – The language that the destination program should be in.

  • comm_type (str) – The type of communication channel that should be used for the test.

  • nrep (int, optional) – The number of times the test run should be repeated. Defaults to 3.

  • max_errors (int, optional) – Maximum number of errors that should be retried. Defaults to 5.

  • matlab_running (bool, optional) – If True, the test will assert that there is an existing Matlab engine before starting, otherwise the test will assert that there is not an existing Matlab engine. Defaults to False.

yggdrasil.tools module

This modules offers various tools.

class yggdrasil.tools.ProxyMeta(classname, bases, attrs)[source]

Bases: type

Metaclass for handling proxy.

class yggdrasil.tools.ProxyObject(wrapped)[source]

Bases: object

Proxy for another object.

next(*args, **kwargs)
class yggdrasil.tools.TimeOut(max_time, key=None)[source]

Bases: object

Class for checking if a period of time has been elapsed.

Parameters
  • max_time (float, bbol) – Maximum period of time that should elapse before ‘is_out’ returns True. If False, ‘is_out’ will never return True. Providing 0 indicates that ‘is_out’ should immediately return True.

  • key (str, optional) – Key that was used to register the timeout. Defaults to None.

max_time

Maximum period of time that should elapsed before ‘is_out’ returns True.

Type

float

start_time

Result of time.time() at start.

Type

float

key

Key that was used to register the timeout.

Type

str

property elapsed

Total time that has elapsed since the start.

Type

float

property is_out

True if there is not any time remaining. False otherwise.

Type

bool

class yggdrasil.tools.YggClass(*args, **kwargs)[source]

Bases: yggdrasil.components.ComponentBase

Base class for Ygg classes.

Parameters
  • name (str) – Name used for component in log messages.

  • uuid (str, optional) – Unique ID for this instance. Defaults to None and is assigned.

  • working_dir (str, optional) – Working directory. If not provided, the current working directory is used.

  • timeout (float, optional) – Maximum time (in seconds) that should be spent waiting on a process. Defaults to 60.

  • sleeptime (float, optional) – Time that class should sleep for when sleep is called. Defaults to 0.01.

  • **kwargs – Additional keyword arguments are passed to the ComponentBase initializer.

name

Class name.

Type

str

uuid

Unique ID for this instance.

Type

str

sleeptime

Time that class should sleep for when sleep called.

Type

float

longsleep

Time that the class will sleep for when waiting for longer tasks to complete (10x longer than sleeptime).

Type

float

timeout

Maximum time that should be spent waiting on a process.

Type

float

working_dir

Working directory.

Type

str

errors

List of errors.

Type

list

sched_out

Output from the last scheduled task with output.

Type

obj

logger

Logger object for this object.

Type

logging.Logger

suppress_special_debug

If True, special_debug log messages are suppressed.

Type

bool

__deepcopy__(memo)[source]

Don’t deep copy since threads cannot be copied.

check_timeout(key=None, key_level=0)[source]

Check timeout for the calling function/method.

Parameters
  • key (str, optional) – Key for timeout that should be checked. Defaults to None and is set by the calling class and function/method (See timeout_key).

  • key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.

Raises

KeyError – If there is not a timeout registered for the specified key.

property critical

Log a critical level message.

property debug

Log a debug level message.

debug_log()[source]

Turn on debugging.

display(msg='', *args, **kwargs)[source]

Print a message, no log.

dummy_log(*args, **kwargs)[source]

Dummy log function that dosn’t do anything.

property error

Log an error level message.

property exception

Log an exception level message.

get_timeout_key(key_level=0, key_suffix=None)[source]

Return a key for a given level in the stack, relative to the function calling get_timeout_key.

Parameters
  • key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that is 2 steps higher in the stack. Higher values use classes and function/methods further up in the stack. Defaults to 0.

  • key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.

Returns

Key identifying calling object and method.

Return type

str

property info

Log an info level message.

property interface_info

Only do info debug message if is interface.

language_info(languages)[source]

Only do info debug message if the language is one of those specified.

property name

Name of the class object.

Type

str

periodic_debug(key, period=10)[source]

Log that should occur periodically rather than with every call.

Parameters
  • key (str) – Key that should be used to identify the debug message.

  • period (int, optional) – Period (in number of messages) that messages should be logged at. Defaults to 10.

Returns

Logging method to be used.

Return type

method

pprint(obj, block_indent=0, indent_str='    ', **kwargs)[source]

Use pprint to represent an object as a string.

Parameters
  • obj (object) – Python object to represent.

  • block_indent (int, optional) – Number of indents that should be placed in front of the entire block. Defaults to 0.

  • indent_str (str, optional) – String that should be used to indent. Defaults to 4 spaces.

  • **kwargs – Additional keyword arguments are passed to pprint.pformat.

Returns

String representation of object using pprint.

Return type

str

printStatus(level='info', return_str=False)[source]

Print the class status.

print_encoded(msg, *args, **kwargs)[source]

Print bytes to stdout, encoding if possible.

Parameters
  • msg (str, bytes) – Message to print.

  • *args – Additional arguments are passed to print.

  • **kwargs – Additional keyword arguments are passed to print.

property print_name

Name of the class object.

Type

str

reset_log()[source]

Resetting logging to prior value.

sched_task(t, func, args=None, kwargs=None, store_output=False, name=None)[source]

Schedule a task that will be executed after a certain time has elapsed.

Parameters
  • t (float) – Number of seconds that should be waited before task is executed.

  • func (object) – Function that should be executed.

  • args (list, optional) – Arguments for the provided function. Defaults to [].

  • kwargs (dict, optional) – Keyword arguments for the provided function. Defaults to {}.

  • store_output (bool, optional) – If True, the output from the scheduled task is stored in self.sched_out. Otherwise, it is not stored. Defaults to False.

  • name (str, optional) – Name for the task.

Returns

The timer object.

Return type

threading.Timer

sleep(t=None)[source]

Have the class sleep for some period of time.

Parameters

t (float, optional) – Time that class should sleep for. If not provided, the attribute ‘sleeptime’ is used.

property special_debug

Log debug level message contingent of supression flag.

start_timeout(t=None, key=None, key_level=0, key_suffix=None)[source]

Start a timeout for the calling function/method.

Parameters
  • t (float, optional) – Maximum time that the calling function should wait before timeing out. If not provided, the attribute ‘timeout’ is used.

  • key (str, optional) – Key that should be associated with the timeout that is created. Defaults to None and is set by the calling class and function/method (See get_timeout_key).

  • key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.

  • key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.

Raises

KeyError – If the key already exists.

stop_timeout(key=None, key_level=0, key_suffix=None, quiet=False)[source]

Stop a timeout for the calling function method.

Parameters
  • key (str, optional) – Key for timeout that should be stopped. Defaults to None and is set by the calling class and function/method (See timeout_key).

  • key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.

  • key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.

  • quiet (bool, optional) – If True, error message on timeout exceeded will be debug log. Defaults to False.

Raises

KeyError – If there is not a timeout registered for the specified key.

property timeout_key

Key identifying calling object and method.

Type

str

verbose_debug(*args, **kwargs)[source]

Log a verbose debug level message.

wait_on_function(function, timeout=None, polling_interval=None, key=None, key_level=0, key_suffix=None, quiet=False)[source]

Wait util a function returns True or a time limit is reached.

Parameters
  • t (float, optional) – Maximum time that the calling function should wait before timeing out. If not provided, the attribute ‘timeout’ is used.

  • key (str, optional) – Key that should be associated with the timeout that is created. Defaults to None and is set by the calling class and function/method (See get_timeout_key).

  • key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.

  • key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.

  • quiet (bool, optional) – If True, error message on timeout exceeded will be debug log. Defaults to False.

Raises

KeyError – If the key already exists.

property warn

Log a warning level message.

property warning

Log a warning level message.

class yggdrasil.tools.YggLoggerAdapter(class_name, instance_name, *args, **kwargs)[source]

Bases: logging.LoggerAdapter

Logger adapter for use with YggClass.

as_str(obj)[source]

Return str version of object if it is not already a string.

Parameters

obj (object) – Object that should be turned into a string.

Returns

String version of provided object.

Return type

str

process(msg, kwargs)[source]

Process logging message.

class yggdrasil.tools.YggPopen(cmd_args, forward_signals=True, for_matlab=False, **kwargs)[source]

Bases: subprocess.Popen

Uses Popen to open a process without a buffer. If not already set, the keyword arguments ‘bufsize’, ‘stdout’, and ‘stderr’ are set to 0, subprocess.PIPE, and subprocess.STDOUT respectively. This sets the output stream to unbuffered and directs both stdout and stderr to the stdout pipe. In addition this class overrides Popen.kill() to allow processes to be killed with CTRL_BREAK_EVENT on windows.

Parameters
  • args (list, str) – Shell command or list of arguments that should be run.

  • forward_signals (bool, optional) – If True, flags will be set such that signals received by the spawning process will be forwarded to the child process. If False, the signals will not be forwarded. Defaults to True.

  • **kwargs – Additional keywords arguments are passed to Popen.

disconnect()[source]

Disconnect objects using resources.

kill(*args, **kwargs)[source]

On windows using CTRL_BREAK_EVENT to kill the process.

yggdrasil.tools.add_line_numbers(lines, for_diff=False)[source]

Add number to lines.

Parameters
  • lines (list) – Lines to number.

  • for_diff (bool, optional) – If True, those lines beginning with removed/notation diff characters (-, ?) will not be numbered. Defaults to False.

Returns

Numbered lines.

Return type

list

yggdrasil.tools.apply_recurse(x, func, **kwargs)[source]

Apply a function recursively to all elements of x if it is a list, tuple, or dictionary.

Parameters
  • x (list, tuple, dict) – Object to apply function to.

  • func (function) – Function to apply to elements of x.

  • **kwargs – Additional keyword arguments are passed to each function call.

Returns

Version of input, but after applying func.

Return type

object

yggdrasil.tools.bytes2str(x, recurse=False)[source]

Convert bytes type to string type.

Parameters
  • x (bytes) – String.

  • recurse (bool, optional) – If True and x is a list, tuple, or dict, the coversion will recurse. Defaults to False.

Returns

Decoded string version of x.

Return type

str

yggdrasil.tools.check_environ_bool(name, valid_values=['true', '1', True, 1])[source]

Check to see if a boolean environment variable is set to True.

Parameters
  • name (str) – Name of environment variable to check.

  • valid_values (list, optional) – Values for the environment variable that indicate it is True. These should all be lower case as the lower case version of the variable contents will be compared to the list. Defaults to [‘true’, ‘1’].

Returns

True if the environment variables is set and is one of the

list valid_values (after being transformed to lower case).

Return type

bool

yggdrasil.tools.display_source(fname, number_lines=False, return_lines=False)[source]

Display source code with syntax highlighting (if available).

Parameters
  • fname (str, list) – Full path(s) to one or more source files.

  • number_lines (bool, optional) – If True, line numbers will be added to the displayed examples. Defaults to False.

  • return_lines (bool, optional) – If True, the lines are returned rather than displayed. Defaults to False.

yggdrasil.tools.display_source_diff(fname1, fname2, number_lines=False, return_lines=False)[source]

Display a diff between two source code files with syntax highlighting (if available).

Parameters
  • fname1 (str) – Name of first source file.

  • fname2 (src) – Name of second source file.

  • number_lines (bool, optional) – If True, line numbers will be added to the displayed examples. Defaults to False.

  • return_lines (bool, optional) – If True, the lines are returned rather than displayed. Defaults to False.

yggdrasil.tools.eval_kwarg(x)[source]

If x is a string, eval it. Otherwise just return it.

Parameters

x (str, obj) – String to be evaluated as an object or an object.

Returns

Result of evaluated string or the input object.

Return type

obj

yggdrasil.tools.find_all(name, path, verification_func=None)[source]

Find all instances of a file with a given name within the directory tree starting at a given path.

Parameters
  • name (str) – Name of the file to be found (with the extension).

  • path (str, None) – Directory where search should start. If set to None on Windows, the current directory and PATH variable are searched.

  • verification_func (function, optional) – Function that returns True when a file is valid and should be returned and False otherwise. Defaults to None and is ignored.

Returns

All instances of the specified file.

Return type

list

yggdrasil.tools.get_YGG_MSG_MAX(comm_type=None)[source]

Get the maximum message size for a given comm type.

Parameters

comm_type (str, optional) – The name of the communication type that the maximum message size should be returned for. Defaults to result of get_default_comm() if not provided.

Returns

Maximum message size (in bytes).

Return type

int

yggdrasil.tools.get_conda_env()[source]

Determine the name of the current conda environment.

Returns

Name of the current conda environment if one is activated. If a

conda environment is not activated, None is returned.

Return type

str

yggdrasil.tools.get_conda_prefix()[source]

Determine the conda path prefix for the current environment.

Returns

Full path to the directory prefix used for the current conda

environment if one exists. If conda cannot be located, None is returned.

Return type

str

yggdrasil.tools.get_default_comm()[source]

Get the default comm that should be used for message passing.

yggdrasil.tools.get_env_prefixes()[source]

Determine the environment path prefix (virtualenv or conda) for the current environment.

Returns

Full path to the directory prefixes used for the current

environments if one (or more) exists. If neither a virtualenv or conda prefix can be located, None is returned.

Return type

list

Raises

RuntimeError – If both virtualenv and conda environments are located.

yggdrasil.tools.get_fds(by_column=None)[source]

Get a list of open file descriptors.

yggdrasil.tools.get_installed_comm(language=None, dont_include_value=False)[source]

Get a list of the communication channel types that are supported by yggdrasil on the current machine. This checks the operating system, supporting libraries, and broker credentials. The order indicates the prefered order of use.

Parameters
  • language (str, optional) – Specific programming language that installed comms should be located for. Defaults to None and all languages supported on the current platform will be checked.

  • dont_include_value (bool, optional) – If True, don’t include the ValueComm in the list returned. Defaults to False.

Returns

The names of the the communication channel types supported on

the current machine.

Return type

list

yggdrasil.tools.get_installed_lang()[source]

Get a list of the languages that are supported by yggdrasil on the current machine. This checks for the necessary interpreters, licenses, and/or compilers.

Returns

The name of languages supported on the current machine.

Return type

list

yggdrasil.tools.get_python_c_library(allow_failure=False, libtype=None)[source]

Determine the location of the Python C API library.

Parameters
  • allow_failure (bool, optional) – If True, the base name will be returned if the file cannot be located. Defaults to False.

  • libtype (str, optional) – Type of library that should be located. Valid values include ‘static’ and ‘shared’. Defaults to ‘shared’ on Unix OSs and ‘static’ on Windows.

Returns

Full path to the library.

Return type

str

Raises
  • ValueError – If libtype is not ‘static’ or ‘shared’.

  • RuntimeError – If the base name for the library cannot be determined.

  • RuntimeError – If the library cannot be located.

yggdrasil.tools.get_shell()[source]

Get the type of shell that yggdrasil was called from.

Returns

Name of the shell.

Return type

str

yggdrasil.tools.get_subprocess_language()[source]

Determine the language of the calling process.

Returns

Name of the programming language responsible for the subprocess.

Return type

str

yggdrasil.tools.get_subprocess_language_driver()[source]

Determine the driver for the langauge of the calling process.

Returns

Class used to handle running a model of the process language.

Return type

ModelDriver

yggdrasil.tools.get_supported_comm(dont_include_value=False)[source]

Get a list of the communication mechanisms supported by yggdrasil.

Parameters

dont_include_value (bool, optional) – If True, don’t include the ValueComm in the list returned. Defaults to False.

Returns

The names of communication mechanisms supported by yggdrasil.

Return type

list

yggdrasil.tools.get_supported_lang()[source]

Get a list of the model programming languages that are supported by yggdrasil.

Returns

The names of programming languages supported by yggdrasil.

Return type

list

yggdrasil.tools.get_supported_platforms()[source]

Get a list of the platforms supported by yggdrasil.

Returns

The name of platforms supported by yggdrasil.

Return type

list

yggdrasil.tools.get_supported_type()[source]

Get a list of the data types that are supported by yggdrasil.

Returns

The names of data types supported by yggdrasil.

Return type

list

yggdrasil.tools.get_venv_prefix()[source]

Determine the virtualenv path prefix for the current environment.

Returns

Full path to the directory prefix used for the current

virtualenv environment if one exists. If virtualenv cannot be located, None is returned.

Return type

str

yggdrasil.tools.import_all_modules(base=None, exclude=None, do_first=None)[source]

Import all yggdrasil modules.

Parameters
  • base (str, optional) – Base module to start from. Defaults to ‘yggdrasil’.

  • exclude (list, optional) – Modules that should not be imported. Defaults to empty list.

  • do_first (list, optional) – Modules that should be import first. Defaults to empty list.

yggdrasil.tools.in_powershell()[source]

Determine if yggdrasil is running from a Windows Powershell.

Returns

True if running from Powershell, False otherwise.

Return type

bool

yggdrasil.tools.is_comm_installed(comm, language=None)[source]

Check to see if yggdrasil can use a communication mechanism on the current machine.

Parameters
  • comm (str) – Communication mechanism to check.

  • language (str, optional) – Specific programming language that communication mechanism should be check for. Defaults to None and all supported languages will be checked.

Returns

True if the communication mechanism can be used on the current

machine, False otherwise.

Return type

bool

yggdrasil.tools.is_lang_installed(lang)[source]

Check to see if yggdrasil can run models written in a programming language on the current machine.

Parameters

lang (str) – Programming language to check.

Returns

True if models in the provided language can be run on the current

machine, False otherwise.

Return type

bool

yggdrasil.tools.is_language_alias(x, language)[source]

Check if a string is an alias for a language.

Parameters
  • x (str) – String to check.

  • language (str, list) – One or more language to check aliases of.

Returns

Returns the version of the language in the provided set if

x is an alias and False otherwise.

Return type

str, bool

yggdrasil.tools.is_subprocess()[source]

Determine if the current process is a subprocess.

Returns

True if YGG_SUBPROCESS environment variable is True, False

otherwise.

Return type

bool

yggdrasil.tools.kill(pid, signum)[source]

Kill process by mapping signal number.

Parameters
  • pid (int) – Process ID.

  • signum (int) – Signal that should be sent.

yggdrasil.tools.locate_file(fname, environment_variable='PATH', directory_list=None, show_alternates=False, verification_func=None)[source]

Locate a file within a set of paths defined by a list or environment variable.

Parameters
  • fname (str, list) – One or more possible names of the file that should be located. If a list is provided, the path for the first entry for which a match could be located will be returned and subsequent entries will not be checked.

  • environment_variable (str) – Environment variable containing the set of paths that should be searched. Defaults to ‘PATH’. If None, this keyword argument will be ignored. If a list is provided, it is assumed to be a list of environment variables that should be searched in the specified order.

  • directory_list (list) – List of paths that should be searched in addition to those specified by environment_variable. Defaults to None and is ignored. These directories will be searched be for those in the specified environment variables.

  • show_alternates (bool, optional) – If True and there is more than one match, the alternate matches will be printed in a warning message. Defaults to False.

  • verification_func (function, optional) – Function that returns True when a file is valid and should be returned and False otherwise. Defaults to None and is ignored.

Returns

Full path to the located file if it was located, False

otherwise.

Return type

bool, str

yggdrasil.tools.locate_path(fname, basedir='/')[source]

Find the full path to a file using where on Windows.

yggdrasil.tools.popen_nobuffer(*args, **kwargs)[source]

Uses Popen to open a process without a buffer. If not already set, the keyword arguments ‘bufsize’, ‘stdout’, and ‘stderr’ are set to 0, subprocess.PIPE, and subprocess.STDOUT respectively. This sets the output stream to unbuffered and directs both stdout and stderr to the stdout pipe.

Parameters
  • args (list, str) – Shell command or list of arguments that should be run.

  • forward_signals (bool, optional) – If True, flags will be set such that signals received by the spawning process will be forwarded to the child process. If False, the signals will not be forwarded. Defaults to True.

  • **kwargs – Additional keywords arguments are passed to Popen.

Returns

Process that was started.

Return type

YggPopen

yggdrasil.tools.pprint_encoded(obj, *args, **kwargs)[source]

Pretty print an object, catching encoding errors as necessary.

Parameters
  • obj (object) – Python object to pprint.

  • *args – Additional arguments are passed to pprint.pformat.

  • **kwargs – Additional keyword arguments are passed to pprint.pformat.

yggdrasil.tools.print_encoded(msg, *args, **kwargs)[source]

Print bytes to stdout, encoding if possible.

Parameters
  • msg (str, bytes) – Message to print.

  • *args – Additional arguments are passed to print.

  • **kwargs – Additional keyword arguments are passed to print.

yggdrasil.tools.remove_path(fpath, timeout=60.0)[source]

Delete a single file.

Parameters
  • fpath (str) – Full path to a file or directory that should be removed.

  • timeout (float, optional) – Time (in seconds) that should be waited before raising an error that a file cannot be removed. Defaults to 60.0.

Raises

RuntimeError – If the product cannot be removed.

yggdrasil.tools.safe_eval(statement, **kwargs)[source]

Run eval with a limited set of builtins and Python libraries/functions.

Parameters
  • statement (str) – Statement that should be evaluated.

  • **kwargs – Additional keyword arguments are variables that are made available to the statement during evaluation.

Returns

Result of the eval.

Return type

object

yggdrasil.tools.sleep(interval)[source]

Sleep for a specified number of seconds.

Parameters

interval (float) – Time in seconds that process should sleep.

yggdrasil.tools.str2bytes(x, recurse=False)[source]

Convert string type to bytes type.

Parameters
  • x (str) – String.

  • recurse (bool, optional) – If True and x is a list, tuple, or dict, the coversion will recurse. Defaults to False.

Returns

Encoded bytes version of x.

Return type

bytes

yggdrasil.tools.timer_context(msg_format, **kwargs)[source]

Context that will time commands executed within it and log a message.

Parameters
  • msg_format (str) – Format string used to format the elapsed time. It should include, at minimum, a ‘{elapsed}’ field. Additional fields may also be present and can be fulfilled by additional keywords.

  • **kwargs – Additional keyword arguments are passed to the format method on msg_format to create the log message.

yggdrasil.tools.track_fds(prefix='')[source]

yggdrasil.units module

yggdrasil.units.add_units(arr, unit_str, dtype=None)[source]

Add units to an array or scalar.

Parameters
  • arr (np.ndarray, float, int) – Scalar or array of data to add units to.

  • unit_str (str) – Unit string.

  • dtype (np.dtype, optional) – Numpy data type that should be maintained for array/qunatity with units. If not provided, this is determined from the array.

Returns

Array with units.

Return type

unyt.unyt_array

yggdrasil.units.are_compatible(units1, units2)[source]

Check if two units are compatible.

Parameters
  • units1 (str) – First units string.

  • units2 (str) – Second units string.

Returns

True if the units are compatible, False otherwise.

Return type

bool

yggdrasil.units.as_unit(ustr)[source]

Get unit object for the string.

Parameters

ustr (str) – Unit string.

Returns

Unit object.

Return type

unyt.Unit

Raises

ValueError – If the string is not a recognized unit.

yggdrasil.units.convert_R_unit_string(r_str)[source]

Convert R unit string to string that the Python package can understand.

Parameters

r_str (str) – R units string to convert.

Returns

Converted string.

Return type

str

yggdrasil.units.convert_from_pandas_timedelta(x)[source]

Covert a pandas.Timedelta instance to a scalar/array with time units.

Parameters

x (pandas.Timedelta) – Timedelta variable to convert.

Returns

Equivalent scalar/array with units.

Return type

object

yggdrasil.units.convert_matlab_unit_string(m_str)[source]

Convert Matlab unit string to string that the Python package can understand.

Parameters

m_str (str) – Matlab units string to convert.

Returns

Converted string.

Return type

str

yggdrasil.units.convert_to(arr, new_units)[source]

Convert quantity with units to new units. Objects without units will be returned with the new units.

Parameters
  • arr (np.ndarray, float, int, unyt.unyt_array) – Quantity with or without units.

  • new_units (str) – New units that should be applied.

Returns

Array with new units.

Return type

unyt.unyt_array

yggdrasil.units.convert_to_pandas_timedelta(x)[source]

Convert variable with time units to a pandas.Timedelta instance.

Parameters

x (object) – Scalar/array with units to convert to a pandas.Timedelta instance.

Returns

Equivalent Timedelta variable.

Return type

pandas.Timedelta

yggdrasil.units.convert_unit_string(orig_str, replacements=None)[source]

Convert unit string to string that the Python package can understand.

Parameters
  • orig_str (str) – Original units string to convert.

  • replacements (dict, optional) – Mapping from unit to another. Defaults to empty dict.

Returns

Converted string.

Return type

str

yggdrasil.units.get_conversion_function(old_units, new_units)[source]

Get a function that will convert a scalar/array from one unit to another.

Parameters
  • old_units (str) – Units to convert from.

  • new_units (str) – Units to convert to.

Returns

Conversion function that takes scalar/array as input

and returns converted scalar/array.

Return type

function

yggdrasil.units.get_data(obj)[source]

Get the array/scalar assocaited with the object.

Parameters

obj (object) – Object to get data for.

Returns

Numpy array representation of the underlying data.

Return type

np.ndarray

yggdrasil.units.get_units(obj)[source]

Get the string representation of the units.

Parameters

obj (object) – Object to get units for.

Returns

Units, empty if input object has none.

Return type

str

yggdrasil.units.get_ureg()[source]

Get the unit registry.

yggdrasil.units.has_units(obj, check_dimensionless=False)[source]

Determine if a Python object has associated units.

Parameters
  • obj (object) – Object to be tested for units.

  • check_dimensionless (bool, optional) – If True, an object with dimensionless units will return True.

Returns

True if the object has units, False otherwise.

Return type

bool

yggdrasil.units.is_null_unit(ustr)[source]

Determines if a string is a null unit.

Parameters

ustr (str) – String to test.

Returns

True if the string is ‘’ or ‘n/a’, False otherwise.

Return type

bool

yggdrasil.units.is_unit(ustr)[source]

Determine if a string is a valid unit.

Parameters

ustr (str) – String representation to test.

Returns

True if the string is a valid unit. False otherwise.

Return type

bool

yggdrasil.yamlfile module

exception yggdrasil.yamlfile.YAMLSpecificationError[source]

Bases: RuntimeError

Error raised when the yaml specification does not meet expectations.

yggdrasil.yamlfile.clone_github_repo(fname, commit=None, local_directory=None)[source]

Clone a GitHub repository, returning the path to the local copy of the file pointed to by the URL if there is one.

Parameters
  • fname (str) – URL to a GitHub repository or a file in a GitHub repository that should be cloned.

  • commit (str, optional) – Commit that should be checked out. Defaults to None and the HEAD of the default branch is used.

  • local_directory (str, optional) – Local directory that the file should be cloned into. Defaults to None and the current working directory will be used.

Returns

Path to the local copy of the repository or file in the

repository.

Return type

str

yggdrasil.yamlfile.complete_partial_integration(existing, name, partial_commtype=None)[source]

Patch input/output channels that are not connected to a stand-in model.

Parameters
  • existing (dict) – Dictionary of existing components.

  • name (str) – Name that should be given to the new model.

  • partial_commtype (dict, optional) – Communicator kwargs that should be be used for the connections to the unpaired channels. Defaults to None and will be ignored.

Returns

Updated dictionary of components.

Return type

dict

Link I/O drivers back to the models they communicate with.

Parameters

existing (dict) – Dictionary of existing components.

Returns

Dictionary with I/O drivers added to models.

Return type

dict

yggdrasil.yamlfile.load_yaml(fname, yaml_param=None, directory_for_clones=None)[source]

Parse a yaml file defining a run.

Parameters
  • fname (str, file, dict) – Path to a YAML file, an open file descriptor to a file containing a YAML, or a loaded YAML document. If fname starts with ‘git:’ then the code will assume the file is in a remote git repository. The remainder of fname can be the full url to the YAML file (http://mygit.repo/foo/bar/yaml/interesting.yml) or just the repo and YAML file (the server is assumed to be github.com if not given) (foo/bar/yam/interesting.yaml will be interpreted as http://github.com/foo/bar/yam/interesting.yml).

  • yaml_param (dict, optional) – Parameters that should be used in mustache formatting of YAML files. Defaults to None and is ignored.

  • directory_for_clones (str, optional) – Directory that git repositories should be cloned into. Defaults to None and the current working directory will be used.

Returns

Contents of yaml file.

Return type

dict

yggdrasil.yamlfile.no_duplicates_constructor(loader, node, deep=False)[source]

Check for duplicate keys.

yggdrasil.yamlfile.parse_component(yml, ctype, existing=None)[source]

Parse a yaml entry for a component, adding it to the list of existing components.

Parameters
  • yml (dict) – YAML dictionary for a component.

  • ctype (str) – Component type. This can be ‘input’, ‘output’, ‘model’, or ‘connection’.

  • existing (dict, optional) – Dictionary of existing components. Defaults to empty dict.

Raises
  • TypeError – If yml is not a dictionary.

  • ValueError – If dtype is not ‘input’, ‘output’, ‘model’, or ‘connection’.

  • ValueError – If the component already exists.

Returns

All components identified.

Return type

dict

yggdrasil.yamlfile.parse_connection(yml, existing)[source]

Parse a yaml entry for a connection between I/O channels.

Parameters
  • yml (dict) – YAML dictionary for a connection.

  • existing (dict) – Dictionary of existing components.

Raises
  • RuntimeError – If the ‘inputs’ entry is not a model output or file.

  • RuntimeError – If neither the ‘inputs’ or ‘outputs’ entries correspond to model I/O channels.

Returns

Updated log of all entries.

Return type

dict

yggdrasil.yamlfile.parse_model(yml, existing)[source]

Parse a yaml entry for a model.

Parameters
  • yml (dict) – YAML dictionary for a model.

  • existing (dict) – Dictionary of existing components.

Returns

Updated log of all entries.

Return type

dict

yggdrasil.yamlfile.parse_yaml(files, complete_partial=False, partial_commtype=None, model_only=False, model_submission=False, yaml_param=None, directory_for_clones=None)[source]

Parse list of yaml files.

Parameters
  • files (str, list) – Either the path to a single yaml file or a list of yaml files.

  • complete_partial (bool, optional) – If True, unpaired input/output channels are allowed and reserved for use (e.g. for calling the model as a function). Defaults to False.

  • partial_commtype (dict, optional) – Communicator kwargs that should be be used for the connections to the unpaired channels when complete_partial is True. Defaults to None and will be ignored.

  • model_only (bool, optional) – If True, the YAML will not be evaluated as a complete integration and only the individual components will be parsed. Defaults to False.

  • model_submission (bool, optional) – If True, the YAML will be evaluated as a submission to the yggdrasil model repository and model_only will be set to True. Defaults to False.

  • yaml_param (dict, optional) – Parameters that should be used in mustache formatting of YAML files. Defaults to None and is ignored.

  • directory_for_clones (str, optional) – Directory that git repositories should be cloned into. Defaults to None and the current working directory will be used.

Raises
  • ValueError – If the yml dictionary is missing a required keyword or has an invalid value.

  • RuntimeError – If one of the I/O channels is not initialized with driver information.

Returns

Dictionary of information parsed from the yamls.

Return type

dict

yggdrasil.yamlfile.prep_yaml(files, yaml_param=None, directory_for_clones=None)[source]

Prepare yaml to be parsed by jsonschema including covering backwards compatible options.

Parameters
  • files (str, list) – Either the path to a single yaml file or a list of yaml files. Entries can also be opened file descriptors for files containing YAML documents or pre-loaded YAML documents.

  • yaml_param (dict, optional) – Parameters that should be used in mustache formatting of YAML files. Defaults to None and is ignored.

  • directory_for_clones (str, optional) – Directory that git repositories should be cloned into. Defaults to None and the current working directory will be used.

Returns

YAML ready to be parsed using schema.

Return type

dict

Module contents

This package provides a framework for integrating models across languages such that they can be run simultaneously, passing input back and forth.