yggdrasil.languages.Python package¶

yggdrasil.languages.Python.YggInterface.YggArrayOutput(name, **kwargs)[source]¶

Get class for handling table-like formatted output as arrays.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiArrayInput(name, **kwargs)[source]¶

Get class for handling table-like formatted input as arrays.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggAsciiTableInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiArrayOutput(name, fmt=None, **kwargs)[source]¶

Get class for handling table-like formatted output as arrays.

Parameters:

name (str) – The name of the message queue where output should be sent.
fmt (str) – A C style format string specifying how each ‘row’ of output should be formated. This should include the newline character.
**kwargs – Additional keyword arguments are passed to YggAsciiTableOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiFileInput(name, **kwargs)[source]¶

Get class for generic ASCII input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiFileOutput(name, **kwargs)[source]¶

Get class for generic ASCII output.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiTableInput(name, as_array=False, **kwargs)[source]¶

Get class for handling table-like formatted input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
as_array (bool, optional) – If True, recv returns the entire table array and can only be called once. If False, recv returns row entries. Default to False.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggAsciiTableOutput(name, fmt=None, as_array=False, **kwargs)[source]¶

Get class for handling table-like formatted output.

Parameters:

name (str) – The name of the message queue where output should be sent.
fmt (str) – A C style format string specifying how each ‘row’ of output should be formated. This should include the newline character.
as_array (bool, optional) – If True, send expects and entire array. If False, send expects the entries for one table row. Defaults to False.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggInit(_type, args=None, kwargs=None)[source]¶

Short interface to identify functions called by another language.

Parameters:

_type (str) – Name of class that should be returned.
args (list, optional) – Additional arguments that should be passed to the class initializer.
kwargs (dict, optional) – Additional keyword arguments that should be passed to the class initializer.

Returns:

An instance of the requested class.

Return type:

obj

yggdrasil.languages.Python.YggInterface.YggInput(name, format_str=None, **kwargs)[source]¶

Get class for handling input from a message queue.

Parameters:

name (str) – The name of the message queue. Combined with the suffix ‘_IN’, it should match an environment variable containing a message queue key.
format_str (str, optional) – C style format string that should be used to deserialize messages that are receieved into a list of python objects. Defaults to None and raw string messages are returned.
**kwargs – Additional keyword arguments are passed to the underlying comm.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggObjInput(name, **kwargs)[source]¶

Get class for handling Obj input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggObjOutput(name, **kwargs)[source]¶

Get class for handling Obj output.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggOutput(name, format_str=None, **kwargs)[source]¶

Get class for handling output to a message queue.

Parameters:

name (str) – The name of the message queue. Combined with the suffix ‘_OUT’, it should match an environment variable containing a message queue key.
format_str (str, optional) – C style format string that should be used to create a message from a list of python ojbects. Defaults to None and raw string messages are sent.
**kwargs – Additional keyword arguments are passed to the underlying comm.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPandasInput(name, **kwargs)[source]¶

Get class for handling Pandas input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPandasOutput(name, **kwargs)[source]¶

Get class for handling Pandas output.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPickleInput(name, **kwargs)[source]¶

Get class for handling pickled input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPickleOutput(name, **kwargs)[source]¶

Get class for handling pickled output.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPlyInput(name, **kwargs)[source]¶

Get class for handling Ply input.

Parameters:

name (str) – The name of the input message queue that input should be received from.
**kwargs – Additional keyword arguments are passed to YggInput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggPlyOutput(name, **kwargs)[source]¶

Get class for handling Ply output.

Parameters:

name (str) – The name of the message queue where output should be sent.
**kwargs – Additional keyword arguments are passed to YggOutput.

Returns:

Communication object.

Return type:

yggdrasil.languages.Python.YggInterface.YggRpcClient(name, outfmt=None, infmt=None, **kwargs)[source]¶

Get class for handling requests and response to an RPC Server from a client.

Parameters:

name (str) – The name of the server queues.
outfmt (str, optional) – Format string used to format variables in a message sent to the request queue. Defautls to ‘%s’.
infmt (str, optional) – Format string used to recover variables from messages received from the response queue. Defautls to ‘%s’.
**kwargs – Additional keyword arguments are passed to InterfaceComm.

Returns:

class:.ClientComm: Communication object.

yggdrasil.languages.Python.YggInterface.YggRpcServer(name, infmt=None, outfmt=None, **kwargs)[source]¶

Get class for handling requests and response for an RPC Server.

Parameters:

name (str) – The name of the server queues.
infmt (str, optional) – Format string used to recover variables from messages received from the request queue. Defaults to ‘%s’.
outfmt (str, optional) – Format string used to format variables in a message sent to the response queue. Defautls to ‘%s’.
**kwargs – Additional keyword arguments are passed to InterfaceComm.

Returns:

class:.ServerComm: Communication object.

yggdrasil.languages.Python.YggInterface.YggTimesync(name='timesync', **kwargs)[source]¶

Get class for handling timestep synchronization requests.

Parameters:

name (str, optional) – The name of the server queues. Defaults to ‘timesync’.
**kwargs – Additional keyword arguments are passed to InterfaceComm.

Returns:

class:.ClientComm: Communication object.

yggdrasil.languages.Python.YggInterface.YggTimesyncServer(name='timesync', **kwargs)[source]¶

Get class for handling timestep synchronization requests on the driver side.

Parameters:

name (str, optional) – The name of the server queues. Defaults to ‘timesync’.
**kwargs – Additional keyword arguments are passed to InterfaceComm.

Returns:

class:.ServerComm: Communication object.

yggdrasil.languages.Python.YggInterface.bufMsgSize()[source]¶: int: Buffer size for average message.

yggdrasil.languages.Python.YggInterface.eof_msg()[source]¶: str: Message signalling end of file.

yggdrasil.languages.Python.YggInterface.maxMsgSize()[source]¶: int: The maximum message size.

Module contents¶

yggdrasil.serialize package¶

Submodules¶

yggdrasil.serialize.AsciiMapSerialize module¶

class yggdrasil.serialize.AsciiMapSerialize.AsciiMapSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for serializing/deserializing name/value mapping.

Parameters:: delimiter (str, optional) – Delimiter that should be used to separate name/value pairs in the map. Defaults to t.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = False¶

default_datatype = {'type': 'object'}¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (bytes) – Message to be deserialized.
Returns:: Deserialized Python dictionary.
Return type:: dict

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (dict) – Python dictionary to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

yggdrasil.serialize.AsciiTableSerialize module¶

class yggdrasil.serialize.AsciiTableSerialize.AsciiTableSerialize(*args, **kwargs)[source]¶

Bases: DefaultSerialize

Class for serialize table output into bytes messages comprising a formatted ASCII table.

Parameters:

format_str (str, optional) – If provided, this string will be used to format messages from a list of arguments and parse messages to get a list of arguments in C printf/scanf style. Defaults to None and messages are assumed to already be bytes.
field_names (list, optional) – The names of fields in the format string. If not provided, names are set based on the order of the fields in the format string.
field_units (list, optional) – The units of fields in the format string. If not provided, all fields are assumed to be dimensionless.
as_array (bool, optional) – If True, each of the arguments being serialized/deserialized will be arrays that are converted to/from bytes in column major (‘F’) order. Otherwise, each argument should be a scalar. Defaults to False.
delimiter (str, optional) – Character(s) that should be used to separate columns. Defaults to ‘t’.
use_astropy (bool, optional) – If True, the astropy package will be used to serialize/deserialize table. Defaults to False.
**kwargs – Additional keyword args are processed as part of the type definition.

format_str¶

Format string used to format/parse bytes messages from/to a list of arguments in C printf/scanf style.

Type:: str

field_names¶

The names of fields in the format string.

Type:: list

field_units¶

The units of fields in the format string.

Type:: list

as_array¶

True or False depending if serialized/deserialized python objects will be arrays or scalars.

Type:: bool

delimiter¶

Character(s) that should be used to separate columns.

Type:: str

use_astropy¶

If True, the astropy package will be used to serialize/deserialize table.

Type:: bool

default_datatype = {'type': 'array'}¶

default_read_meth = 'readline'¶

deserialize_file_header(fd)[source]¶

Deserialize the header information from the file and update the serializer.

Parameters:: fd (file) – File containing header.

file_extensions = ['.txt']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Dictionary of variables to use for testing.
Return type:: dict

has_header = True¶

normalize(args)[source]¶

Normalize a message to conform to the expected datatype.

Parameters:: args (object) – Message arguments.
Returns:: Normalized message.
Return type:: object

property read_meth¶

Method that should be used to read data for deserialization.

Type:: str

serialize_file_header()[source]¶

Return the serialized header information that should be prepended to files serialized using this class.

Returns:: Header string that should be written to the file.
Return type:: bytes

update_field_names()[source]¶: list: Names for each field in the data type.

update_field_units()[source]¶: list: Units for each field in the data type.

update_format_str()[source]¶: Update the format string based on the type definition.

update_serializer(*args, **kwargs)[source]¶

Update serializer with provided information.

Parameters:

skip_type (bool, optional) – If True, everything is updated except the data type. Defaults to False.
**kwargs – Additional keyword arguments are processed as part of they type definition and are parsed for old-style keywords.

Raises:

RuntimeError – If there are keywords that are not valid datatype keywords (currect or old-style).

yggdrasil.serialize.CABOSerialize module¶

class yggdrasil.serialize.CABOSerialize.CABOSerialize(*args, **kwargs)[source]¶

Bases: AsciiMapSerialize

Class for serializing/deserializing CABO parameter files.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = True¶

default_datatype = {'type': 'object'}¶

file_extensions = ['.cab']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (bytes) – Message to be deserialized.
Returns:: Deserialized Python dictionary.
Return type:: dict

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (dict) – Python dictionary to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

classmethod parse_units(x)[source]¶

Parse units.

Parameters:: x (str) – Unit string.
Returns:: Propertly formatted units.
Return type:: str

yggdrasil.serialize.DefaultSerialize module¶

class yggdrasil.serialize.DefaultSerialize.DefaultSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Default class for serializing/deserializing a python object into/from a bytes message.

Parameters:: **kwargs – Additional keyword args are passed to the parent class.

classmethod concatenate(objects, as_array=False, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
as_array (bool, optional) – If True, the objects in the list are complete columns in a table and as_format is set to True. Defaults to False.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

classmethod dict2object(obj, as_array=False, field_names=None, **kwargs)[source]¶

Conver a dictionary to a message object.

Parameters:

obj (dict) – Dictionary to convert to serializable object.
as_array (bool, optional) – If True, the objects in the list are complete columns in a table and as_format is set to True. Defaults to False.
field_names (list, optional) – The field names associated with a table-like data type. Defaults to None. This keyword must be provided if as_array is True.
**kwargs – Additional keyword arguments are ignored.

Returns:

Serializable object.

Return type:

file_extensions = ['.ygg']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶: Method to return a dictionary of testing options for this class.

classmethod object2array(obj, as_array=False, field_names=None, **kwargs)[source]¶

Convert a message object into an array.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in an array form.
as_array (bool, optional) – If True, the objects in the list are complete columns in a table and as_format is set to True. Defaults to False.
field_names (list, optional) – The field names associated with a table-like data type. Defaults to None. This keyword must be provided if as_array is True.
**kwargs – Additional keyword arguments are ignored.

Returns:

Array version of the provided object.

Return type:

np.array

classmethod object2dict(obj, as_array=False, field_names=None, **kwargs)[source]¶

Convert a message object into a dictionary.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in a dictionary form.
as_array (bool, optional) – If True, the objects in the list are complete columns in a table and as_format is set to True. Defaults to False.
field_names (list, optional) – The field names associated with a table-like data type. Defaults to None. This keyword must be provided if as_array is True.
**kwargs – Additional keyword arguments are ignored.

Returns:

Dictionary version of the provided object.

Return type:

yggdrasil.serialize.DirectSerialize module¶

class yggdrasil.serialize.DirectSerialize.DirectSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for directly serializing bytes.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

default_datatype = {'type': 'bytes'}¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the: provided content.
empty (object): Object produced from deserializing an empty: message.

objects (list): List of objects to be serialized/deserialized. extra_kwargs (dict): Extra keyword arguments not used to

construct type definition.

typedef (dict): Type definition resulting from the supplied: kwargs.
dtype (np.dtype): Numpy data types that is consistent with the: determined type definition.

Return type:

yggdrasil.serialize.FunctionalSerialize module¶

class yggdrasil.serialize.FunctionalSerialize.FunctionalSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for serializing/deserializing a Python object into/from a bytes message using defined functions.

Parameters:

encoded_datatype (schema, optional) – JSON schema describing the type that serialized objects should conform to. Defaults to the class attribute default_encoded_datatype. If either func_serialize or func_deserialize are not provided, this needs to be specified in order to serialize non-bytes objects.
func_serialize (func, optional) – Callable object that takes Python objects as input and returns a representation that conforms to encoded_datatype. Defaults to None and the default serialization for encoded_datatype will be used.
func_deserialize (func, optional) – Callable object that takes objects of a type that conforms to encoded_datatype and returns a deserialized Python object. Defaults to None and the default deserialization for encoded_datatype will be used.
**kwargs – Additional keyword args are passed to the parent class’s constructor.

func_deserialize = None¶

func_serialize = None¶

property serializer_info¶

Serializer info.

Type:: dict

yggdrasil.serialize.JSONSerialize module¶

class yggdrasil.serialize.JSONSerialize.JSONSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for serializing a python object into a bytes message using JSON.

Parameters:

indent (str, int, optional) – String or number of spaces that should be used to indent each level within the seiralized structure. Defaults to ‘t’.
sort_keys (bool, optional) – If True, the serialization of dictionaries will be in key sorted order. Defaults to True.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = False¶

default_datatype = {'type': 'object'}¶

file_extensions = ['.json']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (str, bytes) – Message to be deserialized.
Returns:: Deserialized Python object.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (obj) – Python object to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

yggdrasil.serialize.JSONSerialize.decode_json(msg, **kwargs)[source]¶

Decode a Python object from a JSON serialization.

Parameters:

msg (str) – JSON serialization to decode.
**kwargs – Additional keyword arguments are passed to rapidjson.loads.

Returns:

Deserialized Python object.

Return type:

yggdrasil.serialize.JSONSerialize.encode_json(obj, fd=None, indent=None, sort_keys=True, **kwargs)[source]¶

Encode a Python object in JSON format.

Parameters:

obj (object) – Python object to encode.
fd (file, optional) – File descriptor for file that encoded object should be written to. Defaults to None and string is returned.
indent (int, str, optional) – Indentation for new lines in encoded string. Defaults to None.
sort_keys (bool, optional) – If True, the keys will be output in sorted order. Defaults to True.
**kwargs – Additional keyword arguments are passed to rapidjson.dumps.

Returns:

Encoded object.

Return type:

str, bytes

yggdrasil.serialize.JSONSerialize.indent_char2int(indent)[source]¶

Convert a character indent into a number of spaces that should be used. Tabs are set to be equivalent to 4 spaces.

Parameters:: indent (str) – String indent.
Returns:: Number of whitespaces that is equivalent to the provided string.
Return type:: int

yggdrasil.serialize.MatSerialize module¶

class yggdrasil.serialize.MatSerialize.MatSerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for serializing a python object into a bytes message using the Matlab .mat format.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = False¶

default_datatype = {'type': 'object'}¶

file_extensions = ['.mat']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (str, bytes) – Message to be deserialized.
Returns:: Deserialized Python object.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (obj) – Python object to be serialized.
Returns:: Serialized message.
Return type:: bytes, str
Raises:: TypeError – If args is not a dictionary.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

yggdrasil.serialize.ObjSerialize module¶

class yggdrasil.serialize.ObjSerialize.ObjDict[source]¶

Bases: GeometryBase, ObjWavefront

Enhanced dictionary class for storing Obj information.

apply_scalar_map(*args, **kwargs)[source]¶

Set the color of faces in a 3D object based on a scalar map. This creates a copy unless no_copy is True.

Parameters:

scalar_arr (arr) – Scalar values that should be mapped to colors for each face.
color_map (str, optional) – The name of the color map that should be used. Defaults to ‘plasma’.
vmin (float, optional) – Value that should map to the minimum of the colormap. Defaults to min(scalar_arr).
vmax (float, optional) – Value that should map to the maximum of the colormap. Defaults to max(scalar_arr).
scaling (str, optional) – Scaling that should be used to map the scalar array onto the colormap. Defaults to ‘linear’.
scale_by_area (bool, optional) – If True, the elements of the scalar array will be multiplied by the area of the corresponding face. If True, vmin and vmax should be in terms of the scaled array. Defaults to False.
no_copy (bool, optional) – If True, the returned object will not be a copy. Defaults to False.

Returns:

Obj with updated vertex colors.

Return type:

classmethod from_shape(shape, d, conversion=1.0)[source]¶

Create a ply dictionary from a PlantGL shape and descritizer.

Parameters:

scene (openalea.plantgl.scene) – Scene that should be descritized.
d (openalea.plantgl.descritizer) – Descritizer.
conversion (float, optional) – Conversion factor that should be applied to the vertex positions. Defaults to 1.0.

to_geom_args(conversion=1.0, name=None)[source]¶

Get arguments for creating a PlantGL geometry.

Parameters:

conversion (float, optional) – Conversion factor that should be applied to the vertices. Defaults to 1.0.
name (str, optional) – Name that should be given to the created PlantGL symbol. Defaults to None and is ignored.

Returns:

Class, arguments and keyword arguments for PlantGL geometry.

Return type:

class yggdrasil.serialize.ObjSerialize.ObjSerialize(*args, **kwargs)[source]¶

Bases: PlySerialize

Class for serializing/deserializing .obj file formats. Reader adapted from https://www.pygame.org/wiki/OBJFileLoader.

default_datatype = {'type': 'obj'}¶

file_extensions = ['.obj']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (bytes) – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

normalize(args)[source]¶

Normalize a message to conform to the expected datatype.

Parameters:: args (object) – Message arguments.
Returns:: Normalized message.
Return type:: object

yggdrasil.serialize.PandasSerialize module¶

class yggdrasil.serialize.PandasSerialize.PandasSerialize(*args, **kwargs)[source]¶

Bases: AsciiTableSerialize

Class for serializing/deserializing Pandas data frames.

Parameters:

no_header (bool, optional) – If True, headers will not be read or serialized from/to tables. Defaults to False.
str_as_bytes (bool, optional) – If True, strings in columns are read as bytes. Defaults to False.

classmethod apply_field_names(frame, field_names=None)[source]¶

Apply field names as columns to a frame, first checking for a mapping. If there is a direct mapping, the columns are reordered to match the order of the field names. If there is not an overlap in the field names and columns, a one-to-one mapping is assumed, but a warning is issued. If there is a partial overlap, an error is raised.

Parameters:

frame (pandas.DataFrame) – Frame to apply field names to as columns.
field_names (list, optional) – New field names that should be applied. If not provided, the original frame will be returned unaltered.

Returns:

Frame with updated field names.

Return type:

pandas.DataFrame

Raises:

RuntimeError – If there is a partial overlap between the field names and columns.

as_array = True¶

cformat2nptype(*args, **kwargs)[source]¶

Method to convert c format string to numpy data type.

Parameters:

*args – Arguments are passed to serialize.cformat2nptype.
**kwargs – Keyword arguments are passed to serialize.cformat2nptype.

Returns:

Corresponding numpy data type.

Return type:

np.dtype

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = False¶

consolidate_array(out)[source]¶

Consolidate message into a structure numpy array if possible.

Parameters:: out (list, tuple, np.ndarray) – Object to consolidate into a structured numpy array.
Returns:: Structured numpy array containing consolidated message.
Return type:: np.ndarray
Raises:: ValueError – If the array cannot be consolidated.

default_read_meth = 'read'¶

deserialize_file_header(fd)[source]¶

Deserialize the header information from the file and update the serializer.

Parameters:: fd (file) – File containing header.

classmethod dict2object(obj, field_names=None, **kwargs)[source]¶

Conver a dictionary to a message object.

Parameters:

obj (dict) – Dictionary to convert to serializable object.
**kwargs – Additional keyword arguments are ignored.

Returns:

Serializable object.

Return type:

disable_file_header()[source]¶: Set serializer attributes to disable file headers from being included in the serializations.

property empty_msg¶

Object indicating empty message.

Type:: obj

enable_file_header()[source]¶: Set serializer attributes to enable file headers to be included in the serializations.

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (str, bytes) – Message to be deserialized.
Returns:: Deserialized Python object.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (obj) – Python object to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

get_field_names(*args, **kwargs)[source]¶

Get the field names for an array of fields.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

Returns:

Names for each field in the data type.

Return type:

classmethod get_testing_options(not_as_frames=False, no_names=False, no_header=False, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

not_as_frames (bool, optional) – If True, the returned example includes data that is not in a pandas data frame. Defaults to False.
no_names (bool, optional) – If True, an example is returned where the names are not provided to the deserializer. Defaults to False.
no_header (bool, optional) – If True, an example is returned
False. (where a header is not included. Defaults to) –

Returns:

Dictionary of variables to use for testing.

Return type:

initialize_from_message(msg, **kwargs)[source]¶

Initialize the serializer based on recieved message.

Parameters:

msg (object) – Message that serializer should be initialized from.
**kwargs – Additional keyword arguments are treated as metadata that may contain additional information for initializing the serializer.

normalize(args)[source]¶

Normalize a message to conform to the expected datatype.

Parameters:: args (object) – Message arguments.
Returns:: Normalized message.
Return type:: object

classmethod normalize_bytes2unicode(args)[source]¶

Convert columns that are bytes to unicode.

Parameters:: args (pd.DataFrame) – Pandas dataframe to convert columns for.
Returns:: Version of args without bytes in columns.
Return type:: pd.DataFrame

classmethod normalize_unicode2bytes(args)[source]¶

Convert columns that are unicode to bytes.

Parameters:: args (pd.DataFrame) – Pandas dataframe to convert columns for.
Returns:: Version of args without unicode in columns.
Return type:: pd.DataFrame

classmethod object2array(obj, **kwargs)[source]¶

Convert a message object into an array.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in an array form.
**kwargs – Additional keyword arguments are ignored.

Returns:

Array version of the provided object.

Return type:

np.array

classmethod object2dict(obj, **kwargs)[source]¶

Convert a message object into a dictionary.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in a dictionary form.
**kwargs – Additional keyword arguments are ignored.

Returns:

Dictionary version of the provided object.

Return type:

property send_converter¶

serialize_file_header()[source]¶

Return the serialized header information that should be prepended to files serialized using this class.

Returns:: Header string that should be written to the file.
Return type:: bytes

yggdrasil.serialize.PickleSerialize module¶

class yggdrasil.serialize.PickleSerialize.PickleSerialize(*args, **kwargs)[source]¶

Bases: DefaultSerialize

Class for serializing a python object into a bytes message by pickling.

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

file_extensions = ['.pkl']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (str, bytes) – Message to be deserialized.
Returns:: Deserialized Python object.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (obj) – Python object to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_first_frame(msg)[source]¶

Extract one frame from the provided message that may contain one or more frames.

Parameters:

msg (bytes) – Message containing one or more frames.

Returns:

Portion of message containing the first frame. If no frames: are found, an empty string will be returned.

Return type:

bytes

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

is_framed = True¶

yggdrasil.serialize.PlySerialize module¶

class yggdrasil.serialize.PlySerialize.GeometryBase[source]¶

Bases: object

Base class for extening rapidjson geometry classes.

apply_scalar_map(scalar_arr, color_map=None, vmin=None, vmax=None, scaling='linear', scale_by_area=False, no_copy=False, _as_obj=False)[source]¶

Set the color of faces in a 3D object based on a scalar map. This creates a copy unless no_copy is True.

Parameters:

scalar_arr (arr) – Scalar values that should be mapped to colors for each face.
color_map (str, optional) – The name of the color map that should be used. Defaults to ‘plasma’.
vmin (float, optional) – Value that should map to the minimum of the colormap. Defaults to min(scalar_arr).
vmax (float, optional) – Value that should map to the maximum of the colormap. Defaults to max(scalar_arr).
scaling (str, optional) – Scaling that should be used to map the scalar array onto the colormap. Defaults to ‘linear’.
scale_by_area (bool, optional) – If True, the elements of the scalar array will be multiplied by the area of the corresponding face. If True, vmin and vmax should be in terms of the scaled array. Defaults to False.
no_copy (bool, optional) – If True, the returned object will not be a copy. Defaults to False.

Returns:

Geometry with updated vertex colors.

Return type:

classmethod from_scene(scene, d=None, conversion=1.0)[source]¶

Create a geometry dictionary from a PlantGL scene and descritizer.

Parameters:

scene (openalea.plantgl.scene) – Scene that should be descritized.
d (openalea.plantgl.descritizer, optional) – Descritizer. Defaults to openalea.plantgl.all.Tesselator.
conversion (float, optional) – Conversion factor that should be applied to the vertex positions. Defaults to 1.0.

classmethod from_shape(shape, d, conversion=1.0, _as_obj=False)[source]¶

Create a geometry dictionary from a PlantGL shape and descritizer.

Parameters:

scene (openalea.plantgl.scene) – Scene that should be descritized.
d (openalea.plantgl.descritizer) – Descritizer.
conversion (float, optional) – Conversion factor that should be applied to the vertex positions. Defaults to 1.0.

to_geom_args(conversion=1.0, name=None, _as_obj=False)[source]¶

Get arguments for creating a PlantGL geometry.

Parameters:

conversion (float, optional) – Conversion factor that should be applied to the vertices. Defaults to 1.0.
name (str, optional) – Name that should be given to the created PlantGL symbol. Defaults to None and is ignored.

Returns:

Class, arguments and keyword arguments for PlantGL geometry.

Return type:

to_scene(conversion=1.0, name=None)[source]¶

Create a PlantGL scene from a geometry dictionary.

Parameters:

conversion (float, optional) – Conversion factor that should be applied to the vertices. Defaults to 1.0.
name (str, optional) – Name that should be given to the created PlantGL symbol. Defaults to None and is ignored.

Returns:

class yggdrasil.serialize.PlySerialize.PlyDict[source]¶

Bases: GeometryBase, Ply

Enhanced dictionary class for storing Ply information.

class yggdrasil.serialize.PlySerialize.PlySerialize(*args, **kwargs)[source]¶

Bases: SerializeBase

Class for serializing/deserializing .ply file formats.

Parameters:

write_header (bool, optional) – If True, headers will be added to serialized output. Defaults to True.
newline (str, optional) – String that should be used for new lines. Defaults to ‘n’.
prune_duplicates (bool, optional) – If True, serialized meshes in array format will be pruned of duplicates when being normalized into a Ply object. If False, duplicates will not be pruned. Defaults to True.

write_header¶

If True, headers will be added to serialized output.

Type:: bool

newline¶

String that should be used for new lines.

Type:: str

default_rgb¶

Default color in RGB that should be used for missing colors.

Type:: list

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = False¶

default_datatype = {'type': 'ply'}¶

file_extensions = ['.ply']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (bytes) – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

property initialized¶

True if the serializer has been initialized.

Type:: bool

classmethod is_mesh(args)[source]¶

Check if an object is a 3D mesh with x, y, z for vertices in each face in rows.

Parameters:: args (object) – Object to check.
Returns:: True if object is a mesh, false otherwise.
Return type:: bool

normalize(args)[source]¶

Normalize a message to conform to the expected datatype.

Parameters:: args (object) – Message arguments.
Returns:: Normalized message.
Return type:: object

yggdrasil.serialize.SerializeBase module¶

class yggdrasil.serialize.SerializeBase.SerializeBase(*args, **kwargs)[source]¶

Bases: YggClass

Base class for serializing/deserializing a Python object into/from a bytes message.

Parameters:

newline (str, optional) – One or more characters indicating a newline. Defaults to ‘n’.
comment (str, optional) – One or more characters indicating a comment. Defaults to ‘# ‘.
datatype (schema, optional) – JSON schema defining the type of object that the serializer will be used to serialize/deserialize. Defaults to default_datatype.
**kwargs – Additional keyword args are processed as part of the type definition.

initialized¶

True if the serializer has been initialized either by input arguments specifying the type or by infering the type from a processed message.

Type:: bool

Class Attributes:

has_header (bool): True if the serialization has a header when written: to a file.
default_read_meth (str): Default method that data should be read from: a file for deserialization.
is_framed (bool): True if the serialization has a frame allowing: multiple serialized objects to be recovered from a single message.
concats_as_str (bool): True if serialized objects can be concatenated: directly as strings.
encoded_datatype (schema): JSON schema defining the type of object: produced by the class’s func_serialize method. For most classes this will be {‘type’: ‘scalar’, ‘subtype’: ‘string’}, indicating that the method will produce bytes suitable for serialization.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration.

cformat2nptype(*args, **kwargs)[source]¶

Method to convert c format string to numpy data type.

Parameters:

*args – Arguments are passed to serialize.cformat2nptype.
**kwargs – Keyword arguments are passed to serialize.cformat2nptype.

Returns:

Corresponding numpy data type.

Return type:

np.dtype

classmethod concatenate(objects, **kwargs)[source]¶

Concatenate objects to get object that would be recieved if the concatenated serialization were deserialized.

Parameters:

objects (list) – Objects to be concatenated.
**kwargs – Additional keyword arguments are ignored.

Returns:

Set of objects that results from concatenating those provided.

Return type:

concats_as_str = True¶

consolidate_array(out)[source]¶

Consolidate message into a structure numpy array if possible.

Parameters:: out (list, tuple, np.ndarray) – Object to consolidate into a structured numpy array.
Returns:: Structured numpy array containing consolidated message.
Return type:: np.ndarray
Raises:: ValueError – If the array cannot be consolidated.

decode(msg, no_data=False, metadata=None)[source]¶

Decode message parts into header and body.

Parameters:

msg (str, bytes) – Message to be decoded.
no_data (bool, optional) – If True, only the metadata is returned. Defaults to False.
metadata (dict, optional) – Metadata that should be used to deserialize the message instead of the current header content. Defaults to None and is not used.

Returns:

Deserialized message and header information.

Return type:

tuple(obj, dict)

Raises:

ValueError – If msg contains a header, but metadata is also provided as an argument.
TypeError – If msg is not bytes.

default_datatype = {'subtype': 'string', 'type': 'scalar'}¶

default_read_meth = 'read'¶

deserialize(msg, **kwargs)[source]¶

Deserialize a message.

Parameters:

msg (str, bytes) – Message to be deserialized.
**kwargs – Additional keyword arguments are passed to the decode method.

Returns:

Deserialized message and header information.

Return type:

tuple(obj, dict)

Raises:

TypeError – If msg is not bytes type (str on Python 2).

deserialize_file_header(fd)[source]¶

Deserialize the header information from the file and update the serializer.

Parameters:: fd (file) – File containing header.

classmethod dict2object(obj, **kwargs)[source]¶

Conver a dictionary to a message object.

Parameters:

obj (dict) – Dictionary to convert to serializable object.
**kwargs – Additional keyword arguments are ignored.

Returns:

Serializable object.

Return type:

disable_file_header()[source]¶: Set serializer attributes to disable file headers from being included in the serializations.

dump(fd, obj, **kwargs)[source]¶

Serialize to a file.

Parameters:

fd (str, file) – Filename or file-like object to load from.
**kwargs – Additional keyword arguments are passed to the created FileComm used for reading.

property empty_msg¶

Object indicating empty message.

Type:: obj

enable_file_header()[source]¶: Set serializer attributes to enable file headers to be included in the serializations.

encode(data, metadata, no_metadata=False, max_header_size=0)[source]¶

Encode the message with metadata in a header.

Parameters:

data (bytes) – Message data serialized into bytes.
metadata (dict) – Metadata that should be included in the message header.
no_metadata (bool, optional) – If True, no metadata will be added to the serialized message. Defaults to False.
max_header_size (int, optional) – Maximum size that header should occupy in order to be sent in a single message. A value of 0 indicates that any size header is valid. Defaults to 0.

Returns:

Encoded message with header.

Return type:

bytes

file_extensions = ['.txt']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg – Message to be deserialized.
Returns:: Deserialized message.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args – List of arguments to be formatted or numpy array to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

get_field_names(as_bytes=False)[source]¶

Get the field names for an array of fields.

Parameters:: as_bytes (bool, optional) – If True, the field names will be returned as bytes. If False the field names will be returned as unicode. Defaults to False.
Returns:: Names for each field in the data type.
Return type:: list

get_field_units(as_bytes=False)[source]¶

Get the field units for an array of fields.

Parameters:: as_bytes (bool, optional) – If True, the field units will be returned as bytes. If False the field units will be returned as unicode. Defaults to False.
Returns:: Units for each field in the data type.
Return type:: list

get_status_message(nindent=0, extra_lines_before=None, extra_lines_after=None)[source]¶

Return lines composing a status message.

Parameters:

nindent (int, optional) – Number of tabs that should be used to indent each line. Defaults to 0.
extra_lines_before (list, optional) – Additional lines that should be added to the beginning of the default print message. Defaults to empty list if not provided.
extra_lines_after (list, optional) – Additional lines that should be added to the end of the default print message. Defaults to empty list if not provided.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

classmethod get_testing_options(table_example=False, array_columns=False, include_oldkws=False, table_string_type='bytes', no_names=False, no_units=False, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

table_example (bool, optional) – If True, the returned options will be for an array of elements representing a table-like structure. Defaults to False.
array_columns (bool, optional) – If True, table_example is set to True and the returned options will be for an array data type where each element is an array representing a column Defaults to False.
include_oldkws (bool, optional) – If True, old-style keywords will be added to the returned options. This will only have an effect if table_example is True. Defaults to False.
table_string_type (str, optional) – Type that should be used for the string column in the table. Defaults to ‘bytes’.
no_names (bool, optional) – If True, an example is returned where the names are not provided to the deserializer. Defaults to False.
no_units (bool, optional) – If True, units will not be added to the returned array if table_example is True.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the provided content.
empty (object): Object produced from deserializing an empty message.
objects (list): List of objects to be serialized/deserialized. extra_kwargs (dict): Extra keyword arguments not used to construct type definition.
typedef (dict): Type definition resulting from the supplied kwargs.
dtype (np.dtype): Numpy data types that is consistent with the determined type definition.
contents (bytes): Concatenated serialization that will result from deserializing the serialized objects.
contents_recv (list): List of objects that would be deserialized from contents.

Return type:

has_header = False¶

initialize_from_message(msg, serializer=None, **metadata)[source]¶

Initialize the serializer based on recieved message.

Parameters:

msg (object) – Message that serializer should be initialized from.
**kwargs – Additional keyword arguments are treated as metadata that may contain additional information for initializing the serializer.

initialize_from_metadata(metadata)[source]¶

Initialize a serializer based on received metadata. This method will exit early if the serializer has already been intialized.

Parameters:: metadata (dict) – Header information including type info that should be used to initialize the serializer class.

property initialized¶

True if the serializer has been initialized.

Type:: bool

property input_kwargs¶

Get the input keyword arguments used to create this class.

Type:: dict

is_framed = False¶

load(fd, **kwargs)[source]¶

Deserialize from a file.

Parameters:

fd (str, file) – Filename or file-like object to load from.
**kwargs – Additional keyword arguments are passed to the created FileComm used for reading.

Returns:

The deserialized data object or a list of: deserialized data objects if there is more than one.

Return type:

normalize(args)[source]¶

Normalize a message to conform to the expected datatype.

Parameters:: args (object) – Message arguments.
Returns:: Normalized message.
Return type:: object

property numpy_dtype¶

Corresponding structured data type. Will be None unless the type is an array of 1darrays.

Type:: np.dtype

classmethod object2array(obj, **kwargs)[source]¶

Convert a message object into an array.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in an array form.
**kwargs – Additional keyword arguments are ignored.

Returns:

Array version of the provided object or None if one cannot: be created.

Return type:

np.array

classmethod object2dict(obj, **kwargs)[source]¶

Convert a message object into a dictionary.

Parameters:

obj (object) – Object that would be serialized by this class and should be returned in a dictionary form.
**kwargs – Additional keyword arguments are ignored.

Returns:

Dictionary version of the provided object.

Return type:

parse_header(msg)[source]¶

Extract header info from a message.

Parameters:: msg (str) – Message to extract header from.
Returns:: Message properties.
Return type:: dict

property read_meth¶

Method that should be used to read data for deserialization.

Type:: str

classmethod seri_kws()[source]¶: Get a list of valid keyword arguments.

serialize(args, metadata=None, add_serializer_info=False, no_metadata=False, max_header_size=0)[source]¶

Serialize a message.

Parameters:

args (obj) – List of arguments to be formatted or a ready made message.
metadata (dict, optional) – Keyword arguments that should be added to the header. Defaults to None and no header is added.
add_serializer_info (bool, optional) – If True, serializer information will be added to the metadata. Defaults to False.
no_metadata (bool, optional) – If True, no metadata will be added to the serialized message. Defaults to False.
max_header_size (int, optional) – Maximum size that header should occupy in order to be sent in a single message. A value of 0 indicates that any size header is valid. Defaults to 0.

Returns:

Serialized message.

Return type:

bytes, str

Raises:

TypeError – If returned msg is not bytes type (str on Python 2).

serialize_file_header()[source]¶

Return the serialized header information that should be prepended to files serialized using this class.

Returns:: Header string that should be written to the file.
Return type:: bytes

property serializer_info¶

Serializer info.

Type:: dict

property typedef¶

Alias for datatype.

Type:: dict

update_serializer(skip_type=False, seritype=None, datatype=None, from_message=False, **kwargs)[source]¶

Update serializer with provided information.

Parameters:

skip_type (bool, optional) – If True, everything is updated except the data type. Defaults to False.
**kwargs – Additional keyword arguments are processed as part of they type definition and are parsed for old-style keywords.

Raises:

RuntimeError – If there are keywords that are not valid datatype keywords (currect or old-style).

update_typedef_from_oldstyle(typedef)[source]¶

Update a given typedef using an old, table-style serialization spec. Existing typedef values are not overwritten and warnings are raised if the provided serialization spec is not compatible with the type definition.

Parameters:: typedef (dict) – Type definition to update.
Returns:: Updated typedef.
Return type:: dict

yggdrasil.serialize.YAMLSerialize module¶

class yggdrasil.serialize.YAMLSerialize.YAMLSerialize(*args, **kwargs)[source]¶

Bases: JSONSerialize

Class for serializing a python object into a bytes message using YAML.

Parameters:

indent (str, int, optional) – String or number of spaces that should be used to indent each level within the seiralized structure. Defaults to ‘t’.
encoding (str, optional) – Encoding that should be used to serialize the object. Defaults to ‘utf-8’.
default_flow_style (bool, optional) – If True, nested collections will be serialized in the block style. If False, they will always be serialized in the flow style. See PyYAML Documentation.

file_extensions = ['.yaml', '.yml']¶

func_deserialize(msg)[source]¶

Deserialize a message.

Parameters:: msg (str, bytes) – Message to be deserialized.
Returns:: Deserialized Python object.
Return type:: obj

func_serialize(args)[source]¶

Serialize a message.

Parameters:: args (obj) – Python object to be serialized.
Returns:: Serialized message.
Return type:: bytes, str

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:: Dictionary of variables to use for testing.
Return type:: dict

yggdrasil.serialize.YAMLSerialize.decode_yaml(msg, sorted_dict_type=None, **kwargs)[source]¶

Decode a Python object from a YAML serialization.

Parameters:

msg (str) – YAML serialization to decode.
sorted_dict_type (type, optional) – Class that should be used to contain mapping objects while preserving order. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to yaml.load.

Returns:

Deserialized Python object.

Return type:

yggdrasil.serialize.YAMLSerialize.encode_yaml(obj, fd=None, indent=None, sorted_dict_type=None, sort_keys=True, **kwargs)[source]¶

Encode a Python object in YAML format.

Parameters:

obj (object) – Python object to encode.
fd (file, optional) – File descriptor for file that encoded object should be written to. Defaults to None and string is returned.
indent (int, str, optional) – Indentation for new lines in encoded string. Defaults to None.
sort_keys (bool, optional) – If True, dictionaries will be sorted alphabetically by key. Defaults to True.
**kwargs – Additional keyword arguments are passed to yaml.dump.

Returns:

Encoded object.

Return type:

str, bytes

Module contents¶

yggdrasil.drivers package¶

Submodules¶

yggdrasil.drivers.BuildModelDriver module¶

class yggdrasil.drivers.BuildModelDriver.BuildModelDriver(*args, **kwargs)[source]¶

Bases: CompiledModelDriver

Class for running build file compiled drivers.

Parameters:

name (str) – Driver name.
args (str, list) – Executable that should be created (make target) and any arguments for the executable.
target (str, optional) – Make target that should be built to create the model executable. Defaults to None.
target_language (str, optional) – Language that the target is written in. Defaults to None and will be set based on the source files provided.
target_compiler (str, optional) – Compilation tool that should be used to compile the target language. Defaults to None and will be set based on the selected language driver.
target_linker (str, optional) – Compilation tool that should be used to link the target language. Defaults to None and will be set based on the selected language driver.
target_compiler_flags (list, optional) – Compilation flags that should be passed to the target language compiler. Defaults to [].
target_linker_flags (list, optional) – Linking flags that should be passed to the target language linker. Defaults to [].
env_compiler (str, optional) – Environment variable where the compiler executable should be stored for use within the Makefile. If not provided, this will be determined by the target language driver.
env_compiler_flags (str, optional) – Environment variable where the compiler flags should be stored (including those required to compile against the yggdrasil interface). If not provided, this will be determined by the target language driver.
env_linker (str, optional) – Environment variable where the linker executable should be stored for use within the Makefile. If not provided, this will be determined by the target language driver.
env_linker_flags (str, optional) – Environment variable where the linker flags should be stored (including those required to link against the yggdrasil interface). If not provided, this will be determined by the target language driver.
**kwargs – Additional keyword arguments are passed to parent class.

buildfile¶

Path to file containing build instructions.

Type:: str

builddir¶

Path to directory where build products will be saved.

Type:: str

sourcedir¶

Path to directory where source files are located.

Type:: str

compiledir¶

Path to directory where compilation call should be made from. Defaults to working_dir.

Type:: str

target¶

Name of executable that should be created and called.

Type:: str

target_language¶

Language that the target is written in.

Type:: str

target_language_driver¶

Language driver for the target language.

Type:: ModelDriver

target_compiler¶

Compilation tool that should be used to compile the target language.

Type:: str

target_linker¶

Compilation tool that should be used to link the target language.

Type:: str

target_compiler_flags¶

Compilation flags that should be passed to the target language compiler.

Type:: list

target_linker_flags¶

Linking flags that should be passed to the target language linker.

Type:: list

env_compiler¶

Compiler environment variable.

Type:: str

env_compiler_flags¶

Compiler flags environment variable.

Type:: str

env_linker¶

Linker environment variable.

Type:: str

env_linker_flags¶

Linker flags environment variable.

Type:: str

Class Attributes:

built_where_called (bool): If True, it is assumed that compilation: output will be saved in the same directory from which the compilation command is issued.

Raises:: RuntimeError – If neither the IPC or ZMQ C libraries are available.

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

allow_parallel_build = False¶

base_languages = ['c']¶

buildfile_base = None¶

built_where_called = False¶

classmethod call_compiler(src, **kwargs)[source]¶

Compile a source file into an executable or linkable object file, checking for errors.

Parameters:

src (str) – Full path to source file.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Full path to compiled source.

Return type:

cleanup()[source]¶: Remove compiled executable.

compile_dependencies_instance(*args, **kwargs)[source]¶: Compile dependencies specifically for this instance.

compile_model(**kwargs)[source]¶

Compile model executable(s).

Parameters:: **kwargs – Keyword arguments are passed on to the parent class’s method.

executable_type = 'build'¶

classmethod fix_path(path, for_env=False)[source]¶

Update a path.

Parameters:

path (str) – Path that should be formatted.
for_env (bool, optional) – If True, the path is formatted for use in and environment variable. Defaults to False.

Returns:

Updated path.

Return type:

full_language = False¶

classmethod get_buildfile_lock(**kwargs)[source]¶: Get a lock for a buildfile to prevent simultaneous access, creating one as necessary.

classmethod get_language_for_buildfile(buildfile, target=None)[source]¶

Determine the target language based on the contents of a build file.

Parameters:

buildfile (str) – Full path to the build configuration file.
target (str, optional) – Target that will be built. Defaults to None and the default target in the build file will be used.

classmethod get_language_for_source(fname=None, buildfile=None, languages=None, early_exit=False, call_base=False, **kwargs)[source]¶

Determine the language that can be used with the provided source file(s). If more than one language applies to a set of multiple files, the language that applies to the most files is returned.

Parameters:

fname (str, list) – The full path to one or more files. If more than one is provided, they are iterated over.
buildfile (str, optional) – Full path to the build configuration file. Defaults to None and will be searched for.
languages (list, optional) – The list of languages that are acceptable. Defaults to None and any language will be acceptable.
early_exit (bool, optional) – If True, the first language identified will be returned if fname is a list of files. Defaults to False.
source_dir (str, optional) – Full path to the directory containing the source files. Defaults to None and is determiend from fname.
buildfile – Full path to the build configuration file. Defaults to None and will be searched for.
target (str, optional) – The build target. Defaults to None.
call_base (bool, optional) – If True, the base class’s method is called directly. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The language that can operate on the specified file.

Return type:

classmethod get_source_dir(fname=None, source_dir=None)[source]¶

classmethod get_target_language_info(target_language_driver=None, target_language='c', target_compiler=None, target_compiler_flags=None, target_linker=None, target_linker_flags=None, logging_level=None, without_wrapper=False, compiler_flag_kwargs=None, linker_flag_kwargs=None, **kwargs)[source]¶

Get a dictionary of information about language compilation tools.

Parameters:

target_language_driver (ModelDriver, optional) – Driver associated with the target language. If not provided, one will be created based on ‘target_language’.
target_language (str, optional) – Language to get info for. Defaults to ‘c’.
target_compiler (str, optional) – Compilation tool that should be used to compile the target language. Defaults to None and will be set based on the selected language driver.
target_linker (str, optional) – Compilation tool that should be used to link the target language. Defaults to None and will be set based on the selected language driver.
target_compiler_flags (list, optional) – Compilation flags that should be passed to the target language compiler. Defaults to [].
target_linker_flags (list, optional) – Linking flags that should be passed to the target language linker. Defaults to [].
logging_level (int, optional) – The numeric logging level that should be passed as a definition. Defaults to None and is ignored.
without_wrapper (bool, optional) – If True, the returned info will be updated for compilation when a build wrapper was not created. Defaults to False.
compiler_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_compiler_flags method. Defaults to None.
linker_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_linker_flags method. Defaults to None.
**kwargs – Additional keyword arguments are added to the output dictionary.

Returns:

Information about language compilers and linkers.

Return type:

is_build_tool = True¶

classmethod is_source_file(fname)[source]¶

Determine if the provided file name points to a source files for the associated programming language by checking the extension.

Parameters:: fname (str) – Path to file.
Returns:: True if the provided file is a source file, False otherwise.
Return type:: bool

parse_arguments(args, **kwargs)[source]¶

Sort arguments based on their syntax to determine if an argument is a source file, compilation flag, or runtime option/flag that should be passed to the model executable.

Parameters:

args (list) – List of arguments provided.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

set_env(**kwargs)[source]¶

Get environment variables that should be set for the model process.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method.
Returns:: Environment variables for the model process.
Return type:: dict

classmethod set_env_compiler(language_info=None, **kwargs)[source]¶

Get environment variables that should be set for the compilation process.

Parameters:

language_info (dict) – Language compilation tool information.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Environment variables for the model process.

Return type:

set_target_language()[source]¶

Set the language of the target being compiled (usually the same as the language associated with this driver.

Returns:: Name of language.
Return type:: str

sourcedir_as_sourcefile = False¶

property target_language_info¶

Information about the underlying language.

Type:: dict

class yggdrasil.drivers.BuildModelDriver.BuildToolBase(**kwargs)[source]¶

Bases: CompilerBase

Base class for build tools.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

build_language = None¶

classmethod get_flags(**kwargs)[source]¶

Get compilation flags, replacing outfile with target.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method.
Returns:: Compiler flags.
Return type:: list

classmethod get_tool_suffix()[source]¶

Get the string that should be added to tool products based on the tool used.

Returns:

Suffix that should be added to tool products to indicate the: tool used.

Return type:

is_build_tool = True¶

yggdrasil.drivers.CMakeModelDriver module¶

class yggdrasil.drivers.CMakeModelDriver.CMakeBuilder(**kwargs)[source]¶

Bases: LinkerBase

CMake build tool.

classmethod call(*args, **kwargs)[source]¶: Print contents of CMakeCache.txt before raising error.

default_executable = 'cmake'¶

default_flags = []¶

executable_ext = ''¶

classmethod extract_kwargs(kwargs, **kwargs_ex)[source]¶

Extract linker kwargs, leaving behind just compiler kwargs.

Parameters:

kwargs (dict) – Keyword arguments passed to the compiler that should be sorted into kwargs used by either the compiler or linker or both. Keywords that are not used by the compiler will be removed from this dictionary.
**kwargs_ex – Additional keyword arguments are passed to the parent class’s method.

Returns:

Keyword arguments that should be passed to the linker.

Return type:

flag_options = {'builddir': {'key': '--build', 'position': 0}, 'configuration': '--config', 'target': '--target'}¶

classmethod get_executable_command(args, **kwargs)[source]¶

Determine the command required to run the tool using the specified arguments and options.

Parameters:

args (list) – The arguments that should be passed to the tool. If skip_flags is False, these are treated as input files that will be used by the tool.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Output to stdout from the command execution.

Return type:

classmethod get_flags(target=None, builddir=None, **kwargs)[source]¶

Get a list of build flags for building a project using cmake.

Parameters:

target (str, optional) – Target that should be built. Defaults to to None and is ignored.
builddir (str, optional) – Directory containing the build tree. Defaults to None and is set based on outfile is provided or cls.default_builddir if not. Defaults to ‘.’ (which will be the current working directory).
**kwargs – Additional keyword arguments are ignored.

Returns:

Linker flags.

Return type:

classmethod get_output_file(obj, target=None, builddir=None, **kwargs)[source]¶

Determine the appropriate output file that will result when bulding a given directory.

Parameters:

obj (str) – Directory being built or a file in the directory being built.
target (str, optional) – Target that will be used to create the output file. Defaults to None. Target is required in order to determine the name of the file that will be created.
builddir (str, optional) – Directory where build tree should be created. Defaults to None and obj will used (if its a directory) or the directory containing obj will be used (if its a file).
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Full path to file that will be produced.

Return type:

Raises:

RuntimeError – If target is None.

is_gnu = False¶

languages = ['cmake']¶

library_ext = '.so'¶

output_key = None¶

tool_suffix_format = ''¶

toolname = 'cmake'¶

class yggdrasil.drivers.CMakeModelDriver.CMakeConfigure(**kwargs)[source]¶

Bases: BuildToolBase

CMake configuration tool.

add_libraries = False¶

classmethod append_product(products, src, new, **kwargs)[source]¶

Append a product to the specified list along with additional values indicated by cls.product_exts.

Parameters:

products (list) – List of of existing products that new product should be appended to.
src (list) – Input arguments to compilation call that was used to generate the output file (usually one or more source files).
new (str) – New product that should be appended to the list.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

build_language = 'cmake'¶

classmethod call(args, **kwargs)[source]¶

Call the tool with the provided arguments. If the first argument resembles the name of the tool executable, the executable will not be added.

Parameters:

args (list) – The arguments that should be passed to the tool.
**kwargs – Additional keyword arguments are passed to the parent class’s method and the associated linker/archiver’s call method if dont_link is False.

Returns:

Output to stdout from the command execution if skip_flags is: True, produced file otherwise.

Return type:

combine_with_linker = False¶

compile_only_flag = None¶

classmethod create_include(fname, target, driver=None, compiler=None, compiler_flags=None, linker=None, linker_flags=None, library_flags=None, internal_library_flags=None, configuration='Release', verbose=False, **kwargs)[source]¶

Create CMakeList include file with necessary includes, definitions, and linker flags.

Parameters:

fname (str) – File where the include file should be saved.
target (str) – Target that links should be added to.
driver (CompiledModelDriver) – Driver for the language being compiled.
compiler (CompilerBase) – Compiler that should be used to generate the list of compilation flags.
compile_flags (list, optional) – Additional compile flags that should be set. Defaults to [].
linker (LinkerBase) – Linker that should be used to generate the list of compilation flags.
linker_flags (list, optional) – Additional linker flags that should be set. Defaults to [].
library_flags (list, optional) – List of library flags to add. Defaults to [].
internal_library_flags (list, optional) – List of library flags associated with yggdrasil libraries. Defaults to [].
configuration (str, optional) – Build type/configuration that should be built. Defaults to ‘Release’. Only used on Windows to determine the standard library.
verbose (bool, optional) – If True, the contents of the created file are displayed. Defaults to False.
**kwargs – Additional keyword arguments are ignored.

Returns:

Lines that should be added before the executable is: defined in the CMakeLists.txt (e.g. LINK_DIRECTORIES commands).

Return type:

Raises:

ValueError – If a linker or compiler flag cannot be interpreted.

default_archiver = False¶

default_builddir = '.'¶

default_executable = 'cmake'¶

default_flags = []¶

classmethod fix_path(x, is_gnu=False)[source]¶: Fix paths so that they conform to the format expected by the OS and/or build tool.

flag_options = {'builddir': '-B%s', 'configuration': '-DCMAKE_BUILD_TYPE=%s', 'definitions': '-D%s', 'generator': '-G%s', 'platform': '-A%s', 'sourcedir': '', 'toolset': '-T%s'}¶

classmethod generator(return_default=False, default=None, **kwargs)[source]¶

Determine the generator that should be used.

Parameters:

return_default (bool, optional) – If True, the default generator will be returned even if the environment variable is set. Defaults to False.
default (str, optional) – Value that should be returned if a generator cannot be located. Defaults to None.
**kwargs – Keyword arguments are passed to cls.call.

Returns:

Name of the generator.

Return type:

classmethod generator2toolset(generator)[source]¶

Determine the toolset string option that corresponds to the provided generator name.

Parameters:

generator (str) – Name of the generator.

Returns:

Name of the toolset.

Return type:

Raises:

NotImplementedError – If the platform is not windows.
ValueError – If the generator is not a flavor of Visual Studio.
ValueError – If a tool set cannot be located for the specified generator.

classmethod get_executable_command(args, **kwargs)[source]¶

Determine the command required to run the tool using the specified arguments and options.

Parameters:

args (list) – The arguments that should be passed to the tool. If skip_flags is False, these are treated as input files that will be used by the tool.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Output to stdout from the command execution.

Return type:

classmethod get_flags(sourcedir='.', builddir=None, target_compiler=None, target_linker=None, **kwargs)[source]¶

Get a list of configuration/generation flags.

Parameters:

sourcedir (str, optional) – Directory containing the source files to be compiled and the target CMakeLists.txt file. Defaults to ‘.’ (the current working directory).
builddir (str, optional) – Directory that will contain the build tree. Defaults to ‘.’ (this current working directory).
target_compiler (str, optional) – Compiler that should be used by cmake. Defaults to None and the default for the target language will be used.
target_linker (str, optional) – Linker that should be used by cmake. Defaults to None and the default for the target language will be used.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Compiler flags.

Return type:

Raises:

RuntimeError – If dont_link is True and the provide outfile and builddir keyword arguments point to conflicting paths.
ValueError – If ‘include_dirs’ is set (‘sourcedir’ should be used for cmake to specify the location of the source).

classmethod get_output_file(src, dont_link=False, dont_build=None, sourcedir=None, builddir=None, working_dir=None, **kwargs)[source]¶

Determine the appropriate output file or directory that will result when configuring/building a given source directory.

Parameters:

src (str) – Directory containing source files being compiled.
dont_link (bool, optional) – If True, the result assumes that the source is just compiled and not linked. If False, the result will be the final result after linking.
dont_build (bool, optional) – Alias for dont_link. If not None, this keyword overrides the value of dont_link. Defaults to None.
sourcedir (str, optional) – Directory where sources files are located. Defaults to None. If None, src will be used to determine the value.
builddir (str, optional) – Directory where build tree should be created. Defaults to None. If None, sourcedir will be used.
working_dir (str, optional) – Working directory where output file should be located. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are ignored unless dont_link is False; then they are passed to get_linker_output_file

Returns:

Full path to file that will be produced.

Return type:

is_gnu = False¶

is_linker = False¶

languages = ['cmake']¶

output_key = None¶

product_files = ['Makefile', 'CMakeCache.txt', 'cmake_install.cmake', 'CMakeFiles']¶

remove_product_exts = ['CMakeFiles']¶

toolname = 'cmake'¶

class yggdrasil.drivers.CMakeModelDriver.CMakeModelDriver(*args, **kwargs)[source]¶

Bases: BuildModelDriver

Class for running cmake compiled drivers. Before running the cmake command, the cmake commands for setting the necessary compiler & linker flags for the interface’s C/C++ library are written to a file called ‘ygg_cmake.txt’ that should be included in the CMakeLists.txt file (after the target executable has been added).

Parameters:

name (str) – Driver name.
args (str, list) – Executable that should be created (cmake target) and any arguments for the executable.
sourcedir (str, optional) – Source directory to call cmake on. If not provided it is set to working_dir. This should be the directory containing the CMakeLists.txt file. It can be relative to working_dir or absolute.
builddir (str, optional) – Directory where the build should be saved. Defaults to <sourcedir>/build. It can be relative to working_dir or absolute.
configuration (str, optional) – Build type/configuration that should be built. Defaults to ‘Release’.
**kwargs – Additional keyword arguments are passed to parent class.

sourcedir¶

Source directory to call cmake on.

Type:: str

add_libraries¶

If True, interface libraries and dependency libraries are added using CMake’s ADD_LIBRARY directive. If False, interface libraries are found using FIND_LIBRARY.

Type:: bool

configuration¶

Build type/configuration that should be built. This is only used on Windows.

Type:: str

Raises:: RuntimeError – If neither the IPC or ZMQ C libraries are available.

add_libraries = False¶

buildfile_base = 'CMakeLists.txt'¶

property buildfile_orig¶

Full path to where the original CMakeLists.txt file will be stored during compilation of the modified file.

Type:: str

property buildfile_ygg¶

Full path to the verison of the CMakeLists.txt that has been updated w/ yggdrasil compilation flags.

Type:: str

cfg = <yggdrasil.config.YggConfigParser object>¶

compile_model(target=None, **kwargs)[source]¶

Compile model executable(s) and appends any products produced by the compilation that should be removed after the run is complete.

Parameters:

target (str, optional) – Target to build.
**kwargs – Keyword arguments are passed on to the call_compiler method.

default_archiver = None¶

default_archiver_flags = None¶

default_compiler = 'cmake'¶

default_compiler_flags = None¶

default_linker = 'cmake'¶

default_linker_flags = None¶

classmethod fix_path(path, for_env=False, **kwargs)[source]¶

Update a path.

Parameters:

path (str) – Path that should be formatted.
for_env (bool, optional) – If True, the path is formatted for use in and environment variable. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Updated path.

Return type:

classmethod get_language_for_buildfile(buildfile, target=None)[source]¶

Determine the target language based on the contents of a build file.

Parameters:

buildfile (str) – Full path to the build configuration file.
target (str, optional) – Target that will be built. Defaults to None and the default target in the build file will be used.

classmethod get_target_language_info(target_compiler_flags=None, target_linker_flags=None, compiler_flag_kwargs=None, linker_flag_kwargs=None, without_wrapper=False, **kwargs)[source]¶

Get a dictionary of information about language compilation tools.

Parameters:

target_compiler_flags (list, optional) – Compilation flags that should be passed to the target language compiler. Defaults to [].
target_linker_flags (list, optional) – Linking flags that should be passed to the target language linker. Defaults to [].
compiler_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_compiler_flags method. Defaults to None.
linker_flag_kwargs (dict, optional) – Keyword arguments to pass to the get_linker_flags method. Defaults to None.
**kwargs – Keyword arguments are passed to the parent class’s method.

Returns:

Information about language compilers and linkers.

Return type:

inverse_type_map = None¶

language = 'cmake'¶

parse_arguments(args, **kwargs)[source]¶

Sort arguments based on their syntax to determine if an argument is a source file, compilation flag, or runtime option/flag that should be passed to the model executable.

Parameters:

args (list) – List of arguments provided.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

classmethod prune_sh_gcc(path, gcc)[source]¶

Remove instances of sh.exe from the path that are not associated with the selected gcc compiler. This can happen on windows when rtools or git install a version of sh.exe that is added to the path before the compiler.

Parameters:

path (str) – Contents of the path variable.
gcc (str) – Full path to the gcc executable.

Returns:

Modified path that removes the extra instances: of sh.exe.

Return type:

sourcedir_as_sourcefile = True¶

classmethod update_compiler_kwargs(**kwargs)[source]¶

Update keyword arguments supplied to the compiler get_flags method for various options.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Keyword arguments for a get_flags method providing compiler: flags.

Return type:

use_env_vars = False¶

write_wrappers(**kwargs)[source]¶

Write any wrappers needed to compile and/or run a model.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Full paths to any created wrappers.
Return type:: list

yggdrasil.drivers.CModelDriver module¶

class yggdrasil.drivers.CModelDriver.ARArchiver(**kwargs)[source]¶

Bases: ArchiverBase

Archiver class for ar tool.

asan_flags = []¶

compatible_toolsets = ['gnu', 'llvm']¶

default_executable = 'ar'¶

default_executable_env = 'AR'¶

default_flags = []¶

default_flags_env = None¶

is_gnu = True¶

languages = ['c', 'c++', 'fortran']¶

library_ext = '.a'¶

output_first_library = True¶

output_key = ''¶

search_path_envvar = ['LIBRARY_PATH']¶

shared_library_flag = None¶

static_library_flag = 'rcs'¶

toolname = 'ar'¶

toolset = 'gnu'¶

class yggdrasil.drivers.CModelDriver.CCompilerBase(**kwargs)[source]¶

Bases: CompilerBase

Base class for C compilers.

classmethod call(args, **kwargs)[source]¶: Call the compiler with the provided arguments. For yggdrasil C models will always be linked using the C++ linker since some parts of the interface library are written in C++.

default_executable_env = 'CC'¶

default_flags = ['-g', '-Wall']¶

default_flags_env = 'CFLAGS'¶

classmethod get_search_path(*args, **kwargs)[source]¶

Determine the paths searched by the tool for external library files.

Parameters:

*args – Additional arguments are passed to the parent class’s method.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

List of paths that the tools will search.

Return type:

languages = ['c']¶

linker_attributes = {'default_executable_env': 'LD', 'default_flags_env': 'LDFLAGS', 'search_path_envvar': ['LIBRARY_PATH', 'LD_LIBRARY_PATH']}¶

search_path_envvar = ['C_INCLUDE_PATH']¶

search_path_flags = ['-E', '-v', '-xc', '/dev/null']¶

search_regex = ['(?:#include <...> search starts here:)|(?: ([^\\n]+?)(?: \$framework directory\$)?)\\n']¶

search_regex_begin = '#include "..." search starts here:'¶

search_regex_end = 'End of search list.'¶

classmethod set_env(*args, **kwargs)[source]¶

Set environment variables required for compilation.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

Returns:

Environment variables for the model process.

Return type:

class yggdrasil.drivers.CModelDriver.CModelDriver(*args, **kwargs)[source]¶

Bases: CompiledModelDriver

Class for running C models.

classmethod add_extra_vars(direction, x)[source]¶

Add extra variables required for communication.

Parameters:

direction (str) – Direction of channel (‘input’ or ‘output’).
x (dict) – Dictionary describing the variable.

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration.

classmethod allows_realloc(var)[source]¶

Determine if a variable allows the receive call to perform realloc.

Parameters:: var (dict) – Dictionary of variable properties.
Returns:: True if the variable allows realloc, False otherwise.
Return type:: bool

brackets = ('{', '}')¶

classmethod call_linker(obj, language=None, **kwargs)[source]¶

Link several object files to create an executable or library (shared or static), checking for errors.

Parameters:

obj (list) – Object files that should be linked.
language (str, optional) – Language that should be used to link the files. Defaults to None and the language of the current driver is used.
**kwargs – Additional keyword arguments are passed to run_executable.

Returns:

Full path to compiled source.

Return type:

cfg = <yggdrasil.config.YggConfigParser object>¶

classmethod configure(cfg, macos_sdkroot=None, vcpkg_dir=None, **kwargs)[source]¶

Add configuration options for this language. This includes locating any required external libraries and setting option defaults.

Parameters:

cfg (YggConfigParser) – Config class that options should be set for.
macos_sdkroot (str, optional) – Full path to the root directory for the MacOS SDK that should be used. Defaults to None and is ignored.
vcpkg_dir (str, optional) – Full path to the root directory containing a vcpkg installation. This should be the directory that contains the vcpkg executable and any packages installed by vcpkg (in subdirectories). Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

default_archiver = 'ar'¶

default_archiver_flags = None¶

default_compiler = 'gcc'¶

default_compiler_flags = None¶

default_linker = 'gcc'¶

default_linker_flags = None¶

external_libraries = {'czmq': {'include': 'czmq.h', 'language': 'c', 'libtype': 'shared', 'shared': '/lib/x86_64-linux-gnu/libczmq.so'}, 'numpy': {'for_python_api': True, 'include': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h', 'language': 'c', 'libtype': 'header_only'}, 'python': {'for_python_api': True, 'include': '/opt/hostedtoolcache/Python/3.8.17/x64/include/python3.8/Python.h', 'language': 'c', 'libtype': 'shared', 'shared': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/libpython3.8.so', 'standard': True}, 'rapidjson': {'include': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/rapidjson/include/rapidjson/rapidjson.h', 'language': 'c', 'libtype': 'header_only'}, 'zmq': {'include': 'zmq.h', 'language': 'c', 'libtype': 'shared', 'shared': '/lib/x86_64-linux-gnu/libzmq.so'}}¶

classmethod finalize_function_io(direction, x)[source]¶

Finalize info for an input/output channel following function parsing.

Parameters:

direction (str) – Direction of channel (‘input’ or ‘output’)
x (dict) – Channel info.

classmethod format_function_param(key, default=None, **kwargs)[source]¶

Return the formatted version of the specified key.

Parameters:

key (str) – Key in cls.function_param mapping that should be formatted.
default (str, optional) – Format that should be returned if key is not in cls.function_param. Defaults to None.
**kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns:

Formatted string.

Return type:

Raises:

NotImplementedError – If key is not in cls.function_param and default is not set.

function_param = {'and': '&&', 'assign': '{name} = {value};', 'assign_copy': 'memcpy({name}, {value}, {N}*sizeof({native_type}));', 'block_end': '}', 'break': 'break;', 'comment': '//', 'copy_any': 'copy_generic_into(&{name}, {value});', 'copy_array': 'copy_generic_into(&{name}, {value});', 'copy_class': '{name} = copy_python({value});', 'copy_function': '{name} = copy_python({value});', 'copy_instance': 'copy_generic_into(&{name}, {value});', 'copy_obj': '{name} = copy_obj({value});', 'copy_object': 'copy_generic_into(&{name}, {value});', 'copy_ply': '{name} = copy_ply({value});', 'copy_schema': 'copy_generic_into(&{name}, {value});', 'declare': '{type_name} {variable};', 'error': 'printf("{error_msg}\\n"); return -1;', 'exec_begin': 'int main() {', 'exec_end': ' return 0;\n}', 'exec_prefix': '#include <stdbool.h>', 'false': 'false', 'false_flag': '0', 'flag_cond': '{flag_var} >= 0', 'for_begin': 'for ({iter_var} = {iter_begin}; {iter_var} < {iter_end}; {iter_var}++) {{', 'fprintf': 'printf("{message}\\n", {variables});', 'free': 'if ({variable} != NULL) {{ free({variable}); {variable} = NULL; }}', 'free_any': 'free_generic({variable});', 'free_array': 'free_json_array({variable});', 'free_class': 'destroy_python({variable});', 'free_function': 'destroy_python({variable});', 'free_instance': 'free_generic({variable});', 'free_obj': 'free_obj({variable});', 'free_object': 'free_json_object({variable});', 'free_ply': 'free_ply({variable});', 'free_schema': 'free_schema({variable});', 'function_def_begin': '{output_type} {function_name}({input_var}) {{', 'function_def_regex': '(?P<flag_type>.+?)\\s*{function_name}\\s*\$(?P<inputs>(?:[^{{])*?)\$\\s*\\{{(?P<body>(?:.*?\\n?)*?)(?:(?:return *(?P<flag_var>.+?)?;(?:.*?\\n?)*?\\}})|(?:\\}}))', 'if_begin': 'if ({cond}) {{', 'if_elif': '}} else if ({cond}) {{', 'if_else': '}} else {{', 'import': '#include "{filename}"', 'indent': ' ', 'index': '{variable}[{index}]', 'init_any': 'init_generic()', 'init_array': 'init_json_array()', 'init_class': 'init_python()', 'init_function': 'init_python()', 'init_instance': 'init_generic()', 'init_obj': 'init_obj()', 'init_object': 'init_json_object()', 'init_ply': 'init_ply()', 'init_schema': 'init_schema()', 'init_type_1darray': 'create_dtype_1darray("{subtype}", {precision}, {length}, "{units}", {use_generic})', 'init_type_array': 'create_dtype_json_array({nitems}, {items}, {use_generic})', 'init_type_default': 'create_dtype_default("{type}", {use_generic})', 'init_type_empty': 'create_dtype_empty({use_generic})', 'init_type_from_schema': 'create_dtype_from_schema("{schema}", {use_generic})', 'init_type_ndarray': 'create_dtype_ndarray("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_ndarray_arr': 'create_dtype_ndarray_arr("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_obj': 'create_dtype_obj({use_generic})', 'init_type_object': 'create_dtype_json_object({nitems}, {keys}, {values}, {use_generic})', 'init_type_ply': 'create_dtype_ply({use_generic})', 'init_type_pyobj': 'create_dtype_pyobj("{type}", {use_generic})', 'init_type_scalar': 'create_dtype_scalar("{subtype}", {precision}, "{units}", {use_generic})', 'init_type_schema': 'create_dtype_schema({use_generic})', 'input': '{channel} = yggInputType("{channel_name}", {channel_type});', 'inputs_def_regex': '\\s*(?P<native_type>(?:[^\\s\\*])+(\\s+)?(?P<ptr>\\*+)?)(?(ptr)(?(1)(?:\\s*)|(?:\\s+)))(\$(?P<name_ptr>\\*+)?)?(?P<name>.+?)(?(4)(?:\$))(?P<shape>(?:\\[.+?\\])+)?\\s*(?:,|$)(?:\\n)?', 'interface': '#include "{interface_library}"', 'line_end': ';', 'not': '!', 'not_flag_cond': '{flag_var} < 0', 'null': 'NULL', 'output': '{channel} = yggOutputType("{channel_name}", {channel_type});', 'outputs_def_regex': '\\s*(?P<native_type>(?:[^\\s\\*])+(\\s+)?(?P<ptr>\\*+)?)(?(ptr)(?(1)(?:\\s*)|(?:\\s+)))(?P<name>.+?)(?P<shape>(?:\\[.+?\\])+)?\\s*(?:,|$)(?:\\n)?', 'print': 'printf("{message}\\n");', 'print_any': 'display_generic({object});', 'print_array': 'display_json_array({object});', 'print_bytes': 'printf("%s\\n", {object});', 'print_class': 'display_python({object});', 'print_complex': 'print_complex({object});', 'print_float': 'printf("%f\\n", {object});', 'print_function': 'display_python({object});', 'print_instance': 'display_generic({object});', 'print_int': 'printf("%i\\n", {object});', 'print_obj': 'display_obj({object});', 'print_object': 'display_json_object({object});', 'print_ply': 'display_ply({object});', 'print_schema': 'display_schema({object});', 'print_string': 'printf("%s\\n", {object});', 'print_uint': 'printf("%u\\n", {object});', 'print_unicode': 'printf("%s\\n", {object});', 'python_init': 'init_python_API();', 'quote': '"', 'recv_function': 'yggRecvRealloc', 'recv_heap': 'yggRecvRealloc', 'recv_stack': 'yggRecv', 'return': 'return {output_var};', 'send_function': 'yggSend', 'true': 'true', 'true_flag': '1', 'while_begin': 'while ({cond}) {{'}¶

classmethod get_json_type(native_type)[source]¶

Get the JSON type from the native language type.

Parameters:: native_type (str) – The native language type.
Returns:: The JSON type.
Return type:: str, dict

classmethod get_name_declare(var)[source]¶

Determine the name that should be used for declaration.

Parameters:: var (str, dict) – Name of variable or dictionary of information.
Returns:: Modified name for declaration.
Return type:: str

classmethod get_native_type(const=False, **kwargs)[source]¶

Get the native type.

Parameters:

type (str, optional) – Name of yggdrasil extended JSON type or JSONSchema dictionary defining a datatype.
const (bool, optional) – If True, the native type will be marked as constant. Defaults to False.
**kwargs – Additional keyword arguments may be used in determining the precise declaration that should be used.

Returns:

The native type.

Return type:

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

include_channel_obj = True¶

classmethod input2output(var)[source]¶

Perform conversion necessary to turn a variable extracted from a function definition from an input to an output.

Parameters:: var (dict) – Variable definition.
Returns:: Updated variable definition.
Return type:: dict

interface_dependencies = ['rapidjson']¶

interface_directories = ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C']¶

interface_library = 'ygg'¶

interface_map = {'call': 'flag = rpcCall({channel_obj}, {outputs}, {inputs})', 'client': 'yggRpcClientType("{channel_name}", {datatype_out}, {datatype_in})', 'import': '#include "YggInterface.h"', 'input': 'yggInputType("{channel_name}", {datatype})', 'output': 'yggOutputType("{channel_name}", {datatype})', 'recv': 'flag = yggRecv({channel_obj}, {input_refs})', 'send': 'flag = yggSend({channel_obj}, {outputs})', 'server': 'yggRpcServerType("{channel_name}", {datatype_in}, {datatype_out})', 'timesync': 'yggTimesync("{channel_name}", "{time_units}")'}¶

internal_libraries = {'datatypes': {'compiler_flags': ['-fPIC'], 'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/datatypes', 'external_dependencies': ['rapidjson', 'python', 'numpy', 'm'], 'include_dirs': [], 'internal_dependencies': ['regex'], 'language': 'c++', 'libtype': 'object'}, 'regex': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c', 'libtype': 'header_only', 'source': 'regex_posix.h'}, 'regex_posix': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c', 'libtype': 'header_only', 'source': 'regex_posix.h'}, 'regex_win32': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c++', 'libtype': 'object', 'source': 'regex_win32.cpp'}, 'ygg': {'compiler_flags': ['-fPIC'], 'external_dependencies': ['rapidjson', 'python', 'numpy', 'm'], 'include_dirs': ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/communication', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/serialize', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C'], 'internal_dependencies': ['regex', 'datatypes'], 'language': 'c', 'linker_language': 'c++', 'source': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/YggInterface.c'}}¶

inverse_type_map = None¶

is_typed = True¶

language = 'c'¶

language_ext = ['.c', '.h']¶

locked_buildfile = 'datatypes.o'¶

classmethod output2input(var, in_definition=True)[source]¶

Perform conversion necessary to turn an output variable into an corresponding input that can be used to format a function definition.

Parameters:

var (dict) – Variable definition.
in_definition (bool, optional) – If True, the returned dictionary corresponds to an input variable in a function definition. If False, the returned value will correspond to an input to a function. Defaults to True.

Returns:

Updated variable definition.

Return type:

outputs_in_inputs = True¶

classmethod parse_var_definition(io, value, **kwargs)[source]¶

Extract information about input/output variables from a string definition.

Parameters:

io (str) – Description of variables contained in the provided string. Must be ‘inputs’ or ‘outputs’.
value (str) – String containing one or more variable definitions.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

List of information about the variables contained in: the provided string.

Return type:

Raises:

AssertionError – If io is not ‘inputs’ or ‘outputs’.
NotImplementedError – If the def_regex for the specified io is not defined.

classmethod prepare_output_variables(vars_list, in_definition=False, in_inputs=False, for_yggdrasil=False)[source]¶

Concatenate a set of output variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable names to concatenate as output from a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying output variables in a function definition. Defaults to False.
in_inputs (bool, optional) – If True, the output variables should be formated to be included as input variables. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod prepare_variables(vars_list, in_definition=False, for_yggdrasil=False)[source]¶

Concatenate a set of input variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing input/output to/from a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying variables in a function definition. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod requires_length_var(var)[source]¶

Determine if a variable requires a separate length variable.

Parameters:: var (dict) – Dictionary of variable properties.
Returns:: True if a length variable is required, False otherwise.
Return type:: bool

classmethod requires_shape_var(var)[source]¶

Determine if a variable requires a separate shape variable.

Parameters:: var (dict) – Dictionary of variable properties.
Returns:: True if a shape variable is required, False otherwise.
Return type:: bool

classmethod set_env_class(**kwargs)[source]¶

Set environment variables that are instance independent.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method and update_ld_library_path.
Returns:: Environment variables for the model process.
Return type:: dict

standard_libraries = ['m']¶

supported_comm_options = {'ipc': {'platforms': ['MacOS', 'Linux']}, 'zmq': {'libraries': ['zmq', 'czmq']}}¶

supported_comms = ['ipc', 'zmq']¶

type_class_map = {}¶

type_map = {'1darray': '*', 'any': 'generic_t', 'array': 'json_array_t', 'boolean': 'bool', 'bytes': 'string_t', 'class': 'python_class_t', 'comm': 'comm_t*', 'complex': 'complex_X', 'dtype': 'dtype_t*', 'flag': 'int', 'float': 'double', 'function': 'python_function_t', 'instance': 'python_instance_t', 'int': 'intX_t', 'integer': 'int', 'length': 'size_t', 'ndarray': '*', 'null': 'void*', 'number': 'double', 'obj': 'obj_t', 'object': 'json_object_t', 'ply': 'ply_t', 'schema': 'schema_t', 'string': 'char*', 'uint': 'uintX_t', 'unicode': 'unicode_t'}¶

classmethod update_ld_library_path(env, paths_to_add=None, add_to_front=False, add_libpython_dir=False, toolname=None, env_var=None, **kwargs)[source]¶

Update provided dictionary of environment variables so that LD_LIBRARY_PATH includes the interface directory containing the interface libraries.

Parameters:

env (dict) – Dictionary of enviroment variables to be updated.
paths_to_add (list, optional) – Paths that should be added. If not provided, defaults to [cls.get_language_dir()].
add_to_front (bool, optional) – If True, new paths are added to the front, rather than the end. Defaults to False.
add_libpython_dir (bool, optional) – If True, the directory containing the Python C library will be added. Defaults to False.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
env_var (str, optional) – Environment variable where the paths should be added. Defaults to None and is only set for linux (LD_LIBRARY_PATH) and windows (PATH).
**kwargs – Additional keyword arguments are ignored.

Returns:

Updated dictionary of environment variables.

Return type:

classmethod update_python_path(env)[source]¶

Update provided dictionary of environment variables so that PYTHONPATH and PYTHONHOME are set as needed (primarily on windows).

Parameters:: env (dict) – Dictionary of enviroment variables to be updated.
Returns:: Updated dictionary of environment variables.
Return type:: dict

classmethod write_assign_to_output(dst_var, src_var, outputs_in_inputs=False, dont_add_lengths=False, use_length_prefix=False, **kwargs)[source]¶

Write lines assigning a value to an output variable.

Parameters:

dst_var (str, dict) – Name or information dictionary for variable being assigned to.
src_var (str, dict) – Name or information dictionary for value being assigned to dst_var.
outputs_in_inputs (bool, optional) – If True, outputs are passed as input parameters. In some languages, this means that a pointer or reference is passed (e.g. C) and so the assignment should be to the memory indicated rather than the variable. Defaults to False.
dont_add_lengths (bool, optional) – If True, length variables are not added for arrays. Defaults to False.
use_length_prefix (bool, optional) – If True and length variables are added, they will be named using prefixes. Otherwise, suffixes will be used. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines achieving assignment.

Return type:

classmethod write_declaration(var, **kwargs)[source]¶

Return the lines required to declare a variable with a certain type.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
**kwargs – Addition keyword arguments are passed to the parent class’s method.

Returns:

The lines declaring the variable.

Return type:

classmethod write_free(var, **kwargs)[source]¶

Return the lines required to free a variable with a certain type.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines freeing the variable.

Return type:

classmethod write_function_def(function_name, dont_add_lengths=False, use_length_prefix=False, **kwargs)[source]¶

Write a function definition.

Parameters:

function_name (str) – Name fo the function being defined.
dont_add_lengths (bool, optional) – If True, length variables are not added for arrays. Defaults to False.
use_length_prefix (bool, optional) – If True and length variables are added, they will be named using prefixes. Otherwise, suffixes will be used. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines completing the function call.

Return type:

Raises:

ValueError – If outputs_in_inputs is not True and more than one output variable is specified.

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

classmethod write_model_function_call(model_function, flag_var, inputs, outputs, **kwargs)[source]¶

Write lines necessary to call the model function.

Parameters:

model_function (str) – Handle of the model function that should be called.
flag_var (str) – Name of variable that should be used as a flag.
inputs (list) – List of dictionaries describing inputs to the model.
outputs (list) – List of dictionaries describing outputs from the model.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to carry out a call to a model function in: this language.

Return type:

classmethod write_model_recv(channel, recv_var, **kwargs)[source]¶

Write a model receive call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being received from was stored in.
recv_var (dict, list) – Information of one or more variables that receieved information should be stored in.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to carry out a receive call in this language.

Return type:

classmethod write_print_output_var(var, in_inputs=False, **kwargs)[source]¶

Get the lines necessary to print an output variable in this language.

Parameters:

var (dict) – Variable information.
in_inputs (bool, optional) – If True, the output variable is passed in as an input variable to be populated. Defaults to False.
**kwargs – Additional keyword arguments are passed to write_print_var.

Returns:

Lines printing the specified variable.

Return type:

class yggdrasil.drivers.CModelDriver.ClangCompiler(**kwargs)[source]¶

Bases: CCompilerBase

Interface class for clang compiler/linker.

asan_flags = ['-fsanitize=address']¶

combine_with_linker = True¶

compatible_toolsets = ['llvm']¶

default_archiver = 'libtool'¶

default_executable = 'clang'¶

default_flags = ['-g', '-Wall']¶

default_linker = 'clang'¶

flag_options = {'definitions': '-D%s', 'include_dirs': '-I%s', 'isysroot': {'key': '-isysroot', 'prepend': True}, 'mmacosx-version-min': '-mmacosx-version-min=%s', 'sysroot': '--sysroot'}¶

classmethod get_flags(*args, **kwargs)[source]¶: Get a list of compiler flags.

is_gnu = False¶

is_linker = False¶

object_tool = 'otool -L'¶

platforms = ['MacOS', 'Linux', 'Windows']¶

preload_envvar = 'DYLD_INSERT_LIBRARIES'¶

toolname = 'clang'¶

toolset = 'llvm'¶

class yggdrasil.drivers.CModelDriver.ClangLinker(**kwargs)[source]¶

Bases: LDLinker

Interface class for clang linker (calls to ld).

aliases = []¶

asan_flags = ['-fsanitize=address', '-shared-libasan']¶

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

compatible_toolsets = ['llvm']¶

default_executable = 'clang'¶

default_flags = []¶

flag_options = {'library_dirs': '-L%s', 'library_libs': {'allow_duplicate_values': True, 'key': '-l%s'}, 'library_libs_nonstd': '', 'library_rpath': '-rpath', 'linker-version': '-mlinker-version=%s'}¶

classmethod get_flags(*args, **kwargs)[source]¶: Get a list of linker flags.

is_gnu = False¶

languages = ['c']¶

library_ext = '.so'¶

platforms = ['MacOS', 'Linux', 'Windows']¶

search_path_flags = ['-Xlinker', '-v']¶

search_regex = ['\\t([^\\t\\n]+)\\n']¶

search_regex_begin = 'Library search paths:'¶

classmethod tool_version(**kwargs)[source]¶

Determine the version of this tool.

Parameters:: **kwargs – Keyword arguments are passed to cls.call.
Returns:: Version of the tool.
Return type:: str

toolname = 'clang'¶

toolset = 'llvm'¶

class yggdrasil.drivers.CModelDriver.GCCCompiler(**kwargs)[source]¶

Bases: CCompilerBase

Interface class for gcc compiler/linker.

aliases = ['gnu-cc', 'gnu-gcc']¶

asan_flags = ['-fsanitize=address']¶

combine_with_linker = True¶

compatible_toolsets = ['gnu']¶

default_archiver = 'ar'¶

default_executable = 'gcc'¶

default_flags = ['-g', '-Wall']¶

default_linker = 'gcc'¶

dll2a(dll, dst=None, overwrite=False)[source]¶

Convert a window’s .dll library into a static library.

Parameters:

dll (str) – Full path to .dll library to convert.
dst (str, optional) – Full path to location where the new library should be saved. Defaults to None and will be set based on lib or will be placed in the same directory as dll.
overwrite (bool, optional) – If True, the static file will be created even if it already exists. Defaults to False.

Returns:

Full path to new .a static library.

Return type:

classmethod get_flags(*args, **kwargs)[source]¶: Get a list of compiler flags.

is_gnu = True¶

classmethod is_installed()[source]¶

Determine if this tool is installed by looking for the executable.

Returns:: True if the tool is installed, False otherwise.
Return type:: bool

is_linker = False¶

object_tool = 'ldd'¶

platforms = ['MacOS', 'Linux', 'Windows']¶

preload_envvar = 'LD_PRELOAD'¶

toolname = 'gcc'¶

toolset = 'gnu'¶

class yggdrasil.drivers.CModelDriver.GCCLinker(**kwargs)[source]¶

Bases: LDLinker

Interface class for gcc linker (calls to ld).

aliases = ['gnu-cc', 'gnu-gcc']¶

compatible_toolsets = ['gnu']¶

default_executable = 'gcc'¶

default_flags = []¶

flag_options = {'library_dirs': '-L%s', 'library_libs': {'allow_duplicate_values': True, 'key': '-l%s'}, 'library_libs_nonstd': {'allow_duplicate_values': True, 'key': '-l:%s'}, 'library_rpath': '-Wl,-rpath'}¶

is_gnu = True¶

languages = ['c']¶

library_ext = '.so'¶

platforms = ['MacOS', 'Linux', 'Windows']¶

search_path_flags = ['-Xlinker', '--verbose']¶

search_regex = ['SEARCH_DIR\$"=([^"]+)"\$;']¶

toolname = 'gcc'¶

toolset = 'gnu'¶

class yggdrasil.drivers.CModelDriver.LDLinker(**kwargs)[source]¶

Bases: LinkerBase

Linker class for ld tool.

asan_flags = ['-fsanitize=address']¶

default_executable = 'ld'¶

default_executable_env = 'LD'¶

default_flags = []¶

default_flags_env = 'LDFLAGS'¶

classmethod get_flags(*args, **kwargs)[source]¶: Get a list of linker flags.

is_gnu = False¶

languages = ['c']¶

library_ext = '.so'¶

search_path_envvar = ['LIBRARY_PATH', 'LD_LIBRARY_PATH']¶

classmethod tool_version(**kwargs)[source]¶

Determine the version of this tool.

Parameters:: **kwargs – Keyword arguments are passed to cls.call.
Returns:: Version of the tool.
Return type:: str

toolname = 'ld'¶

version_flags = ['-v']¶

class yggdrasil.drivers.CModelDriver.LibtoolArchiver(**kwargs)[source]¶

Bases: ArchiverBase

Archiver class for libtool tool.

asan_flags = []¶

compatible_toolsets = ['clang']¶

default_executable = 'libtool'¶

default_executable_env = 'LIBTOOL'¶

default_flags = []¶

is_gnu = False¶

languages = ['c', 'c++']¶

library_ext = '.a'¶

search_path_envvar = ['LIBRARY_PATH']¶

shared_library_flag = None¶

static_library_flag = '-static'¶

toolname = 'libtool'¶

toolset = 'clang'¶

class yggdrasil.drivers.CModelDriver.MSVCArchiver(**kwargs)[source]¶

Bases: ArchiverBase

Microsoft Visual Studio C Archiver.

compatible_toolsets = ['msvc', 'llvm']¶

default_executable = 'LIB'¶

default_flags = []¶

is_gnu = False¶

languages = ['c', 'c++']¶

library_ext = '.a'¶

output_key = '/OUT:%s'¶

platforms = ['Windows']¶

search_path_envvar = ['LIB']¶

shared_library_flag = None¶

static_library_flag = None¶

toolname = 'LIB'¶

toolset = 'msvc'¶

class yggdrasil.drivers.CModelDriver.MSVCCompiler(**kwargs)[source]¶

Bases: CCompilerBase

Microsoft Visual Studio C Compiler.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

combine_with_linker = True¶

compatible_toolsets = ['msvc']¶

default_archiver = 'LIB'¶

default_executable = 'cl'¶

default_flags = ['/W4', '/Zi', '/FS', '/EHsc', '/bigobj', '/TP', '/nologo', '-D_CRT_SECURE_NO_WARNINGS']¶

default_linker = 'LINK'¶

is_gnu = False¶

linker_attributes = {'default_executable': None, 'default_executable_env': 'LINK', 'default_flags_env': 'LDFLAGS', 'flag_options': {'import_lib': '/IMPLIB:%s', 'library_dirs': '/LIBPATH:%s', 'library_libs': '', 'library_libs_nonstd': ''}, 'output_first': True, 'output_first_library': False, 'output_key': '/OUT:%s', 'search_path_envvar': ['LIB'], 'search_path_flags': None, 'shared_library_flag': '/DLL'}¶

linker_switch = '/link'¶

output_first = True¶

output_key = '/Fo%s'¶

platforms = ['Windows']¶

product_exts = ['.dir', '.ilk', '.pdb', '.sln', '.vcxproj', '.vcxproj.filters', '.exp', '.lib']¶

search_path_envvar = ['INCLUDE']¶

search_path_flags = None¶

classmethod tool_version(**kwargs)[source]¶

Determine the version of this tool.

Parameters:: **kwargs – Keyword arguments are passed to cls.call.
Returns:: Version of the tool.
Return type:: str

toolname = 'cl'¶

toolset = 'msvc'¶

version_flags = []¶

yggdrasil.drivers.CModelDriver.get_OSX_SYSROOT()[source]¶

Determin the path to the OSX SDK.

Returns:

Full path to the SDK directory if one is located. None: otherwise.

Return type:

yggdrasil.drivers.CPPModelDriver module¶

class yggdrasil.drivers.CPPModelDriver.CPPCompilerBase(**kwargs)[source]¶

Bases: CCompilerBase

Base class for C++ compilers.

cpp_std = 'c++14'¶

default_executable = None¶

default_executable_env = 'CXX'¶

default_flags_env = 'CXXFLAGS'¶

default_linker = None¶

classmethod find_standard_flag(flags)[source]¶

Locate the standard flag in a list of flags.

Parameters:: flags (list) – Compilation flags.
Returns:: Index of the standard flag. -1 if not present.
Return type:: int

classmethod handle_standard_flag(flags, skip_standard_flag=False)[source]¶

Add or remove standard flag from a list of flags.

Parameters:

flags (list) – Compilation flags.
skip_standard_flag (bool, optional) – If True, the C++ standard flag will not be added. Defaults to False.

languages = ['c++']¶

search_path_flags = ['-E', '-v', '-xc++', '/dev/null']¶

class yggdrasil.drivers.CPPModelDriver.CPPModelDriver(*args, **kwargs)[source]¶

Bases: CModelDriver

Class for running C++ models.

classmethod add_extra_vars(direction, x)[source]¶

Add extra variables required for communication.

Parameters:

direction (str) – Direction of channel (‘input’ or ‘output’).
x (dict) – Dictionary describing the variable.

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration.

classmethod allows_realloc(var)[source]¶

Determine if a variable allows the receive call to perform realloc.

Parameters:: var (dict) – Dictionary of variable properties.
Returns:: True if the variable allows realloc, False otherwise.
Return type:: bool

base_languages = ['c']¶

cfg = <yggdrasil.config.YggConfigParser object>¶

default_archiver = 'ar'¶

default_archiver_flags = None¶

default_compiler = 'g++'¶

default_compiler_flags = None¶

default_linker = 'g++'¶

default_linker_flags = None¶

dont_declare_channel = True¶

function_param = {'and': '&&', 'assign': '{name} = {value};', 'assign_copy': 'memcpy({name}, {value}, {N}*sizeof({native_type}));', 'block_end': '}', 'break': 'break;', 'comment': '//', 'copy_any': '{name}.CopyFrom({value}, {name}.GetAllocator(), true);', 'copy_array': '{name}.CopyFrom({value}, {name}.GetAllocator(), true);', 'copy_class': '{name} = copy_python({value});', 'copy_function': '{name} = copy_python({value});', 'copy_instance': 'copy_generic_into(&{name}, {value});', 'copy_object': '{name}.CopyFrom({value}, {name}.GetAllocator(), true);', 'copy_schema': '{name}.CopyFrom({value}, {name}.GetAllocator(), true);', 'declare': '{type_name} {variable};', 'error': 'throw "{error_msg}";', 'exec_begin': 'int main() {', 'exec_end': ' return 0;\n}', 'exec_prefix': '#include <iostream>\n#include <exception>\n', 'false': 'false', 'false_flag': '0', 'flag_cond': '{flag_var} >= 0', 'for_begin': 'for ({iter_var} = {iter_begin}; {iter_var} < {iter_end}; {iter_var}++) {{', 'fprintf': 'printf("{message}\\n", {variables});', 'free': 'if ({variable} != NULL) {{ free({variable}); {variable} = NULL; }}', 'free_any': 'free_generic({variable});', 'free_array': 'free_json_array({variable});', 'free_class': 'destroy_python({variable});', 'free_function': 'destroy_python({variable});', 'free_instance': 'free_generic({variable});', 'free_obj': 'free_obj({variable});', 'free_object': 'free_json_object({variable});', 'free_ply': 'free_ply({variable});', 'free_schema': 'free_schema({variable});', 'function_def_begin': '{output_type} {function_name}({input_var}) {{', 'function_def_regex': '(?P<flag_type>.+?)\\s*{function_name}\\s*\$\\s*(?P<inputs>(?:(?:const\\s+[^{{\\&]+\\s+\\&[^{{\\&]+)|(?:[^{{\\&]+))(?:\\s*,\\s*(?:const\\s+[^{{\\&]+\\s+\\&[^{{\\&]+)|(?:[^{{\\&]+))*?)(?:,\\s*(?P<outputs>(?:\\s*(?:[^\\s\\&]+)(?:(?:\\&\\s+)|(?:\\s+(?:\\()?\\&))(?:[^{{])+)+))?\$\\s*\\{{(?P<body>(?:.*?\\n?)*?)(?:(?:return +(?P<flag_var>.+?)?;(?:.*?\\n?)*?\\}})|(?:\\}}))', 'if_begin': 'if ({cond}) {{', 'if_elif': '}} else if ({cond}) {{', 'if_else': '}} else {{', 'import': '#include "{filename}"', 'indent': ' ', 'index': '{variable}[{index}]', 'init_class': 'init_python()', 'init_function': 'init_python()', 'init_instance': 'init_generic()', 'init_type_1darray': 'create_dtype_1darray("{subtype}", {precision}, {length}, "{units}", {use_generic})', 'init_type_array': 'create_dtype_json_array({nitems}, {items}, {use_generic})', 'init_type_default': 'create_dtype_default("{type}", {use_generic})', 'init_type_empty': 'create_dtype_empty({use_generic})', 'init_type_from_schema': 'create_dtype_from_schema("{schema}", {use_generic})', 'init_type_ndarray': 'create_dtype_ndarray("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_ndarray_arr': 'create_dtype_ndarray_arr("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_obj': 'create_dtype_obj({use_generic})', 'init_type_object': 'create_dtype_json_object({nitems}, {keys}, {values}, {use_generic})', 'init_type_ply': 'create_dtype_ply({use_generic})', 'init_type_pyobj': 'create_dtype_pyobj("{type}", {use_generic})', 'init_type_scalar': 'create_dtype_scalar("{subtype}", {precision}, "{units}", {use_generic})', 'init_type_schema': 'create_dtype_schema({use_generic})', 'input': 'YggInput {channel}("{channel_name}", {channel_type});', 'inputs_def_regex': '\\s*(?:const\\s+)?(?P<native_type>(?:[^\\s\\&\\<\\*])+(?:\\<(?P<subtypes>\\s*.+?(?:\\s*,\\s*.+?)*\\s*)\\>)?(\\s+)?(?P<ptr>\\*+)?)(?:\\s*\\&)?(?(ptr)(?(3)(?:\\s*)|(?:\\s+)))(\$(?P<name_ptr>\\*+)?)?(?P<name>[^\\&\\>\\*]+?)(?(5)(?:\$))(?P<shape>(?:\\[.+?\\])+)?\\s*(?:,|$)(?:\\n)?', 'interface': '#include "{interface_library}"', 'line_end': ';', 'not': '!', 'not_flag_cond': '{flag_var} < 0', 'null': 'NULL', 'output': 'YggOutput {channel}("{channel_name}", {channel_type});', 'outputs_def_regex': '\\s*(?P<native_type>(?:[^\\s])+)(\\s+)?(\$)?(?P<ref>\\&)(?(1)(?:\\s*)|(?:\\s+))(?P<name>.+?)(?(2)(?:\$|(?:)))(?P<shape>(?:\\[.+?\\])+)?\\s*(?:,|$)(?:\\n)?', 'print': 'std::cout << "{message}" << std::endl;', 'print_any': 'std::cout << document2string({object}) << std::endl;', 'print_array': 'std::cout << document2string({object}) << std::endl;', 'print_bytes': 'printf("%s\\n", {object});', 'print_class': 'display_python({object});', 'print_complex': 'print_complex({object});', 'print_float': 'printf("%f\\n", {object});', 'print_function': 'display_python({object});', 'print_instance': 'display_generic({object});', 'print_int': 'printf("%i\\n", {object});', 'print_object': 'std::cout << document2string({object}) << std::endl;', 'print_schema': 'std::cout << document2string({object}) << std::endl;', 'print_string': 'printf("%s\\n", {object});', 'print_uint': 'printf("%u\\n", {object});', 'print_unicode': 'printf("%s\\n", {object});', 'python_init': 'init_python_API();', 'quote': '"', 'recv_function': '{channel}.recvRealloc', 'recv_heap': '{channel}.recvRealloc', 'recv_stack': '{channel}.recv', 'return': 'return {output_var};', 'send_function': '{channel}.send', 'true': 'true', 'true_flag': '1', 'try_begin': 'try {', 'try_error_type': 'const std::exception&', 'try_except': '}} catch ({error_type} {error_var}) {{', 'vector_regex': '(?:std\\:\\:)?vector\\<\\s*(?P<type>.*)\\s*\\>', 'while_begin': 'while ({cond}) {{'}¶

classmethod get_json_type(native_type)[source]¶

Get the JSON type from the native language type.

Parameters:: native_type (str) – The native language type.
Returns:: The JSON type.
Return type:: str, dict

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

include_arg_count = True¶

include_channel_obj = False¶

interface_library = 'ygg++'¶

interface_map = {'call': 'flag = {channel_obj}.call({nargs}, {outputs}, {input_refs})', 'client': 'YggRpcClient {channel_obj}("{channel_name}", {datatype_out}, {datatype_in})', 'import': '#include "YggInterface.hpp"', 'input': 'YggInput {channel_obj}("{channel_name}")', 'output': 'YggOutput {channel_obj}("{channel_name}")', 'recv': 'flag = {channel_obj}.recv({nargs}, {input_refs})', 'send': 'flag = {channel_obj}.send({nargs}, {outputs})', 'server': 'YggRpcServer {channel_obj}("{channel_name}", {datatype_in}, {datatype_out})', 'timesync': 'YggTimesync {channel_obj}("{channel_name}", "{time_units}")'}¶

internal_libraries = {'datatypes': {'compiler_flags': ['-fPIC'], 'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/datatypes', 'external_dependencies': ['rapidjson', 'python', 'numpy', 'm'], 'include_dirs': [], 'internal_dependencies': ['regex'], 'language': 'c++', 'libtype': 'object'}, 'regex': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c', 'libtype': 'header_only', 'source': 'regex_posix.h'}, 'regex_posix': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c', 'libtype': 'header_only', 'source': 'regex_posix.h'}, 'regex_win32': {'directory': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/regex', 'external_dependencies': [], 'internal_dependencies': [], 'language': 'c++', 'libtype': 'object', 'source': 'regex_win32.cpp'}, 'ygg++': {'compiler_flags': ['-fPIC'], 'external_dependencies': ['rapidjson', 'python', 'numpy', 'm'], 'include_dirs': ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/communication', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/serialize', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/CPP'], 'internal_dependencies': ['regex', 'datatypes'], 'language': 'c++', 'linker_language': 'c++', 'source': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/CPP/YggInterface.cpp'}}¶

inverse_type_map = None¶

classmethod is_cpp_class(var)[source]¶

Determine if a variable uses a C++ class.

Parameters:: var (dict) – Variable.
Returns:: True if it is a C++ class, False otherwise.
Return type:: bool

classmethod is_std_class(var)[source]¶

Determine if a variable utilizing a C++ stdlib class.

Parameters:: var (dict) – Variable.
Returns:: True if var is a C++ stdlib class, False otherwise.
Return type:: bool

language = 'c++'¶

language_aliases = ['cpp', 'cxx']¶

language_ext = ['.cpp', '.CPP', '.cxx', '.C', '.c++', '.cc', '.cp', '.tcc', '.hpp', '.HPP', '.hxx', '.H', '.h++', '.hh', '.hp', '.h']¶

classmethod output2input(var, in_definition=True)[source]¶

Perform conversion necessary to turn an output variable into an corresponding input that can be used to format a function definition.

Parameters:

var (dict) – Variable definition.
in_definition (bool, optional) – If True, the returned dictionary corresponds to an input variable in a function definition. If False, the returned value will correspond to an input to a function. Defaults to True.

Returns:

Updated variable definition.

Return type:

classmethod prepare_input_variables(vars_list, in_definition=False, for_yggdrasil=False)[source]¶

Concatenate a set of input variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing input to a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying input variables in a function definition. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod requires_length_var(var)[source]¶

Determine if a variable requires a separate length variable.

Parameters:: var (dict) – Dictionary of variable properties.
Returns:: True if a length variable is required, False otherwise.
Return type:: bool

classmethod set_env_class(**kwargs)[source]¶

Set environment variables that are instance independent.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method and update_ld_library_path.
Returns:: Environment variables for the model process.
Return type:: dict

type_map = {'1darray': '*', 'any': 'rapidjson::Document', 'array': 'rapidjson::Document', 'boolean': 'bool', 'bytes': 'string_t', 'class': 'python_class_t', 'comm': 'comm_t*', 'complex': 'complex_X', 'dtype': 'dtype_t*', 'flag': 'int', 'float': 'double', 'function': 'python_function_t', 'instance': 'python_instance_t', 'int': 'intX_t', 'integer': 'int', 'length': 'size_t', 'ndarray': '*', 'null': 'void*', 'number': 'double', 'obj': 'rapidjson::ObjWavefront', 'object': 'rapidjson::Document', 'ply': 'rapidjson::Ply', 'schema': 'rapidjson::Document', 'string': 'char*', 'uint': 'uintX_t', 'unicode': 'unicode_t'}¶

classmethod write_doc2vars(channel, std, var_list)[source]¶

Generate the lines of code required to unpack a document into a list of variables.

Parameters:

channel (str) – Name of variable that the channel that the document was received from is stored in.
std (dict) – Variable information for the received document.
var_list (list) – Variables that the document should be unpacked into.

classmethod write_model_recv(channel, recv_var, **kwargs)[source]¶

Write a model receive call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being received from was stored in.
recv_var (dict, list) – Information of one or more variables that receieved information should be stored in.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to carry out a receive call in this language.

Return type:

classmethod write_model_send(channel, send_var, **kwargs)[source]¶

Write a model send call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being sent to was stored in.
send_var (dict, list) – Information on one or more variables containing information that will be sent.
flag_var (str, optional) – Name of flag variable that the flag should be stored in. Defaults to ‘flag’,
allow_failure (bool, optional) – If True, the returned lines will call a break if the flag is False. Otherwise, the returned lines will issue an error. Defaults to False.

Returns:

Lines required to carry out a send call in this language.

Return type:

classmethod write_try_except(try_contents, except_contents, error_var='e', error_type=None, **kwargs)[source]¶

Return the lines required to complete a try/except block.

Parameters:

try_contents (list) – Lines of code that should be executed inside the try block.
except_contents (list) – Lines of code that should be executed inside the except block.
error_var (str, optional) – Name of variable where the caught error should be stored. Defaults to ‘e’. If ‘…’, the catch clause will catch all errors, but there will not be a name error.
error_type (str, optional) – Name of error type that should be caught. If not provided, defaults to None and will be set based on the class function_param entry for ‘try_error_type’. If ‘…’, the catch clause will catch all errors and error_var will be ignored.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines of code perfoming a try/except block.

classmethod write_vars2doc(channel, var_list, std)[source]¶

Generate the lines of code required to pack a list of variables into a document.

Parameters:

channel (str) – Name of variable that the channel that will be used to send the document is stored in.
var_list (list) – Variables that should be packed into the document.
std (dict) – Variable information for the document that will be generated.

class yggdrasil.drivers.CPPModelDriver.ClangPPCompiler(**kwargs)[source]¶

Bases: CPPCompilerBase, ClangCompiler

Interface class for clang++ compiler.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

default_executable = 'clang++'¶

default_flags = ['-g', '-Wall']¶

default_linker = 'clang++'¶

classmethod get_executable_command(args, skip_flags=False, unused_kwargs=None, **kwargs)[source]¶

Determine the command required to run the tool using the specified arguments and options.

Parameters:

args (list) – The arguments that should be passed to the tool. If skip_flags is False, these are treated as input files that will be used by the tool.
**kwargs – Additional keyword arguments will be passed to the parent class’s method.

Returns:

Output to stdout from the command execution.

Return type:

classmethod get_flags(skip_standard_flag=False, **kwargs)[source]¶

Get a list of compiler flags.

Parameters:

skip_standard_flag (bool, optional) – If True, the C++ standard flag will not be added. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Compiler flags.

Return type:

is_gnu = False¶

is_linker = False¶

toolname = 'clang++'¶

class yggdrasil.drivers.CPPModelDriver.ClangPPLinker(**kwargs)[source]¶

Bases: ClangLinker

Interface class for clang++ linker (calls to ld).

aliases = []¶

default_executable = 'clang++'¶

default_flags = []¶

is_gnu = False¶

languages = ['c++']¶

library_ext = '.so'¶

toolname = 'clang++'¶

toolset = 'llvm'¶

class yggdrasil.drivers.CPPModelDriver.GPPCompiler(**kwargs)[source]¶

Bases: CPPCompilerBase, GCCCompiler

Interface class for G++ compiler/linker.

aliases = ['gnu-c++']¶

default_executable = 'g++'¶

default_flags = ['-g', '-Wall']¶

default_linker = 'g++'¶

classmethod get_flags(skip_standard_flag=False, **kwargs)[source]¶

Get a list of compiler flags.

Parameters:

skip_standard_flag (bool, optional) – If True, the C++ standard flag will not be added. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Compiler flags.

Return type:

is_gnu = True¶

is_linker = False¶

toolname = 'g++'¶

class yggdrasil.drivers.CPPModelDriver.GPPLinker(**kwargs)[source]¶

Bases: GCCLinker

Interface class for clang++ linker (calls to ld).

aliases = ['gnu-c++']¶

default_executable = 'g++'¶

default_flags = []¶

is_gnu = True¶

languages = ['c++']¶

library_ext = '.so'¶

toolname = 'g++'¶

toolset = 'gnu'¶

class yggdrasil.drivers.CPPModelDriver.MSVCPPCompiler(**kwargs)[source]¶

Bases: CPPCompilerBase, MSVCCompiler

Inteface class for MSVC compiler when compiling C++.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

default_archiver = 'LIB'¶

default_executable = 'cl'¶

default_flags = ['/W4', '/Zi', '/FS', '/EHsc', '/bigobj', '/TP', '/nologo', '-D_CRT_SECURE_NO_WARNINGS']¶

default_linker = 'LINK'¶

dont_create_linker = True¶

is_gnu = False¶

search_path_flags = None¶

toolname = 'cl++'¶

yggdrasil.drivers.CompiledModelDriver module¶

class yggdrasil.drivers.CompiledModelDriver.ArchiverBase(**kwargs)[source]¶

Bases: LinkerBase

Base class for archivers.

static_library_flag¶

Flag that should be prepended to the archiver tool arguments to indicated that a static library should be produced.

Type:: str

library_name_key¶

Option key indicating the name of a library that should be linked against.

Type:: str

library_directory_key¶

Option key indicating a directory that should be included in the linker search path for libraries.

Type:: str

library_prefix¶

Prefix that should be added to library paths.

Type:: str

library_ext¶

Extension that should be used for static libraries.

Type:: str

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

flag_options = {}¶

classmethod get_flags(**kwargs)[source]¶

Get a list of flags for this archiver tool.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method.
Returns:: Archiver flags.
Return type:: list

classmethod get_output_file(obj, **kwargs)[source]¶

Determine the appropriate output file that will result when linking a given object file.

Parameters:

obj (str) – Object file being linked that name base will be taken from.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Full path to file that will be produced.

Return type:

library_ext = None¶

static_library_flag = '-static'¶

tooltype = 'archiver'¶

class yggdrasil.drivers.CompiledModelDriver.BuildToolBase(**kwargs)[source]¶

Bases: CompilerBase

Base class for build tools which are used to coordinate compilation.

Parameters:

buildfile (str, optional) – File containing information about the build. Defaults to default_buildfile class attribute, if set.
builddir (str, optional) – Directory where build files should be placed. Defaults to directory where build is called.
sourcedir (str, optional) – Directory where source files are stored. Defaults to directory where build is called.
target (str, optional) – Build target. If not provided, build will be created without a target.

Class Attributes:

default_buildfile = None¶

flag_options = {}¶

tooltype = 'buildtool'¶

class yggdrasil.drivers.CompiledModelDriver.CompilationToolBase(**kwargs)[source]¶

Bases: object

Base class for compilation command line tools.

Class Attributes:

toolname (str): Tool name used for registration and as a default for the: executable. [REQUIRED]

aliases (list): Alternative names that the tool might have. tooltype (str): Tool type. One of ‘compiler’, ‘linker’, or ‘archiver’.

[AUTOMATED]

languages (list): Programming languages that this tool can be used on.: [REQUIRED]
platforms (list): Platforms that the tool is available on. Defaults to: [‘Windows’, ‘MacOS’, ‘Linux’].
default_executable (str): The default tool executable command if: different than the toolname.
default_executable_env (str): Environment variable where the executable: command might be stored.
default_flags (list): Default flags that should be used when calling the: tool (e.g. for verbose output or enhanced warnings).
default_flags_env (str): Environment variable where default flags for: the tools might be stored.

output_key (str): Option key indicating the output file name. output_first (bool): If True, the output key (and its value) are put in

front of the other flags.

flag_options (collections.OrderedDict): Mappings between keyword: arguments passed to get_flags and command line tool flags. Keys indicate the keyword argument that will be processed in Python and values are the string used with the tool on the command line to indicate the desired flag. Flags that contain ‘%s’ will be formatted with values passed to get_flags as the designated keyword argument. Flags that do not contain ‘%s’ will assumed to act on the following argument. If the value passed to get_flags under the designated keyword argument is not a boolean, it will be appended to the flag list following the corresponding flag. If the value is a boolean and it is True, only the flag will be added to the list of flags. The order of entries indicates the order the flags should be added to the list.
search_path_envvar (list): Environment variables containing a list of paths: to search for library files. Either search_path_envvar or search_path_flags must be set. [REQUIRED]
search_path_env (list): Paths relative to the env prefix that should: be searched if the VIRTUAL_ENV or CONDA_PREFIX environment variable is set.
search_path_flags (list): Flags that should be passed to the tool: executable in order to locate the search path. Either search_path_envvar or search_path_flags must be set. [REQUIRED]
search_regex_begin (str): Search string indicating where the set of: paths begins in the output from running the tool executable with the search_path_flags. If None, the search is performed from the very beginning of the output.
search_regex_end (str): Search string indicating where the set of: paths ends in the output from running the tool executable with the search_path_flags. If None, the search is performed to the very end of the output.
search_regex (list): Regex strings that should be used to locate paths: in the output from running the tool executable with the search_path_flags.
product_exts (list): List of extensions that will be added to the: output file’s base to get a list of products that might be produced by calling the compilation tool.
product_files (list): List of file basenames that will be joined with: the directory containing each output to get a list of products that might be produced by calling the compilation tool.
remove_product_exts (list): List of extensions or directories matching: entries in product_exts and product_files that should be removed during cleanup. Be careful when adding files to this list.

asan_flags (list): Flags added when with_asan is specified.

aliases = []¶

classmethod append_flags(out, key, value, **kwargs)[source]¶

Append one ore more flag(s) to a list of flags based on a key value/set.

Parameters:

out (list) – Existing list that the new flag should be appended to.
key (str) – String identifying the type of flag. See create_flag for additional details.
value (object) – Value that should be set in the flag. If a list, multiple flags are added, one for each item in the list.
prepend (bool, optional) – If True, new flags are prepended to the front of the list rather than the end. Defaults to False. This keyword argument is ignoerd if position is provided.
position (int, optional) – If not None, this is the position that the new elements should be added to the existing flags. Positive positions indicate indexes into the existing list of flags. Negative are relative to the end of the existing list such that -1 is the same as just appending the new flags to the end of the list. Defaults to None and prepend takes precedence.
no_duplicates (bool, optional) – If True, the new flags being added will be checked against the existing ones to ensure that there are not duplicates. If False, the new flags are added reguardless of the existing flags. Defaults to False.
allow_duplicate_values (bool, optional) – If True, the same key can be added with the same value more than once. Otherwise, only the first instance of the value is added. Defaults to False.

Raises:

ValueError – If there are unexpected keyword arguments.
ValueError – If no_duplicates is True and the existing list of flags already contains a flag matching the provided flag key.

classmethod append_product(products, src, new, new_dir=None, dont_append_src=False)[source]¶

Append a product to the specified list along with additional values indicated by cls.product_exts.

Parameters:

products (list) – List of of existing products that new product should be appended to.
src (list) – Input arguments to compilation call that was used to generate the output file (usually one or more source files).
new (str) – New product that should be appended to the list.
new_dir (str, optional) – Directory that should be used as base when adding files listed in cls.product_files. Defaults to os.path.dirname(new).
dont_append_src (bool, optional) – If True and src is in the list of products, it will be removed. Defaults to False.

asan_flags = None¶

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

classmethod call(args, language=None, toolname=None, skip_flags=False, dry_run=False, out=None, overwrite=False, products=None, allow_error=False, working_dir=None, additional_args=None, suffix='', cache_key=None, **kwargs)[source]¶

Call the tool with the provided arguments. If the first argument resembles the name of the tool executable, the executable will not be added.

Parameters:

args (list) – The arguments that should be passed to the tool.
language (str, optional) – Language of tool that should be used. If different than the languages supported by the current tool, the correct tool is used instead. Defaults to None and is ignored.
toolname (str, optional) – Name of compilation tool that should be used. Defaults to None and the default tool for the language will be used.
skip_flags (bool, optional) – If True, args is assumed to include any necessary flags. If False, args are assumed to the files that the tool is called on and flags are determined from them. Defaults to False.
dry_run (bool, optional) – If True, the tool won’t be called, but the products will be updated. Defautls to False.
out (str, optional) – Full path to output file that should be created. If None, the path will be determined from the path to the first argument provided. Defaults to None. This keyword argument will be ignored if skip_flags is True.
overwrite (bool, optional) – If True, the existing compile file will be overwritten. Otherwise, it will be kept and this function will return without recompiling the source file.
products (list, optional) – Existing Python list that additional products produced by the compilation should be appended to. Defaults to None and is ignored.
allow_error (bool, optional) – If True and there is an error when call the executable, it will be ignored. If False, errors will result in an exception being raised. Defaults to False.
working_dir (str, optional) – Working directory where tool should be called from. This will also be used to construct the path for the output file. Defaults to None and is ignored.
additional_args (list, optional) – Additional arguments that should be appended to args before continuing. Defaults to None and is ignored.
suffix (str, optional) – Suffix that should be added to the output file (before the extension). Defaults to “”.
cache_key (str, optional) – Key that should be used to cache results so that they may be used multiple times. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to cls.get_executable_command. and tools.popen_nobuffer.

Returns:

Output to stdout from the command execution if skip_flags is: True, produced file otherwise.

Return type:

Raises:

RuntimeError – If there is an error when running the command and allow_error is False.

compatible_toolsets = []¶

classmethod create_flag(key, value)[source]¶

Create a flag from a key/value set.

Parameters:

key (str) – String identifying the type of flag. If key contains ‘%s’, it is assumed that the flag will be produced by formatting the value according to key. If key dosn’t contain ‘%s’, it is assumed that key indicates that value is the following item and they both will be returned. If key is an empty string, then the value is assumed to constitute the entire flag.
value (object) – Value that should be set in the flag. If a list, multiple flags are returned, one for each item in the list.

Returns:

Items representing the flag.

Return type:

default_executable = None¶

default_executable_env = None¶

default_flags = []¶

default_flags_env = None¶

classmethod env_matches_tool(use_sysconfig=False, env=None, with_flags=False)[source]¶

Determine if the executable pointed to by any environment variable matches this compilation tool.

Parameters:

use_sysconfig (bool, optional) – If True, check the sysconfig variables, otherwise check os.environ. Defaults to False.
env (dict, optional) – Existing dictionary that should be updated with variables. Defaults to None and is ignored.
with_flags (bool, optional) – If True, preserve any flags included in the environment variable. Defaults to False.

Returns:

True if the environment variable matches, False otherwise.

Return type:

classmethod file2base(fname)[source]¶

Determine basename from path.

Parameters:: fname (str) – Full or partial path to file.
Returns:: File name without extension.
Return type:: str

flag_options = {}¶

classmethod get_alternate_class(toolname=None, language=None)[source]¶

Return an alternate class to use if the provided toolname dosn’t match the current tool.

Parameters:

toolname (str, optional) – Name of compilation tool that should be used. Defaults to None and the current toolname is assumed.
language (str, optional) – Language that alternate class should support. Defaults to None and the current language will be assumed.

Returns:

The compilation tool that corresponds: to the provided toolname.

Return type:

classmethod get_env_flags()[source]¶

Get a list of flags stored in the environment variables.

Returns:: Flags for the tool.
Return type:: list

classmethod get_env_prefixes()[source]¶

Determine the virtualenv/conda path prefixes.

Returns:

Virtualenv/conda path prefixes. Empty list will be: returned if virtualenv/conda are not active.

Return type:

classmethod get_executable(full_path=False)[source]¶

Determine the executable that should be used to call this tool.

Parameters:: full_path (bool, optional) – If True the full path to the executable file will be returned. Defaults to False.
Returns:: Name of (or path to) the tool executable.
Return type:: str

classmethod get_executable_command(args, skip_flags=False, unused_kwargs=None, use_ccache=False, **kwargs)[source]¶

Determine the command required to run the tool using the specified arguments and options.

Parameters:

args (list) – The arguments that should be passed to the tool. If skip_flags is False, these are treated as input files that will be used by the tool.
skip_flags (bool, optional) – If True, args is assumed to include any necessary flags. If False, args are assumed to the files that the tool is called on and flags are determined from them. Defaults to False.
unused_kwargs (dict, optional) – Existing Python dictionary that unused keyword arguments will be added to. Defaults to None and is initialized to an empty dict.
use_ccache (bool, optional) – If True, ccache will be added to the compilation executable. Defaults to False.
**kwargs – Additional keyword arguments are ignored and stored in unused_kwargs if provided.

Returns:

Output to stdout from the command execution.

Return type:

classmethod get_flags(flags=None, outfile=None, output_first=None, unused_kwargs=None, skip_defaults=False, dont_skip_env_defaults=False, with_asan=False, **kwargs)[source]¶

Get a list of flags for the tool.

Parameters:

flags (list, optional) – User defined flags that should be included. Defaults to empty list.
outfile (str, optional) – If provided, it is appended to the end of the flags following the cls.output_key flag to indicate that this is the name of the output file. Defaults to None and is ignored.
output_first (bool, optional) – If True, output flag(s) will be placed at the front of the returned flags. If False, they are placed at the end. Defaults to None and is set by cls.output_first.
unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to None and is ignored.
skip_defaults (bool, optional) – If True, the default flags will not be added. Defaults to False.
dont_skip_env_defaults (bool, optional) – If skip_defaults is True, and this keyword is True, the flags from the environment variable will be added. Defaults to False.
remove_flags (list, optional) – List of flags to remove. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are ignored and added to unused_kwargs if provided.

Returns:

Flags for the tool.

Return type:

classmethod get_language_ext()[source]¶

Get the extensions associated with the language that this tool can handle.

Returns:: Language file extensions.
Return type:: list

classmethod get_search_path(env_only=False, libtype=None, cfg=None)[source]¶

Determine the paths searched by the tool for external library files.

Parameters:

env_only (bool, optional) – If True, only the search paths as indicated by a virtualenv/conda environment are returned. Defaults to False.
libtype (str, optional) – Library type being searched for. Defaults to None.
cfg (YggConfigParser, optional) – Configuration object currently being updated. Defaults to the global configuration.

Returns:

List of paths that the tools will search.

Return type:

classmethod get_source_products(products, source_products=[])[source]¶

Get the list of products that should be removed without checking for source files based on cls.remove_product_exts.

Parameters:

products (list) – List of products that should be checked against cls.remove_product_exts.
source_products (list, optional) – Existing list of products that new source_products should be added to. Defaults to [].

Returns:

Products that should be removed without checking for source: files.

Return type:

classmethod get_tool_suffix()[source]¶

Get the string that should be added to tool products based on the tool used.

Returns:

Suffix that should be added to tool products to indicate the: tool used.

Return type:

is_build_tool = False¶

is_gnu = False¶

classmethod is_installed()[source]¶

Determine if this tool is installed by looking for the executable.

Returns:: True if the tool is installed, False otherwise.
Return type:: bool

languages = []¶

classmethod locate_file(fname, libtype=None, verbose=False, **kwargs)[source]¶

Locate a library file.

Parameters:

fname (str) – Name of library.
libtype (str, optional) – Library type being searched for. Defaults to None.
verbose (bool, optional) – If True, display information about the success or failure of the search. Defaults to False.
**kwargs – Additional keyword arguments are passed to get_search_path.

Returns:

Full path to located library file.

Return type:

object_tool = None¶

output_first = False¶

output_key = '-o'¶

platforms = ['Windows', 'MacOS', 'Linux']¶

product_exts = []¶

product_files = []¶

remove_product_exts = []¶

classmethod remove_products(src, new)[source]¶

Remove products produced during compilation.

Parameters:

src (list) – Input arguments to compilation call that was used to generate the output file (usually one or more source files).
new (str) – The full path to the primary product of the compilation call (usually an object, executable, or library).

search_path_env = None¶

search_path_envvar = None¶

search_path_flags = None¶

search_regex = ['([^\n]+)']¶

search_regex_begin = None¶

search_regex_end = None¶

classmethod set_env(existing=None, **kwargs)[source]¶

Set environment variables required for compilation.

Parameters:

existing (dict, optional) – Existing dictionary of environment variables that new variables should be added to. Defaults to a copy of os.environ.
**kwargs – Additional keyword arguments are ignored.

Returns:

Environment variables for the model process.

Return type:

source_product_exts = []¶

tool_suffix_format = '_%sx'¶

classmethod tool_version(**kwargs)[source]¶

Get the version of the compilation tool.

Returns:: Version.
Return type:: str

toolname = None¶

toolset = None¶

tooltype = None¶

version_flags = ['--version']¶

classmethod write_wrappers(**kwargs)[source]¶

Write any wrappers needed to compile and/or run a model.

Parameters:: **kwargs – Keyword arguments are ignored (only included to allow cascade from child classes).
Returns:: Full paths to any created wrappers.
Return type:: list

class yggdrasil.drivers.CompiledModelDriver.CompilationToolMeta(name, bases, class_dict)[source]¶

Bases: type

Meta class for registering compilers.

class yggdrasil.drivers.CompiledModelDriver.CompiledModelDriver(*args, **kwargs)[source]¶

Bases: ModelDriver

Base class for models written in compiled languages.

Parameters:

name (str) – Driver name.
args (str or list) – The model executable and any arguments that should be passed to the model executable.
source_files (list, optional) – Source files that should be compiled into an executable. Defaults to an empty list and the driver will search for a source file based on the model executable (the first model argument).
compiler (str, optional) – Command or path to executable that should be used to compile the model. If not provided, the compiler will be determined based on configuration options for the language (if present) and the registered compilers that are available on the current operating system.
compiler_flags (list, optional) – Flags that should be passed to the compiler during compilation. If nto provided, the compiler flags will be determined based on configuration options for the language (if present), the compiler defaults, and the default_compiler_flags class attribute.
linker (str, optional) – Command or path to executable that should be used to link the model. If not provided, the linker will be determined based on configuration options for the language (if present) and the registered linkers that are available on the current operating system
linker_flags (list, optional) – Flags that should be passed to the linker during compilation. If not provided, the linker flags will be determined based on configuration options for the language (if present), the linker defaults, and the default_linker_flags class attribute.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.
with_asan (bool, optional) – If True, the model will be compiled and linked with the address sanitizer enabled (if there is one available for the selected compiler).
**kwargs – Additional keyword arguments are passed to parent class.

Class Attributes:

default_compiler (str): Name of compiler that will be used if not: set explictly by instance or config file.
default_compiler_flags (list): Flags that should be passed to the: compiler by default for this language.
default_linker (str): Name of linker that will be used if not: set explictly by instance or config file
default_linker_flags (list): Flags that should be passed to the: linker by default for this language.
allow_parallel_build (bool): If True, a file can be compiled by: two processes simultaneously. If False, it cannot and an MPI barrier will be used to prevent simultaneous compilation. Defaults to False.

source_files¶

Source files.

Type:: list

compiler¶

Name or path to the compiler that should be used.

Type:: str

compiler_flags¶

Compiler flags.

Type:: list

linker¶

Name or path to the linker that should be used.

Type:: str

linker_flags¶

Linker flags.

Type:: list

compiler_tool¶

Compiler tool that will be used.

Type:: CompilerBase

linker_tool¶

Linker tool that will be used.

Type:: LinkerBase

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

allow_parallel_build = False¶

buildfile_locked(dry_run=False)[source]¶: Context manager for locked build file.

classmethod call_compiler(src, language=None, toolname=None, dont_build=None, **kwargs)[source]¶

Compile a source file into an executable or linkable object file, checking for errors.

Parameters:

src (str) – Full path to source file.
out (str, optional) – Full path to the output object file that should be created. Defaults to None and is created from the provided source file.
flags (list, optional) – Compilation flags. Defaults to []. If compiler_flags is present, flags is replaced by compiler_flags.
compiler_flags (list, optional) – Alternative to flags. Ignored if not provided.
dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable. Defaults to False.
dont_build (bool, optional) – If True, cmake configuration/generation will be run, but the project will not be built. Defaults to None. If provided, this overrides dont_link.
overwrite (bool, optional) – If True, the existing compile file will be overwritten. Otherwise, it will be kept and this function will return without recompiling the source file.
language (str, optional) – Language that should be used to compile the files. Defaults to None and the language of the current driver is used.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
products (list, optional) – Existing Python list that additional products produced by the compilation should be appended to. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to run_executable. and call_linker if dont_link is False.

Returns:

Full path to compiled source.

Return type:

Raises:

RuntimeError – If there is an error in calling the compiler.
RuntimeError – If the compilation command dosn’t yield the specified output file.

classmethod call_linker(obj, language=None, toolname=None, **kwargs)[source]¶

Link several object files to create an executable or library (shared or static), checking for errors.

Parameters:

obj (list) – Object files that should be linked.
language (str, optional) – Language that should be used to link the files. Defaults to None and the language of the current driver is used.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
**kwargs – Additional keyword arguments are passed to run_executable.

Returns:

Full path to compiled source.

Return type:

classmethod cleanup_dependencies(products=None, verbose=False, **kwargs)[source]¶: Cleanup dependencies.

classmethod compile_dependencies(toolname=None, dep=None, **kwargs)[source]¶: Compile any required internal libraries, including the interface.

compile_dependencies_instance(*args, **kwargs)[source]¶: Compile dependencies specifically for this instance.

compile_model(source_files=None, skip_interface_flags=False, **kwargs)[source]¶

Compile model executable(s).

Parameters:

source_files (list, optional) – Source files that will be compiled. Defaults to None and is set to the source_files attribute.
skip_interface_flags (bool, optional) – If True, interface flags will not be added. This includes the logger flag specifying the current logging level. Defaults to False.
**kwargs – Keyword arguments are passed on to the call_compiler method.

Returns:

Compiled model file path.

Return type:

classmethod configuration_steps()[source]¶

Get a list of configuration steps with tuples of flags and boolean values.

Returns:

Pairs of descriptions and states for: different steps in the configuration all steps must be True for the language to be configured.

Return type:

OrderedDict

classmethod configure(cfg, **kwargs)[source]¶

Add configuration options for this language.

Parameters:

cfg (CisConfigParser) – Config class that options should be set for.
**kwargs – Additional keyword arguments are used to set tool configuration options (e.g. ‘compiler’).

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod configure_executable_type(cfg)[source]¶

Add configuration options specific in the executable type before the libraries are configured.

Parameters:

cfg (CisConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod configure_libraries(cfg)[source]¶

Add configuration options for external libraries in this language.

Parameters:

cfg (CisConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod configure_library(cfg, k)[source]¶

Add configuration options for an external library.

Parameters:

cfg (YggConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

default_archiver = None¶

default_archiver_flags = None¶

default_compiler = None¶

default_compiler_flags = None¶

default_linker = None¶

default_linker_flags = None¶

classmethod executable_command(args, exec_type='compiler', toolname=None, **kwargs)[source]¶

Compose a command for running a program using the compiler for this language and the provied arguments. If not already present, the compiler command and compiler flags are prepended to the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it. For the compiler, this means the source files, for the linker, this means the object files.
exec_type (str, optional) – Type of executable command that will be returned. If ‘compiler’, a command using the compiler is returned, if ‘linker’, a command using the linker is returned, and if ‘direct’, the raw args being provided are returned. Defaults to ‘compiler’.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
**kwargs – Additional keyword arguments are passed to either get_linker_flags or get_compiler_flags.

Returns:

Arguments composing the command required to run the program: from the command line using the compiler for this language.

Return type:

Raises:

ValueError – If exec_type is not ‘compiler’, ‘linker’, or ‘direct’.

executable_type = 'compiler'¶

external_libraries = {}¶

classmethod get_available_tools(tooltype)[source]¶

Return the registry of compilation tools for this language.

Parameters:: tooltype (str) – Name of tool type that should be returned. Values include ‘compiler’, ‘linker’, or ‘archiver’.
Returns:: Registry of tools for this language.
Return type:: dict

classmethod get_buildfile_lock(fname=None, context=None, instance=None, **kwargs)[source]¶

Get a lock for a buildfile to prevent simultaneous access, creating one as necessary.

Parameters:

name (str) – Build file.
context (threading.Context) – Threading context.
instance (ModelDriver) – Driver instance that should be used.
**kwargs – Additional keyword arguments are passed to the FileLock initialization.

Returns:

Lock for the buildfile.

Return type:

FileLock

classmethod get_compiler_flags(toolname=None, compiler=None, **kwargs)[source]¶

Determine the flags required by the current compiler.

Parameters:

toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
compiler (CompilerBase, optional) – Compilation tool class for the compiler that should be used. Defaults to None and is set based on toolname.
**kwargs – Keyword arguments are passed to cls.update_compiler_kwargs first and then the compiler’s get_flags method.

Returns:

Flags for the compiler.

Return type:

classmethod get_dependency_include_dirs(dep, toolname=None, default=None)[source]¶

Get the include directories for a dependency.

Parameters:

dep (str) – Name of internal or external dependency or full path to the library.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

Returns:

Full paths to the directories containing the dependency’s: header(s).

Return type:

Raises:

ValueError – If the include directory cannot be determined for the specified dependency and default is None.

classmethod get_dependency_info(dep, toolname=None, default=None)[source]¶

Get the dictionary of information associated with a dependency.

Parameters:

dep (str) – Name of internal or external dependency.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
default (dict, optional) – Information dictionary that should be returned if dep cannot be located. Defaults to None and an error will be raised if dep cannot be found.

Returns:

Dependency info.

Return type:

classmethod get_dependency_library(dep, default=None, libtype=None, commtype=None, toolname=None, disable_python_c_api=False, with_asan=False)[source]¶

Get the library location for a dependency.

Parameters:

dep (str) – Name of internal or external dependency or full path to the library.
default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.
libtype (str, optional) – Library type that should be returned. Valid values are ‘static’ and ‘shared’. Defaults to None and will be set on the dependency’s libtype (if it has one) or the _default_libtype parameter (if it doesn’t).
commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used in the names of internal libraries.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.
with_asan (bool, optional) – If True, the model will be compiled and linked with the address sanitizer enabled (if there is one available for the selected compiler).

Returns:

Full path to the library file. For header only libraries,: an empty string will be returned.

Return type:

Raises:

ValueError – If libtype is not ‘static’ or ‘shared’.
ValueError – If the path to the library cannot be determined for the specified dependency and default is None.

classmethod get_dependency_object(dep, default=None, commtype=None, toolname=None, disable_python_c_api=False, with_asan=False)[source]¶

Get the location of an object file for a dependency.

Parameters:

dep (str) – Name of internal or external dependency or full path to the object file.
default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.
commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used in the names of internal libraries.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.
with_asan (bool, optional) – If True, the model will be compiled and linked with the address sanitizer enabled (if there is one available for the selected compiler).

Returns:

Full path to the object file.

Return type:

classmethod get_dependency_order(deps, toolname=None, disable_python_c_api=False)[source]¶

Get the correct dependency order, including any dependencies for the direct dependencies.

Parameters:

deps (list) – Dependencies in order.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.

Returns:

Dependency order.

Return type:

classmethod get_dependency_source(dep, toolname=None, default=None)[source]¶

Get the path to the library source files (or header files) for a dependency.

Parameters:

dep (str) – Name of internal or external dependency or full path to the library.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
default (str, optional) – Default that should be used if a value cannot be determined form internal/external dependencies or if dep is not a valid file. Defaults to None and is ignored.

Returns:

Full path to the library source file. For header only libraries: this will be the header location.

Return type:

classmethod get_external_libraries(no_comm_libs=False)[source]¶

Determine the external libraries that are required based on the default comm.

Parameters:

no_comm_libs (bool, optional) – If True, libraries for the installed comms are not included in the returned list. Defaults to False.

Returns:

The names of external libraries required by the interface: library, including the dependency libraries for the installed comm libraries.

Return type:

classmethod get_internal_suffix(commtype=None, disable_python_c_api=False, with_asan=False)[source]¶

Determine the suffix that should be used for internal libraries.

Parameters:

commtype (str, optional) – If provided, this is the communication type that should be used for the model. If None, the default comm is used.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.
with_asan (bool, optional) – If True, the model will be compiled and linked with the address sanitizer enabled (if there is one available for the selected compiler).

Returns:

Suffix that should be added to internal libraries to: differentiate between different dependencies.

Return type:

classmethod get_linker_flags(toolname=None, **kwargs)[source]¶

Determine the flags required by the current linker.

Parameters:

toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
**kwargs – Keyword arguments are passed to cls.update_linker_kwargs first and then the linker’s get_flags method.

Returns:

Flags for the linker.

Return type:

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

classmethod get_tool(*args, **kwargs)[source]¶

Get tool from a driver class.

Parameters:

*args – Arguments are passed to the get_tool_static method.
**kwargs – Keyword arguments are passed to the get_tool_static method.

Returns:

Class providing an interface to the specified: compilation tool.

Return type:

get_tool_instance(*args, **kwargs)[source]¶

Get tool from a driver instance.

Parameters:

*args – Arguments are passed to the get_tool_static method.
**kwargs – Keyword arguments are passed to the get_tool_static method.

Returns:

Class providing an interface to the specified: compilation tool.

Return type:

static get_tool_static(cls, tooltype, toolname=None, return_prop='tool', default=False, language=None)[source]¶

Get the class associated with the specified compilation tool for this language.

Parameters:

cls (class, instance) – Compiled driver class or instance of compiled driver class to get tool for.
tooltype (str) – Type of compilation tool that should be returned.
toolname (str, optional) – Name of the tool that should be returned. Defaults to None and the tool name associated with the provided class/instance will be used.
return_prop (str, optional) – Value that should be returned. If ‘tool’, the tool is returned. If ‘name’, the tool name is returned. If ‘flags’, the tool flags are returned. Defaults to ‘tool’.
default (object, optiona) – Tool that should be returned if one cannot be identified. If False, an error will be raised when a tool cannot be located. Defaults to False.
language (str, optional) – Language of tools that should be returned. Defaults to None if not provided.

Returns:

Class providing an interface to the specified: compilation tool.

Return type:

Raises:

NotImplementedError – If a tool is not specified.
ValueError – If return_prop is not ‘tool’, ‘name’, or ‘flags’.

classmethod identify_source_files(args=None, working_dir=None, source_files=None, **kwargs)[source]¶

Determine the source file based on model arguments.

Parameters:

args (list, optional) – Arguments provided.
working_dir (str, optional) – Working directory.
source_files (list, optional) – Source files in the model.
**kwargs – Additional keyword arguments are ignored.

Returns:

Source files.

Return type:

internal_libraries = {}¶

is_build_tool = False¶

classmethod is_language_installed()[source]¶

Determine if the interpreter/compiler for the associated programming language is installed.

Returns:: True if the language interpreter/compiler is installed.
Return type:: bool

classmethod is_library_installed(lib, cfg=None)[source]¶

Determine if a dependency is installed by check for the appropriate config options setting the path to the library files.

Parameters:

lib (str) – Name of the library that should be checked.
cfg (CisConfigParser, optional) – Config class that should be checked. Defaults to yggdrasil.config.ygg_cfg if not provided.

Returns:

True if the library is installed, False otherwise.

Return type:

classmethod is_standard_library(dep)[source]¶

Determine if a dependency is a standard library.

Parameters:

dep (str, tuple) – Dependency name or tuple with language and dependency name.

Returns:

True if the dependency is a standard library, false: otherwise.

Return type:

classmethod is_tool_installed(tooltype)[source]¶

Determine if a compilation tool of a certain is installed for this language.

Parameters:: tooltype (str) – Type of tool to check for. Supported values include ‘compiler’, ‘linker’, & ‘archiver’.
Returns:: True if a tool of the specified type is installed.
Return type:: bool

kwargs_in_suffix = ['with_asan', 'disable_python_c_api', 'commtype']¶

classmethod language_executable(toolname=None)[source]¶

Command required to compile/run a model written in this language from the command line.

Parameters:

toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns:

Name of (or path to) compiler/interpreter executable required: to run the compiler/interpreter from the command line.

Return type:

classmethod language_version(toolname=None, **kwargs)[source]¶

Determine the version of this language.

Parameters:

**kwargs – Keyword arguments are passed to cls.run_executable.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.

Returns:

Version of compiler/interpreter for this language.

Return type:

locked_buildfile = None¶

model_command()[source]¶

Return the command that should be used to run the model.

Returns:

Any commands/arguments needed to run the model from the: command line.

Return type:

classmethod mpi_partner_cleanup(self)[source]¶: Actions cleaning up an MPIPartnerModel.

classmethod mpi_partner_init(self)[source]¶: Actions initializing an MPIPartnerModel.

parse_arguments(args, **kwargs)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:

args (list) – List of arguments provided.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

classmethod partner_buildfile_lock(self)[source]¶: Actions completing buildfile lock on MPIPartnerModels.

run_model(**kwargs)[source]¶

Run the model. Unless overridden, the model will be run using run_executable.

Parameters:: **kwargs – Keyword arguments are passed to run_executable.

classmethod select_suffix_kwargs(kwargs)[source]¶

Select suffix kwargs.

Parameters:: kwargs (dict) – Keyword arguments to select suffix kwargs from.
Returns:: Selected suffix kwargs.
Return type:: dict

set_env(for_compile=False, compile_kwargs=None, toolname=None, **kwargs)[source]¶

Get environment variables that should be set for the model process.

Parameters:

for_compile (bool, optional) – If True, environment variables are set that are necessary for compiling. Defaults to False.
compile_kwargs (dict, optional) – Keyword arguments that should be passed to the compiler’s set_env method. Defaults to empty dict.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Environment variables for the model process.

Return type:

classmethod set_env_compiler(compiler=None, toolname=None, **kwargs)[source]¶

Get environment variables that should be set for the compilation process.

Parameters:

compiler (CompilerBase, optional) – Compiler that set_env shoudl be called for. If not provided, the default compiler for this language will be used.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Environment variables for the model process.

Return type:

set_target_language()[source]¶

Set the language of the target being compiled (usually the same as the language associated with this driver.

Returns:: Name of language.
Return type:: str

standard_libraries = []¶

classmethod update_compiler_kwargs(for_api=False, for_model=False, commtype=None, toolname=None, directory=None, include_dirs=None, definitions=None, skip_interface_flags=False, **kwargs)[source]¶

Update keyword arguments supplied to the compiler get_flags method for various options.

Parameters:

dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable. Defaults to False.
for_api (bool, optional) – If True, flags are added that are required for compiling internal api libraries in this language. This includes external communication libraries. Defaults to False.
for_model (bool, optional) – If True, flags are added that are required for including the interface library. Defaults to False.
commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used if for_model is True.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
include_dirs (list, optional) – If provided, each list element will be added as an included directory flag. Defaults to None and is initialized as an empty list.
definitions (list, optional) – If provided, each list element will be added as a defintion. Defaults to None and is initialized to an empty list.
skip_interface_flags (bool, optional) – If True, interface flags will not be added. Defaults to False.
internal_dependencies (list, optional) – If provided, a list of names of internal libraries that are required or linkable object files for dependencies. Defaults to an empty list.
external_dependencies (list, optional) – If provided, a list of names of external libraries that are required or linkable object files for dependencies. Defaults to an empty list.
**kwargs – Additional keyword arguments are passed to the compiler class’s ‘get_flags’ method and get_linker_flags if dont_link is False.

Returns:

Keyword arguments for a get_flags method providing compiler: flags.

Return type:

classmethod update_linker_kwargs(for_api=False, for_model=False, commtype=None, toolname=None, libtype='object', skip_interface_flags=False, use_library_path_internal=False, **kwargs)[source]¶

Update keyword arguments supplied to the linker/archiver get_flags method for various options.

Parameters:

for_api (bool, optional) – If True, flags are added that are required for linking internal api libraries in this language. This includes external communication libraries. Defaults to False.
for_model (bool, optional) – If True, flags are added that are required for including the interface library. Defaults to False.
commtype (str, optional) – If provided, this is the communication type that should be used for the model and flags for just that comm type will be included. If None, flags for all installed comm types will be included. Default to None. This keyword is only used if for_model is True.
toolname (str, optional) – Name of compiler tool that should be used. Defaults to None and the default compiler for the language will be used.
libtype (str, optional) – Library type that should be created by the linker/archiver. Valid values are ‘static’, ‘shared’, or ‘object’. Defaults to ‘object’.
skip_interface_flags (bool, optional) – If True, interface flags will not be added. Defaults to False.
libraries (list, optional) – Full paths to libraries that should be linked against. Defaults to an empty list.
internal_dependencies (list, optional) – If provided, a list of names of internal libraries that are required or linkable object files for dependencies. Defaults to an empty list.
external_dependencies (list, optional) – If provided, a list of names of external libraries that are required or linkable object files for dependencies. Defaults to an empty list.
use_library_path_internal (bool, optional) – If True, internal dependencies are included as full paths. Defaults to False.
**kwargs – Additional keyword arguments are passed to the linker (or archiver if static is True) ‘get_flags’ method.

Returns:

Keyword arguments for a get_flags method providing linker/: archiver flags.

Return type:

write_wrappers(**kwargs)[source]¶

Write any wrappers needed to compile and/or run a model.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Full paths to any created wrappers.
Return type:: list

class yggdrasil.drivers.CompiledModelDriver.CompilerBase(**kwargs)[source]¶

Bases: CompilationToolBase

Base class for compilers.

Parameters:

linker (str, optional) – Name of the linker that should be used for linking compiled objects. Defaults to None if not provided and default_linker will be used.
archiver (str, optional) – Name of the archiver that should be used for combining compiled objects into a static library. Defaults to None if not provided and default_archiver will be used.
linker_flags (list, optional) – Flags that should be used when linking compiled objects. Defaults to default_linker_flags if not provided.
archiver_flags (list, optional) – Flags that should be used for combining compiled objects into a static library. Defaults to default_archiver_flags if not provided.

Class Attributes:

compile_only_flag (str): Flag that should prepended to compiler/linker: combination tool arguments to indicate that only compilation should be performed.
default_linker (str): Name of linker that should be used after compiling: with this compiler. If not set, it is assumed that this compiler is also a linker.
default_archiver (str): Name of archiver that should be used to create: a static library after compiling with this compiler. If not set, it is assumed that this compiler is also a linker.
default_linker_flags (list): Flags that should be used with the linker: if no other flags are provided.
default_archiver_flags (list): Flags that should be used with the: archiver if no other flags are provided.
linker_switch (str): Flag to indicate beginning of flags that should be: passed to the linker from a call to a compiler/linker combination tools (e.g. /link on Windows).

object_ext (str): Extension that should be used for object files. is_linker (bool): If True, the tool also serves as a linker and a

separate linker class will be automatically generating from the linker_attributes dictionary. This will be set to True if no_separate_linking is True.

no_separate_linking (bool): If True, the tool severs as linker but: cannot be called for just compilation or linking alone.
linker_attributes (dict): Attributes that should be added to the linker: class created for this tool if is_linker is True.
linker_base_classes (tuple): Base classes that should be used to create: the default linker from the compiler tool. If None, (LinkerBase, ) is used if no_separate_linking is False and (DummyLinkerBase, ) is used if no_separate_linking is True.
combine_with_linker (bool): If True, the compiler and linker flags can: be combined and passed to the compiler executable to perform both operations in succession. If False, the compilation and linking steps must be performed separately. If None, this is determined by checking if the compiler and linker names match.

classmethod archiver()[source]¶

Get the associated archiver class.

Returns:: Archiver class associated with this compiler.
Return type:: ArchiverToolBase

classmethod asan_library()[source]¶: Return the address sanitizer library.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

classmethod call(args, dont_link=None, skip_flags=False, out=None, libtype=None, additional_objs=None, **kwargs)[source]¶

Call the tool with the provided arguments. If the first argument resembles the name of the tool executable, the executable will not be added.

Parameters:

args (list) – The arguments that should be passed to the tool.
dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable or library. Defaults to True if cls.no_separate_linking is True or libtype is ‘object’ and False otherwise.
skip_flags (bool, optional) – If True, args is assumed to include any necessary flags. If False, args are assumed to the files that the tool is called on and flags are determined from them. Defaults to False.
out (str, optional) – Full path to output file that should be created. If None, the path will be determined from the path to the first arguments provided. Defaults to None. This keyword argument will be ignored if skip_flags is True.
libtype (str, optional) – Library type that should be created by the linker/archiver. Defaults to None.
additional_objs (list, optional) – Additional linkable object files that should be supplied to the linker/archiver if dont_link is False. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to the parent class’s method and the associated linker/archiver’s call method if dont_link is False.

Returns:

Output to stdout from the command execution if skip_flags is: True, produced file otherwise.

Return type:

combine_with_linker = None¶

compile_only_flag = '-c'¶

default_archiver = None¶

default_archiver_flags = None¶

default_linker = None¶

default_linker_flags = None¶

flag_options = {'definitions': '-D%s', 'include_dirs': '-I%s'}¶

classmethod get_flags(dont_link=None, add_linker_switch=False, libtype=None, logging_level=None, disable_python_c_api=False, **kwargs)[source]¶

Get a list of compiler flags.

Parameters:

dont_link (bool, optional) – If True, the command will result in a linkable object file rather than an executable or library. Defaults to True if cls.no_separate_linking is True or libtype is ‘object’ and False otherwise.
add_linker_switch (bool, optional) – If True, the linker_switch flag will be added to the flags even if dont_link is True as long as the flag is not None. Defaults to False.
libtype (str, optional) – Library type that should be created by the linker/archiver. Defaults to None.
logging_level (int, optional) – Logging level that should be passed as a definition to the C compiler. Defaults to None and will be ignored.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method and get_linker_flags if dont_link is False.

Returns:

Compiler flags.

Return type:

Raises:

ValueError – If dont_link is False and cls.combine_with_linker is False.

classmethod get_library_tool(libtype=None, **kwargs)[source]¶

Determine the tool that should be used based on the provided arguments.

Parameters:

libtype (str, optional) – Library type that should be created by the linker/archiver. If ‘static’, the archiver is returned. If ‘shared’ or any other value, the linker is returned. Defaults to None.
**kwargs – Additional keyword arguments are ignored.

Returns:

Linker/archiver that should be used.

Return type:

classmethod get_output_file(src, dont_link=False, working_dir=None, libtype=None, no_src_ext=False, no_tool_suffix=False, suffix='', **kwargs)[source]¶

Determine the appropriate output file that will result when compiling a given source file.

Parameters:

src (str) – Source file being compiled that name base will be taken from.
dont_link (bool, optional) – If True, the result assumes that the source is just compiled and not linked. If False, the result will be the final result after linking. Defaults to None and will be set to True if libtype is ‘object’ and False otherwise.
working_dir (str, optional) – Working directory where output file should be located. Defaults to None and is ignored.
libtype (str, optional) – Library type that should be created by the linker/archiver. Defaults to None.
no_src_ext (bool, optional) – If True, the source extension will not be added to the object file name. Defaults to False. Ignored if dont_link is False.
no_tool_suffix (bool, optional) – If True, the tool suffix will not be added to the object file name. Defaults to False.
suffix (str, optional) – Suffix that should be added to the output file (before the extension). Defaults to “”.
**kwargs – Additional keyword arguments are ignored unless dont_link is False; then they are passed to the linker’s get_output_file method.

Returns:

Full path to file that will be produced.

Return type:

classmethod init_asan_env(out)[source]¶: Add environment variables to preload the ASAN libraries.

is_linker = True¶

classmethod linker()[source]¶

Get the associated linker class.

Returns:: Linker class associated with this compiler.
Return type:: CompilationToolBase

linker_attributes = {}¶

linker_base_classes = None¶

linker_switch = None¶

classmethod locate_file(fname, **kwargs)[source]¶

Locate a library file.

Parameters:

fname (str) – Name of library.
**kwargs – Additional keyword arguments are passed to get_search_path.

Returns:

Full path to located library file.

Return type:

no_separate_linking = False¶

object_ext = '.o'¶

classmethod preload_env(libs, env)[source]¶

Get environment variables necessary to preload libraries.

Parameters:

libs (list) – One or more libaries to preload.
env (dict) – Dictionary to add environment variables to.

Returns:

Environment variable options.

Return type:

search_path_env = ['include']¶

tooltype = 'compiler'¶

class yggdrasil.drivers.CompiledModelDriver.DummyLinkerBase(**kwargs)[source]¶

Bases: LinkerBase

Base class for a dummy linker in the case that the linking step cannot be split into a separate call.

classmethod call(*args, **kwargs)[source]¶: Raises an error to ward off calling the dummy linker.

classmethod get_flags(**kwargs)[source]¶: Raises an error to ward off getting flags for the dummy linker.

is_dummy = True¶

toolname = 'dummy'¶

class yggdrasil.drivers.CompiledModelDriver.GFortranLinker(**kwargs)¶

Bases: LinkerBase

aliases = []¶

compatible_toolsets = ['gnu']¶

default_executable = 'gfortran'¶

default_executable_env = 'FC'¶

default_flags = []¶

default_flags_env = 'LFLAGS'¶

is_gnu = True¶

languages = ['fortran']¶

library_ext = '.so'¶

platforms = ['MacOS', 'Linux', 'Windows']¶

search_path_envvar = ['LIBRARY_PATH', 'LD_LIBRARY_PATH']¶

toolname = 'gfortran'¶

toolset = 'gnu'¶

class yggdrasil.drivers.CompiledModelDriver.LinkerBase(**kwargs)[source]¶

Bases: CompilationToolBase

Base class for linkers.

shared_library_flag¶

Flag that should be prepended to the linker tool arguments to indicate that a shared/dynamic library should be produced instead of an executable.

Type:: str

library_name_key¶

Option key indicating the name of a library that should be linked against.

Type:: str

library_directory_key¶

Option key indicating a directory that should be included in the linker search path for libraries.

Type:: str

library_prefix¶

Prefix that should be added to library paths.

Type:: str

library_ext¶

Extension that should be used for shared libraries.

Type:: str

executable_ext¶

Extension that should be used for executables.

Type:: str

output_first_library¶

If True, the output key (and its value) are put in front of the other flags when building a library. A value of None causes the output_first attribute to be used (unless explicitly set in the method call).

Type:: bool

all_library_ext = ['.so', '.a']¶

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

executable_ext = '.out'¶

classmethod extract_kwargs(kwargs, compiler=None, add_kws_link=[], add_kws_both=[])[source]¶

Extract linker kwargs, leaving behind just compiler kwargs.

Parameters:

kwargs (dict) – Keyword arguments passed to the compiler that should be sorted into kwargs used by either the compiler or linker or both. Keywords that are not used by the compiler will be removed from this dictionary.
compiler (CompilerBase, optional) – Compiler tool that linker kwargs are being extracted in order to call. Defaults to None and is ignored.
add_kws_link (list, optional) – Addition keywords that should be added to the list of those reserved for the linker. Defaults to [].
add_kws_both (list, optional) – Additional keywords that should be added to the list of those that are valid for both the linker and compiler. Defaults to [].

Returns:

Keyword arguments that should be passed to the linker.

Return type:

flag_options = {'library_dirs': '-L%s', 'library_libs': {'allow_duplicate_values': True, 'key': '-l%s'}, 'library_libs_nonstd': {'allow_duplicate_values': True, 'key': '-l:%s'}}¶

classmethod get_flags(build_library=False, skip_library_libs=False, use_library_path=False, **kwargs)[source]¶

Get a list of linker flags.

Parameters:

libraries (list, optional) – Full paths to libraries that should be linked against. Defaults to an empty list.
library_dirs (list, optional) – Directories that should be searched for libraries. Defaults to an empty list.
library_libs (list, optional) – Names of libraries that should be linked against. Defaults to an empty list.
library_libs_nonstd (list, optional) – Names of libraries w/ non-standard naming conventions that should be linked against. Defaults to an empty list.
library_flags (list, optional) – Existing list that library flags should be appended to instead of the returned flags if skip_library_libs is True. Defaults to [].
build_library (bool, optional) – If True, a shared library is built. If False, an executable is created. Defaults to False.
skip_library_libs (bool, optional) – If True, the library_libs will not be added to the returned flags. Instead, any additional required library flags will be appended to the provided library_flags list which should then be added to the compilation command by the user in the appropriate location. Defaults to False.
use_library_path (bool, optional) – If True, the included libraries will be added to the output list as complete paths rather than as separate flags for library and library search directory. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Linker flags.

Return type:

classmethod get_output_file(obj, build_library=False, working_dir=None, suffix='', no_tool_suffix=False, **kwargs)[source]¶

Determine the appropriate output file that will result when linking a given object file.

Parameters:

obj (str) – Object file being linked that name base will be taken from.
build_library (bool, optional) – If True, a shared library path is returned. If False, an executable file name is returned. Defaults to False.
working_dir (str, optional) – Working directory where output file should be located. Defaults to None and is ignored.
suffix (str, optional) – Suffix that should be added to the output file (before the extension). Defaults to “”.
no_tool_suffix (bool, optional) – If True, the tool suffix will not be added to the object file name. Defaults to False.
**kwargs – Additional keyword arguments are ignored.

Returns:

Full path to file that will be produced.

Return type:

classmethod is_standard_libname(libname)[source]¶

Determine if the provided file name conforms to the standards expected by this linker.

Parameters:: libname (str) – Library file name to check.
Returns:: True if the name conforms, False otherwise.
Return type:: bool

classmethod libpath2libname(libpath)[source]¶

Determine the library name from the library path.

Parameters:: libpath (str) – Full or partial path to library.
Returns:: Library name.
Return type:: str

library_ext = None¶

library_prefix = 'lib'¶

output_first_library = None¶

search_path_env = ['lib']¶

shared_library_flag = '-shared'¶

tooltype = 'linker'¶

class yggdrasil.drivers.CompiledModelDriver.LockedFile(fname, context, when_to_lock='init')[source]¶

Bases: object

Class for locking files during compilation.

property message¶

Message form.

Type:: str

class yggdrasil.drivers.CompiledModelDriver.MSVCLinker(**kwargs)¶

Bases: LinkerBase

aliases = []¶

compatible_toolsets = ['msvc']¶

default_executable = 'LINK'¶

default_executable_env = 'LINK'¶

default_flags = []¶

default_flags_env = 'LDFLAGS'¶

flag_options = {'import_lib': '/IMPLIB:%s', 'library_dirs': '/LIBPATH:%s', 'library_libs': '', 'library_libs_nonstd': ''}¶

is_gnu = False¶

languages = ['c']¶

library_ext = '.so'¶

output_first = True¶

output_first_library = False¶

output_key = '/OUT:%s'¶

platforms = ['Windows']¶

search_path_envvar = ['LIB']¶

search_path_flags = None¶

shared_library_flag = '/DLL'¶

toolname = 'LINK'¶

toolset = 'msvc'¶

class yggdrasil.drivers.CompiledModelDriver.MakeLinker(**kwargs)¶

Bases: DummyLinkerBase

aliases = []¶

default_executable = 'make'¶

default_executable_env = None¶

default_flags = []¶

executable_ext = ''¶

is_gnu = False¶

languages = ['make']¶

library_ext = '.so'¶

platforms = ['MacOS', 'Linux']¶

tool_suffix_format = ''¶

toolname = 'make'¶

toolset = None¶

class yggdrasil.drivers.CompiledModelDriver.NMakeLinker(**kwargs)¶

Bases: DummyLinkerBase

aliases = []¶

default_executable = 'nmake'¶

default_executable_env = None¶

default_flags = []¶

executable_ext = ''¶

is_gnu = False¶

languages = ['make']¶

library_ext = '.so'¶

platforms = ['Windows']¶

tool_suffix_format = ''¶

toolname = 'nmake'¶

toolset = None¶

yggdrasil.drivers.CompiledModelDriver.find_compilation_tool(tooltype, language, allow_failure=False, dont_check_installation=False)[source]¶

Return the prioritized class for a compilation tool of a certain type that can handle the specified language.

Parameters:

tooltype (str) – Type of tool. Valid values include ‘compiler’, ‘linker’, and ‘archiver’.
allow_failure (bool, optional) – If True and a tool cannot be located, None will be returned. Otherwise, an error will be raised if a tool cannot be located. Defaults to False.
dont_check_installation (bool, optional) – If True, the first tool in the registry will be returned even if it is not installed. Defaults to False.

Returns:

Name of the determined tool type.

Return type:

Raises:

RuntimeError – If a tool cannot be located for the specified language on the current platform and allow_failure is False.

yggdrasil.drivers.CompiledModelDriver.get_compatible_tool(tool, tooltype, language, default=False)[source]¶

Get a compatible compilation tool that can be used in conjunction with the one provided based on the registry of compilation toolsets.

Parameters:

tool (CompilationToolBase, str) – Compilation tool or name of a compilation tool to get compatible counterpart to.
tooltype (str) – Type of compilation tool that should be returned.
language (str) – Language that compilation tool should handle.
default (CompilationToolBase, optional) – Default tool that should be returned if not compatible tool can be located. Defaults to False and an error will be raised if a tool cannot be located.

Returns:

Compatible compilation tool class.

Return type:

yggdrasil.drivers.CompiledModelDriver.get_compilation_tool(tooltype, name, default=False)[source]¶

Return the class providing information about a compilation tool.

Parameters:

tooltype (str) – Type of tool. Valid values include ‘compiler’, ‘linker’, and ‘archiver’.
name (str) – Name or path to the desired compilation tool.
default (object, optional) – Value that should be returned if a tool cannot be located. If False, an error will be raised. Defaults to False.

Returns:

Class providing access to the specified tool.

Return type:

Raises:

ValueError – If a tool with the provided name cannot be located.

yggdrasil.drivers.CompiledModelDriver.get_compilation_tool_registry(tooltype, init_languages=None)[source]¶

Return the registry containing compilation tools of the specified type.

Parameters:

tooltype (str) – Type of tool. Valid values include ‘compiler’, ‘linker’, and ‘archiver’.
init_languages (list, optional) – List of languages that should be imported prior to returning the registry, thereby populating the compilation tools for that language. Defaults to None and is ignored.

Returns:

Registry for specified type.

Return type:

collections.OrderedDict

Raises:

ValueError – If tooltype is not a valid value (i.e. ‘compiler’, ‘linker’, or ‘archiver’).

yggdrasil.drivers.ConnectionDriver module¶

Module for funneling messages from one comm to another.

class yggdrasil.drivers.ConnectionDriver.ConnectionDriver(*args, **kwargs)[source]¶

Bases: Driver

Class that continuously passes messages from one comm to another.

Parameters:

name (str) – Name that should be used to set names of input/output comms.
inputs (list, optional) – One or more dictionaries containing keyword arguments for constructing input communicators. Defaults to an empty dictionary if not provided.
outputs (list, optional) – One or more dictionaries containing keyword arguments for constructing output communicators. Defaults to an empty dictionary if not provided.
input_pattern (str, optional) –
The communication pattern that should be used to handle incoming messages when there is more than one input communicators present. Defaults to ‘cycle’. Options include:

’cycle’: Receive from the next available input communicator. ‘gather’: Receive lists of messages with one element from each

communicator where a message is only returned when there is a message from each.
output_pattern (str, optional) –
The communication pattern that should be used to handling outgoing messages when there is more than one output communicator present. Defaults to ‘broadcast’. Options include:

’cycle’: Rotate through output comms, sending one message to
each.

’broadcast’: Send the same message to each comm. ‘scatter’: Send part of message (must be a list) to each comm.
transform (str, func, optional) – Function or string specifying function that should be used to translate messages from the input communicator before passing them to the output communicator. If a string, the format should be “<package.module>:<function>” so that <function> can be imported from <package>. Defaults to None and messages are passed directly. This can also be a list of functions/strings that will be called on the messages in the order they are provided.
timeout_send_1st (float, optional) – Time in seconds that should be waited before giving up on the first send. Defaults to self.timeout.
single_use (bool, optional) – If True, the driver will be stopped after one loop. Defaults to False.
onexit (str, optional) – Class method that should be called when a model that the connection interacts with exits, but before the connection driver is shut down. Defaults to None.
**kwargs – Additonal keyword arguments are passed to the parent class.

icomm_kws¶

Keyword arguments for the input communicator.

Type:: dict

ocomm_kws¶

Keyword arguments for the output communicator.

Type:: dict

icomm¶

Input communicator.

Type:: CommBase

ocomm¶

Output communicator.

Type:: CommBase

nrecv¶

Number of messages received.

Type:: int

nproc¶

Number of messages processed.

Type:: int

nsent¶

Number of messages sent.

Type:: int

state¶

Descriptor of last action taken.

Type:: str

transform¶

Function that will be used to translate messages from the input communicator before passing them to the output communicator.

Type:: func

timeout_send_1st¶

Time in seconds that should be waited before giving up on the first send.

Type:: float

single_use¶

If True, the driver will be stopped after one loop.

Type:: bool

onexit¶

Class method that should be called when the corresponding model exits, but before the driver is shut down.

Type:: str

after_loop()[source]¶: Actions to perform after sending messages.

after_loop_process()[source]¶: Actions to preform after loop for process.

before_loop()[source]¶: Actions to perform prior to sending messages.

property can_run_remotely¶

True if process should be run remotely.

Type:: bool

cleanup()[source]¶: Ensure that the communicators are closed.

close_comm()[source]¶: Close the communicators.

property close_state¶

State of the connection at close.

Type:: str

confirm_input(timeout=None)[source]¶: Confirm receipt of messages from input comm.

confirm_output(timeout=None)[source]¶: Confirm receipt of messages from output comm.

do_terminate()[source]¶: Stop the driver by closing the communicators.

drain_input(timeout=None)[source]¶: Drain messages from input comm.

drain_output(timeout=None, dont_confirm_eof=False)[source]¶: Drain messages from output comm.

get_flag_attr(attr)[source]¶: Return the flag attribute.

graceful_stop(timeout=None, **kwargs)[source]¶

Stop the driver, first waiting for the input comm to be empty.

Parameters:

timeout (float, optional) – Max time that should be waited. Defaults to None and is set to attribute timeout.
**kwargs – Additional keyword arguments are passed to the parent class’s graceful_stop method.

property is_comm_closed¶

Returns True if both communicators are closed.

Type:: bool

property is_comm_open¶

Returns True if both communicators are open.

Type:: bool

property is_valid¶

Returns True if the connection is open and the parent class is valid.

Type:: bool

property model_env¶

Mapping between model name and opposite comm environment variables that need to be provided to the model.

Type:: dict

property n_msg¶

Number of messages waiting in input communicator.

Type:: int

property nproc¶

Number of messages processed.

Type:: int

property nrecv¶

Number of messages received.

Type:: int

property nsent¶

Number of messages sent.

Type:: int

on_eof(msg)[source]¶

Actions to take when EOF received.

Parameters:: msg (CommMessage) – Message object that provided the EOF.
Returns:: Value that should be returned by recv_message on EOF.
Return type:: CommMessage, bool

on_message(msg)[source]¶

Process a message.

Parameters:: msg (bytes, str) – Message to be processed.
Returns:: Processed message.
Return type:: bytes, str

on_model_exit(direction, name, errors=False)[source]¶: Drain input and then close it.

on_model_exit_remote(direction, name, errors=False)[source]¶

Drain input and then close it (on the remote process).

Parameters:

direction (str) – Direction of model.
name (str) – Name of model exiting.
errors (list, optional) – Errors generated by the model. Defaults to False.

Returns:

True if all of the input/output models have signed: off; False otherwise.

Return type:

open_comm()[source]¶: Open the communicators.

printStatus(beg_msg='', end_msg='', verbose=False, return_str=False)[source]¶

Print information on the status of the ConnectionDriver.

Parameters:

beg_msg (str, optional) – Additional message to print at beginning.
end_msg (str, optional) – Additional message to print at end.
verbose (bool, optional) – If True, the status of individual comms will be displayed. Defaults to False.
return_str (bool, optional) – If True, the message string is returned. Defaults to False.

recv_message(**kwargs)[source]¶

Get a new message to send.

Parameters:: **kwargs – Additional keyword arguments are passed to the appropriate recv method.
Returns:: False if no more messages, message otherwise.
Return type:: CommMessage, bool

remove_model(direction, name)[source]¶

Remove a model from the list of models.

Parameters:

direction (str) – Direction of model.
name (str) – Name of model exiting.

Returns:

True if all of the input/output models have signed: off; False otherwise.

Return type:

run_loop()[source]¶: Run the driver. Continue looping over messages until there are not any left or the communication channel is closed.

send_eof(**kwargs)[source]¶

Send EOF message.

Returns:: Success or failure of send.
Return type:: bool

send_message(msg, **kwargs)[source]¶

Send a single message.

Parameters:

msg (CommMessage) – Message being sent.
*kwargs – Keyword arguments are passed to the output comm send method.

Returns:

Success or failure of send.

Return type:

set_close_state(state)[source]¶: Set the close state if its not already set.

set_flag_attr(attr, value=True)[source]¶: Set a flag.

start()[source]¶: Open connection before running.

property state¶

Current state of the connection.

Type:: str

update_serializer(msg)[source]¶: Update the serializer for the output comm based on input.

wait_for_route(timeout=None)[source]¶: Wait until messages have been routed.

class yggdrasil.drivers.ConnectionDriver.RemoteTaskLoop(*args, **kwargs)[source]¶

Bases: YggTaskLoop

Class to handle running tasks on the connection loop process.

after_loop()[source]¶: Actions performed after the loop.

close()[source]¶: Close the queues.

is_open()[source]¶

run_task_local(task, args, kwargs)[source]¶: Run task on the current process.

run_task_remote(task, args, kwargs)[source]¶: Run task on the connection loop process.

target()[source]¶: Complete all pending tasks.

exception yggdrasil.drivers.ConnectionDriver.TaskThreadError[source]¶: Bases: RuntimeError

yggdrasil.drivers.ConnectionDriver.run_remotely(method)[source]¶: Decorator for methods that should be run remotely.

yggdrasil.drivers.DSLModelDriver module¶

class yggdrasil.drivers.DSLModelDriver.DSLModelDriver(*args, **kwargs)[source]¶

Class for running domain specific lanugage models.

after_loop()[source]¶: Actions to perform after run_loop has finished. Mainly checking if there was an error and then handling it.

base_languages = ['python']¶

executable_type = 'dsl'¶

full_language = False¶

function_param = None¶

is_dsl = True¶

classmethod is_language_installed()[source]¶

Determine if this model driver is installed on the current machine.

Returns:

Truth of if this model driver can be run on the current: machine.

Return type:

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

classmethod model_wrapper(*args, **kwargs)[source]¶: Model wrapper.

property model_wrapper_args¶

Positional arguments for the model wrapper.

Type:: tuple

property model_wrapper_kwargs¶

Keyword arguments for the model wrapper.

Type:: dict

classmethod model_wrapper_no_forward(*args, **kwargs)[source]¶

queue_close()[source]¶: Close the queue for messages from the model process.

queue_recv()[source]¶: Receive a message from the model process.

run_model(return_process=True, **kwargs)[source]¶

Run the model. Unless overridden, the model will be run using run_executable.

Parameters:

return_process (bool, optional) – If True, the process running the model is returned. If False, the process will block until the model finishes running. Defaults to True.
**kwargs – Keyword arguments are passed to run_executable.

yggdrasil.drivers.Driver module¶

class yggdrasil.drivers.Driver.Driver(*args, **kwargs)[source]¶

Bases: YggTaskLoop

Base class for all drivers.

Parameters:

name (str) – Driver name.
yml (dict, optional) – Dictionary of yaml specification options for this driver. Defaults to empty dict.
env (dict, optional) – Dictionary of environment variables that should be set when the driver starts. Defaults to {}.
comm_env (dict, optional) – Dictionary of environment variables for paired IO communication drivers. Defaults to {}.
namespace (str, optional) – Namespace for set of drivers running together. If not provided, the config option (‘rmq’, ‘namespace’) is used.
rank (int, optional) – Rank of the integration. Defaults to None.

name¶

Driver name.

Type:: str

env¶

Dictionary of environment variables.

Type:: dict

comm_env¶

Dictionary of environment variables for paired IO communication drivers.

Type:: dict

yml¶

Dictionary of yaml specification options for this driver.

Type:: dict

namespace¶

Namespace for set of drivers running together.

Type:: str

rank¶

Rank of the integration.

Type:: int

do_terminate()[source]¶: Actions that should stop the driver.

graceful_stop()[source]¶: Gracefully stop the driver.

property is_valid¶

True if the driver is functional.

Type:: bool

stop()[source]¶: Stop the driver.

terminate()[source]¶: Stop the driver, without attempting to allow it to finish.

yggdrasil.drivers.DummyModelDriver module¶

class yggdrasil.drivers.DummyModelDriver.DummyModelDriver(*args, **kwargs)[source]¶

Class that stands-in to act as a model utilizing unmatched input/output channels.

base_languages = ['python']¶

before_loop()[source]¶: Actions before loop.

before_start()[source]¶: Actions to perform before the run starts.

cfg = <yggdrasil.config.YggConfigParser object>¶

comms_implicit = True¶

classmethod configuration_steps()[source]¶

Get a list of configuration steps with tuples of flags and boolean values.

Returns:

Pairs of descriptions and states for: different steps in the configuration all steps must be True for the language to be configured.

Return type:

OrderedDict

default_interpreter = 'dummy'¶

default_interpreter_flags = []¶

executable_type = 'other'¶

full_language = False¶

inverse_type_map = None¶

classmethod is_language_installed()[source]¶

Determine if this model driver is installed on the current machine.

Returns:

Truth of if this model driver can be run on the current: machine.

Return type:

language = 'dummy'¶

language_ext = []¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

no_executable = True¶

run_loop()[source]¶: Loop to check if model is still running and forward output.

property service_partner¶

YAML representation of the dummy model that should stand-in for the model client-side.

Type:: dict

yggdrasil.drivers.DuplicatedModelDriver module¶

class yggdrasil.drivers.DuplicatedModelDriver.DuplicatedModelDriver(*args, **kwargs)[source]¶

Bases: Driver

Base class for Model drivers and for running executable based models.

Parameters:

name (str) – Unique name used to identify the model. This will be used to report errors associated with the model.
*args – Additional arguments are passed to the models in the set.
**kwargs – Additional keyword arguments are passed to the models in the set.

Attributes:

Raises:: RuntimeError – If both with_strace and with_valgrind are True.

after_loop(*args, **kwargs)[source]¶: Actions to perform after run_loop has finished.

cleanup(*args, **kwargs)[source]¶: Actions to perform to clean up the thread after it has stopped.

property errors¶

Errors returned by model copies.

Type:: list

classmethod get_base_name(name)[source]¶

Get the name of the base model.

Parameters:: name (str) – Model name.
Returns:: Base model name.
Return type:: str

classmethod get_yaml_copies(yml)[source]¶

Get a list of yamls for creating duplicate models for the model described by the provided yaml.

Parameters:: yml (dict) – Input parameters for creating a model driver.
Returns:: Copies of input parameters for creating duplicate models.
Return type:: list

graceful_stop(*args, **kwargs)[source]¶: Gracefully stop the driver.

property io_errors¶

Errors produced by input/output drivers to this model.

Type:: list

name_format = '%s_copy%d'¶

printStatus(*args, **kwargs)[source]¶: Print the class status.

run_loop()[source]¶: Loop to check if model is still running and forward output.

start(*args, **kwargs)[source]¶: Start thread/process and print info.

stop(*args, **kwargs)[source]¶: Stop the driver.

terminate(*args, **kwargs)[source]¶: Set the terminate event and wait for the thread/process to stop.

yggdrasil.drivers.ExecutableModelDriver module¶

class yggdrasil.drivers.ExecutableModelDriver.ExecutableModelDriver(*args, **kwargs)[source]¶

Bases: ModelDriver

Class for running executable based models.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

cfg = <yggdrasil.config.YggConfigParser object>¶

comms_implicit = True¶

classmethod configuration_steps()[source]¶

Get a list of configuration steps with tuples of flags and boolean values.

Returns:

Pairs of descriptions and states for: different steps in the configuration all steps must be True for the language to be configured.

Return type:

OrderedDict

classmethod executable_command(args, unused_kwargs=None, **kwargs)[source]¶

Compose a command for running a program using the exectuable for this language (compiler/interpreter) with the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it.
unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are ignored.

Returns:

Arguments composing the command required to run the program: from the command line using the executable for this language.

Return type:

executable_type = 'other'¶

full_language = False¶

inverse_type_map = None¶

classmethod is_language_installed()[source]¶

Determine if the interpreter/compiler for the associated programming language is installed.

Returns:: True if the language interpreter/compiler is installed.
Return type:: bool

language = 'executable'¶

classmethod language_version(version_flags=None, **kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

no_executable = True¶

yggdrasil.drivers.FileInputDriver module¶

class yggdrasil.drivers.FileInputDriver.FileInputDriver(*args, **kwargs)[source]¶

Bases: InputDriver

Class that sends messages read from a file.

Parameters:

name (str) – Name of the queue that messages should be sent to.
args (str) – Path to the file that messages should be read from.
**kwargs – Additional keyword arguments are passed to the parent class.

yggdrasil.drivers.FileOutputDriver module¶

class yggdrasil.drivers.FileOutputDriver.FileOutputDriver(*args, **kwargs)[source]¶

Bases: OutputDriver

Class to handle output of received messages to a file.

Parameters:

name (str) – Name of the output queue to receive messages from.
args (str) – Path to the file that messages should be written to.
**kwargs – Additional keyword arguments are passed to the parent class.

yggdrasil.drivers.FortranModelDriver module¶

class yggdrasil.drivers.FortranModelDriver.FortranCompilerBase(**kwargs)[source]¶

Bases: CompilerBase

Base class for Fortran compilers.

classmethod append_product(products, src, new, new_dir=None, dont_append_src=False)[source]¶

Append a product to the specified list along with additional values indicated by cls.product_exts.

Parameters:

products (list) – List of of existing products that new product should be appended to.
src (list) – Input arguments to compilation call that was used to generate the output file (usually one or more source files).
new (str) – New product that should be appended to the list.
new_dir (str, optional) – Directory that should be used as base when adding files listed in cls.product_files. Defaults to os.path.dirname(new).
dont_append_src (bool, optional) – If True and src is in the list of products, it will be removed. Defaults to False.

default_archiver = None¶

default_executable = None¶

default_executable_env = 'FC'¶

default_flags = ['-g', '-Wall', '-cpp', '-pedantic-errors', '-ffree-line-length-0']¶

default_flags_env = 'FFLAGS'¶

default_linker = None¶

classmethod get_flags(**kwargs)[source]¶

Get a list of flags for the tool.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Flags for the tool.
Return type:: list

languages = ['fortran']¶

linker_attributes = {'default_flags_env': 'LFLAGS', 'search_path_envvar': ['LIBRARY_PATH', 'LD_LIBRARY_PATH']}¶

product_exts = ['mod']¶

search_path_envvar = []¶

class yggdrasil.drivers.FortranModelDriver.FortranModelDriver(*args, **kwargs)[source]¶

Bases: CompiledModelDriver

Class for running Fortran models.

Parameters:

standard (str, optional) – Fortran standard that should be used. Defaults to ‘f2003’.
**kwargs – Additional keyword arguments are passed to parent class.

classmethod allows_realloc(var, from_native_type=False)[source]¶

Determine if a variable allows the receive call to perform realloc.

Parameters:

var (dict) – Dictionary of variable properties.
from_native_type (bool, optional) – If True, the reallocability of the variable will be determined from the native type. Defaults to False.

Returns:

True if the variable allows realloc, False otherwise.

Return type:

base_languages = ['c']¶

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

cfg = <yggdrasil.config.YggConfigParser object>¶

compile_model(**kwargs)[source]¶

Compile model executable(s).

Parameters:

**kwargs – Keyword arguments are passed to the parent class’s
method. –

Returns:

Compiled model file path.

Return type:

default_archiver = 'ar'¶

default_archiver_flags = None¶

default_compiler = 'gfortran'¶

default_compiler_flags = None¶

default_linker = 'gfortran'¶

default_linker_flags = None¶

default_type_precision = {'integer': 4, 'real': 4}¶

classmethod escape_quotes(x)[source]¶

Escape quotes in a string.

Parameters:: x (str) – String to escape quotes in.
Returns:: x with escaped quotes.
Return type:: str

external_libraries = {}¶

classmethod format_function_param_index(extra=None, **kwargs)[source]¶

Return the formatted version of the index key.

Parameters:

extra (dict, optional) – Variable dictionary specifying the variable name, datatype, etc. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns:

Formatted string.

Return type:

classmethod format_function_param_len(extra=None, **kwargs)[source]¶

Return the formatted version of the len key.

Parameters:

extra (dict, optional) – Variable dictionary specifying the variable name, datatype, etc. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns:

Formatted string.

Return type:

function_param = {'and': '.and.', 'assign': '{name} = {value}', 'block_end': 'END', 'break': 'EXIT', 'comment': '!', 'continuation_after': ' &', 'continuation_before': '&', 'continuation_break': (',', '{', ' '), 'copy_any': 'call copy_generic_into({name}, {value})', 'copy_array': '{name} = yggarr(copy_generic(ygggeneric({value})))', 'copy_class': '{name} = copy_python({value})', 'copy_function': '{name} = yggpyfunc(copy_python(yggpython({value})))', 'copy_generic': 'call copy_generic_into({name}, {value})', 'copy_instance': '{name} = yggpyinst(copy_generic(ygggeneric({value})))', 'copy_obj': '{name} = copy_obj({value})', 'copy_object': '{name} = yggmap(copy_generic(ygggeneric({value})))', 'copy_ply': '{name} = copy_ply({value})', 'copy_schema': '{name} = yggschema(copy_generic(ygggeneric({value})))', 'declare': '{type_name} :: {variable}', 'definition_regex': '\\s*(?P<type>(?:(?: )|(?:.))+)\\s*::\\s*(?P<name>.+)(?:\\s*=\\s*(?P<value>.+))?\\n', 'error': 'write(*, \'("{error_msg}")\')\nstop 1', 'escaped_double_quote': '""', 'exec_begin': 'PROGRAM main\n use iso_c_binding\n use fygg', 'exec_end': ' stop\nEND PROGRAM main', 'false': '.false.', 'flag_cond': '{flag_var}', 'for_begin': 'DO {iter_var} = {iter_begin}, {iter_end}', 'for_end': 'END DO', 'fprintf': 'write(*, \'("{message}")\') {variables}', 'free': 'DEALLOCATE({variable})', 'free_any': 'call free_generic({variable})', 'free_array': 'call free_generic(ygggeneric({variable}))', 'free_class': 'call free_python({variable})', 'free_function': 'call free_python(yggpython({variable}))', 'free_instance': 'call free_generic(ygggeneric({variable}))', 'free_obj': 'call free_obj({variable})', 'free_object': 'call free_generic(ygggeneric({variable}))', 'free_ply': 'call free_ply({variable})', 'free_schema': 'call free_generic(ygggeneric({variable}))', 'function_call_noout': 'call {function_name}({input_var})', 'function_def_begin': 'FUNCTION {function_name}({input_var}) result({output_var})', 'function_def_end': 'END FUNCTION {function_name}', 'function_def_regex': '(?P<procedure_type>(?i:(?:subroutine)|(?:function)))\\s+{function_name}\\s*\$(?P<inputs>(?:[^\\(]*?))\$\\s*(?:result\\s*\$(?P<flag_var>.+)\$)?\\s*\\n(?P<preamble>(?:[^:]*?\\n)*?)(?P<definitions>(?:(?:(?: )|(?:.))+\\s*::\\s*(?:.+)\\n)+)(?P<body>(?:.*?\\n?)*?)(?i:end\\s+(?P=procedure_type))\\s+{function_name}', 'if_begin': 'IF ({cond}) THEN', 'if_elif': 'ELSE IF ({cond}) THEN', 'if_else': 'ELSE', 'if_end': 'END IF', 'import': '#include "{filename}"', 'import_nofile': 'use {function}', 'indent': ' ', 'index': '{variable}({index})', 'init_any': 'init_generic()', 'init_array': 'yggarr(init_generic())', 'init_class': 'init_python()', 'init_function': 'yggpyfunc(init_python())', 'init_instance': 'yggpyinst(init_generic())', 'init_obj': 'init_obj()', 'init_object': 'yggmap(init_generic())', 'init_ply': 'init_ply()', 'init_schema': 'yggschema(init_generic())', 'init_type_1darray': 'create_dtype_1darray("{subtype}", {precision}, {length}, "{units}", {use_generic})', 'init_type_array': 'create_dtype_json_array({nitems}, {items}, {use_generic})', 'init_type_default': 'create_dtype_default("{type}", {use_generic})', 'init_type_empty': 'create_dtype_empty({use_generic})', 'init_type_from_schema': 'create_dtype_from_schema("{schema}", {use_generic})', 'init_type_ndarray': 'create_dtype_ndarray("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_ndarray_arr': 'create_dtype_ndarray("{subtype}", {precision}, {ndim}, {shape}, "{units}", {use_generic})', 'init_type_obj': 'create_dtype_obj({use_generic})', 'init_type_object': 'create_dtype_json_object({nitems}, {keys}, {values}, {use_generic})', 'init_type_ply': 'create_dtype_ply({use_generic})', 'init_type_pyobj': 'create_dtype_pyobj("{type}", {use_generic})', 'init_type_scalar': 'create_dtype_scalar("{subtype}", {precision}, "{units}", {use_generic})', 'init_type_schema': 'create_dtype_schema({use_generic})', 'input': '{channel} = ygg_input_type("{channel_name}", {channel_type})', 'inputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)(?:\\n)?', 'interface': 'use fygg', 'len': 'size({variable},1)', 'not': '.not.', 'not_flag_cond': '.not.{flag_var}', 'null': 'c_null_ptr', 'or': '.or.', 'output': '{channel} = ygg_output_type("{channel_name}", {channel_type})', 'outputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)(?:\\n)?', 'print': 'write(*, \'("{message}")\')', 'print_any': 'call display_generic({object})', 'print_array': 'call display_generic(ygggeneric({object}))', 'print_class': 'call display_python({object})', 'print_function': 'call display_python(yggpython({object}))', 'print_generic': 'write(*, *) {object}', 'print_instance': 'call display_generic(ygggeneric({object}))', 'print_null': 'call display_null({object})', 'print_obj': 'call display_obj({object})', 'print_object': 'call display_generic(ygggeneric({object}))', 'print_ply': 'call display_ply({object})', 'print_schema': 'call display_generic(ygggeneric({object}))', 'python_init': 'call init_python_API()', 'quote': "'", 'recv_function': 'ygg_recv_var_realloc', 'recv_heap': 'ygg_recv_var_realloc', 'recv_stack': 'ygg_recv_var', 'send_function': 'ygg_send_var', 'subroutine_def_begin': 'SUBROUTINE {function_name}({input_var}) ', 'subroutine_def_end': 'END SUBROUTINE {function_name}', 'true': '.true.', 'type_regex': '(type\$)?(?P<type>[^,\\(]+)(?(1)(?:\$))(?:\\s*\$\\s*(?:kind\\s*=\\s*(?:(?P<precision>\\d*)|(?P<precision_var>.+?)))?\\s*,?\\s*(?:len\\s*=\\s*(?:(?P<length>(?:\\d+))|(?P<length_var>.+?)))?\$)?(?:\\s*,\\s*dimension\$(?:(?P<shape>(?:\\d)+(?:,\\s*(?:\\d)+)*?)|(?P<shape_var>.+?(?:,\\s*.+)*?))\$)?(?:\\s*,\\s*(?P<pointer>pointer))?(?:\\s*,\\s*(?P<target>target))?(?:\\s*,\\s*(?P<allocatable>allocatable))?(?:\\s*,\\s*(?P<parameter>parameter))?(?:\\s*,\\s*intent\$(?P<intent>.*?)\$)?', 'while_begin': 'DO WHILE ({cond})', 'while_end': 'END DO'}¶

classmethod get_internal_suffix(commtype=None, **kwargs)[source]¶

Determine the suffix that should be used for internal libraries.

Parameters:

commtype (str, optional) – If provided, this is the communication type that should be used for the model. If None, the default comm is used.
**kwargs – Additional keyword arguments will be passed to the parent class’s method.

Returns:

Suffix that should be added to internal libraries to: differentiate between different dependencies.

Return type:

classmethod get_inverse_type_map()[source]¶

Get the inverse type map.

Returns:: Mapping from native type to JSON type.
Return type:: dict

classmethod get_json_type(native_type)[source]¶

Get the JSON type from the native language type.

Parameters:: native_type (str) – The native language type.
Returns:: The JSON type.
Return type:: str, dict

classmethod get_native_type(**kwargs)[source]¶

Get the native type.

Parameters:

type (str, optional) – Name of yggdrasil extended JSON type or JSONSchema dictionary defining a datatype.
**kwargs – Additional keyword arguments may be used in determining the precise declaration that should be used.

Returns:

The native type.

Return type:

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

global_scope_macro = '#define WITH_GLOBAL_SCOPE(COMM) call set_global_comm(); COMM; call unset_global_comm()'¶

include_channel_obj = True¶

interface_inside_exec = True¶

interface_library = 'fygg'¶

interface_map = {'call': 'flag = ygg_rpc_call({channel_obj}, [yggarg({outputs})], [yggarg({inputs})])', 'client': 'ygg_rpc_client_type("{channel_name}", {datatype_out}, {datatype_in})', 'import': 'use fygg', 'input': 'ygg_input_type("{channel_name}", {datatype})', 'output': 'ygg_output_type("{channel_name}", {datatype})', 'recv': 'flag = ygg_recv_var({channel_obj}, [yggarg({inputs})])', 'send': 'flag = ygg_send_var({channel_obj}, [yggarg({outputs})])', 'server': 'ygg_rpc_server_type("{channel_name}", {datatype_in}, {datatype_out})', 'timesync': 'yggTimesync("{channel_name}", "{time_units}")'}¶

internal_libraries = {'c_wrappers': {'external_dependencies': [('c', 'rapidjson'), ('c', 'python'), ('c', 'numpy'), ('c', 'm')], 'include_dirs': ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/fortran'], 'internal_dependencies': [('c', 'ygg')], 'language': 'c', 'libtype': 'object', 'source': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/fortran/c_wrappers.c'}, 'fygg': {'external_dependencies': [('c', 'rapidjson'), ('c', 'python'), ('c', 'numpy'), ('c', 'm'), 'stdc++'], 'include_dirs': ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/communication', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C/serialize', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/C'], 'internal_dependencies': [('c', 'regex'), ('c', 'datatypes'), ('c', 'ygg'), 'c_wrappers'], 'libtype': 'static', 'source': '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/fortran/YggInterface.f90'}}¶

inverse_type_map = None¶

is_typed = True¶

language = 'fortran'¶

language_ext = ['.f90', '.f77', '.f', '.h']¶

locked_buildfile = 'fygg.mod'¶

max_line_width = 72¶

outputs_in_inputs = True¶

classmethod parse_function_definition(model_file, model_function, **kwargs)[source]¶

Get information about the inputs & outputs to a model from its defintition if possible.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
**kwargs – Additional keyword arugments are passed to the parent class’s method.

Returns:

Parameters extracted from the function definitions.

Return type:

classmethod prepare_variables(vars_list, for_yggdrasil=False, **kwargs)[source]¶

Concatenate a set of input variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing input/output to/from a function call.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Concatentated variables list.

Return type:

classmethod set_env_class(**kwargs)[source]¶

Set environment variables that are instance independent.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method and update_ld_library_path.
Returns:: Environment variables for the model process.
Return type:: dict

classmethod split_line(line, length=None, force_split=False)[source]¶

Split a line as close to (or before) a given character as possible.

Parameters:

line (str) – Line to split.
length (int, optional) – Maximum length of split lines. Defaults to cls.max_line_width if not provided.
force_split (bool, optional) – If True, force a split to occur at the specified length. Defauts to False.

Returns:

Set of lines resulting from spliting the provided line.

Return type:

standard_libraries = ['stdc++']¶

supported_comm_options = {'ipc': {'platforms': ['MacOS', 'Linux']}, 'zmq': {'libraries': [('c', 'zmq'), ('c', 'czmq')]}}¶

type_map = {'1darray': '*', '1darray_pointer': '{type}{precision}_1d', 'any': 'ygggeneric', 'array': 'yggarr', 'boolean': 'logical(kind = X)', 'bytes': 'character(len = X)', 'class': 'yggpython', 'comm': 'yggcomm', 'complex': 'complex(kind = X)', 'dtype': 'yggdtype', 'flag': 'logical', 'float': 'real(kind = X)', 'function': 'yggpyfunc', 'instance': 'yggpyinst', 'int': 'integer(kind = X)', 'integer': 'integer', 'ndarray': '*', 'ndarray_pointer': '{type}{precision}_{ndim}d', 'null': 'yggnull', 'number': 'real(kind = 8)', 'obj': 'yggobj', 'object': 'yggmap', 'ply': 'yggply', 'schema': 'yggschema', 'string': 'character(len = X)', 'uint': 'ygguintX', 'unicode': "character(kind = selected_char_kind('ISO_10646'), len = X)"}¶

types_in_funcdef = False¶

classmethod update_io_from_function(model_file, model_function, **kwargs)[source]¶

Update inputs/outputs from the function definition.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Flag variable used by the model. If None, the: model does not use a flag variable.

Return type:

dict, None

classmethod write_declaration(var, **kwargs)[source]¶

Return the lines required to declare a variable with a certain type.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines declaring the variable.

Return type:

classmethod write_executable(lines, **kwargs)[source]¶

Return the lines required to complete a program that will run the provided lines.

Parameters:

lines (list) – Lines of code to be wrapped as an executable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines of code wrapping the provided lines with the: necessary code to run it as an executable (e.g. C/C++’s main).

Return type:

lines

classmethod write_executable_import(module=False, **kwargs)[source]¶

Add import statements to executable lines.

Parameters:

module (str, optional) – If provided, the include statement importing code will be wrapped in a module of the provided name. Defaults to False and the include will not be wrapped.
**kwargs – Keyword arguments for import statement.

Returns:

Lines required to complete the import.

Return type:

classmethod write_function_def(function_name, **kwargs)[source]¶

Write a function definition.

Parameters:

function_name (str) – Name fo the function being defined.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines completing the function call.

Return type:

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

classmethod write_model_recv(channel, recv_var, **kwargs)[source]¶

Write a model receive call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being received from was stored in.
recv_var (dict, list) – Information of one or more variables that receieved information should be stored in.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to carry out a receive call in this language.

Return type:

classmethod write_model_send(channel, send_var, **kwargs)[source]¶

Write a model send call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being sent to was stored in.
send_var (dict, list) – Information on one or more variables containing information that will be sent.
flag_var (str, optional) – Name of flag variable that the flag should be stored in. Defaults to ‘flag’,
allow_failure (bool, optional) – If True, the returned lines will call a break if the flag is False. Otherwise, the returned lines will issue an error. Defaults to False.

Returns:

Lines required to carry out a send call in this language.

Return type:

classmethod write_print_var(var, **kwargs)[source]¶

Get the lines necessary to print a variable in this language.

Parameters:

var (dict) – Variable information.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines printing the specified variable.

Return type:

classmethod write_type_decl(name, datatype, **kwargs)[source]¶

Get lines declaring the datatype within the language.

Parameters:

name (str) – Name of variable that should be declared.
datatype (dict) – Type definition.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to define a type declaration.

Return type:

classmethod write_type_def(name, datatype, **kwargs)[source]¶

Get lines declaring the data type within the language.

Parameters:

name (str) – Name of variable that definition should be stored in.
datatype (dict) – Type definition.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines required to define a type definition.

Return type:

zero_based = False¶

class yggdrasil.drivers.FortranModelDriver.GFortranCompiler(**kwargs)[source]¶

Bases: FortranCompilerBase

Interface class for gfortran compiler/linker.

combine_with_linker = True¶

compatible_toolsets = ['gnu', 'llvm']¶

default_archiver = 'ar'¶

default_executable = 'gfortran'¶

default_flags = ['-g', '-Wall', '-cpp', '-pedantic-errors', '-ffree-line-length-0', '-x', 'f95-cpp-input']¶

default_linker = 'gfortran'¶

flag_options = {'definitions': '-D%s', 'include_dirs': '-I%s', 'module-dir': '-J%s', 'module-search-path': '-I%s', 'standard': '-std=%s'}¶

is_gnu = True¶

platforms = ['MacOS', 'Linux', 'Windows']¶

toolname = 'gfortran'¶

toolset = 'gnu'¶

yggdrasil.drivers.InputDriver module¶

class yggdrasil.drivers.InputDriver.InputDriver(*args, **kwargs)[source]¶

Driver for receiving input from another model’s comm via a local comm.

Parameters:

name (str) – The name of the local message comm that the driver should connect to.
args (str) – The name of the other message comm that the driver should connect to.
**kwargs – Additional keyword arguments are passed to parent class.

yggdrasil.drivers.InterpretedModelDriver module¶

class yggdrasil.drivers.InterpretedModelDriver.InterpretedModelDriver(*args, **kwargs)[source]¶

Bases: ModelDriver

Base class for models written in interpreted languages.

Parameters:

name (str) – Driver name.
args (str or list) – Argument(s) for running the model on the command line. This driver will check to see if there is an interpreter included in the args provided. If not, one will be added unless skip_interpreter is True.
interpreter (str, optional) – Name or path of interpreter executable that should be used to run the model. If not provided, the interpreter will be determined based on configuration options for the language (if present) and the default_interpreter class attribute.
interpreter_flags (list, optional) – Flags that should be passed to the interpreter when running the model. If not provided, the flags are determined based on configuration options for the language (if present) and the default_interpreter_flags class attribute.
skip_interpreter (bool, optional) – If True, no interpreter will be added to the arguments. This should only be used for subclasses that will not be invoking the model via the command line. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class.

Class Attributes:

default_interpreter (str): Name of interpreter that will be used if not: set explicitly by instance or config file. Defaults to the language name if not set.
default_interpreter_flags (list): Flags that will be passed to the: interpreter when running the model by default if not set explicitly by instance or config file.

interpreter¶

Name or path to the interpreter that will be used.

Type:: str

interpreter_flags¶

Flags that will be passed to the interpreter when running a model.

Type:: list

path_env_variable¶

Name of the environment variable containing path information for the interpreter for this language.

Type:: str

paths_to_add¶

Paths that should be added to the path_env_variable for this language on the process the model is run in.

Type:: list

comm_atexit¶

Function taking a comm instance as input that performs any necessary operations during exit. If None, no additional actions are taken.

Type:: function

comm_linger¶

If True, interface comms will linger during close. This should only be required if the language will disreguard Python threads at exit (e.g. when using a Matlab engine).

Type:: bool

decode_format (function: Function decoding format string created in this language. If None, no additional actions are taken.

recv_converters¶

Mapping between the names of message types (e.g. ‘array’, ‘pandas’) and functions that should be used to prepare such objects for return when they are received.

Type:: dict

send_converters¶

Mapping between the names of message types (e.g. ‘array’, ‘pandas’) and functions that should be used to prepare such objects for sending.

Type:: dict

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

classmethod are_dependencies_installed(**kwargs)[source]¶

Determine if the dependencies are installed for the interface (not including dependencies needed by a particular communication type).

Returns:: True if the dependencies are installed. False otherwise.
Return type:: bool

comm_atexit = None¶

comm_linger = False¶

decode_format = None¶

default_interpreter = None¶

default_interpreter_flags = []¶

classmethod executable_command(args, exec_type='interpreter', unused_kwargs=None, skip_interpreter_flags=False, interpreter=None, interpreter_flags=None, **kwargs)[source]¶

Compose a command for running a program in this language with the provied arguments. If not already present, the interpreter command and interpreter flags are prepended to the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it.
exec_type (str, optional) – Type of executable command that will be returned. If ‘interpreter’, a command using the interpreter is returned and if ‘direct’, the raw args being provided are returned. Defaults to ‘interpreter’.
skip_interpreter_flags (bool, optional) – If True, interpreter flags will not be added to the command after the interpreter. Defaults to False. Interpreter flags will not be added, reguardless of this keyword, if the first element of args is already an interpreter.
unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to {}.
**kwargs – Additional keyword arguments are ignored.

Returns:

Arguments composing the command required to run the program: from the command line using the interpreter for this language.

Return type:

Raises:

ValueError – If exec_type is not ‘interpreter’ or ‘direct’.

executable_type = 'interpreter'¶

classmethod get_interpreter()[source]¶

Command required to run a model written in this language from the command line.

Returns:: Name of (or path to) interpreter executable.
Return type:: str

classmethod get_interpreter_flags()[source]¶

Get the flags that should be passed to the interpreter when using it to run a model on the command line.

Returns:

Flags that should be passed to the interpreter on the command: line.

Return type:

classmethod is_interpreter(cmd)[source]¶

Determine if a command line argument is an interpreter.

Parameters:: cmd (str) – Command that should be checked.
Returns:: True if the command is an interpreter, False otherwise.
Return type:: bool

classmethod language2python(pyobj)[source]¶

Prepare an object from the target language for receipt in Python.

Parameters:

pyobj (object) – Python object transformed from the target language.

Returns:

Python object in a form that conforms with the: expected Python type.

Return type:

classmethod language_executable()[source]¶

Command required to compile/run a model written in this language from the command line.

Returns:

Name of (or path to) compiler/interpreter executable required: to run the compiler/interpreter from the command line.

Return type:

parse_arguments(*args, **kwargs)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

path_env_variable = None¶

paths_to_add = ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/MATLAB', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/SBML', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/Python', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/R', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/LPy', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/dummy', '/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/Julia']¶

classmethod python2language(pyobj)[source]¶

Prepare a python object for transformation in target language.

Parameters:

pyobj (object) – Python object.

Returns:

Python object in a form that is friendly to the: target language.

Return type:

recv_converters = {}¶

run_model(*args, **kwargs)[source]¶

Run the model. Unless overridden, the model will be run using run_executable.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

send_converters = {}¶

set_env(**kwargs)[source]¶

Get environment variables that should be set for the model process.

Returns:: Environment variables for the model process.
Return type:: dict

yggdrasil.drivers.JuliaModelDriver module¶

class yggdrasil.drivers.JuliaModelDriver.JuliaModelDriver(*args, **kwargs)[source]¶

Class for running Julia models.

base_languages = ['python']¶

cfg = <yggdrasil.config.YggConfigParser object>¶

default_interpreter = 'julia'¶

default_interpreter_flags = []¶

function_param = {'and': '&&', 'assign': 'global {name} = {value}', 'block_end': 'end', 'break': 'break', 'comment': '#', 'error': 'error("{error_msg}")', 'false': 'false', 'first_index': 1, 'for_begin': 'for {iter_var} in {iter_begin}:{iter_end}', 'fprintf': '@printf("{message}", {variables})', 'function_def_begin': 'function {function_name}({input_var})', 'function_def_regex': '\\n?( *)function +{function_name}(?P<outtype>\\:\\:[^\$]+?)? *\\((?P<inputs>(?:[^\$]+?)?)\\) *(?P<body>(?:(?:\\1(?: *)(?!return)[^ ].*?\\n)|(?: *\\n))*)(?:\\1(?: *)(?:return *)?(?P<outputs>.*?) *\\n)?', 'if_begin': 'if {cond}', 'if_elif': 'elseif {cond}', 'if_else': 'else', 'import': 'include("{filename}")', 'import_nofile': 'using {function}', 'indent': ' ', 'index': '{variable}[{index}]', 'input': '{channel} = Yggdrasil.YggInterface("YggInput", "{channel_name}")', 'inputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'interface': 'using {interface_library}', 'istype': '{variable} isa {type}', 'len': 'length({variable})', 'not': '!', 'or': '||', 'output': '{channel} = Yggdrasil.YggInterface("YggOutput", "{channel_name}")', 'outputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'print': 'println("{message}")', 'print_generic': 'println({object})', 'python_interface': '{channel} = Yggdrasil.YggInterface("{python_interface}", "{channel_name}")', 'python_interface_format': '{channel} = Yggdrasil.YggInterface("{python_interface}", "{channel_name}", "{format_str}")', 'quote': '"', 'recv_function': '{channel}.recv', 'return': 'return {output_var}', 'send_function': '{channel}.send', 'true': 'true', 'try_begin': 'try', 'try_except': 'catch', 'while_begin': 'while {cond}'}¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_dependencies = ['PyCall', 'Unitful']¶

interface_library = 'Yggdrasil'¶

interface_map = {'call': 'flag, {inputs} = {channel_obj}.call({outputs})', 'client': 'Yggdrasil.YggInterface("YggRpcClient", "{channel_name}")', 'import': 'import Yggdrasil: commtype', 'input': 'Yggdrasil.YggInterface("YggInput", "{channel_name}")', 'output': 'Yggdrasil.YggInterface("YggOutput", "{channel_name}")', 'recv': 'flag, {inputs} = {channel_obj}.recv()', 'send': 'flag = {channel_obj}.send({outputs})', 'server': 'Yggdrasil.YggInterface("YggRpcServer", "{channel_name}")', 'timesync': 'Yggdrasil.YggInterface("YggTimesync", "{channel_name}")'}¶

inverse_type_map = None¶

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

language = 'julia'¶

language_ext = ['.jl']¶

classmethod set_env_class(**kwargs)[source]¶

Get environment variables that should be set for the model process.

Returns:: Environment variables for the model process.
Return type:: dict

type_map = {'1darray': 'Array{X}', 'array': 'Array', 'boolean': 'Bool', 'bytes': 'CodeUnits{UInt8,String}', 'complex': 'Complex', 'float': 'Float', 'int': 'Int', 'ndarray': 'Array{X}', 'null': 'Nothing', 'obj': 'ObjDict', 'object': 'Dict{AbstractString,Any}', 'ply': 'PlyDict', 'schema': 'Dict', 'string': 'String', 'uint': 'Uint', 'unicode': 'String'}¶

classmethod write_executable_import(**kwargs)[source]¶

Add import statements to executable lines.

Parameters:: **kwargs – Keyword arguments for import statement.
Returns:: Lines required to complete the import.
Return type:: list

classmethod write_finalize_oiter(var)[source]¶

Get the lines necessary to finalize an array after iteration.

Parameters:: var (dict, str) – Name or information dictionary for the variable being initialized.
Returns:: The lines finalizing the variable.
Return type:: list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

zero_based = False¶

yggdrasil.drivers.LPyModelDriver module¶

class yggdrasil.drivers.LPyModelDriver.LPyModelDriver(*args, **kwargs)[source]¶

Bases: PythonModelDriver

Class for running LPy models.

cfg = <yggdrasil.config.YggConfigParser object>¶

default_interpreter = '/opt/hostedtoolcache/Python/3.8.17/x64/bin/python'¶

default_interpreter_flags = ['/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/yggdrasil/languages/LPy/lpy_model.py']¶

executable_type = 'dsl'¶

full_language = False¶

function_param = None¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_dependencies = ['openalea.lpy']¶

inverse_type_map = None¶

is_dsl = True¶

language = 'lpy'¶

language_ext = ['.lpy']¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

supported_comms = ['rmq_async', 'ipc', 'buffer', 'rest', 'value', 'rmq', 'mpi', 'zmq']¶

yggdrasil.drivers.MPIPartnerModel module¶

class yggdrasil.drivers.MPIPartnerModel.MPIPartnerModel(*args, **kwargs)[source]¶

Bases: ModelDriver

Class for shadowing a model run on another MPI process.

base_languages = []¶

before_start(**kwargs)[source]¶: Actions to perform before the run starts.

cfg = <yggdrasil.config.YggConfigParser object>¶

cleanup(*args, **kwargs)[source]¶: Remove compile executable.

comms_implicit = True¶

classmethod configuration_steps()[source]¶

Get a list of configuration steps with tuples of flags and boolean values.

Returns:

Pairs of descriptions and states for: different steps in the configuration all steps must be True for the language to be configured.

Return type:

OrderedDict

executable_type = 'other'¶

full_language = False¶

graceful_stop()[source]¶: Gracefully stop the driver.

init_mpi()[source]¶: Initialize MPI communicator.

inverse_type_map = None¶

classmethod is_comm_installed(**kwargs)[source]¶

Determine if a comm is installed for the associated programming language.

Parameters:: **kwargs – Keyword arguments are ignored.
Returns:: True if a comm is installed for this language.
Return type:: bool

classmethod is_language_installed()[source]¶

Determine if this model driver is installed on the current machine.

Returns:

Truth of if this model driver can be run on the current: machine.

Return type:

kill_process()[source]¶: Kill the process running the model, checking return code.

language = 'mpi'¶

classmethod language_executable(**kwargs)[source]¶

Command required to compile/run a model written in this language from the command line.

Returns:

Name of (or path to) compiler/interpreter executable required: to run the compiler/interpreter from the command line.

Return type:

language_ext = []¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

property model_process_complete¶

Has the process finished or not. Returns True if the process has not started.

Type:: bool

run_loop()[source]¶: Loop to check if model is still running.

run_model(**kwargs)[source]¶: Dummy stand-in for ModelDriver run_model method.

stop_mpi_partner(**kwargs)[source]¶: Send a message to stop the MPI partner model on the main process.

yggdrasil.drivers.MakeModelDriver module¶

class yggdrasil.drivers.MakeModelDriver.MakeCompiler(**kwargs)[source]¶

Bases: BuildToolBase

Make configuration tool.

Parameters:

makefile (str, optional) – Path to make file either absolute, relative to makedir (if provided), or relative to working_dir. Defaults to Makefile.
makedir (str, optional) – Directory where make should be invoked from if it is not the same as the directory containing the makefile. Defaults to directory containing makefile if provided, otherwise working_dir.
**kwargs – Additional keyword arguments are passed to parent class.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

build_language = 'make'¶

combine_with_linker = True¶

compile_only_flag = None¶

default_archiver = False¶

default_executable = 'make'¶

default_flags = ['--always-make']¶

default_linker = 'make'¶

flag_options = {'makefile': {'key': '-f', 'position': 0}}¶

classmethod get_executable_command(args, target=None, **kwargs)[source]¶

Determine the command required to run the tool using the specified arguments and options.

Parameters:

args (list) – The arguments that should be passed to the tool. If skip_flags is False, these are treated as input files that will be used by the tool.
target (str, optional) – Target that should be built. Defaults to to None and is set to the base name of first element in the provided arguments.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Output to stdout from the command execution.

Return type:

classmethod get_flags(target=None, **kwargs)[source]¶

Get compilation flags, replacing outfile with target.

Parameters:

target (str, optional) – Target that should be built. Defaults to to None and is ignored.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Compiler flags.

Return type:

classmethod get_output_file(src, target=None, **kwargs)[source]¶

Determine the appropriate output file that will result when compiling a target.

Parameters:

src (str) – Make target or source file being compiled that will be used to determine the path to the output file.
target (str, optional) – Target that will be used to create the output file instead of src if provided. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Full path to file that will be produced.

Return type:

is_gnu = False¶

is_linker = True¶

languages = ['make']¶

linker_attributes = {'executable_ext': '', 'tool_suffix_format': ''}¶

no_separate_linking = True¶

output_key = ''¶

platforms = ['MacOS', 'Linux']¶

classmethod tool_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.call.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

toolname = 'make'¶

class yggdrasil.drivers.MakeModelDriver.MakeModelDriver(*args, **kwargs)[source]¶

Bases: BuildModelDriver

Class for running make file compiled drivers. Before running the make command, the necessary compiler & linker flags for the interface’s C/C++ library are stored the environment variables CFLAGS and LDFLAGS respectively. These should be used in the make file to correctly compile with the interface’s C/C++ libraries.

Parameters:

name (str) – Driver name.
args (str, list) – Executable that should be created (make target) and any arguments for the executable.
makefile (str, optional) – Path to make file either absolute, relative to makedir (if provided), or relative to working_dir. Defaults to Makefile.
makedir (str, optional) – Directory where make should be invoked from if it is not the same as the directory containing the makefile. Defaults to directory containing makefile if provided, otherwise working_dir.
**kwargs – Additional keyword arguments are passed to parent class.

makefile¶

Path to make file either relative to makedir or absolute.

Type:: str

makedir¶

Directory where make should be invoked from.

Type:: str

Raises:: RuntimeError – If neither the IPC or ZMQ C libraries are available.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

buildfile_base = 'Makefile'¶

built_where_called = True¶

cfg = <yggdrasil.config.YggConfigParser object>¶

compile_model(target=None, **kwargs)[source]¶

Compile model executable(s).

Parameters:

target (str, optional) – Target to build.
**kwargs – Keyword arguments are passed on to the parent class’s method.

default_archiver = None¶

default_archiver_flags = None¶

default_compiler = 'make'¶

default_compiler_flags = None¶

default_linker = 'make'¶

default_linker_flags = None¶

classmethod fix_path(path, for_env=False, **kwargs)[source]¶

Update a path.

Parameters:

path (str) – Path that should be formatted.
for_env (bool, optional) – If True, the path is formatted for use in and environment variable. Defaults to False.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Updated path.

Return type:

classmethod get_buildfile_lock(**kwargs)[source]¶: Get a lock for a buildfile to prevent simultaneous access, creating one as necessary.

classmethod get_language_for_buildfile(buildfile, target=None)[source]¶

Determine the target language based on the contents of a build file.

Parameters:

buildfile (str) – Full path to the build configuration file.
target (str, optional) – Target that will be built. Defaults to None and the default target in the build file will be used.

classmethod get_target_language_info(*args, **kwargs)[source]¶

Get a dictionary of information about language compilation tools.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

Returns:

Information about language compilers and linkers.

Return type:

inverse_type_map = None¶

language = 'make'¶

parse_arguments(args, **kwargs)[source]¶

Sort arguments based on their syntax to determine if an argument is a source file, compilation flag, or runtime option/flag that should be passed to the model executable.

Parameters:

args (list) – List of arguments provided.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

class yggdrasil.drivers.MakeModelDriver.NMakeCompiler(**kwargs)[source]¶

Bases: MakeCompiler

compile_only_flag = None¶

default_executable = 'nmake'¶

default_flags = ['/NOLOGO']¶

default_linker = 'nmake'¶

flag_options = {'makefile': '/f'}¶

is_gnu = False¶

is_linker = True¶

platforms = ['Windows']¶

toolname = 'nmake'¶

yggdrasil.drivers.MatlabModelDriver module¶

class yggdrasil.drivers.MatlabModelDriver.MatlabModelDriver(*args, **kwargs)[source]¶

Base class for running Matlab models.

Parameters:

name (str) – Driver name.
args (str or list) – Argument(s) for running the model in matlab. Generally, this should be the full path to a Matlab script.
use_symunit (bool, optional) – If True, input/output variables with units will be represented in Matlab using symunit. Defaults to False.
**kwargs – Additional keyword arguments are passed to parent class’s __init__ method.

started_matlab¶

True if the driver had to start a new matlab engine. False otherwise.

Type:: bool

screen_session¶

Screen session that Matlab was started in.

Type:: str

mlengine¶

Matlab engine used to run script.

Type:: object

mlsession¶

Name of the Matlab session that was started.

Type:: str

Raises:: RuntimeError – If Matlab is not installed.

Note

Matlab models that call exit will shut down the shared engine.

after_loop()[source]¶: Actions to perform after run_loop has finished. Mainly checking if there was an error and then handling it.

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

base_languages = ['python']¶

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

before_start()[source]¶: Actions to perform before the run loop.

cfg = <yggdrasil.config.YggConfigParser object>¶

check_exits()[source]¶

Check to make sure the program dosn’t contain any exits as exits will shut down the Matlab engine as well as the program.

Raises:: RuntimeError – If there are any exit calls in the file.

cleanup()[source]¶: Close the Matlab session and engine.

classmethod comm_atexit(comm)[source]¶

Operations performed on comm at exit including draining receive.

Parameters:: comm (CommBase) – Communication object.

comm_linger = False¶

classmethod configure(cfg, disable_engine=None, hide_matlab_libiomp=None)[source]¶

Add configuration options for this language. This includes locating any required external libraries and setting option defaults.

Parameters:

cfg (YggConfigParser) – Config class that options should be set for.
disable_engine (bool, optional) – Disable the use of the Python engine for Matlab. If not provided, this argument will be ignored.
hide_matlab_libiomp (bool, optional) – Hide the Matlab version of libiomp by moving it and recording the name of the file. If False, the file will be restored if it exists. If not provided, this argument will be ignored.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod decode_format(format_str)[source]¶

Method for decoding format strings created in this language.

Parameters:: format_str (str) – Encoded format string.
Returns:: Decoded format string.
Return type:: str

default_interpreter = 'matlab'¶

default_interpreter_flags = ['-nodisplay', '-nosplash', '-nodesktop', '-nojvm', '-r']¶

classmethod executable_command(args, **kwargs)[source]¶

Compose a command for running a program in this language with the provied arguments. If not already present, the interpreter command and interpreter flags are prepended to the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it.
**kwargs – Additional keyword arguments are ignored.

Returns:

Arguments composing the command required to run the program: from the command line using the interpreter for this language.

Return type:

function_param = {'and': '&&', 'assign': '{name} = {value};', 'block_end': 'end', 'break': 'break;', 'comment': '%', 'eol': ';', 'error': "error('{error_msg}');", 'expand_mult': '{name} = {value}{{:}};', 'false': 'false', 'first_index': 1, 'for_begin': 'for {iter_var} = {iter_begin}:{iter_end}', 'fprintf': "fprintf('{message}', {variables});", 'function_def_begin': 'function {output_var} = {function_name}({input_var})', 'function_def_regex': 'function *(\\[ *)?(?P<outputs>.*?)(?(1)\\]) *= *{function_name} *\$(?P<inputs>(?:.|\\n)*?)\$\\n(?:(?P<body>(?:\\s*if(?:.*?\\n?)*?end;?)|(?:\\s*for(?:.*?\\n?)*?end;?)|(?:\\s*parfor(?:.*?\\n?)*?end;?)|(?:\\s*switch(?:.*?\\n?)*?end;?)|(?:\\s*try(?:.*?\\n?)*?end;?)|(?:\\s*while(?:.*?\\n?)*?end;?)|(?:\\s*arguments(?:.*?\\n?)*?end;?)|(?:(?:.*?\\n?)*?))(?:\\s*end;?))?', 'functions_defined_last': True, 'if_begin': 'if ({cond})', 'if_elif': 'elseif ({cond})', 'if_else': 'else', 'indent': ' ', 'index': '{variable}{{{index}}}', 'input': "{channel} = YggInterface('YggInput', '{channel_name}');", 'inputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:(?:,(?: *... *\\n)?)|$)', 'istype': "isa({variable}, '{type}')", 'len': 'length({variable})', 'line_end': ';', 'multiple_outputs': '[{outputs}]', 'not': 'not', 'output': "{channel} = YggInterface('YggOutput', '{channel_name}');", 'outputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'print': "disp('{message}');", 'print_generic': 'disp({object});', 'python_interface': "{channel} = YggInterface('{python_interface}', '{channel_name}');", 'python_interface_format': "{channel} = YggInterface('{python_interface}', '{channel_name}', '{format_str}');", 'quote': "'", 'recv_function': '{channel}.recv', 'send_function': '{channel}.send', 'true': 'true', 'try_begin': 'try', 'try_except': 'catch {error_var}', 'while_begin': 'while ({cond})'}¶

classmethod get_matlab_info()[source]¶

Determine the root directory where Matlab is installed and the version that is installed (if Matlab is installed at all). This will fail if Matlab is not installed, cannot be started, or does not operate as expected.

Returns:: Matlab root directory and Matlab version string.
Return type:: tuple
Raises:: RuntimeError – If Matlab cannot be started or the root directory or release cannot be determiend.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_map = {'call': '[flag, {inputs}] = {channel_obj}.call({outputs})', 'client': "YggInterface('YggRpcClient', '{channel_name}')", 'import': '', 'input': "YggInterface('YggInput', '{channel_name}')", 'output': "YggInterface('YggOutput', '{channel_name}')", 'recv': '[flag, {inputs}] = {channel_obj}.recv()', 'send': 'flag = {channel_obj}.send({outputs})', 'server': "YggInterface('YggRpcServer', '{channel_name}')", 'timesync': "YggInterface('YggTimesync', '{channel_name}')"}¶

inverse_type_map = None¶

language = 'matlab'¶

language_ext = ['.m']¶

classmethod language_version(skip_config=False)[source]¶

Determine the version of this language.

Parameters:: skip_config (bool, optional) – If True, the config option for the version (if it exists) will be ignored and the version will be determined fresh.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

parse_arguments(args)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:: args (list) – List of arguments provided.

path_env_variable = 'MATLABPATH'¶

recv_converters = {'pandas': 'array'}¶

classmethod run_code(lines, **kwargs)[source]¶

Run code by first writing it as an executable and then calling the driver.

Parameters:

lines (list) – Lines of code to be wrapped as an executable.
**kwargs – Additional keyword arguments are passed to the write_executable method.

classmethod run_executable(args, dont_wrap_error=False, fname_wrapper=None, matlab_engine=None, **kwargs)[source]¶

Run a program using the executable for this language and the provided arguments.

Parameters:

args (list) – The program that should be run and any arguments that should be provided to it.
dont_wrap_error (bool, optional) – If False, the executable will be wrapped in a try/catch block to prevent errors from stopping Matlab shutdown. If True, the command will be executed as is with the Matlab interpreter. Defaults to False.
fname_wrapper (str, optional) – File where wrapper should be saved. If not provided, one is created. Defaults to None.
matlab_engine (MatlabEngine, optional) – Matlab engine that should be used to run the command. If not provided, the Matlab interpreter is used instead. Defaults to None.
**kwargs – Additional keyword arguments are passed to cls.executable_command and tools.popen_nobuffer.

Returns:

Output to stdout from the run command.

Return type:

Raises:

RuntimeError – If the language is not installed.
RuntimeError – If there is an error when running the command.

run_loop()[source]¶: Loop to check if model is still running and forward output.

send_converters = {'pandas': <function consolidate_array>, 'table': <function consolidate_array>}¶

set_env()[source]¶

Get environment variables that should be set for the model process.

Returns:: Environment variables for the model process.
Return type:: dict

start_matlab_engine()[source]¶: Start matlab session and connect to it.

type_map = {'1darray': 'mat', 'array': 'cell', 'boolean': 'logical', 'bytes': 'char (utf-8)', 'complex': 'complex', 'float': 'single, double', 'int': 'intX', 'ndarray': 'mat', 'null': 'NaN', 'obj': 'containers.Map', 'object': 'containers.Map', 'ply': 'containers.Map', 'schema': 'containers.Map', 'string': 'char', 'uint': 'uintX', 'unicode': 'char'}¶

version_flags = ["fprintf('R%s', version('-release')); exit();"]¶

classmethod write_error_wrapper(fname, try_lines, env=None, matlab_engine=None)[source]¶

Write a wrapper for the model that encloses it in a try except so that the error can be propagated appropriately.

Parameters:

fname (str) – File where the wrapper should be written.
try_lines (list) – List of lines to go in the try block.
model_file (str) – Path to model that should be wrapped.
env (dict, optional) – Dictionary of environment variables that should be set before calling the model. Defaults to None and is ignored.
matlab_engine (MatlabEngine, optional) – Matlab engine that will be used to call the wrapper. If not provided, it is assumed the error will be called using the Matlab interpreter on the command line. Defautls to None.

Raises:

classmethod write_finalize_iiter(var, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

classmethod write_finalize_oiter(var)[source]¶

Get the lines necessary to finalize an array after iteration.

Parameters:: var (dict, str) – Name or information dictionary for the variable being initialized.
Returns:: The lines finalizing the variable.
Return type:: list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

zero_based = False¶

class yggdrasil.drivers.MatlabModelDriver.MatlabProcess(*args, **kwargs)[source]¶

Bases: YggClass

Add features to mimic subprocess.Popen while running Matlab function asynchronously.

Parameters:

target (func) – Matlab function that should be called.
args (list, tuple) – Arguments that should be passed to target.
kwargs (dict, optional) – Keyword arguments that should be passed to target. Defaults to empty dict.
name (str, optional) – A name for the process. Generated if not provided.
matlab_engine (MatlabEngine, optional) – MatlabEngine that should be used to get errors. Defaults to None and errors will not be recovered unless passed through stdout and stderr before shutdown.

stdout¶

File like string buffer that stdout from target will be written to.

Type:: StringIO

stderr¶

File like string buffer that stderr from target will be written to.

Type:: StringIO

target¶

Matlab function that should be called.

Type:: func

args¶

Arguments that should be passed to target.

Type:: list, tuple

kwargs¶

Keyword arguments that should be passed to target.

Type:: dict

future¶

Future result from async function. This will be None until start is called.

Type:: MatlabFutureResult

matlab_engine¶

MatlabEngine that should be used to get errors.

Type:: MatlabEngine

Raises:: RuntimeError – If Matlab is not installed.

is_alive()[source]¶: bool: Is the async call funning.

is_cancelled()[source]¶: bool: Was the async call cancelled or not.

is_done()[source]¶: bool: Is the async call still running.

is_started()[source]¶: bool: Has start been called.

kill(*args, **kwargs)[source]¶: Cancel the async call.

on_matlab_error()[source]¶: Actions performed on error in Matlab engine.

poll(*args, **kwargs)[source]¶: Fake poll.

print_output()[source]¶: Print output from stdout and stderr.

property returncode¶

Return code.

Type:: int

start()[source]¶: Start asychronous call.

property stderr_line¶

Output to stderr from function call.

Type:: str

property stdout_line¶

Output to stdout from function call.

Type:: str

yggdrasil.drivers.MatlabModelDriver.connect_matlab_engine(matlab_session, first_connect=False)[source]¶

Connect to Matlab engine.

Parameters:

matlab_session (str) – Name of the Matlab session that should be connected to.
first_connect (bool, optional) – If True, this is the first time Python is connecting to the Matlab shared engine and certain environment variables should be set. Defaults to False.

Returns:

Matlab engine that was connected.

Return type:

MatlabEngine

yggdrasil.drivers.MatlabModelDriver.install_matlab_engine()[source]¶: Install the MATLAB engine API for Python.

yggdrasil.drivers.MatlabModelDriver.is_matlab_running()[source]¶

Determine if there is a Matlab engine running.

Returns:: True if there is a Matlab engine running, False otherwise.
Return type:: bool

yggdrasil.drivers.MatlabModelDriver.kill_all()[source]¶: Kill all Matlab shared engines.

yggdrasil.drivers.MatlabModelDriver.locate_matlab_engine_processes()[source]¶

Get all of the active matlab sharedEngine processes.

Returns:: Active matlab sharedEngine processes.
Return type:: list

yggdrasil.drivers.MatlabModelDriver.locate_matlabroot()[source]¶

Find directory that servers as matlab root.

Returns:: Full path to matlabroot directory.
Return type:: str

yggdrasil.drivers.MatlabModelDriver.start_matlab_engine(skip_connect=False, timeout=None)[source]¶

Start a Matlab shared engine session inside a detached screen session.

Parameters:

skip_connect (bool, optional) – If True, the engine is not connected. Defaults to False.
timeout (int, optional) – Time (in seconds) that should be waited for Matlab to start up. Defaults to None and is set from the config option (‘matlab’, ‘startup_waittime_s’).

Returns:

Information on the started session including the name of the: screen session running matlab, the created engine object, the name of the matlab session, and the matlab engine process.

Return type:

Raises:

RuntimeError – If Matlab is not installed.

yggdrasil.drivers.MatlabModelDriver.stop_matlab_engine(screen_session, matlab_engine, matlab_session, matlab_process, keep_engine=False)[source]¶

Stop a Matlab shared engine session running inside a detached screen session.

Parameters:

screen_session (str) – Name of the screen session that the shared Matlab session was started in.
matlab_engine (MatlabEngine) – Matlab engine that should be stopped.
matlab_session (str) – Name of Matlab session that the Matlab engine is connected to.
matlab_process (psutil.Process) – Process running the Matlab shared engine.
keep_engine (bool, optional) – If True, the references to the engine will be removed so it is not deleted. Defaults to False.

Raises:

RuntimeError – If Matlab is not installed.

yggdrasil.drivers.ModelDriver module¶

class yggdrasil.drivers.ModelDriver.ModelDriver(*args, **kwargs)[source]¶

Bases: Driver

Base class for Model drivers and for running executable based models.

Parameters:

name (str) – Unique name used to identify the model. This will be used to report errors associated with the model.
args (str or list) – The path to the file containing the model program that will be run by the driver for the model’s language and/or a list of arguments that should be passed as input to the model program or language executable (e.g. source code or configuration file for a domain specific language).
products (list, optional) – Paths to files created by the model that should be cleaned up when the model exits. Entries can be absolute paths or paths relative to the working directory. Defaults to [].
function (str, optional) – If provided, an integrated model is created by wrapping the function named here. The function must be located within the file specified by the source file listed in the first argument. If not provided, the model must contain it’s own calls to the yggdrasil interface.
iter_function_over (array, optional) – Variable(s) that should be received or sent as an array, but iterated over. Defaults to an empty array and is ignored.
source_products (list, optional) – Files created by running the model that are source files. These files will be removed without checking their extension so users should avoid adding files to this list unless they are sure they should be deleted. Defaults to [].
is_server (bool, dict, optional) – If True, the model is assumed to be a server for one or more client models and an instance of yggdrasil.drivers.ServerDriver is started. The corresponding channel that should be passed to the yggdrasil API will be the name of the model. If is_server is a dictionary, it should contain an ‘input’ key and an ‘output’ key. These are required to be the names of existing input and output channels in the model that will be co-opted by the server. (Note: This requires that the co-opted output channel’s send method is called once for each time the co-opted input channel’s recv method is called. If used with the function parameter, is_server must be a dictionary. Defaults to False.
client_of (str, list, optional) – The names of one or more models that this model will call as a server. If there are more than one, this should be specified as a sequence collection (list). The corresponding channel(s) that should be passed to the yggdrasil API will be the name of the server model joined with the name of the client model with an underscore <server_model>_<client_model>. There will be one channel created for each server the model is a client of. Defaults to empty list. Use of client_of with function is not currently supported.
timesync (bool, str, optional) – If set, the model is assumed to call a send then receive of the state at each timestep for syncronization with other models that are also integrating in time. If a string is provided, it is assumed to be the name of the server that will handle timestep synchronization. If a boolean is provided, the name of the server will be assumed to be ‘timestep’. Defaults to False.
overwrite (bool, optional) – If True, any existing model products (compilation products, wrapper scripts, etc.) are removed prior to the run. If False, the products are not removed. Defaults to True. Setting this to False can improve the performance, particularly for models that take a long time to compile, but this should only be done once the model has been fully debugged to ensure that each run is tested on a clean copy of the model. The value of this keyword also determines whether or not products are removed after a run.
preserve_cache (bool, optional) – If True model products will be kept following the run, otherwise all products will be cleaned up. Defaults to False. This keyword is superceeded by overwrite.
with_strace (bool, optional) – If True, the command is run with strace (on Linux) or dtrace (on MacOS). Defaults to False.
strace_flags (list, optional) – Flags to pass to strace (or dtrace). Defaults to [].
with_valgrind (bool, optional) – If True, the command is run with valgrind. Defaults to False.
valgrind_flags (list, optional) – Flags to pass to valgrind. Defaults to [].
model_index (int, optional) – Index of model in list of models being run. Defaults to 0.
copy_index (int, optional) – Index of model in set of copies. Defaults to -1 indicating there is only one copy of the model.
outputs_in_inputs (bool, optional) – If True, outputs from wrapped model functions are passed by pointer as inputs for modification and the return value will be a flag. If False, outputs are limited to return values. Defaults to the value of the class attribute outputs_in_inputs.
logging_level (str, optional) – The level of logging messages that should be displayed by the model. Defaults to the logging level as determined by the configuration file and environment variables.
allow_threading (bool, optional) – If True, comm connections will be set up so that the model-side comms can be used by more than one thread. Defaults to False.
copies (int, optional) – The number of copies of the model that should be created. Defaults to 1.
repository_url (str, optional) – URL for the git repository containing the model source code. If provided, relative paths in the model YAML definition will be considered relative to the repository root directory.
repository_commit (str, optional) – Commit that should be checked out in the model repository specified by repository_url. If not provided, the most recent commit on the default branch will be used.
description (str, optional) – Description of the model. This parameter is only used in the model repository or when providing the model as a service.
contact_email (str, optional) – Email address that should be used to contact the maintainer of the model. This parameter is only used in the model repository.
validation_command (str, optional) – Path to a validation command that can be used to verify that the model ran as expected. A non-zero return code is taken to indicate failure.
dependencies (list, optional) – A list of packages required by the model that are written in the same language as the model. If the package requires dependencies outside the language of the model. use the additional_dependencies parameter to provide them. If you need a version of the package from a specific package manager, a mapping with ‘package’ and ‘package_manager’ fields can be provided instead of just the name of the package.
additional_dependencies (dict, optional) – A mapping between languages and lists of packages in those languages that are required by the model.
with_debugger (str, optional) – Debugger tool that should be used to run models. This string should include the tool executable and any flags that should be passed to it.
**kwargs – Additional keyword arguments are passed to parent class.

Class Attributes:

language (str): Primary name for the programming language that this: compiler should be used for. [REQUIRED]
language_aliases (list): Additional/alternative names that the language: may be known by.
language_ext (list): Extensions for programs written in the target: language. [REQUIRED]
base_languages (list): Other programming languages that this driver: and the interpreter for the target language are dependent on (e.g. Matlab models require Python).
executable_type (str): ‘compiler’ or ‘interpreter’ to indicate the type: of the executable for the language. [AUTOMATED]
interface_library (list): Name of the library containing the yggdrasil: interface for the target language. [REQUIRED]
interface_directories (list): Directories containing code in the: interface library for the target language.
interface_dependencies (list): List of names of libraries that are: required to use the interface on the current platform. This dosn’t include libraries required by specific communication types which are described by supported_comm_options.
supported_comms (list): Name of comms supported in the target language.: [REQUIRED]
supported_comm_options (dict): Options for the supported comms like the: platforms they are available on and the external libraries required to use them. [REQUIRED]
external_libraries (dict): Information on external libraries required: for running models in the target language using yggdrasil.
internal_libraries (dict): Information on internal libraries required: for running models in the target language using yggdrasil.
type_map (dict): Mapping of yggdrasil extended JSON types to: datatypes in the target programming language. [REQUIRED]
function_param (dict): Options specifying how different operations: would be encoded in the target language (e.g. if statements, for loops, while loops). [REQUIRED]
version_flags (list): Flags that should be called with the language: executable to determine the version of the compiler/interpreter. Defaults to [’–version’].
outputs_in_inputs (bool): If True, outputs are passed by pointer as: inputs for modification and the return value should be a flag. Defaults to False.
include_arg_count (bool): If True, the number of arguments passed: to send/recv calls is prepended to the arguments to the function. Defaults to False.
include_channel_obj (bool): If True, the channel object is passed as: input to the send/recv calls (after the argument count if it is also present due to include_arg_count being True). Defaults to False.

is_typed (bool): True if the language is typed, False otherwise. brackets (tuple): A pair of opening and clossing characters that

are used by the language to mark blocks. Set to None and ignored by default.

no_executable (bool): True if there is not an executable associated: with the language driver. Defaults to False.
comms_implicit (bool): True if the comms installed for this driver: are not explicitly defined (depend on input parameters). Defaults to False.

args¶

Argument(s) for running the model on the command line.

Type:: list

model_file¶

Full path to the model executable or interpretable script.

Type:: str

model_args¶

Runtime arguments for running the model on the command line.

Type:: list

model_src¶

Full path to the model source code. For interpreted languages, this will be the same as model_file.

Type:: str

model_function_info¶

Parameters recovered by parsing the provided model function definition.

Type:: dict

overwrite¶

If True, any existing compilation products will be overwritten by compilation and cleaned up following the run. Otherwise, existing products will be used and will remain after the run.

Type:: bool

products¶

Files created by running the model. This includes compilation products such as executables and/or object files.

Type:: list

source_products¶

Files created by running the model that are source files. These files will be removed without checking their extension so users should avoid adding files to this list unless they are sure they should be deleted.

Type:: list

wrapper_products¶

Files created in order to wrap the model.

Type:: list

process¶

Process used to run the model.

Type:: yggdrasil.tools.YggPopen

function¶

The name of the model function that should be wrapped.

Type:: str

iter_function_over¶

Variable(s) that should be received or sent as an array, but iterated over.

Type:: array

is_server¶

If True, the model is assumed to be a server and an instance of yggdrasil.drivers.ServerDriver is started. If a dict, the input/output channels with the specified names in the dict will be replaced with a server.

Type:: bool, dict

client_of¶

The names of server models that this model is a client of.

Type:: list

timesync¶

If set, the name of the server performing timestep synchronization for the model.

Type:: str

with_strace¶

If True, the command is run with strace or dtrace.

Type:: bool

strace_flags¶

Flags to pass to strace/dtrace.

Type:: list

with_valgrind¶

If True, the command is run with valgrind.

Type:: bool

valgrind_flags¶

Flags to pass to valgrind.

Type:: list

model_index¶

Index of model in list of models being run.

Type:: int

copy_index¶

Index of model in set of copies.

Type:: int

modified_files¶

List of pairs of originals and copies of files that should be restored during cleanup.

Type:: list

allow_threading¶

If True, comm connections will be set up so that the model-side comms can be used by more than one thread.

Type:: bool

copies¶

The number of copies of the model that should be created.

Type:: int

repository_url¶

URL for the git repository containing the model source code. If provided, relative paths in the model YAML definition will be considered relative to the repository root directory.

Type:: str

repository_commit¶

Commit that should be checked out in the model repository specified by repository_url.

Type:: str

description¶

Description of the model. This parameter is only used in the model repository or when providing the model as a service.

Type:: str

contact_email¶

Email address that should be used to contact the maintainer of the model. This parameter is only used in the model repository.

Type:: str

validation_command¶

Path to a validation command that can be used to verify that the model ran as expected. A non-zero return code is taken to indicate failure.

Type:: str

dependencies¶

A list of packages required by the model that are written in the same language as the model. If the package requires dependencies outside the language of the model, use the additional_dependencies parameter to provide them. If you need a version of the package from a specific package manager, a mapping

with ‘package’ and ‘package_manager’ fields can be provided instead of just the name of the package.

Type:: list

additional_dependencies¶

A mapping between languages and lists of packages in those languages that are required by the model.

Type:: dict

Raises:: RuntimeError – If both with_strace and with_valgrind are True.

classmethod add_extra_vars(direction, x)[source]¶

Add extra variables required for communication.

Parameters:

direction (str) – Direction of channel (‘input’ or ‘output’)
x (dict) – Dictionary describing the variable.

after_loop()[source]¶: Actions to perform after run_loop has finished. Mainly checking if there was an error and then handling it.

static after_registration(cls, cfg=None, second_pass=False)[source]¶

Operations that should be performed to modify class attributes after registration. For compiled languages this includes selecting the default compiler. The order of precedence is the config file ‘compiler’ option for the language, followed by the environment variable set by _compiler_env, followed by the existing class attribute.

Parameters:

cfg (YggConfigParser, optional) – Config class that should be used to set options for the driver. Defaults to None and yggdrasil.config.ygg_cfg is used.
second_pass (bool, optional) – If True, the class as already been registered. Defaults to False.

classmethod are_base_languages_installed(missing=None)[source]¶

Determine if the base languages are installed.

Parameters:: missing (list, optional) – A pre-existing list that missing base languages should be appended to.
Returns:: True if the base langauges are installed. False otherwise.
Return type:: bool

classmethod are_dependencies_installed()[source]¶

Determine if the dependencies are installed for the interface (not including dependencies needed by a particular communication type).

Returns:: True if the dependencies are installed. False otherwise.
Return type:: bool

base_languages = []¶

before_loop()[source]¶: Actions before loop.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration including things like platform dependent properties and checking environment variables for default settings.

before_start(no_queue_thread=False, **kwargs)[source]¶

Actions to perform before the run starts.

Parameters:

no_queue_thread (bool, optional) – If True, the queue_thread is not created/started. Defaults to False.
**kwargs – Keyword arguments are pased to run_model.

brackets = None¶

classmethod channels2vars(channels)[source]¶

Convert a list of channels to a list of variables.

Parameters:: channels (list) – List of channel dictionaries.
Returns:: List of variables.
Return type:: list

classmethod check_flag_var(info, outputs_in_inputs=None)[source]¶

Check if the flag variable should be treated as an output.

Parameters:

info (dict) – Information about the function.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.

check_mpi_request(name)[source]¶

Check if a request has been completed.

Parameters:

name (str) – Name that request was registered under.

Returns:

Received message or False if the request does not: exist or is not complete.

Return type:

bool, str

cleanup()[source]¶: Remove compile executable.

classmethod cleanup_dependencies(products=[], verbose=False, **kws)[source]¶: Cleanup dependencies.

cleanup_products()[source]¶: Remove products created in order to run the model.

comms_implicit = False¶

classmethod configuration_steps()[source]¶

Get a list of configuration steps with tuples of flags and boolean values.

Returns:

Pairs of descriptions and states for: different steps in the configuration all steps must be True for the language to be configured.

Return type:

OrderedDict

classmethod configure(cfg, **kwargs)[source]¶

Add configuration options for this language.

Parameters:

cfg (CisConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod configure_executable_type(cfg)[source]¶

Add configuration options specific in the executable type before the libraries are configured.

Parameters:

cfg (YggConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod configure_libraries(cfg)[source]¶

Add configuration options for external libraries in this language.

Parameters:

cfg (YggConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

dont_declare_channel = False¶

enqueue_output_loop()[source]¶: Keep passing lines to queue.

classmethod escape_quotes(x)[source]¶

Escape quotes in a string.

Parameters:: x (str) – String to escape quotes in.
Returns:: x with escaped quotes.
Return type:: str

classmethod executable_command(args, unused_kwargs=None, **kwargs)[source]¶

Compose a command for running a program using the exectuable for this language (compiler/interpreter) with the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it.
unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are ignored.

Returns:

Arguments composing the command required to run the program: from the command line using the executable for this language.

Return type:

executable_type = None¶

classmethod expand_server_io(inputs, outputs, client_comms=[])[source]¶

Update inputs/outputs w/ information about server that will be using them.

Parameters:

inputs (list) – List of model inputs including types.
outputs (list) – List of model outputs including types.
client_comms (list, optional) – List of the names of client comms that should be removed from the list of outputs. Defaults to [].

external_libraries = {}¶

classmethod finalize_function_io(direction, x)[source]¶

Finalize info for an input/output channel following function parsing.

Parameters:

direction (str) – Direction of channel (‘input’ or ‘output’)
x (dict) – Channel info.

static finalize_registration(cls)[source]¶: Operations that should be performed after a class has been fully initialized and registered.

classmethod format_function_param(key, default=None, replacement=None, ignore_method=False, **kwargs)[source]¶

Return the formatted version of the specified key.

Parameters:

key (str) – Key in cls.function_param mapping that should be formatted.
default (str, optional) – Format that should be returned if key is not in cls.function_param. Defaults to None.
replacement (str, optional) – Format that should be used instead of the one in cls.function_param. Defaults to None.
**kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns:

Formatted string.

Return type:

Raises:

NotImplementedError – If key is not in cls.function_param and default is not set.

full_language = True¶

function_param = None¶

classmethod get_all_language_ext()[source]¶: Return the list of all language extensions.

classmethod get_inverse_type_map()[source]¶

Get the inverse type map.

Returns:: Mapping from native type to JSON type.
Return type:: dict

get_io_env(input_drivers=None, output_drivers=None)[source]¶

Get environment variables set by the input/output drivers.

Parameters:

input_drivers (list, optional) – Input drivers. Defaults to the yaml entry if not provided.
output_drivers (list, optional) – Output drivers. Defaults to the yaml entry if not provided.

Returns:

Environment variables.

Return type:

classmethod get_json_type(native_type)[source]¶

Get the JSON type from the native language type.

Parameters:: native_type (str) – The native language type.
Returns:: The JSON type.
Return type:: str, dict

classmethod get_language_dir()[source]¶: Return the langauge directory.

classmethod get_language_ext()[source]¶: Return the language extension, including from the base classes.

classmethod get_language_for_source(fname, languages=None, early_exit=False, **kwargs)[source]¶

Determine the language that can be used with the provided source file(s). If more than one language applies to a set of multiple files, the language that applies to the most files is returned.

Parameters:

fname (str, list) – The full path to one or more files. If more than one
languages (list, optional) – The list of languages that are acceptable. Defaults to None and any language will be acceptable.
early_exit (bool, optional) – If True, the first language identified will be returned if fname is a list of files. Defaults to False.
**kwargs – Additional keyword arguments are passed to recursive calls.

Returns:

The language that can operate on the specified file.

Return type:

classmethod get_map_language_ext()[source]¶: Return the mapping of all language extensions.

classmethod get_name_declare(var)[source]¶

Determine the name that should be used for declaration.

Parameters:: var (str, dict) – Name of variable or dictionary of information.
Returns:: Modified name for declaration.
Return type:: str

classmethod get_native_type(return_json=False, **kwargs)[source]¶

Get the native type.

Parameters:

return_json (bool, optional) – If True, the returned value will also include the JSON datatype.
**kwargs – Additional keyword arguments may be used in determining the precise declaration that should be used.

Returns:

The native type.

Return type:

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

graceful_stop()[source]¶: Gracefully stop the driver.

property has_sent_messages¶

True if output has been received from the model.

Type:: bool

classmethod identify_source_files(args=None, working_dir=None, **kwargs)[source]¶

Determine the source file based on model arguments.

Parameters:

args (list, optional) – Arguments provided.
working_dir (str, optional) – Working directory.
**kwargs – Additional keyword arguments are ignored.

Returns:

Source files.

Return type:

include_arg_count = False¶

include_channel_obj = False¶

init_from_function(args)[source]¶: Initialize model parameters based on the wrapped function.

init_mpi()[source]¶: Initialize MPI communicator.

classmethod input2output(var)[source]¶

Perform conversion necessary to turn a variable extracted from a function definition from an input to an output.

Parameters:: var (dict) – Variable definition.
Returns:: Updated variable definition.
Return type:: dict

classmethod install_dependency(package=None, package_manager=None, arguments=None, command=None, always_yes=False, command_kwargs=None)[source]¶

Install a dependency.

Parameters:

package (str) – Name of the package that should be installed. If the package manager supports it, this can include version requirements.
package_manager (str, optional) – Package manager that should be used to install the package.
arguments (str, optional) – Additional arguments that should be passed to the package manager.
command (list, optional) – Command that should be used to install the package.
always_yes (bool, optional) – If True, the package manager will not ask users for input during installation. Defaults to False.
command_kwargs (dict, optional) – Keyword arguments that should be passed to the subprocess call for the installation.

classmethod install_model_dependencies(dependencies, always_yes=False)[source]¶

Install any dependencies required by the model.

Parameters:

dependencies (list) – Dependencies that should be installed.
always_yes (bool, optional) – If True, the package manager will not ask users for input during installation. Defaults to False.

interface_dependencies = []¶

interface_directories = []¶

interface_inside_exec = False¶

interface_library = None¶

internal_libraries = {}¶

inverse_type_map = None¶

property io_errors¶

Errors produced by input/output drivers to this model.

Type:: list

classmethod is_comm_installed(commtype=None, skip_config=False, **kwargs)[source]¶

Determine if a comm is installed for the associated programming language.

Parameters:

commtype (str, optional) – If provided, this method will only test for installation of the specified communication type. Defaults to None and will check for any installed comm.
skip_config (bool, optional) – If True, the config list of comms installed for this language will not be used to determine if the comm is installed and the class attribute supported_comm_options will be processed. Defaults to False and config options are used in order to improve performance after initial configuration.
platforms (list, optional) – Platforms on which the comm can be installed. Defaults to None and is ignored unless there is a value for the commtype in supported_comm_options. This keyword argument is ignored if skip_config is False.
libraries (list, optional) – External libraries that are required by the specified commtype. Defaults to None and is ignored unless there is a value for the commtype in supported_comm_options. This keyword argument is ignored if skip_config is False.
**kwargs – Additional keyword arguments are passed to either is_comm_installed for the base languages, supported languages, or is_library_installed as appropriate.

Returns:

True if a comm is installed for this language.

Return type:

classmethod is_configured()[source]¶

Determine if the appropriate configuration has been performed (e.g. installation of supporting libraries etc.)

Returns:: True if the language has been configured.
Return type:: bool

classmethod is_disabled()[source]¶

is_dsl = False¶

classmethod is_installed()[source]¶

Determine if this model driver is installed on the current machine.

Returns:

Truth of if this model driver can be run on the current: machine.

Return type:

classmethod is_interface_installed()[source]¶

Determine if the interface library for the associated programming language is installed.

Returns:: True if the interface library is installed.
Return type:: bool

classmethod is_language_installed()[source]¶

Determine if the interpreter/compiler for the associated programming language is installed.

Returns:: True if the language interpreter/compiler is installed.
Return type:: bool

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

classmethod is_source_file(fname)[source]¶

Determine if the provided file name points to a source files for the associated programming language by checking the extension.

Parameters:: fname (str) – Path to file.
Returns:: True if the provided file is a source file, False otherwise.
Return type:: bool

is_typed = False¶

kill_process()[source]¶: Kill the process running the model, checking return code.

language = None¶

language_aliases = []¶

classmethod language_executable(**kwargs)[source]¶

Command required to compile/run a model written in this language from the command line.

Returns:

Name of (or path to) compiler/interpreter executable required: to run the compiler/interpreter from the command line.

Return type:

language_ext = None¶

classmethod language_version(version_flags=None, **kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

max_line_width = None¶

model_command()[source]¶

Return the command that should be used to run the model.

Returns:

Any commands/arguments needed to run the model from the: command line.

Return type:

property model_process_complete¶

Has the process finished or not. Returns True if the process has not started.

Type:: bool

property model_process_returncode¶

Return code for the model process where non-zero values indicate that there was an error.

Type:: int

classmethod mpi_partner_cleanup(self)[source]¶: Actions cleaning up an MPIPartnerModel.

classmethod mpi_partner_init(self)[source]¶: Actions initializing an MPIPartnerModel.

property n_sent_messages¶

Number of messages sent by the model via each connection.

Type:: dict

no_executable = False¶

property numeric_logging_level¶

Logging level for the model.

Type:: int

classmethod output2input(var, in_definition=True)[source]¶

Perform conversion necessary to turn an output variable into an corresponding input that can be used to format a function definition.

Parameters:

var (dict) – Variable definition.
in_definition (bool, optional) – If True, the returned dictionary corresponds to an input variable in a function definition. If False, the returned value will correspond to an input to a function. Defaults to True.

Returns:

Updated variable definition.

Return type:

outputs_in_inputs = False¶

parse_arguments(args, default_model_dir=None)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:

args (list) – List of arguments provided.
default_model_dir (str, optional) – Path to directory that should be used to normalize the model file path if it is not absolute. Defaults to None and is set to the working_dir.

classmethod parse_function_definition(model_file, model_function, contents=None, match=None, expected_outputs=[], outputs_in_inputs=None)[source]¶

Get information about the inputs & outputs to a model from its defintition if possible.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
contents (str, optional) – String containing the function definition. If not provided, the function definition is read from model_file.
match (re.Match, optional) – Match object for the function regex. If not provided, a search is performed using function_def_regex.
expected_outputs (list, optional) – List of names or variable information dictionaries for outputs that are expected to be extracted from the function’s definition. This variable is only used if outputs_in_inputs is True and outputs are not extracted from the function’s defintion using the regex for this language. Defaults to [].
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.

Returns:

Parameters extracted from the function definitions.

Return type:

classmethod parse_var_definition(io, value, outputs_in_inputs=None)[source]¶

Extract information about input/output variables from a string definition.

Parameters:

io (str) – Description of variables contained in the provided string. Must be ‘inputs’ or ‘outputs’.
value (str) – String containing one or more variable definitions.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.

Returns:

List of information about the variables contained in: the provided string.

Return type:

Raises:

AssertionError – If io is not ‘inputs’ or ‘outputs’.
NotImplementedError – If the def_regex for the specified io is not defined.

classmethod prepare_input_variables(vars_list, in_definition=False, for_yggdrasil=False)[source]¶

Concatenate a set of input variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing input to a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying input variables in a function definition. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod prepare_output_variables(vars_list, in_definition=False, in_inputs=False, for_yggdrasil=False)[source]¶

Concatenate a set of output variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing output from a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying output variables in a function definition. Defaults to False.
in_inputs (bool, optional) – If True, the output variables should be formated to be included as input variables. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod prepare_variables(vars_list, in_definition=False, for_yggdrasil=False)[source]¶

Concatenate a set of input variables such that it can be passed as a single string to the function_call parameter.

Parameters:

vars_list (list) – List of variable dictionaries containing info (e.g. names) that should be used to prepare a string representing input/output to/from a function call.
in_definition (bool, optional) – If True, the returned sequence will be of the format required for specifying variables in a function definition. Defaults to False.
for_yggdrasil (bool, optional) – If True, the variables will be prepared in the formated expected by calls to yggdarsil send/recv methods. Defaults to False.

Returns:

Concatentated variables list.

Return type:

classmethod preparse_function(yml)[source]¶

Extract information about inputs and outputs based on the function being wrapped.

Parameters:: yml (dict) – Options that will be used to initialize the model.
Returns:: Information about the parsed function.
Return type:: dict

python_interface = {'array_input': 'YggArrayInput', 'array_output': 'YggArrayOutput', 'pandas_input': 'YggPandasInput', 'pandas_output': 'YggPandasOutput', 'table_input': 'YggAsciiTableInput', 'table_output': 'YggAsciiTableOutput'}¶

queue_close()[source]¶: Close the queue for messages from the model process.

queue_recv()[source]¶: Receive a message from the model process.

recv_mpi(tag=0, dont_block=False)[source]¶: Receive an MPI message.

remove_products()[source]¶: Delete products produced during the process of running the model.

restore_files()[source]¶: Restore modified files to their original form.

classmethod run_code(lines, process_kwargs={}, **kwargs)[source]¶

Run code by first writing it as an executable and then calling the driver.

Parameters:

lines (list) – Lines of code to be wrapped as an executable.
process_kwargs (dict, optional) – Keyword arguments that should be passed to run_model. Defaults to {}.
**kwargs – Additional keyword arguments are passed to the write_executable method.

classmethod run_executable(args, return_process=False, debug_flags=None, **kwargs)[source]¶

Run a program using the executable for this language and the provided arguments.

Parameters:

args (list) – The program that should be run and any arguments that should be provided to it.
return_process (bool, optional) – If True, the process class is returned without checking the process output. If False, communicate is called on the process and the output is parsed for errors. Defaults to False.
debug_flags (list, optional) – Debug executable and flags that should be prepended to the executable command. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to cls.executable_command and tools.popen_nobuffer.

Returns:

Output to stdout from the run command if return_process is: False, the process if return_process is True.

Return type:

Raises:

RuntimeError – If the language is not installed.
RuntimeError – If there is an error when running the command.

run_finally()[source]¶: Actions to perform in finally clause of try/except wrapping run.

run_loop()[source]¶: Loop to check if model is still running and forward output.

run_model(command=None, return_process=True, **kwargs)[source]¶

Run the model. Unless overridden, the model will be run using run_executable.

Parameters:

command (list, optional) – Command to run. Defaults to None and is created by the model_command method.
return_process (bool, optional) – If True, the process running the model is returned. If False, the process will block until the model finishes running. Defaults to True.
**kwargs – Keyword arguments are passed to run_executable.

run_validation()[source]¶: Run the validation script for the model.

send_mpi(msg, tag=0, dont_block=False)[source]¶: Send an MPI message.

set_break_flag(*args, **kwargs)[source]¶: Stop the model loop.

set_env(existing=None, **kwargs)[source]¶

Get environment variables that should be set for the model process.

Parameters:

existing (dict, optional) – Existing dictionary of environment variables that new variables should be added to. Defaults to a copy of os.environ.
**kwargs – Additional keyword arguments are passed to set_env_class.

Returns:

Environment variables for the model process.

Return type:

classmethod set_env_class(existing=None, **kwargs)[source]¶

Set environment variables that are instance independent.

Parameters:

existing (dict, optional) – Existing dictionary of environment variables that new variables should be added to. Defaults to a copy of os.environ.
**kwargs – Additional keyword arguments are ignored.

Returns:

Environment variables for the model process.

Return type:

classmethod split_line(line, length=None, force_split=False)[source]¶

Split a line as close to (or before) a given character as possible.

Parameters:

line (str) – Line to split.
length (int, optional) – Maximum length of split lines. Defaults to cls.max_line_width if not provided.
force_split (bool, optional) – If True, force a split to occur at the specified length. Defauts to False.

Returns:

Set of lines resulting from spliting the provided line.

Return type:

classmethod split_variables(var_str)[source]¶

Split variable string include individual variables.

Parameters:: var_str (str) – String containing multiple variables.
Returns:: Split variables.
Return type:: list

stop_mpi_partner(msg=None, dest=0, tag=None)[source]¶: Send a message to stop the MPI partner model on the main process.

supported_comm_options = {}¶

supported_comms = []¶

type_map = None¶

types_in_funcdef = True¶

classmethod update_io_from_function(model_file, model_function, inputs=[], outputs=[], contents=None, outputs_in_inputs=None, iter_function_over=[])[source]¶

Update inputs/outputs from the function definition.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
inputs (list, optional) – List of model inputs including types. Defaults to [].
outputs (list, optional) – List of model outputs including types. Defaults to [].
contents (str, optional) – Contents of file to parse rather than re-reading the file. Defaults to None and is ignored.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.
iter_function_over (array, optional) – Variable(s) that should be received or sent as an array, but iterated over. Defaults to an empty array and is ignored.

Returns:

Flag variable used by the model. If None, the: model does not use a flag variable.

Return type:

dict, None

version_flags = ['--version']¶

wait_on_mpi_request(name, timeout=False)[source]¶

Wait for a request to be completed.

Parameters:

name (str) – Name that request was registered under.

Returns:

Received message or False if the request does not: exist or is not complete.

Return type:

bool, str

wait_process(timeout=None, key=None, key_suffix=None)[source]¶

Wait for some amount of time for the process to finish.

Parameters:

timeout (float, optional) – Time (in seconds) that should be waited. Defaults to None and is infinite.
key (str, optional) – Key that should be used to register the timeout. Defaults to None and set based on the stack trace.

Returns:

True if the process completed. False otherwise.

Return type:

classmethod write_assign_to_output(dst_var, src_var, copy=False, outputs_in_inputs=False, **kwargs)[source]¶

Write lines assigning a value to an output variable.

Parameters:

dst_var (str, dict) – Name or information dictionary for variable being assigned to.
src_var (str, dict) – Name or information dictionary for value being assigned to dst_var.
copy (bool, optional) – If True, the assigned value is copied during assignment. Defaults to False.
outputs_in_inputs (bool, optional) – If True, outputs are passed as input parameters. In some languages, this means that a pointer or reference is passed (e.g. C) and so the assignment should be to the memory indicated rather than the variable. Defaults to False.

Returns:

Lines achieving assignment.

Return type:

classmethod write_channel_decl(var, **kwargs)[source]¶

Write a channel declaration.

Parameters:

var (dict) – Information dictionary for the channel. being declared.
**kwargs – Additional keyword arguments are passed to class’s write_declaration.

Returns:

The lines declaring the variable.

Return type:

classmethod write_channel_def(key, datatype=None, **kwargs)[source]¶

Write an channel definition.

Parameters:

key (str) – Entry in cls.function_param that should be used.
datatype (dict, optional) – Data type associated with the channel. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed as parameters to format_function_param.

Returns:

Lines required to declare and define an output channel.

Return type:

classmethod write_declaration(var, value=None, requires_freeing=None, definitions=None, is_argument=False)[source]¶

Return the lines required to declare a variable with a certain type.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
value (str, optional) – Value that should be assigned to the variable after it is declared.
requires_freeing (list, optional) – Existing list that variables requiring freeing should be appended to. Defaults to None and is ignored.
definitions (list, optional) – Existing list that variable definitions should be added to. Defaults to None if not provided and definitions will be included in the returned lines.
dont_define (bool, optional) – If True, the variable will not be defined. Defaults to False.
is_argument (bool, optional) – If True, the variable being declared is an input argument. Defaults to False.

Returns:

The lines declaring the variable.

Return type:

classmethod write_executable(lines, prefix=None, suffix=None, function_definitions=None, imports=None, model_name=None)[source]¶

Return the lines required to complete a program that will run the provided lines.

Parameters:

lines (list) – Lines of code to be wrapped as an executable.
prefix (list, optional) – Lines of code that should proceed the wrapped code. Defaults to None and is ignored. (e.g. C/C++ include statements).
suffix (list, optional) – Lines of code that should follow the wrapped code. Defaults to None and is ignored.
function_definitions (list, optional) – Lines of code defining functions that will beused by the code contained in lines. Defaults to None and is ignored.
imports (list, optional) – Kwargs for packages that should be imported for use by the executable. Defaults to None and is ignored.
model_name (str, optional) – Name given to the model. Defaults to None.

Returns:

Lines of code wrapping the provided lines with the: necessary code to run it as an executable (e.g. C/C++’s main).

Return type:

lines

classmethod write_executable_import(model_name=None, **kwargs)[source]¶

Add import statements to executable lines.

Parameters:: **kwargs – Keyword arguments for import statement.
Returns:: Lines required to complete the import.
Return type:: list

classmethod write_expand_single_element(output_var, add_cond=False)[source]¶

Write lines allowing extraction of the only element from a single element array as a stand-alone variable if the variable is an array and only has one element.

Parameters:

output_var (str) – Name of the variable that should be conditionally expanded.
add_cond (list, optional) – Additional conditions that must be satisfied for the array element to be extracted. Defaults to False and is ignored.

Returns:

Lines added the conditional expansion of single element: arrays.

Return type:

classmethod write_finalize_iiter(var)[source]¶

Get the lines necessary to finalize an input array for iteration.

Parameters:: var (dict, str) – Name or information dictionary for the variable finalized.
Returns:: The lines finalizing the variable.
Return type:: list

classmethod write_finalize_oiter(var, value=None, requires_freeing=None)[source]¶

Get the lines necessary to finalize an array after iteration.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
requires_freeing (list, optional) – Existing list of variables requiring freeing. Defaults to None and is ignored.

Returns:

The lines finalizing the variable.

Return type:

classmethod write_for_loop(iter_var, iter_begin, iter_end, loop_contents)[source]¶

Return the lines required to complete a for loop.

Parameters:

iter_var (str) – Name of variable that iterator should use.
iter_begin (int) – Beginning of iteration.
iter_end (int) – End of iteration.
loop_contents (list) – Lines of code that should be executed inside the loop.

Returns:

Lines of code performing a loop.

Return type:

classmethod write_free(var, **kwargs)[source]¶

Return the lines required to free a variable with a certain type.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
**kwargs – Additional keyword arguments are passed to format_function_param.

Returns:

The lines freeing the variable.

Return type:

classmethod write_function_call(function_name, inputs=[], outputs=[], include_arg_count=False, outputs_in_inputs=False, **kwargs)[source]¶

Write a function call.

Parameters:

function_name (str) – Name of the function being called.
inputs (list, optional) – List of inputs to the function. Defaults to [].
outputs (list, optional) – List of outputs from the function. Defaults to [].
include_arg_count (bool, optional) – If True, the count of input arguments is included as the first argument. Defaults to False.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.
**kwargs – Additional keyword arguments are passed to cls.format_function_param.

Returns:

Lines completing the function call.

Return type:

classmethod write_function_def(function_name, inputs=[], outputs=[], input_var=None, output_var=None, function_contents=[], outputs_in_inputs=False, opening_msg=None, closing_msg=None, print_inputs=False, print_outputs=False, skip_interface=False, function_keys=None, verbose=False, **kwargs)[source]¶

Write a function definition.

Parameters:

function_name (str) – Name fo the function being defined.
inputs (list, optional) – List of inputs to the function. Defaults to []. Ignored if input_var provided.
outputs (list, optional) – List of outputs from the function. Defaults to []. If not provided, no return call is added to the function body. Ignored if output_var provided.
input_var (str, optional) – Full string specifying input in the function definition. If not provided, this will be created based on the contents of the inputs variable.
output_var (str, optional) – Full string specifying output in the function definition. If not provided, this will be created based on the contents of the outputs variable.
function_contents (list, optional) – List of lines comprising the body of the function. Defaults to [].
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to False.
opening_msg (str, optional) – String that should be printed before the function contents (and inputs if print_inputs is True). Defaults to None and is ignored.
closing_msg (str, optional) – String that should be printed after the function contents (and outputs if print_outputs is True). Defaults to None and is ignored.
print_inputs (bool, optional) – If True, the input variables will be printed before the function contents. Defaults to False.
print_outputs (bool, optional) – If True, the output variables will be printed after the function contents. Defaults to False.
skip_interface (bool, optional) – If True, the line including the interface will be skipped. Defaults to False.
function_keys (tuple, optional) – 2 element tuple that specifies the keys for the function_param entries that should be used to begin & end a function definition. Defaults to (‘function_def_begin’, function_def_end’).
verbose (bool, optional) – If True, the contents of the created file are displayed. Defaults to False.
**kwargs – Additional keyword arguments are passed to cls.format_function_param.

Returns:

Lines completing the function call.

Return type:

Raises:

NotImplementedError – If the function_param attribute for the class is not defined.

classmethod write_if_block(cond, block_contents, else_block_contents=False)[source]¶

Return the lines required to complete a conditional block.

Parameters:

cond (str) – Conditional that should determine block execution.
block_contents (list) – Lines of code that should be executed inside the block.
else_block_contents (list, optional) – Lines of code that should be executed inside the else clause of the block. Defaults to False if not provided and an else clause is omitted.

Returns:

Lines of code performing conditional execution of a block.

Return type:

classmethod write_initialize(var, value=None, requires_freeing=None)[source]¶

Get the code necessary to initialize a variable.

Parameters:

var (dict, str) – Name or information dictionary for the variable being declared.
value (str, optional) – Value that should be assigned to the variable after it is declared.
requires_freeing (list, optional) – Existing list that variables requiring freeing should be appended to. Defaults to None and is ignored.

Returns:

The lines initializing the variable.

Return type:

classmethod write_initialize_oiter(var, value=None, requires_freeing=None)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
requires_freeing (list, optional) – Existing list that variables requiring freeing should be appended to. Defaults to None and is ignored.

Returns:

The lines initializing the variable.

Return type:

classmethod write_model_function_call(model_function, flag_var, inputs, outputs, outputs_in_inputs=None, on_failure=None, format_not_flag_cond=None, format_flag_cond=None, iter_function_idx=None)[source]¶

Write lines necessary to call the model function.

Parameters:

model_function (str) – Handle of the model function that should be called.
flag_var (str) – Name of variable that should be used as a flag.
inputs (list) – List of dictionaries describing inputs to the model.
outputs (list) – List of dictionaries describing outputs from the model.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to the class attribute outputs_in_inputs.
on_failure (list, optional) – Lines to be executed if the model call fails. Defaults to an error message. This variable is only used if flag_var is not None and outputs_in_inputs is True.
format_not_flag_cond (str, optional) – Format string that produces a conditional expression that evaluates to False when the model flag indicates a failure. Defaults to None and the class’s value for ‘not_flag_cond’ in function_param is used if it exists. If it does not exist, format_flag_cond is used.
format_flag_cond (str, optional) – Format string that produces a conditional expression that evaluates to True when the model flag indicates a success. Defaults to None and the defaults class’s value for ‘flag_cond’ in function_param is used if it exists. If it does not exist, the flag is directly evaluated as if it were a boolean.
iter_function_idx (dict, optional) – Variable that serves as an index to iterate over variables. Defaults to None.

Returns:

Lines required to carry out a call to a model function in: this language.

Return type:

classmethod write_model_recv(channel, recv_var, flag_var='flag', iter_var=None, allow_failure=False, alt_recv_function=None, include_arg_count=None)[source]¶

Write a model receive call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being received from was stored in.
recv_var (dict, list) – Information of one or more variables that receieved information should be stored in.
flag_var (str, optional) – Name of flag variable that the flag should be stored in. Defaults to ‘flag’,
iter_var (str, optional) – Name of flag signifying when the model is in it’s first iteration. If allow_failure is True and iter_var is provided, an error will be raised if iter_var is True. Defaults to None.
allow_failure (bool, optional) – If True, the returned lines will call a break if the flag is False. Otherwise, the returned lines will issue an error. Defaults to False.
alt_recv_function (str, optional) – Alternate receive function format string. Defaults to None and is ignored.
include_arg_count (bool, optional) – If True, the arguments to the receive call will be proceeded with an argument count. If not provided, the class attribute of the same name will be used.

Returns:

Lines required to carry out a receive call in this language.

Return type:

classmethod write_model_send(channel, send_var, flag_var='flag', allow_failure=False, alt_send_function=None, include_arg_count=None)[source]¶

Write a model send call include checking the return flag.

Parameters:

channel (str) – Name of variable that the channel being sent to was stored in.
send_var (dict, list) – Information on one or more variables containing information that will be sent.
flag_var (str, optional) – Name of flag variable that the flag should be stored in. Defaults to ‘flag’,
allow_failure (bool, optional) – If True, the returned lines will call a break if the flag is False. Otherwise, the returned lines will issue an error. Defaults to False.
alt_send_function (str, optional) – Alternate send function format string. Defaults to None and is ignored.
include_arg_count (bool, optional) – If True, the arguments to the send call will be proceeded with an argument count. If not provided, the class attribute of the same name will be used.

Returns:

Lines required to carry out a send call in this language.

Return type:

classmethod write_model_wrapper(model_file, model_function, inputs=[], outputs=[], model_flag=None, outputs_in_inputs=None, verbose=False, copies=1, iter_function_over=[], verbose_model=False, skip_update_io=False, model_name=None)[source]¶

Return the lines required to wrap a model function as an integrated model.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
inputs (list, optional) – List of model inputs including types. Defaults to [].
outputs (list, optional) – List of model outputs including types. Defaults to [].
model_flag (dict, optional) – Information about the flag that should be used to track the success of yggdrasil send/recv calls. This should only be provided if update_io_from_function has already been called. Defaults to None and is determined by update_io_from_function.
outputs_in_inputs (bool, optional) – If True, the outputs are presented in the function definition as inputs. Defaults to the class attribute outputs_in_inputs.
verbose (bool, optional) – If True, the contents of the created file are displayed. Defaults to False.
copies (int, optional) – Number of times the model driver is duplicated. If more than one, no error will be raised in the event there is never a call the the function. Defaults to 1.
iter_function_over (array, optional) – Variable(s) that should be received or sent as an array, but iterated over. Defaults to an empty array and is ignored.
skip_update_io (bool, optional) – If True, update_io_from_function will not be called. Defaults to False.
verbose_model (bool, optional) – If True, print statements will be added after every line in the model. Defaults to False.
model_name (str, optional) – Name given to the model. Defaults to None.

Returns:

Lines of code wrapping the provided model with the necessary: code to run it as part of an integration.

Return type:

classmethod write_print_input_var(var, **kwargs)[source]¶

Get the lines necessary to print an input variable in this language.

Parameters:

var (dict) – Variable information.
**kwargs – Additional keyword arguments are passed to write_print_var.

Returns:

Lines printing the specified variable.

Return type:

classmethod write_print_output_var(var, in_inputs=False, **kwargs)[source]¶

Get the lines necessary to print an output variable in this language.

Parameters:

var (dict) – Variable information.
in_inputs (bool, optional) – If True, the output variable is passed in as an input variable to be populated. Defaults to False.
**kwargs – Additional keyword arguments are passed to write_print_var.

Returns:

Lines printing the specified variable.

Return type:

classmethod write_print_var(var, prefix_msg=None)[source]¶

Get the lines necessary to print a variable in this language.

Parameters:

var (dict) – Variable information.
prefix_msg (str, optional) – Message that should be printed before the variable. Defaults to None and is ignored.

Returns:

Lines printing the specified variable.

Return type:

classmethod write_try_except(try_contents, except_contents, error_var='e', error_type=None)[source]¶

Return the lines required to complete a try/except block.

Parameters:

try_contents (list) – Lines of code that should be executed inside the try block.
except_contents (list) – Lines of code that should be executed inside the except block.
error_var (str, optional) – Name of variable where the caught error should be stored. Defaults to ‘e’.
error_type (str, optional) – Name of error type that should be caught. If not provided, defaults to None and will be set based on the class function_param entry for ‘try_error_type’.

Returns:

Lines of code perfoming a try/except block.

classmethod write_type_decl(name, datatype, name_base=None, requires_freeing=None, definitions=None, no_decl=False)[source]¶

Get lines declaring the datatype within the language.

Parameters:

name (str) – Name of variable that should be declared.
datatype (dict) – Type definition.
requires_freeing (list, optional) – List that variables requiring freeing should be appended to. Defaults to None.
definitions (list, optional) – Existing list that variable definitions should be added to. Defaults to None if not provided and definitions will be included in the returned lines.
no_decl (bool, optional) – If True, the variable is not declared, but supporting variables will be. Defaults to False.

Returns:

Lines required to define a type declaration.

Return type:

classmethod write_type_def(name, datatype, name_base=None, use_generic=False)[source]¶

Get lines declaring the data type within the language.

Parameters:

name (str) – Name of variable that definition should be stored in.
datatype (dict) – Type definition.
use_generic (bool, optional) – If True variables serialized and/or deserialized by the type will be assumed to be generic objects. Defaults to False.

Returns:

Lines required to define a type definition.

Return type:

classmethod write_while_loop(cond, loop_contents)[source]¶

Return the lines required to complete a for loop.

Parameters:

cond (str) – Conditional that should determine loop execution.
loop_contents (list) – Lines of code that should be executed inside the loop.

Returns:

Lines of code performing a loop.

Return type:

write_wrappers(**kwargs)[source]¶

Write any wrappers needed to compile and/or run a model.

Parameters:: **kwargs – Keyword arguments are ignored (only included to allow cascade from child classes).
Returns:: Full paths to any created wrappers.
Return type:: list

zero_based = True¶

yggdrasil.drivers.ModelDriver.remove_product(product, check_for_source=False, **kwargs)[source]¶

Delete a single product after checking that the product is not (or does not contain, in the case of directories), source files.

Parameters:

product (str) – Full path to a file or directory that should be removed.
check_for_source (bool, optional) – If True, the specified product will be checked to ensure that no source files are present. If a source file is present, a RuntimeError will be raised. Defaults to False.
**kwargs – Additional keyword arguments are passed to tools.remove_path.

Raises:

RuntimeError – If the specified product is a source file and check_for_source is False.
RuntimeError – If the specified product is a directory that contains a source file and check_for_source is False.
RuntimeError – If the product cannot be removed.

yggdrasil.drivers.ModelDriver.remove_products(products, source_products)[source]¶

Delete products produced during the process of running the model.

Parameters:

products (list) – List of products that should be removed after checking that they are not source files.
source_products (list) – List of products that should be removed without checking that they are not source files.

yggdrasil.drivers.OSRModelDriver module¶

class yggdrasil.drivers.OSRModelDriver.OSRModelDriver(*args, **kwargs)[source]¶

Bases: ExecutableModelDriver

Class for running OpenSimRoot model.

Parameters:

sync_vars_in (list, optional) – Variables that should be synchronized from other models. Defaults to [].
sync_vars_out (list, optional) – Variables that should be synchronized to other models. Defaults to [].
copy_xml_to_osr (bool, optional) – If True, the XML file(s) will be copied to the OSR repository InputFiles direcitory before running. This is necessary if the XML file(s) use any of the files located there since OSR always assumes the included file paths are relative. Defaults to False.
update_interval (float, optional) – Max simulation interval at which synchronization should occur (in days). Defaults to 1.0 if not provided. If the XML input file loads additional export modules that output at a shorter rate, the existing table of values will be extrapolated.

static after_registration(cls, **kwargs)[source]¶: Operations that should be performed to modify class attributes after registration.

base_languages = ['c++']¶

cfg = <yggdrasil.config.YggConfigParser object>¶

classmethod cleanup_dependencies(*args, **kwargs)[source]¶: Cleanup dependencies.

classmethod clone_repository(dest=None)[source]¶

Clone the OpenSimRoot repository.

Parameters:: dest (str, optional) – Full path to location where the repository should be cloned. Defaults to ‘$TEMP/OpenSimRoot’.
Returns:: Full path to location where the repository was cloned.
Return type:: str

classmethod compile_dependencies(target='OpenSimRootYgg', toolname=None, **kwargs)[source]¶

Compile the OpenSimRoot executable with the yggdrasil flag set.

Parameters:

target (str, optional) – Make target that should be build. Defaults to ‘OpenSimRootYgg’ (the yggdrasil-instrumented version of the OSR executable).
toolname (str, optional) – C++ compiler that should be used. Forced to be ‘cl.exe’ on windows. Otherwise the default C++ compiler will be used.
**kwargs – Additional keyword arguments are passed to the compile_dependencies methods of the base classes.

classmethod configure_executable_type(cfg)[source]¶

Add configuration options specific in the executable type before the libraries are configured.

Parameters:

cfg (CisConfigParser) – Config class that options should be set for.

Returns:

Section, option, description tuples for options that could not: be set.

Return type:

classmethod executable_command(args, **kwargs)[source]¶

Compose a command for running a program using the exectuable for this language (compiler/interpreter) with the provided arguments.

Parameters:

args (list) – The program that returned command should run and any arguments that should be provided to it.
unused_kwargs (dict, optional) – Existing dictionary that unused keyword arguments should be added to. Defaults to None and is ignored.
**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Arguments composing the command required to run the program: from the command line using the executable for this language.

Return type:

executable_path = '/tmp/OpenSimRoot/OpenSimRoot/StaticBuild/OpenSimRootYgg'¶

executable_type = 'dsl'¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_dependencies = ['make']¶

inverse_type_map = None¶

classmethod is_language_installed()[source]¶

Determine if the interpreter/compiler for the associated programming language is installed.

Returns:: True if the language interpreter/compiler is installed.
Return type:: bool

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

language = 'osr'¶

classmethod language_executable(**kwargs)[source]¶

Command required to compile/run a model written in this language from the command line.

Returns:

Name of (or path to) compiler/interpreter executable required: to run the compiler/interpreter from the command line.

Return type:

language_ext = ['.xml']¶

classmethod language_version(version_flags=None, **kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

parse_arguments(*args, **kwargs)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:

*args – Additional arguments are passed to the parent class’s method.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

repository = '/tmp/OpenSimRoot'¶

repository_branch = 'volatile_active'¶

repository_url = 'https://gitlab.com/langmm/OpenSimRoot.git'¶

classmethod set_env_class(**kwargs)[source]¶

Set environment variables that are instance independent.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method.
Returns:: Environment variables for the model process.
Return type:: dict

wrap_xml(src, dst)[source]¶

Wrap the input XML, adding tags for the inputs/outputs specified for this model.

Parameters:

src (str) – Full path to the XML file that should be wrapped.
dst (str) – Full path to the location where the updated XML should be saved.

Returns:

Full path to XML file produced.

Return type:

write_wrappers(**kwargs)[source]¶

Write any wrappers needed to compile and/or run a model.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Full paths to any created wrappers.
Return type:: list

yggdrasil.drivers.OutputDriver module¶

class yggdrasil.drivers.OutputDriver.OutputDriver(*args, **kwargs)[source]¶

Driver for sending output to another model’s comm via a local comm.

Parameters:

name (str) – The name of the local message comm that the driver should connect to.
args (str) – The name of the other message comme that the driver should connect to.
**kwargs – Additional keyword arguments are passed to parent class.

yggdrasil.drivers.PyTorchModelDriver module¶

class yggdrasil.drivers.PyTorchModelDriver.PyTorchModelDriver(*args, **kwargs)[source]¶

Bases: DSLModelDriver

Class for handling PyTorch models.

cfg = <yggdrasil.config.YggConfigParser object>¶

default_interpreter = 'pytorch'¶

default_interpreter_flags = []¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_dependencies = ['torch']¶

inverse_type_map = None¶

language = 'pytorch'¶

language_ext = ['.py']¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

classmethod model_wrapper(model_file, weights_file, inputs=[], outputs=[], env=None, working_dir=None, input_transform=None, output_transform=None)[source]¶: Model wrapper.

property model_wrapper_args¶

Positional arguments for the model wrapper.

Type:: tuple

property model_wrapper_kwargs¶

Keyword arguments for the model wrapper.

Type:: dict

yggdrasil.drivers.PythonModelDriver module¶

class yggdrasil.drivers.PythonModelDriver.PythonModelDriver(*args, **kwargs)[source]¶

Class for running Python models.

cfg = <yggdrasil.config.YggConfigParser object>¶

default_interpreter = '/opt/hostedtoolcache/Python/3.8.17/x64/bin/python'¶

default_interpreter_flags = []¶

static finalize_registration(cls)[source]¶: Operations that should be performed after a class has been fully initialized and registered.

classmethod format_function_param(key, default=None, **kwargs)[source]¶

Return the formatted version of the specified key.

Parameters:

key (str) – Key in cls.function_param mapping that should be formatted.
default (str, optional) – Format that should be returned if key is not in cls.function_param. Defaults to None.
**kwargs – Additional keyword arguments are used in formatting the request function parameter.

Returns:

Formatted string.

Return type:

Raises:

NotImplementedError – If key is not in cls.function_param and default is not set.

function_param = {'and': 'and', 'assign': '{name} = {value}', 'block_end': '', 'break': 'break', 'comment': '#', 'error': 'raise Exception("{error_msg}")', 'exec_begin': 'def main():', 'exec_suffix': 'if __name__ == "__main__":\n main()', 'false': 'False', 'for_begin': 'for {iter_var} in range({iter_begin}, {iter_end}):', 'fprintf': 'print("{message}" % ({variables}))', 'function_def_begin': 'def {function_name}({input_var}):', 'function_def_regex': '\\n?( *)def +{function_name} *\$(?P<inputs>(?:.|\\n)*?)\$? *:(?P<body>(?:\\1(?: )+(?!return).*\\n)|(?: *\\n))*(?:\\1(?: )+return *(?P<outputs>.*)?)?', 'if_begin': 'if ({cond}):', 'if_elif': 'elif ({cond}):', 'if_else': 'else:', 'import': 'from {filename} import {function}', 'import_nofile': 'import {function}', 'indent': ' ', 'index': '{variable}[{index}]', 'input': "{channel} = ygg.YggInput('{channel_name}')", 'inputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'interface': 'import {interface_library} as ygg', 'istype': 'isinstance({variable}, {type})', 'len': 'len({variable})', 'multiple_outputs': '[{outputs}]', 'not': 'not', 'output': "{channel} = ygg.YggOutput('{channel_name}')", 'outputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'print': 'print("{message}")', 'print_generic': 'from yggdrasil.tools import print_encoded; print_encoded({object})', 'python_interface': "{channel} = ygg.{python_interface}('{channel_name}')", 'python_interface_format': "{channel} = ygg.{python_interface}('{channel_name}', '{format_str}')", 'quote': '"', 'recv_function': '{channel}.recv', 'return': 'return {output_var}', 'send_function': '{channel}.send', 'true': 'True', 'try_begin': 'try:', 'try_error_type': 'BaseException', 'try_except': 'except {error_type} as {error_var}:', 'while_begin': 'while ({cond}):'}¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

classmethod install_dependency(package=None, package_manager=None, **kwargs)[source]¶

Install a dependency.

Parameters:

package (str) – Name of the package that should be installed. If the package manager supports it, this can include version requirements.
package_manager (str, optional) – Package manager that should be used to install the package.
**kwargs – Additional keyword arguments are passed to the parent class.

interface_library = 'yggdrasil.interface.YggInterface'¶

interface_map = {'call': 'flag, {inputs} = {channel_obj}.call({outputs})', 'client': 'YggRpcClient("{channel_name}")', 'import': 'from yggdrasil.languages.Python.YggInterface import {commtype}', 'input': 'YggInput("{channel_name}")', 'output': 'YggOutput("{channel_name}")', 'recv': 'flag, {inputs} = {channel_obj}.recv()', 'send': 'flag = {channel_obj}.send({outputs})', 'server': 'YggRpcServer("{channel_name}")', 'timesync': 'YggTimesync("{channel_name}")'}¶

inverse_type_map = None¶

classmethod is_language_installed()[source]¶

Determine if this model driver is installed on the current machine.

Returns:

Truth of if this model driver can be run on the current: machine.

Return type:

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

language = 'python'¶

language_ext = ['.py']¶

run_validation()[source]¶: Run the validation script for the model.

set_env(**kwargs)[source]¶

Get environment variables that should be set for the model process.

Parameters:: **kwargs – Additional keyword arguments are passed to the parent class’s method.
Returns:: Environment variables for the model process.
Return type:: dict

supported_comm_options = {'ipc': {'libraries': ['sysv_ipc'], 'platforms': ['MacOS', 'Linux']}, 'mpi': {'libraries': ['mpi4py']}, 'rest': {'libraries': ['requests']}, 'rmq': {'libraries': ['pika']}, 'zmq': {'libraries': ['zmq']}}¶

supported_comms = ['rmq_async', 'ipc', 'buffer', 'rest', 'value', 'rmq', 'mpi', 'zmq']¶

type_map = {'1darray': 'numpy.ndarray', 'array': 'list', 'boolean': 'bool', 'bytes': 'bytes', 'complex': 'numpy.complexX', 'float': 'numpy.floatX', 'int': 'numpy.intX', 'ndarray': 'numpy.ndarray', 'null': 'None', 'obj': 'ObjDict', 'object': 'dict', 'ply': 'PlyDict', 'schema': 'dict', 'string': 'str', 'uint': 'numpy.uintX', 'unicode': 'str'}¶

classmethod write_finalize_oiter(var)[source]¶

Get the lines necessary to finalize an array after iteration.

Parameters:: var (dict, str) – Name or information dictionary for the variable being initialized.
Returns:: The lines finalizing the variable.
Return type:: list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

yggdrasil.drivers.RModelDriver module¶

class yggdrasil.drivers.RModelDriver.RModelDriver(*args, **kwargs)[source]¶

Base class for running R models.

Parameters:

name (str) – Driver name.
args (str or list) – Argument(s) for running the model in matlab. Generally, this should be the full path to a Matlab script.
**kwargs – Additional keyword arguments are passed to parent class’s __init__ method.

base_languages = ['python']¶

brackets = ('{', '}')¶

classmethod call_compiler(package_dir, toolname=None, flags=None, language='c++', verbose=False, use_ccache=False, disable_python_c_api=False, with_asan=False)[source]¶

Build an R package w/ the yggdrasil compilers.

Parameters:

package_dir (str) – Full path to the package directory.
toolname (str, optional) – Compilation tool that should be used. Defaults to None and the default tools will be used.
flags (list, optional) – Additional flags that should be passed to the build command. Defaults to [].
language (str, optional) – Language that the package is written in (e.g. c, fortran). Defautls to ‘c++’.
verbose (bool, optional) – If True, information about the build process will be displayed. Defaults to False.
use_ccache (bool, optional) – If True, ccache will be added to the compilation executable. Defaults to False.
disable_python_c_api (bool, optional) – If True, the Python C API will be disabled. Defaults to False.

cfg = <yggdrasil.config.YggConfigParser object>¶

classmethod comm_atexit(comm)[source]¶

Operations performed on comm at exit including draining receive.

Parameters:: comm (CommBase) – Communication object.

default_interpreter = 'Rscript'¶

default_interpreter_flags = ['--default-packages=methods,utils']¶

function_param = {'and': '&&', 'assign': '{name} <- {value}', 'assign_mult': '{name} %<-% {value}', 'block_end': '}', 'comment': '#', 'error': 'stop("{error_msg}")', 'false': 'FALSE', 'first_index': 1, 'for_begin': 'for ({iter_var} in {iter_begin}:{iter_end}) {{', 'fprintf': 'print(sprintf("{message}", {variables}))', 'function_def_begin': '{function_name} <- function({input_var}) {{', 'function_def_regex': '{function_name} *(?:(?:\\<-)|(?:=)) *function\$(?P<inputs>(?:.|(?:\\r?\\n))*?)\$\\s*\\{{(?P<body>(?:.|(?:\\r?\\n))*?)(?:return\$(list\\()?(?P<outputs>(?:.|(?:\\r?\\n))*?)(?(3)\$)\\)(?:.|(?:\\r?\\n))*?\\}})|(?:\\}})', 'if_begin': 'if ({cond}) {{', 'if_elif': '}} else if ({cond}) {{', 'if_else': '}} else {{', 'import': 'reticulate::py_config(); source("{filename}")', 'indent': ' ', 'index': '{variable}[[{index}]]', 'input': '{channel} <- YggInterface("YggInput", "{channel_name}")', 'inputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'interface': 'library(yggdrasil)', 'interface_regex': "(?P<indent>[ \\t]*)(?P<variable>[a-zA-Z\\_]+[a-zA-Z\\_0-9]*)\\s*\\<\\-\\s*YggInterface\$\\'(?P<class>[a-zA-Z\\_]+[a-zA-Z\\_0-9]*)\\'\\s*\\,\\s*\\'(?P<channel>[a-zA-Z\\_]+[a-zA-Z\\_0-9]*)\\'\\s*(?P<args>[^\$]*)\\)", 'istype': 'is({variable}, "{type}")', 'len': 'length({variable})', 'multiple_outputs': 'c({outputs})', 'multiple_outputs_def': 'list({outputs})', 'not': '!', 'output': '{channel} <- YggInterface("YggOutput", "{channel_name}")', 'outputs_def_regex': '\\s*(?P<name>.+?)\\s*(?:,|$)', 'print': 'print("{message}")', 'print_generic': 'print({object})', 'python_interface': '{channel} <- YggInterface("{python_interface}", "{channel_name}")', 'python_interface_format': '{channel} <- YggInterface("{python_interface}", "{channel_name}", "{format_str}")', 'quote': '"', 'recv_function': '{channel}$recv', 'return': 'return({output_var})', 'send_function': '{channel}$send', 'true': 'TRUE', 'try_begin': 'tryCatch({', 'try_end': '})', 'try_except': '}}, error = function({error_var}) {{', 'while_begin': 'while ({cond}) {{'}¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

classmethod install_dependency(package, package_manager=None, **kwargs)[source]¶

Install a dependency.

Parameters:

package (str) – Name of the package that should be installed. If the package manager supports it, this can include version requirements.
package_manager (str, optional) – Package manager that should be used to install the package.
**kwargs – Additional keyword arguments are passed to the parent class.

interface_dependencies = ['units', 'R6', 'Rcpp', 'bit64', 'zeallot', 'reticulate']¶

interface_library = 'yggdrasil'¶

interface_map = {'call': 'c(flag, {inputs}) %<-% {channel_obj}$call({outputs})', 'client': "YggInterface('YggRpcClient', '{channel_name}')", 'import': 'library(yggdrasil)', 'input': "YggInterface('YggInput', '{channel_name}')", 'output': "YggInterface('YggOutput', '{channel_name}')", 'recv': 'c(flag, {inputs}) %<-% {channel_obj}$recv()', 'send': 'flag = {channel_obj}$send({outputs})', 'server': "YggInterface('YggRpcServer', '{channel_name}')", 'timesync': "YggInterface('YggTimesync', '{channel_name}')"}¶

inverse_type_map = None¶

classmethod is_library_installed(lib, **kwargs)[source]¶

Determine if a dependency is installed.

Parameters:

lib (str) – Name of the library that should be checked.
**kwargs – Additional keyword arguments are ignored.

Returns:

True if the library is installed, False otherwise.

Return type:

language = 'R'¶

language_aliases = ['r']¶

language_ext = ['.R']¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to the parent class’s method.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

classmethod python2language(pyobj)[source]¶

Prepare a python object for transformation in R.

Parameters:: pyobj (object) – Python object.
Returns:: Python object in a form that is R friendly.
Return type:: object

run_model(*args, **kwargs)[source]¶

Run the model. Unless overridden, the model will be run using run_executable.

Parameters:

*args – Arguments are passed to the parent class’s method.
**kwargs – Keyword arguments are passed to the parent class’s method.

send_converters = {'table': <function consolidate_array>}¶

set_env(**kwargs)[source]¶

Get environment variables that should be set for the model process.

Returns:: Environment variables for the model process.
Return type:: dict

type_map = {'1darray': 'list', 'array': 'list', 'boolean': 'logical', 'bytes': 'char (utf-8)', 'complex': 'complex', 'float': 'double', 'int': 'integer, bit64::integer64', 'ndarray': 'list', 'null': 'NA', 'number': 'double', 'obj': 'ObjDict', 'object': 'list', 'ply': 'PlyDict', 'schema': 'list', 'string': 'character', 'uint': 'integer', 'unicode': 'char'}¶

classmethod write_executable_import(**kwargs)[source]¶

Add import statements to executable lines.

Parameters:: **kwargs – Keyword arguments for import statement.
Returns:: Lines required to complete the import.
Return type:: list

classmethod write_expand_single_element(output_var, add_cond=False)[source]¶

Write lines allowing extraction of the only element from a single element array as a stand-alone variable if the variable is an array and only has one element.

Parameters:

output_var (str) – Name of the variable that should be conditionally expanded.
add_cond (str, optional) – Additional condition that must be satisfied for the array element to be extracted. Defaults to False and is ignored.

Returns:

Lines added the conditional expansion of single element: arrays.

Return type:

classmethod write_finalize_oiter(var)[source]¶

Get the lines necessary to finalize an array after iteration.

Parameters:: var (dict, str) – Name or information dictionary for the variable being initialized.
Returns:: The lines finalizing the variable.
Return type:: list

classmethod write_initialize_oiter(var, value=None, **kwargs)[source]¶

Get the lines necessary to initialize an array for iteration output.

Parameters:

var (dict, str) – Name or information dictionary for the variable being initialized.
value (str, optional) – Value that should be assigned to the variable.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

The lines initializing the variable.

Return type:

classmethod write_model_wrapper(model_file, model_function, **kwargs)[source]¶

Return the lines required to wrap a model function as an integrated model.

Parameters:

model_file (str) – Full path to the file containing the model function’s declaration.
model_function (str) – Name of the model function.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines of code wrapping the provided model with the necessary: code to run it as part of an integration.

Return type:

zero_based = False¶

yggdrasil.drivers.RPCRequestDriver module¶

class yggdrasil.drivers.RPCRequestDriver.RPCRequestDriver(*args, **kwargs)[source]¶

Class for handling client side RPC type communication.

Parameters:

model_request_name (str) – The name of the channel used by the client model to send requests.
**kwargs – Additional keyword arguments are passed to parent class.

response_drivers¶

Response drivers created for each request.

Type:: dict

before_loop()[source]¶: Send client sign on to server response driver.

property clients¶

Clients that are connected.

Type:: list

close_comm()[source]¶: Close response drivers.

close_response_drivers()[source]¶: Close response driver.

property model_env¶

Mapping between model name and opposite comm environment variables that need to be provided to the model.

Type:: dict

property nclients¶

Number of clients that are connected.

Type:: int

on_eof(msg)[source]¶

On EOF, decrement number of clients. Only send EOF if the number of clients drops to 0.

Parameters:: msg (CommMessage) – Message object that provided the EOF.
Returns:: Value that should be returned by recv_message on EOF.
Return type:: CommMessage, bool

printStatus(*args, **kwargs)[source]¶: Also print response drivers.

prune_response_drivers()[source]¶: Promote errors from response drivers.

remove_model(direction, name)[source]¶

Remove a model from the list of models.

Parameters:

direction (str) – Direction of model.
name (str) – Name of model exiting.

Returns:

True if all of the input/output models have signed: off; False otherwise.

Return type:

run_loop()[source]¶: Run the driver. Continue looping over messages until there are not any left or the communication channel is closed.

send_message(msg, **kwargs)[source]¶

Start a response driver for a request message and send message with header.

Parameters:

msg (CommMessage) – Message being sent.
**kwargs – Keyword arguments are passed to parent class send_message.

Returns:

Success or failure of send.

Return type:

property servers_recvd¶

Names of server models that have returned responses.

Type:: list

yggdrasil.drivers.RPCResponseDriver module¶

class yggdrasil.drivers.RPCResponseDriver.RPCResponseDriver(*args, **kwargs)[source]¶

Class for handling client side RPC type communication.

Parameters:

model_response_address (str) – The address of the channel used by the client model to receive responses.
msg_id (str) – ID associate with the request message this driver was created to respond to.
**kwargs – Additional keyword arguments are passed to parent class.

msg_id¶

ID associate with the request message this driver was created to respond to.

Type:: str

response_drivers¶

Response drivers created for each request.

Type:: list

property response_address¶

Address of response comm.

Type:: str

send_eof()[source]¶

Send EOF message.

Returns:: Success or failure of send.
Return type:: bool

send_message(msg, **kwargs)[source]¶

Propagate the request_id.

Parameters:

msg (CommMessage) – Message being sent.
**kwargs – Keyword arguments are passed to parent class send_message.

Returns:

Success or failure of send.

Return type:

yggdrasil.drivers.SBMLModelDriver module¶

class yggdrasil.drivers.SBMLModelDriver.SBMLModelDriver(*args, **kwargs)[source]¶

Bases: DSLModelDriver

Class for running SBML models.

Parameters:

start_time (float, optional) – Time that simulation should be started from. If ‘reset’ is True, the start time will always be the provided value, otherwise, the start time will be the end of the previous call after the first call. Defaults to 0.0.
steps (int, optional) – Number of steps that should be output. Defaults to None.
reset (bool, optional) – If True, the simulation will be reset to it’s initial values before each call (including the start time). Defaults to False.
selections (list, optional) – Variables to include in the output. Defaults to None and the time/floating selections will be returned.
integrator (str, optional) – Name of integrator that should be used. Valid options include [‘cvode’, ‘gillespie’, ‘rk4’, ‘rk45’]. Defaults to ‘cvode’.
integrator_settings (dict, optional) – Settings for the integrator. Defaults to empty dict.
variable_step (bool, optional) – If True, the output steps will have a variable spacing. Defaults to False.
only_output_final_step (bool, optional) – If True, only the final timestep is output. Defaults to False.
skip_start_time (bool, optional) – If True, the results for the initial time step will not be output. Defaults to False. This option is ignored if only_output_final_step is True.

classmethod call_model(model, curr_time, end_time, steps, reset=False, start_time=None, selections=None, variable_step=False)[source]¶: Call the model.

cfg = <yggdrasil.config.YggConfigParser object>¶

default_interpreter = 'sbml'¶

default_interpreter_flags = []¶

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

**kwargs – Additional keyword arguments are passed to the parent class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:: kwargs (dict): Keyword arguments for driver instance. deps (list): Dependencies to install.

Return type:

interface_dependencies = ['roadrunner']¶

inverse_type_map = None¶

language = 'sbml'¶

language_ext = ['.xml']¶

classmethod language_version(**kwargs)[source]¶

Determine the version of this language.

Parameters:: **kwargs – Keyword arguments are passed to cls.run_executable.
Returns:: Version of compiler/interpreter for this language.
Return type:: str

classmethod model_wrapper(model_file, start_time, steps, inputs=[], outputs=[], integrator=None, integrator_settings={}, env=None, working_dir=None, reset=False, selections=None, variable_step=False, skip_start_time=False, only_output_final_step=False)[source]¶: Model wrapper.

property model_wrapper_args¶

Positional arguments for the model wrapper.

Type:: tuple

property model_wrapper_kwargs¶

Keyword arguments for the model wrapper.

Type:: dict

classmethod setup_model(model_file, inputs=[], outputs=[], integrator=None, integrator_settings={}, selections=None)[source]¶: Set up model class instance.

yggdrasil.drivers.TimeSyncModelDriver module¶

class yggdrasil.drivers.TimeSyncModelDriver.TimeSyncModelDriver(*args, **kwargs)[source]¶

Bases: DSLModelDriver

Class for synchronizing states for timesteps between two models.

Parameters:

synonyms (dict, optional) –
Mapping from model names to mappings from base variables names to information about one or more alternate variable names used by the named model that should be converted to the base variable. Values for providing information about alternate variables can either be strings (implies equivalence with the base variable in everything but name and units) or mappings with the keys:

alt (str, list): Name of one or more variables used by
the model that should be used to calculate the named base variable.

alt2base (function): Callable object that takes the
alternate variables named by the ‘alt’ property as input and returns the base variable.

base2alt (function): Callable object that takes the base
variable as input and returns the alternate variables named by the ‘alt’ property.

Defaults to an empty dictionary.
aggregation (str, dict, optional) – Method(s) that should be used to aggregate synonymous variables across models. This can be a single method that should be used for all synonymous variables or a dictionary mapping between synonymous variables and the method that should be used by that variable. If a variable is not present in the dictionary or a values is not provided, ‘mean’ will be used. See the documentation for pandas.DataFrame.aggregate for available options.
interpolation (str, dict, optional) – Method(s) or keyword arguments that should be used to interpolate missing timesteps. This can be a single method or a dictionary mapping between model name and the interpolation methods (or keyword arguments) that should be used for variables from that model. Defaults to ‘index’. See the documentation for pandas.DataFrame.interpolate for available options.
additional_variables (dict, optional) – Mapping from model name to a list of variables from other models that are not provided by the model, but should still be returned to the model. Defaults to empty dictionary.

cfg = <yggdrasil.config.YggConfigParser object>¶

classmethod check_for_data(time, tables, table_units, table_lock, open_clients)[source]¶

Check for a time in the tables to determine if there is sufficient data available to calculate the state.

Parameters:

time (pandas.Timedelta) – Time that state is requested at.
tables (dict) – Mapping from model name to pandas DataFrames containing variables supplied by the model.
table_units (dict) – Mapping from model name to dictionaries mapping from variable names to units.
table_lock (RLock) – Thread-safe lock for accessing table.
open_clients (list) – Clients that are still open.

Returns:

True if there is sufficient data, False otherwise.

Return type:

default_interpreter = 'timesync'¶

default_interpreter_flags = []¶

executable_type = 'other'¶

inverse_type_map = None¶

language = 'timesync'¶

classmethod merge(tables, table_units, table_lock, open_clients, synonyms, interpolation, aggregation)[source]¶

Merge tables from models to get data.

Parameters:

tables (dict) – Mapping from model name to pandas DataFrames containing variables supplied by the model.
table_units (dict) – Mapping from model name to dictionaries mapping from variable names to units.
table_lock (RLock) – Thread-safe lock for accessing table.
open_clients (list) – Clients that are still open.
synonyms (dict) – Dictionary mapping from base variables to alternate variables and mapping functions used to convert between the variables. Defaults to empty dict and no conversions are performed.
interpolation (dict) – Mapping from model name to the interpolation kwargs that should be used. Defaults to empty dictionary.
aggregation (dict) – Mapping from variable name to the aggregation method that should be used. Defaults to empty dictionary.

classmethod model_wrapper(name, synonyms, interpolation, aggregation, additional_variables, env=None)[source]¶: Model wrapper.

property model_wrapper_args¶

Positional arguments for the model wrapper.

Type:: tuple

parse_arguments(args, **kwargs)[source]¶

Sort model arguments to determine which one is the executable and which ones are arguments.

Parameters:

args (list) – List of arguments provided.
**kwargs – Additional keyword arguments are ignored.

classmethod response_loop(client_model, request_id, rpc, time, internal_variables, external_variables, tables, table_units, table_lock, synonyms, interpolation, aggregation)[source]¶

Check for available data and send response if it is available.

Parameters:

client_model (str) – Name of model that made the request.
request_id (str) – ID associated with request that should be responded to.
rpc (ServerComm) – Server RPC comm that should be used to reply to the request when the data is available.
time (pandas.Timedelta) – Time to get variables at.
internal_variables (list) – Variables that model is requesting that it also calculates.
external_variables (list) – Variables that model is requesting that will be provided by other models.
tables (dict) – Mapping from model name to pandas DataFrames containing variables supplied by the model.
table_units (dict) – Mapping from model name to dictionaries mapping from variable names to units.
table_lock (RLock) – Thread-safe lock for accessing table.
synonyms (dict) – Dictionary mapping from base variables to alternate variables and mapping functions used to convert between the variables. Defaults to empty dict and no conversions are performed.
interpolation (dict) – Mapping from model name to the interpolation kwargs that should be used. Defaults to empty dictionary.
aggregation (dict) – Mapping from variable name to the aggregation method that should be used. Defaults to empty dictionary.

Module contents¶

IO and Model drivers.

yggdrasil.drivers.create_driver(driver=None, name=None, args=None, **kwargs)[source]¶

Dynamically create a driver based on a string and other driver properties.

Parameters:

driver (str) – Name of the driver that should be created.
name (str) – Name to give the driver.
args (object, optional) – Second argument for drivers which take a minimum of two arguments. If None, the driver is assumed to take a minimum of one argument. Defaults to None.
**kwargs – Additional keyword arguments are passed to the driver class.

Returns:

Instance of the requested driver.

Return type:

yggdrasil.communication package¶

Subpackages¶

Submodules¶

yggdrasil.communication.AsciiFileComm module¶

class yggdrasil.communication.AsciiFileComm.AsciiFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a file on disk.

Parameters:

name (str) – The environment variable where communication address is stored.
comment (str, optional) – String indicating a comment. If ‘read_meth’ is ‘readline’ and this is provided, lines starting with a comment will be skipped.
**kwargs – Additional keywords arguments are passed to parent class.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the: provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a test

file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by sending: the messages in ‘send’.

Return type:

yggdrasil.communication.AsciiMapComm module¶

class yggdrasil.communication.AsciiMapComm.AsciiMapComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a ASCII map on disk.

Parameters:

name (str) – The environment variable where file path is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.AsciiTableComm module¶

class yggdrasil.communication.AsciiTableComm.AsciiTableComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a file on disk.

yggdrasil.communication.AsyncComm module¶

class yggdrasil.communication.AsyncComm.AsyncComm(wrapped, daemon=False, async_recv_method='recv', async_send_method='send_message', async_recv_kwargs=None, async_send_kwargs=None)[source]¶

Bases: ProxyObject, ComponentBaseUnregistered

Class for handling asynchronous I/O.

Parameters:

name (str) – The name of the message queue.
async_recv_method (str, optional) – The method that should be used to receive message into the backlog. Defaults to ‘recv’.
async_send_method (dict, optional) – The method that should be used to send message in the backlog. Defaults to ‘send_message’.
async_recv_kwargs (dict, optional) – Keyword arguments to pass to calls receiving messages into the backlog. Defaults to {}.
async_send_method – Keyword arguments to pass to calls sending message from the backlog. Defaults to {}.
**kwargs – Additional keyword arguments are passed to CommBase.

backlog_ready¶

Event set when there is a message in the recv backlog.

Type:: multitasking.Event

add_backlog(payload)[source]¶

Add a message to the backlog of messages.

Parameters:: payload (tuple) – Arguments and keyword argumetns for send or data from receive.

async_recv_kwargs¶

async_recv_method¶

async_send_kwargs¶

async_send_method¶

atexit()[source]¶: Close operations.

property backlog_buffer¶

Messages that have been received.

Type:: list

backlog_ready¶

property backlog_thread¶

Task that will handle sinding or receiving backlogged messages.

Type:: tools.YggTask

close(linger=False)[source]¶

Close the connection.

Parameters:: linger (bool, optional) – If True, drain messages before closing the comm. Defaults to False.

close_on_eof_recv¶

daemon¶

property errors¶

Errors raised by the wrapped comm or async thread.

Type:: list

finalize_message(msg, **kwargs)[source]¶

Perform actions to decipher a message.

Parameters:

msg (CommMessage) – Initial message object to be finalized.
**kwargs – Keyword arguments are passed to the request comm’s finalize_message method.

Returns:

Deserialized and annotated message.

Return type:

property is_closed¶

True if the connection is closed.

Type:: bool

property is_confirmed_send¶

True if all sent messages have been confirmed.

Type:: bool

property is_open¶

True if the backlog is open.

Type:: bool

property is_open_backlog¶

True if the backlog thread is running.

Type:: bool

property is_open_direct¶

True if the direct comm is not None.

Type:: bool

property n_msg¶

The number of messages in the connection.

Type:: int

property n_msg_backlog¶

Number of messages in the backlog.

Type:: int

property n_msg_backlog_recv¶

Number of messages in the receive backlog.

Type:: int

property n_msg_backlog_send¶

Number of messages in the send backlog.

Type:: int

property n_msg_direct¶

Number of messages currently being routed.

Type:: int

property n_msg_direct_recv¶

Number of messages currently being routed in recv.

Type:: int

property n_msg_direct_send¶

Number of messages currently being routed in send.

Type:: int

property n_msg_recv¶

Number of messages in the receive backlog.

Type:: int

property n_msg_recv_drain¶

Number of messages in the receive backlog and direct comm.

Type:: int

property n_msg_send¶

Number of messages in the send backlog.

Type:: int

property n_msg_send_drain¶

Number of messages in the send backlog and direct comm.

Type:: int

next(*args, **kwargs)¶

open()[source]¶: Open the connection by connecting to the queue.

pop_backlog()[source]¶

Pop a message from the front of the backlog.

Returns:

First backlogged send arguments/keyword arguments: or received data.

Return type:

precheck(*args, **kwargs)[source]¶

printStatus(*args, **kwargs)[source]¶: Print status of the communicator.

purge()[source]¶: Purge all messages from the comm.

recv(timeout=None, return_message_object=False, dont_finalize=False, **kwargs)[source]¶

Receive a message.

Parameters:

*args – All arguments are passed to comm _recv method.
return_message_object (bool, optional) – If True, the full wrapped CommMessage message object is returned instead of the tuple. Defaults to False.
dont_finalize (bool, optional) – If True, finalize_message will not be called even if async_recv_method is ‘recv_message’. Defaults to False.
**kwargs – All keywords arguments are passed to comm _recv method.

Returns:

Success or failure of receive and received: message.

Return type:

tuple (bool, obj)

recv_array(*args, **kwargs)[source]¶: Alias for recv_array on wrapped comm.

recv_backlog()[source]¶: Check for any messages in the queue and add them to the recv backlog.

recv_dict(*args, **kwargs)[source]¶: Alias for recv_dict on wrapped comm.

recv_direct()[source]¶: Receive a message directly from the underlying comm.

recv_message(timeout=None, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to the response comm’s recv_message method.
**kwargs – Keyword arguments are passed to the response comm’s recv_message method.

Returns:

Received message.

Return type:

recv_nolimit(*args, **kwargs)[source]¶: Alias for recv_nolimit on wrapped comm.

run_backlog_recv()[source]¶: Continue buffering received messages.

run_backlog_send()[source]¶: Continue trying to send buffered messages.

send(*args, dont_prepare=False, **kwargs)[source]¶

Send a message to the backlog.

Parameters:

*args – All arguments are assumed to be part of the message.
**kwargs – All keywords arguments are passed to comm _send method.

Returns:

Success or failure of sending the message.

Return type:

send_array(*args, **kwargs)[source]¶: Alias for send_array on wrapped comm.

send_backlog()[source]¶: Send a message from the send backlog to the queue.

send_dict(*args, **kwargs)[source]¶: Alias for send_dict on wrapped comm.

send_direct(*args, **kwargs)[source]¶: Send a message directly to the underlying comm.

send_eof(*args, **kwargs)[source]¶

Send the EOF message as a short message.

Parameters:

*args – All arguments are passed to comm send.
**kwargs – All keywords arguments are passed to comm send.

Returns:

Success or failure of send.

Return type:

send_message(msg, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
**kwargs – Additional keyword arguments are passed to _safe_send.

Returns:

Success or failure of send.

Return type:

send_nolimit(*args, **kwargs)[source]¶: Alias for send_nolimit on wrapped comm.

yggdrasil.communication.BufferComm module¶

class yggdrasil.communication.BufferComm.BufferComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O to an in-memory buffer.

Parameters:

name (str) – The name of the message queue.
address (LockedBuffer, optional) – Existing buffer that should be used. If not provided, a new buffer is created.
buffer_task_method (str, optional) – Type of tasks that buffer will be used to share information between. Defaults to ‘thread’.
**kwargs – Additional keyword arguments are passed to CommBase.

address¶

Buffer containing messages.

Type:: LockedBuffer

buffer_task_method¶

Type of tasks that buffer will be used to share information between.

Type:: str

bind()[source]¶: Bind to new buffer.

property is_confirmed_recv¶

True if all received messages have been confirmed.

Type:: bool

property is_confirmed_send¶

True if all sent messages have been confirmed.

Type:: bool

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

property n_msg_recv¶

Number of messages in the receive backlog.

Type:: int

property n_msg_send¶

Number of messages in the send backlog.

Type:: int

no_serialization = True¶

open()[source]¶: Open the buffer.

purge()[source]¶: Purge all messages from the comm.

class yggdrasil.communication.BufferComm.LockedBuffer(*args, **kwargs)[source]¶

Bases: Queue

Buffer intended to be shared between threads/processes.

append(x)[source]¶: Add an element to the queue.

clear()[source]¶: Remove all elements from the queue.

close(join=False)[source]¶: Close the buffer.

property closed¶

True if the queue is closed, False otherwise.

Type:: bool

disconnect()[source]¶: Disconnect from the aliased object by replacing it with a dummy object.

pop(index=0, default=None)[source]¶: Remove the first element from the queue.

yggdrasil.communication.CABOFileComm module¶

class yggdrasil.communication.CABOFileComm.CABOFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a CABO parameter file on disk.

yggdrasil.communication.ClientComm module¶

class yggdrasil.communication.ClientComm.ClientComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling Client side communication.

Parameters:

name (str) – The environment variable where communication address is stored.
request_commtype (str, optional) – Comm class that should be used for the request comm. Defaults to None.
response_kwargs (dict, optional) – Keyword arguments for the response comm. Defaults to empty dict.
direct_connection (bool, optional) – If True, the comm will be directly connected to a ServerComm. Defaults to False.
**kwargs – Additional keywords arguments are passed to the output comm.

response_kwargs¶

Keyword arguments for the response comm.

Type:: dict

request_order¶

Order of request IDs.

Type:: list

responses¶

Mapping between request IDs and response messages.

Type:: dict

ocomm¶

Request comm.

Type:: Comm

icomm¶

Response comm.

Type:: Comm

atexit()[source]¶: Close operations.

call(*args, **kwargs)[source]¶

Do RPC call. The request message is sent to the output comm and the response is received from the input comm.

Parameters:

*args – Arguments are passed to output comm send method.
**kwargs – Keyword arguments are passed to output comm send method

Returns:

Output from input comm recv method.

Return type:

obj

call_nolimit(*args, **kwargs)[source]¶: Alias for call.

close(*args, **kwargs)[source]¶: Close the connection.

create_response_comm()[source]¶: Create a response comm based on information from the last header.

disconnect(*args, **kwargs)[source]¶: Disconnect the comm.

drain_messages(direction='send', **kwargs)[source]¶: Sleep while waiting for messages to be drained.

property filter¶

filter for the communicator.

Type:: FilterBase

finalize_message(msg, **kwargs)[source]¶

Perform actions to decipher a message.

Parameters:

msg (CommMessage) – Initial message object to be finalized.
**kwargs – Keyword arguments are passed to the response comm’s finalize_message method.

Returns:

Deserialized and annotated message.

Return type:

get_status_message(nindent=0, **kwargs)[source]¶

Return lines composing a status message.

Parameters:

nindent (int, optional) – Number of tabs that should be used to indent each line. Defaults to 0.
*kwargs – Additional arguments are passed to the parent class’s method.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

property is_closed¶

True if the connection is closed.

Type:: bool

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

property maxMsgSize¶

Maximum size of a single message that should be sent.

Type:: int

property n_msg_direct¶

Number of messages currently being routed.

Type:: int

property n_msg_send¶

The number of messages in the connection.

Type:: int

property n_msg_send_drain¶

The number of outgoing messages in the connection to drain.

Type:: int

classmethod new_comm_kwargs(name, request_commtype=None, **kwargs)[source]¶

Initialize communication with new comms.

Parameters:

name (str) – Name for new comm.
request_commtype (str, optional) – Name of class for new output comm. Defaults to None.

open()[source]¶: Open the connection.

property opp_address¶

Address for opposite comm.

Type:: str

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

property opp_comms¶

Name/address pairs for opposite comms.

Type:: dict

prepare_message(*args, **kwargs)[source]¶

Perform actions preparing to send a message.

Parameters:

*args – Components of the outgoing message.
**kwargs – Keyword arguments are passed to the request comm’s prepare_message method.

Returns:

Serialized and annotated message.

Return type:

recv_message(*args, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to the response comm’s recv_message method.
**kwargs – Keyword arguments are passed to the response comm’s recv_message method.

Returns:

Received message.

Return type:

rpcCall(*args, **kwargs)[source]¶: Alias for RPCComm.call

rpcRecv(*args, **kwargs)[source]¶: Alias for RPCComm.recv

rpcSend(*args, **kwargs)[source]¶: Alias for RPCComm.send

send_message(msg, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
**kwargs – Additional keyword arguments are passed to the request comm’s send_message method.

Returns:

Success or failure of send.

Return type:

yggdrasil.communication.CommBase module¶

class yggdrasil.communication.CommBase.CommBase(*args, **kwargs)[source]¶

Bases: YggClass

Class for handling I/O.

Parameters:

name (str) – The environment variable where communication address is stored.
address (str, optional) – Communication info. Default to None and address is taken from the environment variable.
direction (str, optional) – The direction that messages should flow through the connection. ‘send’ if the connection will send messages, ‘recv’ if the connecton will receive messages. Defaults to ‘send’.
is_interface (bool, optional) – Set to True if this comm is a Python interface binding. Defaults to False.
language (str, optional) – Programming language of the calling model. Defaults to ‘python’.
env (dict, optional) – Environment variable that should be used. Defaults to os.environ if not provided.
partner_model (str, optional) – Name of model that this comm is partnered with. Default to None, indicating that the partner is not a model.
partner_language (str, optional) – Programming language of this comm’s partner comm. Defaults to ‘python’.
partner_mpi_ranks (list, optional) – Ranks of processes of this comm’s partner comm(s). Defaults to [].
datatype (schema, optional) – JSON schema (with expanded core types defined by yggdrasil) that constrains the type of data that should be sent/received by this object. Defaults to {‘type’: ‘bytes’}. Additional information on specifying datatypes can be found here.
field_names (list, optional) – [DEPRECATED] Field names that should be used to label fields in sent/received tables. This keyword is only valid for table-like datatypes. If not provided, field names are created based on the field order.
field_units (list, optional) – [DEPRECATED] Field units that should be used to convert fields in sent/received tables. This keyword is only valid for table-like datatypes. If not provided, all fields are assumed to be unitless.
as_array (bool, optional) – [DEPRECATED] If True and the datatype is table-like, tables are sent/recieved with either columns rather than row by row. Defaults to False.
( (default_file) – class:.DefaultSerialize, optional): Class with serialize and deserialize methods that should be used to process sent and received messages or a dictionary describing a serializer that obeys the serializer schema.
format_str (str, optional) – String that should be used to format/parse messages. Default to None.
dont_open (bool, optional) – If True, the connection will not be opened. Defaults to False.
is_interface – Set to True if this comm is a Python interface binding. Defaults to False.
recv_timeout (float, optional) – Time that should be waited for an incoming message before returning None. Defaults to 0 (no wait). A value of False indicates that recv should block.
close_on_eof_recv (bool, optional) – If True, the comm will be closed when it receives an end-of-file messages. Otherwise, it will remain open. Defaults to True.
close_on_eof_send (bool, optional) – If True, the comm will be closed after it sends an end-of-file messages. Otherwise, it will remain open. Defaults to False.
single_use (bool, optional) – If True, the comm will only be used to send/recv a single message. Defaults to False.
reverse_names (bool, optional) – If True, the suffix added to the comm with be reversed. Defaults to False.
no_suffix (bool, optional) – If True, no directional suffix will be added to the comm name. Defaults to False.
allow_multiple_comms (bool, optional) – If True, initialize the comm such that mulitiple comms can connect to the same address. Defaults to False.
is_client (bool, optional) – If True, the comm is one of many potential clients that will be sending messages to one or more servers. Defaults to False.
is_response_client (bool, optional) – If True, the comm is a client-side response comm. Defaults to False.
is_server (bool, optional) – If True, the commis one of many potential servers that will be receiving messages from one or more clients. Defaults to False.
is_response_server (bool, optional) – If True, the comm is a server-side response comm. Defaults to False.
recv_converter (func, optional) – Converter that should be used on received objects. Defaults to None.
send_converter (func, optional) – Converter that should be used on sent objects. Defaults to None.
vars (list, optional) – Names of variables to be sent/received by this comm. Defaults to [].
length_map (dict, optional) – Map from pointer variable names to the names of variables where their length will be stored. Defaults to {}.
comm (str, optional) – The comm that should be created. This only serves as a check that the correct class is being created. Defaults to None.
( – class:.FilterBase, optional): Filter that will be used to determine when messages should be sent/received. Ignored if not provided.
( – class:.TransformBase, optional): One or more transformations that will be applied to messages that are sent/received. Ignored if not provided.
is_default (bool, optional) – If True, this comm was created to handle all input/output variables to/from a model. Defaults to False. This variable is used internally and should not be set explicitly in the YAML.
outside_loop (bool, optional) – If True, and the comm is an input/outputs to/from a model being wrapped. The receive/send calls for this comm will be outside the loop for the model. Defaults to False.
dont_copy (bool, optional) – If True, the comm will not be duplicated in the even a model is duplicated via the ‘copies’ parameter. Defaults to False except for in the case that a model is wrapped and the comm is inside the loop or that a model is a RPC input to a model server.
( – class:FileComm, optional): Comm information for a file that input should be drawn from (for input comms) or that output should be sent to (for output comms) in the event that a yaml does not pair the comm with another model comm or a file.
default_value (object, optional) – Value that should be returned in the event that a yaml does not pair the comm with another model comm or a file.
for_service (bool, optional) – If True, this comm bridges the gap to an integration running as a service, possibly on a remote machine. Defaults to False.
**kwargs – Additional keywords arguments are passed to parent class.

Class Attributes:: is_file (bool): True if the comm accesses a file. _maxMsgSize (int): Maximum size of a single message that should be sent. address_description (str): Description of the information constituting

an address for this communication mechanism.

name¶

The environment variable where communication address is stored.

Type:: str

address¶

Communication info.

Type:: str

direction¶

The direction that messages should flow through the connection.

Type:: str

is_interface¶

True if this comm is a Python interface binding.

Type:: bool

language¶

Language that this comm is being called from.

Type:: str

env¶

Environment variable that should be used.

Type:: dict

partner_model¶

Name of model that this comm is partnered with.

Type:: str

partner_language¶

Programming language of this comm’s partner comm.

Type:: str

partner_mpi_ranks¶

Ranks of processes of this comm’s partner comm(s).

Type:: list

serializer (: class:.DefaultSerialize): Object that will be used to serialize/deserialize messages to/from python objects.

recv_timeout¶

Time that should be waited for an incoming message before returning None.

Type:: float

close_on_eof_recv¶

If True, the comm will be closed when it receives an end-of-file messages. Otherwise, it will remain open.

Type:: bool

close_on_eof_send¶

If True, the comm will be closed after it sends an end-of-file messages. Otherwise, it will remain open.

Type:: bool

single_use¶

If True, the comm will only be used to send/recv a single message.

Type:: bool

allow_multiple_comms¶

If True, initialize the comm such that mulitiple comms can connect to the same address.

Type:: bool

is_client¶

If True, the comm is one of many potential clients that will be sending messages to one or more servers.

Type:: bool

is_response_client¶

If True, the comm is a client-side response comm.

Type:: bool

is_server¶

If True, the comm is one of many potential servers that will be receiving messages from one or more clients.

Type:: bool

is_response_server¶

If True, the comm is a server-side response comm.

Type:: bool

recv_converter¶

Converter that should be used on received objects.

Type:: func

send_converter¶

Converter that should be used on sent objects.

Type:: func

filter (: class:.FilterBase): Callable class that will be used to determine when messages should be sent/received.

Raises:

RuntimeError – If the comm class is not installed.
AddressError – If there is not an environment variable with the specified name.
ValueError – If directions is not ‘send’ or ‘recv’.

add_work_comm(comm)[source]¶

Add work comm to dict.

Parameters:: ( (comm) – class:.CommBase): Comm that should be added.
Raises:: KeyError – If there is already a comm associated with the key.

address_description = None¶

property any_files¶

True if the comm interfaces with any files.

Type:: bool

apply_transform(msg_in, for_empty=False, header=False)[source]¶

Evaluate the transform to alter the emssage being sent/received.

Parameters:

msg_in (object) – Message being transformed.
for_empty (bool, optional) – If True, the transformation is being used to check for an empty message and errors will be caught. Defaults to False.
header (dict, optional) – Header keyword arguments associated with a message. Defaults to False and is ignored.
typedef (dict, optiona) – Type to transform. Default to None and will be determined by the serializer if receiving.

Returns:

Transformed message.

Return type:

apply_transform_to_type(typedef)[source]¶

Evaluate the transform to alter the type definition.

Parameters:: typedef (dict) – Type definition to transform.
Returns:: Transformed type definition.
Return type:: dict

atexit()[source]¶: Close operations.

bind()[source]¶: Bind in place of open.

chunk_message(msg)[source]¶

Yield chunks of message of size maxMsgSize

Parameters:: msg (str, bytes) – Raw message bytes to be chunked.
Returns:: Chunks of message.
Return type:: str

classmethod cleanup_comms()[source]¶: Cleanup registered comms of this class.

close(linger=False, **kwargs)[source]¶

Close the connection.

Parameters:

linger (bool, optional) – If True, drain messages before closing the comm. Defaults to False.
**kwargs – Additional keyword arguments are passed to linger method if linger is True.

close_in_thread(no_wait=False, timeout=None)[source]¶

In a new thread, close the comm when it is empty.

Parameters:

no_wait (bool, optional) – If True, don’t wait for closing thread to stop.
timeout (float, optional) – Time that should be waited for the comm to close. Defaults to None and is set to self.timeout. If False, this will block until the comm is closed.

classmethod close_registry_entry(value)[source]¶: Close a registry entry.

coerce_to_dict(msg, key_order, metadata)[source]¶

Convert a message to a dictionary.

Parameters:

msg (object) – Message to convert to a dictionary.
key_order (list) – Key order to use for the output dictionary.
metadata (dict) – Header data to accompany the message.

Returns:

Converted message.

Return type:

classmethod comm_count()[source]¶: int: Number of communication connections.

classmethod comm_registry()[source]¶: dict: Registry of comms of this class.

confirm(direction=None, noblock=False)[source]¶: Confirm message.

confirm_recv(noblock=False)[source]¶: Confirm that message was received.

confirm_send(noblock=False)[source]¶: Confirm that sent message was received.

create_work_comm(work_comm_name=None, **kwargs)[source]¶

Create a temporary work comm.

Parameters:

work_comm_name (str, optional) – Name that should be used for the work comm. If not provided, one is created from the header id and the comm class.
**kwargs – Keyword arguments for new_comm that should override work_comm_kwargs.

Returns:

class:.CommBase: Work comm.

property create_work_comm_kwargs¶

Keyword arguments for a new work comm.

Type:: dict

deserialize(*args, **kwargs)[source]¶: Deserialize a message using the associated deserializer.

drain_messages(direction=None, timeout=None, variable=None)[source]¶: Sleep while waiting for messages to be drained.

drain_server_signon_messages(**kwargs)[source]¶: Drain server signon messages. This should only be used for testing purposes.

property empty_bytes_msg¶

Empty serialized message.

Type:: str

property empty_obj_recv¶

Empty message object.

Type:: obj

property eof_msg¶

Message indicating EOF.

Type:: str

evaluate_filter(*msg_in)[source]¶

Evaluate the filter to determine how the message should be handled.

Parameters:: *msg_in (object) – Parts of message being evaluated.
Returns:: True if the filter evaluates to True, False otherwise.
Return type:: bool

extract_key_order(kwargs)[source]¶

Extract the key order from keyword arguments.

Parameters:: kwargs (dict) – Keyword arguments.
Returns:: Key order.
Return type:: list

finalize_message(msg, skip_processing=False, skip_python2language=False, after_finalize_message=None)[source]¶

Perform actions to decipher a message. The order of steps is

Transform the message

Filter

python2language

Close comm on EOF if close_on_eof_recv set

Check for empty recv after processing

Mark comm as used and close if single use

Apply after_finalize_message functions

Parameters:

msg (CommMessage) – Initial message object to be finalized.
skip_processing (bool, optional) – If True, filters, transformations, and after_finalize_message funciton applications will not be performed. Defaults to False.
skip_python2language (bool, optional) – If True, python2language will not be applied. Defaults to False.
after_finalize_message (list, optional) – A set of function that should be applied to received CommMessage objects following the standard finalization. Defaults to None and is ignored.

Returns:

Deserialized and annotated message.

Return type:

property full_model_name¶

Name of the model using the comm w/ copy suffix.

Type:: str

property get_response_comm_kwargs¶

Keyword arguments to use for a response comm.

Type:: dict

get_status_message(nindent=0, extra_lines_before=None, extra_lines_after=None)[source]¶

Return lines composing a status message.

Parameters:

nindent (int, optional) – Number of tabs that should be used to indent each line. Defaults to 0.
extra_lines_before (list, optional) – Additional lines that should be added to the beginning of the default print message. Defaults to empty list if not provided.
extra_lines_after (list, optional) – Additional lines that should be added to the end of the default print message. Defaults to empty list if not provided.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

classmethod get_testing_options(serializer=None, test_dir=None, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

serializer (str, optional) – The name of the serializer that should be used. If not provided, the _default_serializer class attribute will be used.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the: provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a test

file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by sending: the messages in ‘send’.

Return type:

get_work_comm(header, **kwargs)[source]¶

Get temporary work comm, creating as necessary.

Parameters:

header (dict) – Information that will be sent in the message header to the work comm.
**kwargs – Additional keyword arguments are passed to header2workcomm.

Returns:

class:.CommBase: Work comm.

property get_work_comm_kwargs¶

Keyword arguments for an existing work comm.

Type:: dict

header2workcomm(header, work_comm_name=None, **kwargs)[source]¶

Get a work comm based on header info.

Parameters:

header (dict) – Information that will be sent in the message header to the work comm.
work_comm_name (str, optional) – Name that should be used for the work comm. If not provided, one is created from the header id and the comm class.
**kwargs – Additional keyword arguments are added to the returned dictionary.

Returns:

class:.CommBase: Work comm.

property is_closed¶

True if the connection is closed.

Type:: bool

property is_confirmed¶

True if all messages have been confirmed.

Type:: bool

property is_confirmed_recv¶

True if all received messages have been confirmed.

Type:: bool

property is_confirmed_send¶

True if all sent messages have been confirmed.

Type:: bool

is_empty(msg, emsg)[source]¶

Check that a message matches an empty message object.

Parameters:

msg (object) – Message object.
emsg (object) – Empty message object.

Returns:

True if the object is empty, False otherwise.

Return type:

is_empty_recv(msg)[source]¶

Check if a received message object is empty.

Parameters:: msg (obj) – Message object.
Returns:: True if the object is empty, False otherwise.
Return type:: bool

is_eof(msg)[source]¶

Determine if a message is an EOF.

Parameters:: msg (obj) – Message object to be tested.
Returns:: True if the message indicates an EOF, False otherwise.
Return type:: bool

is_file = False¶

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

classmethod is_registered(key)[source]¶: bool: True if the comm is registered, False otherwise.

language_atexit()[source]¶: Close operations specific to the language.

linger(active_confirm=False)[source]¶: Wait for messages to drain.

linger_close(**kwargs)[source]¶: Wait for messages to drain, then close.

property maxMsgSize¶

Maximum size of a single message that should be sent.

Type:: int

property model_copies¶

Number of copies of the model using the comm.

Type:: int

property model_env¶

Mapping between model name and opposite comm environment variables that need to be provided to the model.

Type:: dict

property model_name¶

Name of the model using the comm.

Type:: str

property n_msg¶

The number of messages in the connection.

Type:: int

property n_msg_recv¶

The number of incoming messages in the connection.

Type:: int

property n_msg_recv_drain¶

The number of incoming messages in the connection to drain.

Type:: int

property n_msg_send¶

The number of outgoing messages in the connection.

Type:: int

property n_msg_send_drain¶

The number of outgoing messages in the connection to drain.

Type:: int

classmethod new_comm(name, *args, **kwargs)[source]¶: Initialize communication with new queue.

classmethod new_comm_kwargs(*args, **kwargs)[source]¶: Get keyword arguments for new comm.

new_server(srv_address)[source]¶

Create a new server.

Parameters:: srv_address (str) – Address of server comm.

no_serialization = False¶

open()[source]¶: Open the connection.

property opp_address¶

Address for opposite comm.

Type:: str

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

property opp_comms¶

Name/address pairs for opposite comms.

Type:: dict

property opp_name¶

Name that should be used for the opposite comm.

Type:: str

precheck(direction)[source]¶

Check that comm is ready for action in specified direction, raising errors if it is not.

Parameters:: direction (str) – Check that comm is ready to perform this action.

prepare_header(header_kwargs)[source]¶: Prepare header kwargs for the communicator.

prepare_message(*args, header_kwargs=None, skip_serialization=False, skip_processing=False, skip_language2python=False, after_prepare_message=None, flag=None)[source]¶

Perform actions preparing to send a message. The order of steps is

Convert the message based on the language

Isolate the message if there is only one

Check if the message is EOF

Check if the message should be filtered

Transform the message

Apply after_prepare_message functions

Serialize the message

Create a work comm if the message is too large to be sent all at once

Parameters:

*args – Components of the outgoing message.
header_kwargs (dict, optional) – Header options that should be set.
skip_serialization (bool, optional) – If True, serialization will not be performed. Defaults to False.
skip_processing (bool, optional) – If True, filters, transformations, and after_prepare_message function applications will not be performed. Defaults to False.
skip_language2python (bool, optional) – If True, language2python will be skipped. Defaults to False.
after_prepare_message (list, optional) – Functions that should be applied after transformation, but before serialization. Defaults to None and is ignored.
flag (int, optional) – Flag that should be added to the message before any additional processing is performed. Defaults to None and is ignored.

Returns:

Serialized and annotated message.

Return type:

printStatus(*args, level='info', return_str=False, **kwargs)[source]¶: Print status of the communicator.

purge()[source]¶: Purge all messages from the comm.

recv(*args, return_message_object=False, **kwargs)[source]¶

Receive a message.

Parameters:

*args – All arguments are passed to comm _recv method.
return_message_object (bool, optional) – If True, the full wrapped CommMessage message object is returned instead of the tuple. Defaults to False.
**kwargs – All keywords arguments are passed to comm _recv method.

Returns:

Success or failure of receive and received: message. If return_message_object is True, the CommMessage object will be returned instead.

Return type:

tuple (bool, obj)

recv_array(*args, **kwargs)[source]¶: Alias for recv.

recv_dict(*args, **kwargs)[source]¶

Return a received message as a dictionary of fields. If there are not any fields specified, the fields will have the form ‘f0’, ‘f1’, ‘f2’, …

Parameters:

*args – Arguments are passed to recv.
**kwargs – Keyword arguments are passed to recv.

Returns:

Success/failure of receive and a dictionar of: message fields.

Return type:

tuple(bool, dict)

Raises:

recv_message(*args, skip_deserialization=False, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to _safe_recv.
skip_deserialization (bool, optional) – If True, deserialization is not performed. Defaults to False.
**kwargs – Additional keyword arguments are passed to _safe_recv.

Returns:

Received message.

Return type:

recv_nolimit(*args, **kwargs)[source]¶: Alias for recv.

classmethod register_comm(key, value)[source]¶: Register a comm.

remove_work_comm(key, in_thread=False, linger=False)[source]¶

Close and remove a work comm.

Parameters:

key (str) – Key of comm that should be removed.
in_thread (bool, optional) – If True, close the work comm in a thread. Defaults to False.
linger (bool, optional) – If True, drain messages before closing the comm. Defaults to False.

send(*args, **kwargs)[source]¶

Send a message.

Parameters:

*args – All arguments are assumed to be part of the message.
**kwargs – All keywords arguments are passed to prepare_message or send_message.

Returns:

Success or failure of send.

Return type:

send_array(*args, **kwargs)[source]¶: Alias for send.

send_dict(args_dict, **kwargs)[source]¶

Send a message with fields specified in the input dictionary.

Parameters:

args_dict (dict) – Dictionary of arguments to send.
**kwargs – Additiona keyword arguments are passed to send.

Returns:

Success/failure of send.

Return type:

Raises:

RuntimeError – If the field order can not be determined.

send_eof(*args, **kwargs)[source]¶

Send the EOF message as a short message.

Parameters:

*args – All arguments are passed to comm send.
**kwargs – All keywords arguments are passed to comm send.

Returns:

Success or failure of send.

Return type:

send_message(msg, skip_safe_send=False, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
skip_safe_send (bool, optional) – If True, no actual send will take place. Defaults to False.
**kwargs – Additional keyword arguments are passed to _safe_send.

Returns:

Success or failure of send.

Return type:

send_nolimit(*args, **kwargs)[source]¶: Alias for send.

serialize(*args, **kwargs)[source]¶: Serialize a message using the associated serializer.

server_exists(srv_address)[source]¶

Determine if a server exists.

Parameters:

srv_address (str) – Address of server comm.

Returns:

True if a server with the provided address exists, False: otherwise.

Return type:

signoff_from_server()[source]¶: Remove a client from the server.

signon_to_server()[source]¶: Add a client to an existing server or create one.

classmethod underlying_comm_class()[source]¶: str: Name of underlying communication class.

classmethod unregister_comm(key, dont_close=False)[source]¶: Unregister a comm.

update_message_from_serializer(msg)[source]¶

Update a message with information about the serializer.

Parameters:: msg (CommMessage) – Incoming message.

update_serializer_from_message(msg)[source]¶

Update the serializer based on information stored in a message.

Parameters:: msg (CommMessage) – Outgoing message.

wait_for_confirm(timeout=None, direction=None, active_confirm=False, noblock=False)[source]¶: Sleep until all messages are confirmed.

wait_for_workers(timeout=None)[source]¶: Sleep until all workers are closed or have been used.

workcomm2header(work_comm, **kwargs)[source]¶

Get header information from a comm.

Parameters:

( (work_comm) – class:.CommBase): Work comm that header describes.
**kwargs – Additional keyword arguments are added to the header.

Returns:

Header information that will be sent with a message.

Return type:

class yggdrasil.communication.CommBase.CommMessage(msg=None, length=0, flag=None, args=None, header=None)[source]¶

Bases: object

Class for passing messages around with additional information required to send/receive them.

msg¶

The serialized message including the header.

Type:: bytes

length¶

The size of the message.

Type:: int

flag¶

Indicates the result of processing the message. Values are: FLAG_FAILURE: Processing was unsuccessful. FLAG_SUCCESS: Processing was successful. FLAG_SKIP: The message should be skipped. FLAG_EOF: The message indicates that there will be no more messages.

Type:: int

args¶

The unserialized message (post-transformation).

Type:: object

header¶

Parameters sent in the header of the message.

Type:: dict

additional_messages¶

Messages that should be sent along with this message as in the case that the message was an iterator.

Type:: list

worker¶

Worker communicator that should be used to send worker messages in the case that the original message had to be split.

Type:: CommBase

worker_messages¶

Messages that should be sent via the worker comm comm as the original message had to be split due to its size.

Type:: list

sent¶

True if the message has been sent, False otherwise.

Type:: bool

singular¶

True if there was only one argument.

Type:: bool

add_message(*args, **kwargs)[source]¶

Add a message to the list of additional messages that should be sent following this one.

Parameters:

*args – Arguments are passed to the CommMessage constructor.
*kwargs – Keyword arguments are passed to the CommMessage constructor.

add_worker_message(*args, **kwargs)[source]¶

Add a message to the list of messages that should be sent via work comm following this one.

Parameters:

*args – Arguments are passed to the CommMessage constructor.
*kwargs – Keyword arguments are passed to the CommMessage constructor.

additional_messages¶

apply_function(x)[source]¶

Apply a function to the message.

Parameters:: x (function) – Function to apply.

args¶

finalized¶

flag¶

header¶

length¶

msg¶

send_worker_messages(**kwargs)[source]¶

Send the worker messages via the worker comm.

Parameters:: **kwargs – Keyword arguments are passed to the send_message method of the worker comm for each message.
Returns:: Success of the send operations.
Return type:: bool

sent¶

sinfo¶

singular¶

stype¶

property tuple_args¶

Form that arguments were originally supplied.

Type:: tuple

worker¶

worker_messages¶

class yggdrasil.communication.CommBase.CommServer(*args, **kwargs)[source]¶

Bases: YggTaskLoop

Basic server object to keep track of clients.

cli_count¶

Number of clients that have connected to this server.

Type:: int

add_client()[source]¶: Increment the client count.

remove_client()[source]¶: Decrement the client count, closing the server if all clients done.

class yggdrasil.communication.CommBase.CommTaskLoop(*args, **kwargs)[source]¶

Bases: YggTaskLoop

Task loop for comms to ensure cleanup.

Parameters:

( (comm) – class:.CommBase): Comm class that thread is for.
name (str, optional) – Name for the thread. If not provided, one is created by combining the comm name and the provided suffix.
suffix (str, optional) – Suffix that should be added to comm name to name the thread. Defaults to ‘CommTask’.
**kwargs – Additional keyword arguments are passed to the parent class.

comm (: class:.CommBase): Comm class that thread is for.

on_main_terminated()[source]¶: Actions taken on the backlog thread when the main thread stops.

exception yggdrasil.communication.CommBase.IncompleteBaseComm[source]¶

Bases: Exception

An exception class for methods that are incomplete for base classes.

exception yggdrasil.communication.CommBase.NeverMatch[source]¶

Bases: Exception

An exception class that is never raised by any code anywhere

yggdrasil.communication.CommBase.cleanup_comms(commtype, close_func=None)[source]¶

Clean up comms of a certain type.

Parameters:: commtype (str) – Comm class that should be cleaned up.
Returns:: Number of comms closed.
Return type:: int

yggdrasil.communication.CommBase.get_comm_registry(commtype)[source]¶

Get the comm registry for a comm class.

Parameters:: commtype (str) – Comm class to get registry for.
Returns:: Dictionary of registered comm objects.
Return type:: dict

yggdrasil.communication.CommBase.is_registered(commtype, key)[source]¶

Determine if a comm object has been registered under the specified key.

Parameters:

commtype (str) – Comm class to check for the key under.
key (str) – Key that should be checked.

yggdrasil.communication.CommBase.register_comm(commtype, key, value)[source]¶

Add a comm object to the global registry.

Parameters:

commtype (str) – Comm class to register the object under.
key (str) – Key that should be used to register the object.
value (obj) – Object being registered.

yggdrasil.communication.CommBase.unregister_comm(commtype, key, dont_close=False)[source]¶

Remove a comm object from the global registry and close it.

Parameters:

commtype (str) – Comm class to check for key under.
key (str) – Key for object that should be removed from the registry.
dont_close (bool, optional) – If True, the comm will be removed from the registry, but it won’t be closed. Defaults to False.

Returns:

True if an object was closed.

Return type:

yggdrasil.communication.DedicatedFileBase module¶

class yggdrasil.communication.DedicatedFileBase.DedicatedFileBase(*args, **kwargs)[source]¶

Bases: FileComm

Base class for handling I/O via a dedicated library.

property concats_as_str¶

True if concatenating file contents result in a valid file.

Type:: bool

deserialize(msg, **kwargs)[source]¶: Don’t deserialize for dedicated comms since using a serializer is inefficient.

file_flush()[source]¶: Flush the file.

file_seek(pos, whence=0)[source]¶

Move in the file to the specified position.

Parameters:

pos (int) – Position (in bytes) to move file to.
whence (int, optional) – Flag indicating position that pos is relative to. 0 for the beginning of the file, 1 for from the current location, and 2 from the end of the file.

property file_size¶

Current size of file.

Type:: int

file_tell()[source]¶: int: Current position in the file.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Items:

kwargs (dict): Keyword arguments for comms tested with: the provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a

test file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by: sending the messages in ‘send’.

Return type:

property is_open¶

True if the connection is open.

Type:: bool

property remaining_bytes¶

Remaining bytes in the file.

Type:: int

property requires_refresh¶

True if a refresh is necessary.

Type:: bool

serialize(obj, **kwargs)[source]¶: Don’t serialize for dedicated comms since using a serializer is inefficient.

yggdrasil.communication.DefaultComm module¶

class yggdrasil.communication.DefaultComm.DefaultComm(*args, **kwargs)[source]¶

Bases: CommBase

Simple wrapper for default class that allows it to be registered.

yggdrasil.communication.ExcelFileComm module¶

class yggdrasil.communication.ExcelFileComm.ExcelFileComm(*args, **kwargs)[source]¶

Class for handling I/O from/to an Excel file.

advance_in_series(series_index=None)[source]¶

Advance to a certain file in a series.

Parameters:: series_index (int, optional) – Index of file in the series that should be moved to. Defaults to None and call will advance to the next file in the series.
Returns:: True if the file was advanced in the series, False otherwise.
Return type:: bool

concats_as_str = True¶

property current_sheet¶

Current sheet name.

Type:: str

file_seek(*args, **kwargs)[source]¶: Move in the file to the specified position.

property file_size¶

Current size of file.

Type:: int

classmethod get_testing_options(sheets=None, columns=None, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

sheets (list, optional) – Sheet names that should be specified.
columns (list, optional) – Column names that should be specified.

Returns:

Dictionary of variables to use for testing. Items:

kwargs (dict): Keyword arguments for comms tested with: the provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a

test file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by: sending the messages in ‘send’.

Return type:

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

no_serialization = True¶

pop_current_sheet()[source]¶: Remove the current sheet from the list of remaining sheets.

read_header()[source]¶: Read header lines from the file and update serializer info.

write_header()[source]¶: Write header lines to the file based on the serializer info.

yggdrasil.communication.FileComm module¶

class yggdrasil.communication.FileComm.FileComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O from/to a file on disk.

>>> x = FileComm('test_send', address='test_file.txt', direction='send')
>>> x.send('Test message')
True
>>> with open('test_file.txt', 'r') as fd:
...     print(fd.read())
Test message
>>> x = FileComm('test_recv', address='test_file.txt', direction='recv')
>>> x.recv()
(True, b'Test message')

Parameters:

name (str) – The environment variable where communication address is stored.
read_meth (str, optional) – Method that should be used to read data from the file. Defaults to ‘read’. Ignored if direction is ‘send’.
append (bool, optional) – If True and writing, file is openned in append mode. If True and reading, file is kept open even if the end of the file is reached to allow for another process to write to the file in append mode. Defaults to False.
in_temp (bool, optional) – If True, the path will be considered relative to the platform temporary directory. Defaults to False.
is_series (bool, optional) – If True, input/output will be done to a series of files. If reading, each file will be processed until the end is reached. If writing, each output will be to a new file in the series. The addressed is assumed to contain a format for the index of the file. Defaults to False.
count (int, optional) – When reading a file, read the file this many of times. Defaults to 0.
wait_for_creation (float, optional) – Time (in seconds) that should be waited before opening for the file to be created if it dosn’t exist. Defaults to 0 s and file will attempt to be opened immediately.
**kwargs – Additional keywords arguments are passed to parent class.

fd¶

File that should be read/written.

Type:: file

read_meth¶

Method that should be used to read data from the file.

Type:: str

append¶

If True and writing, file is openned in append mode.

Type:: bool

in_temp¶

If True, the path will be considered relative to the platform temporary directory.

Type:: bool

is_series¶

If True, input/output will be done to a series of files. If reading, each file will be processed until the end is reached. If writing, each output will be to a new file in the series.

Type:: bool

platform_newline¶

String indicating a newline on the current platform.

Type:: str

Raises:: ValueError – If the read_meth is not one of the supported values.

advance_in_file(file_pos)[source]¶

Advance to a certain position in the current file.

Parameters:: file_pos (int) – Position that should be moved to in the current. file.

advance_in_series(series_index=None)[source]¶

Advance to a certain file in a series.

Parameters:: series_index (int, optional) – Index of file in the series that should be moved to. Defaults to None and call will advance to the next file in the series.
Returns:: True if the file was advanced in the series, False otherwise.
Return type:: bool

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration.

change_position(file_pos, series_index=None, header_was_read=None, header_was_written=None)[source]¶

Change the position in the file/series.

Parameters:

file_pos (int) – Position that should be moved to in the file.
series_index (int, optinal) – Index of the file in the series that should be moved to. Defaults to None and will be set to the current series index.
header_was_read (bool, optional) – Status of if header has been read or not. Defaults to None and will be set to the current value.
header_was_written (bool, optional) – Status of if header has been written or not. Defaults to None and will be set to the current value.

classmethod close_registry_entry(value)[source]¶: Close a registry entry.

property concats_as_str¶

True if concatenating file contents result in a valid file.

Type:: bool

property current_address¶

Address of file currently being used.

Type:: str

disable_header()[source]¶: Turn off header so that it will not be written.

dump(obj, **kwargs)[source]¶

Serialize to a file.

Parameters:: **kwargs – Keyword arguments are passed to the send call.
Raises:: SerializationError – If the send call fails.

enable_header()[source]¶: Turn on header so that it will be written.

property fd¶: Associated file identifier.

file_flush()[source]¶: Flush the file.

file_seek(pos, whence=0)[source]¶

Move in the file to the specified position.

Parameters:

pos (int) – Position (in bytes) to move file to.
whence (int, optional) – Flag indicating position that pos is relative to. 0 for the beginning of the file, 1 for from the current location, and 2 from the end of the file.

property file_size¶

Current size of file.

Type:: int

file_tell()[source]¶: int: Current position in the file.

get_series_address(index=None)[source]¶

Get the address of a file in the series.

Parameters:: index (int, optional) – Index in series to get address for. Defaults to None and the current index is used.
Returns:: Address for the file in the series.
Return type:: str

classmethod get_test_contents(data, **kwargs)[source]¶

Method for returning the serialized form of a set of test data.

Parameters:: data (list) – List of test data objects to serialize.
Returns:: Serialized test data.
Return type:: bytes

classmethod get_testing_options(read_meth=None, serializer=None, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

read_meth (str, optional) – Read method that will be used by the test class. Defaults to None and will be set by the serialier.
**kwargs – Additional keyword arguments are passed to the parent class’s method and the serializers methods for determining the default read_meth and concatenating the sent objects into the objects that are expected to be received.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the: provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a test

file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by sending: the messages in ‘send’.

Return type:

is_file = True¶

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

load(return_message_object=False, **kwargs)[source]¶

Deserialize all contents from a file.

Parameters:

**kwargs – Keyword arguments are passed to recv calls.

Returns:

The deserialized data object or a list of: deserialized data objects if there is more than one.

Return type:

Raises:

SerializationError – If the first recv call fails.

property n_msg_recv¶

The number of messages in the file.

Type:: int

classmethod new_comm_kwargs(*args, **kwargs)[source]¶: Initialize communication with new queue.

open()[source]¶: Open the file.

property open_mode¶

Mode that should be used to open the file.

Type:: str

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

prepare_message(*args, **kwargs)[source]¶

Perform actions preparing to send a message.

Parameters:

*args – Components of the outgoing message.
**kwargs – Keyword arguments are passed to the parent class’s method.

Returns:

Serialized and annotated message.

Return type:

purge()[source]¶: Purge all messages from the comm.

read_header()[source]¶: Read header lines from the file and update serializer info.

record_position()[source]¶: Record the current position in the file/series.

property registry_key¶

String used to register the socket.

Type:: str

property remaining_bytes¶

Remaining bytes in the file.

Type:: int

classmethod remove_companion_files(address)[source]¶

Remove companion files that are created when writing to a file

Parameters:: address (str) – Address for the filename.

remove_file()[source]¶: Remove the file.

reset_position(truncate=False)[source]¶

Move to the front of the file and allow header to be read again.

Parameters:: truncate (bool, optional) – If True, the file will be truncated after moving to the beginning, effectively erasing the file. Defaults to False.

serialize(obj, **kwargs)[source]¶: Serialize a message using the associated serializer.

series_file_size(fname)[source]¶: int: Size of file in series.

classmethod underlying_comm_class()[source]¶: str: Name of underlying communication class.

write_header()[source]¶: Write header lines to the file based on the serializer info.

yggdrasil.communication.FileComm.convert_file(src, dst, src_type=None, dst_type=None, src_kwargs=None, dst_kwargs=None, transform=None)[source]¶

Convert from one file type to another.

Parameters:

src (str) – Path to source file to convert.
dst (str) – Path to destination file that should be created.
src_type (str, dict, optional) – Name of source file type. If not provided, an attempt will be made to identify the file type from the extension.
dst_type (str, dict, optional) – Name of destination file type. If not provided, an attempt will be made to identify the file type from the extension.
transform (dict, optional) – Transform parameters for transforming messages between the soruce and destination file.

Raises:

IOError – If the source file does not exist.
IOError – If the destination file exists.

yggdrasil.communication.FileComm.is_file_like(x)[source]¶

Check if an object is file-like via duck typing.

Parameters:: x (object) – Object to check.
Returns:: True if file-like, False otherwise.
Return type:: bool

yggdrasil.communication.ForkComm module¶

class yggdrasil.communication.ForkComm.ForkComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for receiving/sending messages from/to multiple comms.

Parameters:

name (str) – The environment variable where communication address is stored.
comm_list (list, optional) – The list of options for the comms that should be bundled. If not provided, the bundle will be empty.
pattern (str, optional) –
The communication pattern that should be used to handle outgoing/incoming messages. Options include:

’cycle’: Receive from or send to the next available comm in
the list (default for receiving comms).

’broadcast’: [SEND ONLY] Send the same message to each comm
(default for sending comms).

’scatter’: [SEND ONLY] Send part of message (must be a list)
to each comm.

’gather’: [RECV ONLY] Receive lists of messages from each
comm where a message is only returned when there is a message from each.
**kwargs – Additional keyword arguments are passed to the parent class.

comm_list¶

Comms included in this fork.

Type:: list

curr_comm_index¶

Index comm that next receive will be from.

Type:: int

property any_files¶

True if the comm interfaces with any files.

Type:: bool

bind()[source]¶: Bind in place of open.

child_keys = ['serializer_class', 'serializer_kwargs', 'format_str', 'field_names', 'field_units', 'as_array', 'partner_copies']¶

close(*args, **kwargs)[source]¶: Close the connection.

close_in_thread(*args, **kwargs)[source]¶: In a new thread, close the comm when it is empty.

coerce_to_dict(msg, key_order, metadata)[source]¶

Convert a message to a dictionary.

Parameters:

msg (object) – Message to convert to a dictionary.
key_order (list) – Key order to use for the output dictionary.
metadata (dict) – Header data to accompany the message.

Returns:

Converted message.

Return type:

confirm_recv(noblock=False)[source]¶: Confirm that message was received.

confirm_send(noblock=False)[source]¶: Confirm that sent message was received.

property curr_comm¶

Current comm.

Type:: CommBase

disconnect()[source]¶: Disconnect attributes that are aliases.

drain_server_signon_messages(**kwargs)[source]¶: Drain server signon messages. This should only be used for testing purposes.

property empty_obj_recv¶

Empty message object.

Type:: obj

finalize_message(msg, **kwargs)[source]¶

Perform actions to decipher a message.

Parameters:

msg (CommMessage) – Initial message object to be finalized.
**kwargs – Keyword arguments are passed to the forked comm’s finalize_message method.

Returns:

Deserialized and annotated message.

Return type:

property get_response_comm_kwargs¶

Keyword arguments to use for a response comm.

Type:: dict

get_status_message(**kwargs)[source]¶

Return lines composing a status message.

Parameters:

**kwargs – Keyword arguments are passed on to the parent class’s method.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

property is_confirmed_recv¶

True if all received messages have been confirmed.

Type:: bool

property is_confirmed_send¶

True if all sent messages have been confirmed.

Type:: bool

property is_open¶

True if the connection is open.

Type:: bool

property last_comm¶

Last comm that was used.

Type:: CommBase

property maxMsgSize¶

Maximum size of a single message that should be sent.

Type:: int

property model_env¶

Mapping between model name and opposite comm environment variables that need to be provided to the model.

Type:: dict

property n_msg_direct¶

Number of messages currently being routed.

Type:: int

property n_msg_direct_recv¶

Number of messages currently being routed in recv.

Type:: int

property n_msg_direct_send¶

Number of messages currently being routed in send.

Type:: int

property n_msg_recv¶

The number of incoming messages in the connection.

Type:: int

property n_msg_recv_drain¶

The number of incoming messages in the connection to drain.

Type:: int

property n_msg_send¶

The number of outgoing messages in the connection.

Type:: int

property n_msg_send_drain¶

The number of outgoing messages in the connection to drain.

Type:: int

classmethod new_comm_kwargs(name, *args, **kwargs)[source]¶: Get keyword arguments for new comm.

noprop_keys = ['send_converter', 'recv_converter', 'filter', 'transform']¶

open()[source]¶: Open the connection.

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

property opp_comms¶

Name/address pairs for opposite comms.

Type:: dict

prepare_message(*args, **kwargs)[source]¶

Perform actions preparing to send a message.

Parameters:

*args – Components of the outgoing message.
**kwargs – Additional keyword arguments are passed to the prepare_message methods for the forked comms.

Returns:

Serialized and annotated message.

Return type:

purge()[source]¶: Purge all messages from the comm.

recv_message(*args, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to the forked comm’s recv_message method.
**kwargs – Keyword arguments are passed to the forked comm’s recv_message method.

Returns:

Received message.

Return type:

send_message(msg, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
**kwargs – Additional keyword arguments are passed to _safe_send.

Returns:

Success or failure of send.

Return type:

property suppress_special_debug¶

update_serializer_from_message(msg)[source]¶

Update the serializer based on information stored in a message.

Parameters:: msg (CommMessage) – Outgoing message.

class yggdrasil.communication.ForkComm.ForkedCommMessage(msg, comm_list, pattern='broadcast', **kwargs)[source]¶

Bases: CommMessage

Class for forked comm messages.

Parameters:

msg (CommBase.CommMessage) – Message being distributed.
comm_list (list) – List of communicators that the message is being distributed to.
**kwargs – Additional keyword arguments are passed to the ‘prepare_message’ method for each communicator.

orig¶

yggdrasil.communication.ForkComm.get_comm_name(name, i)[source]¶

Get the name of the ith comm in the series.

Parameters:

name (str) – Name of the fork comm.
i (int) – Index of comm in fork bundle.

Returns:

Name of ith comm in fork bundle.

Return type:

yggdrasil.communication.IPCComm module¶

class yggdrasil.communication.IPCComm.IPCComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O via IPC message queues.

q¶

Message queue.

Type:: sysv_ipc.MessageQueue

Developer Notes:: The default size limit for IPC message queues is 2048 bytes on Mac operating systems so it is important that implementation of this communication mechanism properly split and send messages larger than this limit as more than one message.

address_description = 'An IPC message queue key.'¶

atexit()[source]¶: Close operations.

bind()[source]¶: Bind to random queue if address is generate.

classmethod close_registry_entry(value)[source]¶: Close a registry entry.

confirm_recv(noblock=False)[source]¶: Confirm that message was received.

confirm_send(noblock=False)[source]¶: Confirm that sent message was received.

property is_open¶

True if the queue is not None.

Type:: bool

property n_msg_recv¶

Number of messages in the queue to recv.

Type:: int

property n_msg_send¶

Number of messages in the queue to send.

Type:: int

classmethod new_comm_kwargs(*args, **kwargs)[source]¶: Initialize communication with new queue.

open()[source]¶: Open the queue.

open_after_bind()[source]¶: Open the connection by getting the queue from the bound address.

purge()[source]¶: Purge all messages from the comm.

class yggdrasil.communication.IPCComm.IPCServer(*args, **kwargs)[source]¶

Bases: CommServer

IPC server object for cleaning up server queue.

terminate(*args, **kwargs)[source]¶: Also set break flag.

yggdrasil.communication.IPCComm.get_queue(qid=None)[source]¶

Create or return a sysv_ipc.MessageQueue and register it.

Parameters:: qid (int, optional) – If provided, ID for existing queue that should be returned. Defaults to None and a new queue is returned.
Returns:: Message queue.
Return type:: sysv_ipc.MessageQueue

yggdrasil.communication.IPCComm.ipc_queues(by_id=False)[source]¶

Get a list of active IPC queues.

Returns:: List of IPC queues.
Return type:: list

yggdrasil.communication.IPCComm.ipcrm(options=[])[source]¶

Remove IPC constructs using the ipcrm command.

Parameters:: options (list) – List of flags that should be used. Defaults to an empty list.

yggdrasil.communication.IPCComm.ipcrm_queues(queue_keys=None, by_id=False)[source]¶

Delete existing IPC queues.

Parameters:: queue_keys (list, str, optional) – A list of keys for queues that should be removed. Defaults to all existing queues.

yggdrasil.communication.IPCComm.ipcs(options=[])[source]¶

Get the output from running the ipcs command.

Parameters:: options (list) – List of flags that should be used. Defaults to an empty list.
Returns:: Captured output.
Return type:: str

yggdrasil.communication.IPCComm.remove_queue(mq)[source]¶

Remove a sysv_ipc.MessageQueue and unregister it.

Parameters:: mq (sysv_ipc.MessageQueue) –
Raises:: KeyError – If the provided queue is not registered.

yggdrasil.communication.ImageFileBase module¶

class yggdrasil.communication.ImageFileBase.BMPFileComm(*args, **kwargs)¶: Bases: PILFileBase

class yggdrasil.communication.ImageFileBase.EPSFileComm(*args, **kwargs)¶: Bases: PILFileBase

class yggdrasil.communication.ImageFileBase.GIFFileComm(*args, **kwargs)¶: Bases: PILFileBase

class yggdrasil.communication.ImageFileBase.JPEGFileComm(*args, **kwargs)¶: Bases: PILFileBase

class yggdrasil.communication.ImageFileBase.PILFileBase(*args, **kwargs)[source]¶

Class for handling I/O from/to a JPEG file.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Items:

kwargs (dict): Keyword arguments for comms tested with: the provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a

test file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by: sending the messages in ‘send’.

Return type:

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

class yggdrasil.communication.ImageFileBase.PNGFileComm(*args, **kwargs)¶: Bases: PILFileBase

class yggdrasil.communication.ImageFileBase.TIFFFileComm(*args, **kwargs)¶: Bases: PILFileBase

yggdrasil.communication.JSONFileComm module¶

class yggdrasil.communication.JSONFileComm.JSONFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a JSON file on disk.

Parameters:

name (str) – The environment variable where file path is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.MPIComm module¶

exception yggdrasil.communication.MPIComm.MPIClosedError[source]¶

Bases: BaseException

Exception to raise when the MPI connection has been closed.

class yggdrasil.communication.MPIComm.MPIComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O via MPI communicators.

Parameters:

tag_start (int, optional) – Tag that MPI messages should start with. Defaults to 0.
tag_stride (int, optional) – Amount that tag should be advanced after each message to avoid conflicts w/ other MPIComm communicators. Defaults to 1.
partner_mpi_ranks (list, optional) – Rank of MPI processes that partner models are running on. Defaults to None.

tag¶

Tag that should be used for the next MPI message.

Type:: int

tag_stride¶

Amount that tag should be advanced after each each message to avoid conflicts w/ other MPIComm communicators.

Type:: int

add_request(on_empty=False, **kwargs)[source]¶: Add a request to the queue.

address_description = 'The partner communicator ID(s).'¶

advance_tag(request)[source]¶

Advance to the next tag.

Parameters:: request (MPIRequest, MPIMultiRequest) – Request advancing the tag.

bind()[source]¶: Bind to random queue if address is generate.

cache_tag(request)[source]¶

Store a tag for an uncompleted request.

Parameters:: request (MPIRequest, MPIMultiRequest) – Request to cache.

cancel_requests()[source]¶: Cancel requests that have not yet been completed.

confirm_recv(noblock=False)[source]¶: Confirm that message was received.

confirm_send(noblock=False)[source]¶: Confirm that sent message was received.

property create_work_comm_kwargs¶

Keyword arguments for a new work comm.

Type:: dict

classmethod format_address(ranks, tag_start, tag_stride)[source]¶

Format an MPI address.

Parameters:

ranks (tuple) – Ranks of the partner MPI processes.
tag_start (int) – Tag that the comm starts at.
tag_stride (int) – Tag that the comm advances by.

Returns:

Formatted address.

Return type:

property get_response_comm_kwargs¶

Keyword arguments to use for a response comm.

Type:: dict

get_tag(rank=None)[source]¶

Get the next tag for a rank.

Parameters:: rank (int) – Rank to get tag for.
Returns:: Tag that should be used next for the rank.
Return type:: int

property is_open¶

True if the queue is not None.

Type:: bool

property model_env¶

Mapping between model name and opposite comm environment variables that need to be provided to the model.

Type:: dict

property n_msg_recv¶

Number of messages in the queue to recv.

Type:: int

property n_msg_send¶

Number of messages in the queue to send.

Type:: int

next_rank()[source]¶: Get the rank that should be used next.

open()[source]¶: Open the queue.

property opp_address¶

Address for opposite comm.

Type:: str

classmethod parse_address(address)[source]¶

Parse an MPI address for information about the partner process ranks and how the tags should be iterated.

Parameters:

address (str) – Address to parse.

Returns:

The ranks, starting tag, and tag stride contained in the: address.

Return type:

purge()[source]¶: Purge all messages from the comm.

recv_message(*args, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to the forked comm’s recv_message method.
**kwargs – Keyword arguments are passed to the forked comm’s recv_message method.

Returns:

Received message.

Return type:

send_message(msg, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
**kwargs – Additional keyword arguments are passed to _safe_send.

Returns:

Success or failure of send.

Return type:

property tag¶

Tag for the next message.

Type:: int

class yggdrasil.communication.MPIComm.MPIMultiRequest(*args, **kwargs)[source]¶

Bases: MPIRequest

Container for MPI request for multiple partner comms.

cancel()[source]¶: Cancel a request.

property complete¶

True if the request has been completed, False otherwise.

Type:: bool

make_request(previous_requests=None)[source]¶: Complete a request.

remainder¶

class yggdrasil.communication.MPIComm.MPIRequest(comm, direction, address, tag, **kwargs)[source]¶

Bases: object

Container for MPI request.

address¶

cancel()[source]¶: Cancel a request.

comm¶

property complete¶

True if the request has been completed, False otherwise.

Type:: bool

property data¶

Data returned by a request.

Type:: object

direction¶

make_request(payload=None)[source]¶: Complete a request.

req¶

size¶

size_req¶

tag¶

yggdrasil.communication.MatFileComm module¶

class yggdrasil.communication.MatFileComm.MatFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a Matlab .mat file on disk.

Parameters:

name (str) – The environment variable where file path is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.NetCDFFileComm module¶

class yggdrasil.communication.NetCDFFileComm.NetCDFFileComm(*args, **kwargs)[source]¶

Class for handling I/O from/to an netCDF file.

Parameters:

read_attributes (bool, optional) – If True, the attributes are read in as well as the variables. Defaults to False.
variables (list, optional) – List of variables to read in. If not provided, all variables will be read.
version (int, optional) – Version of netCDF format that should be used. Defaults to 1. Options are 1 (classic format) and 2 (64-bit offset format).
**kwargs – Additional keywords arguments are passed to parent class.

property fd¶: Associated file identifier.

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Returns:

Dictionary of variables to use for testing. Items:

kwargs (dict): Keyword arguments for comms tested with: the provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a

test file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by: sending the messages in ‘send’.

Return type:

transform_type_recv(x)[source]¶

transform_type_send(x)[source]¶

yggdrasil.communication.ObjFileComm module¶

class yggdrasil.communication.ObjFileComm.ObjFileComm(*args, **kwargs)[source]¶

Bases: PlyFileComm

Class for handling I/O from/to a .obj file on disk.

Parameters:

name (str) – The environment variable where communication address is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.PandasFileComm module¶

class yggdrasil.communication.PandasFileComm.PandasFileComm(*args, **kwargs)[source]¶

Bases: AsciiTableComm

Class for handling I/O from/to a pandas csv file on disk.

Parameters:

name (str) – The environment variable where communication address is stored.
delimiter (str, optional) – String that should be used to separate columns. Defaults to ‘t’.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.PickleFileComm module¶

class yggdrasil.communication.PickleFileComm.PickleFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a pickled file on disk.

Parameters:

name (str) – The environment variable where file path is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.PlyFileComm module¶

class yggdrasil.communication.PlyFileComm.PlyFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a .ply file on disk.

Parameters:

name (str) – The environment variable where communication address is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.RESTComm module¶

class yggdrasil.communication.RESTComm.RESTComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O via a RESTful API. The provided address should be an HTTP address to a server running a flask app that has been equipped to respond to send/receive calls via the add_comm_server_to_app method.

Parameters:

params (dict, optional) – Parameters that should be passed via URL. Defaults to None and is ignored.
cookies (dict, optional) – Cookies to send to the server. Defaults to None and is ignored.
**kwargs – Additional keyword arguments will be passed to the base class.

atexit()[source]¶: Close operations.

bind(*args, **kwargs)[source]¶: Bind to address based on information provided.

property is_open¶

True if the connection is open.

Type:: bool

property n_msg_recv¶

The number of incoming messages in the connection.

Type:: int

property n_msg_send¶

The number of outgoing messages in the connection.

Type:: int

open(*args, **kwargs)[source]¶: Open the connection.

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

purge()[source]¶: Purge all messages from the comm.

yggdrasil.communication.RESTComm.add_comm_server_to_app(app)[source]¶

Add methods for handling send/receive calls server-side.

Parameters:: app (flask.Flask) – Flask app to add methods to.

yggdrasil.communication.RMQAsyncComm module¶

class yggdrasil.communication.RMQAsyncComm.RMQAsyncComm(*args, **kwargs)[source]¶

Bases: RMQComm

Class for handling asynchronous RabbitMQ communications. It is not recommended to use this class as it can leave hanging threads if not closed propertly. The normal RMQComm will cover most use cases.

Parameters:

name (str) – The environment variable where the comm address is stored.
dont_open (bool, optional) – If True, the connection will not be opened. Defaults to False.
**kwargs – Additional keyword arguments are passed to CommBase.

times_connected¶

Number of times that this connections has been established.

Type:: int

rmq_thread¶

Thread used to run IO loop.

Type:: multitasking.YggTask

add_on_cancel_callback()[source]¶

add_on_channel_close_callback()[source]¶: This method tells pika to call the on_channel_closed method if RabbitMQ unexpectedly closes the channel.

atexit()[source]¶: Close operations.

bind()[source]¶: Declare queue to get random new queue.

check_for_close()[source]¶: bool: Check if close has been called from the main process.

close_channel()[source]¶: Close the channel if it exists.

close_connection(*args, **kwargs)[source]¶: Close the connection.

connect()[source]¶: Establish the connection.

enable_delivery_confirmations()[source]¶: Turn on delivery confirmations.

property is_open¶

True if the connection and channel are open.

Type:: bool

property n_msg_recv¶

Number of messages in the queue.

Type:: int

property n_msg_send¶

Number of messages in the queue.

Type:: int

new_run_thread(name=None)[source]¶: Get a new thread for running.

on_basic_qos_ok(unused_frame)[source]¶: Actions to perform one the qos is set.

on_bindok(unused_frame, userdata)[source]¶: Actions to perform once the queue is succesfully bound. Start consuming messages.

on_cancelok(unused_frame, userdata)[source]¶: Actions to perform after succesfully cancelling consumption. Closes the channel.

on_channel_closed(channel, reason)[source]¶: Actions to perform when the channel is closed. Close the connection.

on_channel_closeok(_unused_frame)[source]¶

on_channel_open(channel)[source]¶: Actions to perform after a channel is opened. Add the channel close callback and setup the exchange.

on_connection_closed(connection, reason)[source]¶: Actions that must be taken when the connection is closed. Set the channel to None. If the connection is meant to be closing, stop the IO loop. Otherwise, wait 5 seconds and try to reconnect.

on_connection_open(connection)[source]¶: Actions that must be taken when the connection is opened. Add the close connection callback and open the RabbitMQ channel.

on_connection_open_error(unused_connection, err)[source]¶: Actions that must be taken when the connection fails to open.

on_consumer_cancelled(method_frame)[source]¶

on_delivery_confirmation(method_frame)[source]¶: Actions performed when a sent message is confirmed.

on_exchange_declareok(unused_frame, userdata)[source]¶: Actions to perform once an exchange is succesfully declared. Set up the queue.

on_message(_unused_channel, basic_deliver, properties, body)[source]¶: Buffer received messages.

on_queue_declareok(method_frame, userdata)[source]¶: Actions to perform once the queue is succesfully declared. Bind the queue.

open_channel()[source]¶: Open a RabbitMQ channel.

publish_message()[source]¶: Publish the next message in the list.

reconnect()[source]¶: Try to re-establish a connection and resume a new IO loop.

reset_for_reconnection()[source]¶: Reset variables in preparation for reconnection.

property rmq_lock¶: Lock associated with RMQ ioloop thread.

run_thread()[source]¶: Connect to the connection and begin the IO loop.

set_qos()[source]¶

setup_exchange(exchange_name)[source]¶: Setup the exchange.

setup_queue(queue_name)[source]¶: Set up the message queue.

start_consuming()[source]¶

start_publishing()[source]¶: Enable confirmations and begin sending messages.

start_run_thread()[source]¶: Start the run thread and wait for it to finish.

stop(call_on_thread=False)[source]¶: Stop the ioloop.

stop_consuming()[source]¶

class yggdrasil.communication.RMQAsyncComm.RMQTaskLoop(*args, **kwargs)[source]¶

Bases: YggTaskLoop

Task loop for RMQ consumer.

atexit()[source]¶: Actions performed when python exits.

yggdrasil.communication.RMQComm module¶

class yggdrasil.communication.RMQComm.RMQComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling basic RabbitMQ communications.

connection¶

RabbitMQ connection.

Type:: pika.Connection

channel¶

RabbitMQ channel.

Type:: pika.Channel

Raises:: RuntimeError – If a connection cannot be established.

Developer Notes:: It is not advised that new language implement a RabbitMQ communication interface. Rather RMQ communication is included explicitly for connections between models that are not co-located on the same machine and are used by the yggdrasil framework connections on the Python side.

address_description = 'AMPQ queue address of the form ``<url>_RMQPARAM_<exchange>_RMQPARAM_<queue>`` where ``url`` is the broker address (see explanation `here <https://pika.readthedocs.io/en/stable/examples/using_urlparameters.html>`_), ``exchange`` is the name of the exchange on the queue that should be used, and ``queue`` is the name of the queue.'¶

atexit()[source]¶: Close operations.

bind()[source]¶: Declare queue to get random new queue.

close_channel()[source]¶: Close the channel if it exists.

close_connection(*args, **kwargs)[source]¶: Close the connection.

close_queue(skip_unbind=False)[source]¶: Close the queue if the channel exists.

property create_work_comm_kwargs¶

Keyword arguments for a new work comm.

Type:: dict

property exchange¶

AMQP exchange.

Type:: str

get_queue_result()[source]¶: Get the fram from passive queue declare.

property get_work_comm_kwargs¶

Keyword arguments for an existing work comm.

Type:: dict

property is_confirmed_recv¶

True if all received messages have been confirmed.

Type:: bool

property is_confirmed_send¶

True if all sent messages have been confirmed.

Type:: bool

property is_open¶

True if the connection and channel are open.

Type:: bool

property n_msg_recv¶

Number of messages in the queue.

Type:: int

property n_msg_send¶

Number of messages in the queue.

Type:: int

classmethod new_comm_kwargs(name, user=None, password=None, host=None, virtual_host=None, port=None, exchange=None, queue='', **kwargs)[source]¶

Initialize communication with new connection.

Parameters:

name (str) – Name of new connection.
user (str, optional) – RabbitMQ server username. Defaults to config option ‘user’ in section ‘rmq’ if it exists and ‘guest’ if it does not.
password (str, optional) – RabbitMQ server password. Defaults to config option ‘password’ in section ‘rmq’ if it exists and ‘guest’ if it does not.
host (str, optional) – RabbitMQ server host. Defaults to config option ‘host’ in section ‘rmq’ if it exists and _localhost if it does not. If _localhost, the output of socket.gethostname() is used.
virtual_host (str, optional) – RabbitMQ server virtual host. Defaults to config option ‘vhost’ in section ‘rmq’ if it exists and ‘/’ if it does not.
port (str, optional) – Port on host to use. Defaults to config option ‘port’ in section ‘rmq’ if it exists and ‘5672’ if it does not.
exchange (str, optional) – RabbitMQ exchange. Defaults to config option ‘namespace’ in section ‘rmq’ if it exits and ‘’ if it does not.
queue (str, optional) – Name of the queue that messages will be send to or received from. If an empty string, the queue will be a random string and exclusive to a receiving comm. Defaults to ‘’.
**kwargs – Additional keywords arguments are returned as keyword arguments for the new comm.

Returns:

Arguments and keyword arguments for new comm.

Return type:

tuple(tuple, dict)

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

purge()[source]¶: Remove all messages from the associated queue.

property queue¶

AMQP queue.

Type:: str

property url¶

AMQP server address.

Type:: str

class yggdrasil.communication.RMQComm.RMQServer(*args, **kwargs)[source]¶

Bases: CommServer

RMQ server object for cleaning up server connections.

terminate(*args, **kwargs)[source]¶: Also set break flag.

yggdrasil.communication.RMQComm.check_rmq_server(url=None, **kwargs)[source]¶

Check that connection to a RabbitMQ server is possible.

Parameters:

url (str, optional) – Address of RMQ server that includes all the necessary information. If this is provided, the remaining arguments are ignored. Defaults to None and the connection parameters are taken from the other arguments.
username (str, optional) – RMQ server username. Defaults to config value.
password (str, optional) – RMQ server password. Defaults to config value.
host (str, optional) – RMQ hostname or IP Address to connect to. Defaults to config value.
port (str, optional) – RMQ server TCP port to connect to. Defaults to config value.
vhost (str, optional) – RMQ virtual host to use. Defaults to config value.

Returns:

True if connection to RabbitMQ server is possible, False: otherwise.

Return type:

yggdrasil.communication.RMQComm.get_rmq_parameters(url=None, user=None, username=None, password=None, host=None, virtual_host=None, vhost=None, port=None, exchange=None, queue='')[source]¶

Get RabbitMQ connection parameters.

Parameters:

url (str, optional) – Address of RMQ server that includes all the necessary information. If this is provided, the remaining arguments are ignored. Defaults to None and the connection parameters are taken from the other arguments.
user (str, optional) – RabbitMQ server username. Defaults to config option ‘user’ in section ‘rmq’ if it exists and ‘guest’ if it does not.
username (str, optional) – Alias for user.
password (str, optional) – RabbitMQ server password. Defaults to config option ‘password’ in section ‘rmq’ if it exists and ‘guest’ if it does not.
host (str, optional) – RabbitMQ server host. Defaults to config option ‘host’ in section ‘rmq’ if it exists and _localhost if it does not. If _localhost, the output of socket.gethostname() is used.
virtual_host (str, optional) – RabbitMQ server virtual host. Defaults to config option ‘vhost’ in section ‘rmq’ if it exists and ‘/’ if it does not.
vhost (str, optional) – Alias for virtual_host.
port (str, optional) – Port on host to use. Defaults to config option ‘port’ in section ‘rmq’ if it exists and ‘5672’ if it does not.
exchange (str, optional) – RabbitMQ exchange. Defaults to config option ‘namespace’ in section ‘rmq’ if it exits and ‘’ if it does not.
queue (str, optional) – Name of the queue that messages will be send to or received from. If an empty string, the queue will be a random string and exclusive to a receiving comm. Defaults to ‘’.

Returns:

The connection url, exchange, & queue.

Return type:

yggdrasil.communication.SequenceFileBase module¶

class yggdrasil.communication.SequenceFileBase.BAMFileComm(*args, **kwargs)¶: Bases: PySamFileBase

class yggdrasil.communication.SequenceFileBase.BCFFileComm(*args, **kwargs)¶: Bases: PySamFileBase

class yggdrasil.communication.SequenceFileBase.BioPythonFileBase(*args, **kwargs)[source]¶

Bases: SequenceFileBase

Base class for nucleotide/protein sequence I/O using biopython.

dict2record(x)[source]¶: Covert a dictionary to a SeqRecord.

classmethod get_testing_options(piecemeal=False, **kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

piecemeal (bool, optional) – If True, the test data will be piecemeal.

Returns:

Dictionary of variables to use for testing. Items:

kwargs (dict): Keyword arguments for comms tested with: the provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a

test file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by: sending the messages in ‘send’.

Return type:

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

record2dict(x)[source]¶: Convert a SeqRecord to a dictionary.

property records_iterator¶

SeqRecord iterator.

Type:: iterator

class yggdrasil.communication.SequenceFileBase.CRAMFileComm(*args, **kwargs)¶: Bases: PySamFileBase

class yggdrasil.communication.SequenceFileBase.FASTAFileComm(*args, **kwargs)¶: Bases: BioPythonFileBase

class yggdrasil.communication.SequenceFileBase.FASTQFileComm(*args, **kwargs)¶: Bases: BioPythonFileBase

class yggdrasil.communication.SequenceFileBase.PySamFileBase(*args, **kwargs)[source]¶

Bases: SequenceFileBase

Base class for nucleotide/protein sequence I/O using pysam.

advance_in_series(series_index=None)[source]¶

Advance to a certain file in a series.

Parameters:: series_index (int, optional) – Index of file in the series that should be moved to. Defaults to None and call will advance to the next file in the series.
Returns:: True if the file was advanced in the series, False otherwise.
Return type:: bool

concats_as_str = True¶

create_index(address, overwrite=False)[source]¶: Create an index file to accompany the file begin written/read if one is required.

property current_region¶

Current region.

Type:: dict

classmethod dict2header(x)[source]¶

classmethod dict2region(x, header=None)[source]¶

classmethod get_testing_options(test_dir=None, **kwargs)[source]¶: Method to return a dictionary of testing options for this class.

classmethod header2dict(x)[source]¶

property header_size¶

classmethod index_filename(address)[source]¶: Get the name of the index file.

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

property open_mode¶

Mode that should be used to open the file.

Type:: str

classmethod record2dict(x)[source]¶

classmethod region2dict(x)[source]¶

classmethod remove_companion_files(address)[source]¶

Remove companion files that are created when writing to a file

Parameters:: address (str) – Address for the filename.

series_file_size(fname)[source]¶: int: Size of file in series.

class yggdrasil.communication.SequenceFileBase.SAMFileComm(*args, **kwargs)¶: Bases: PySamFileBase

class yggdrasil.communication.SequenceFileBase.SequenceFileBase(*args, **kwargs)[source]¶

Base class for nucleotide/protein sequence I/O.

static before_registration(cls)[source]¶: Operations that should be performed to modify class attributes prior to registration.

class yggdrasil.communication.SequenceFileBase.VCFFileComm(*args, **kwargs)¶: Bases: PySamFileBase

yggdrasil.communication.ServerComm module¶

class yggdrasil.communication.ServerComm.Request(response_address, request_id)[source]¶

Bases: object

request_id¶

response_address¶

class yggdrasil.communication.ServerComm.ServerComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling Server side communication.

Parameters:

name (str) – The environment variable where communication address is stored.
request_commtype (str, optional) – Comm class that should be used for the request comm. Defaults to None.
response_kwargs (dict, optional) – Keyword arguments for the response comm. Defaults to empty dict.
direct_connection (bool, optional) – If True, the comm will be directly connected to a ServerComm. Defaults to False.
**kwargs – Additional keywords arguments are passed to the input comm.

response_kwargs¶

Keyword arguments for the response comm.

Type:: dict

icomm¶

Request comm.

Type:: Comm

ocomm¶

Response comms for each request.

Type:: OrderedDict

property all_clients_connected¶

True if all expected clients have connected. False otherwise.

Type:: bool

close(*args, **kwargs)[source]¶: Close the connection.

property close_on_eof_recv¶

True if the comm will close when EOF is received.

Type:: bool

create_response_comm(header)[source]¶: Create a response comm based on information from the last header.

disconnect(*args, **kwargs)[source]¶: Disconnect the comm.

drain_messages(direction='recv', **kwargs)[source]¶: Sleep while waiting for messages to be drained.

drain_server_signon_messages(**kwargs)[source]¶: Drain server signon messages. This should only be used for testing purposes.

property filter¶

filter for the communicator.

Type:: FilterBase

finalize_message(msg, **kwargs)[source]¶

Perform actions to decipher a message.

Parameters:

msg (CommMessage) – Initial message object to be finalized.
**kwargs – Keyword arguments are passed to the request comm’s finalize_message method.

Returns:

Deserialized and annotated message.

Return type:

get_status_message(nindent=0, **kwargs)[source]¶

Return lines composing a status message.

Parameters:

nindent (int, optional) – Number of tabs that should be used to indent each line. Defaults to 0.
*kwargs – Additional arguments are passed to the parent class’s method.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

property is_closed¶

True if the connection is closed.

Type:: bool

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

property maxMsgSize¶

Maximum size of a single message that should be sent.

Type:: int

property n_msg_direct¶

Number of messages currently being routed.

Type:: int

property n_msg_recv¶

The number of messages in the connection.

Type:: int

property n_msg_recv_drain¶

The number of messages in the connection to drain.

Type:: int

classmethod new_comm_kwargs(name, request_commtype=None, **kwargs)[source]¶

Initialize communication with new comms.

Parameters:

name (str) – Name for new comm.
request_commtype (str, optional) – Name of class for new input comm. Defaults to None.

open()[source]¶: Open the connection.

property open_clients¶

Available open clients.

Type:: list

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

property opp_comms¶

Name/address pairs for opposite comms.

Type:: dict

prepare_message(*args, **kwargs)[source]¶

Perform actions preparing to send a message.

Parameters:

*args – Components of the outgoing message.
**kwargs – Keyword arguments are passed to the request comm’s prepare_message method.

Returns:

Serialized and annotated message.

Return type:

purge()[source]¶: Purge input and output comms.

recv_from(*args, **kwargs)[source]¶

Receive a message from the input comm and open a new response comm for output using address from the header, returning the response_id.

Parameters:

*args – Arguments are passed to input comm recv method.
**kwargs – Keyword arguments are passed to input comm recv method.

Returns:

Success or failure of recv call,: output from input comm recv method, and response_id that response should be sent to.

Return type:

tuple(bool, obj, str)

recv_message(*args, **kwargs)[source]¶

Receive a message.

Parameters:

*args – Arguments are passed to the request comm’s recv_message method.
**kwargs – Keyword arguments are passed to the request comm’s recv_message method.

Returns:

Received message.

Return type:

rpcRecv(*args, **kwargs)[source]¶: Alias for RPCComm.recv

rpcSend(*args, **kwargs)[source]¶: Alias for RPCComm.send

send_message(msg, **kwargs)[source]¶

Send a message encapsulated in a CommMessage object.

Parameters:

msg (CommMessage) – Message to be sent.
**kwargs – Additional keyword arguments are passed to the response comm’s send_message method.

Returns:

Success or failure of send.

Return type:

send_to(response_id, *args, **kwargs)[source]¶

Send a message to a specific response comm.

Parameters:

response_id (str) – ID used to register the response comm.
*args – Arguments are passed to output comm send method.
**kwargs – Keyword arguments are passed to output comm send method.

Returns:

Output from output comm send method.

Return type:

obj

yggdrasil.communication.ValueComm module¶

class yggdrasil.communication.ValueComm.ValueComm(*args, **kwargs)[source]¶

Bases: CommBase

classmethod get_testing_options(**kwargs)[source]¶

Method to return a dictionary of testing options for this class.

Parameters:

serializer (str, optional) – The name of the serializer that should be used. If not provided, the _default_serializer class attribute will be used.

Returns:

Dictionary of variables to use for testing. Key/value pairs:

kwargs (dict): Keyword arguments for comms tested with the: provided content.

send (list): List of objects to send to test file. recv (list): List of objects that will be received from a test

file that was sent the messages in ‘send’.

contents (bytes): Bytes contents of test file created by sending: the messages in ‘send’.

Return type:

classmethod is_installed(language=None)[source]¶

Determine if the necessary libraries are installed for this communication class.

Parameters:: language (str, optional) – Specific language that should be checked for compatibility. Defaults to None and all languages supported on the current platform will be checked. If set to ‘any’, the result will be True if this comm is installed for any of the supported languages.
Returns:: Is the comm installed.
Return type:: bool

property is_open¶

True if the connection is open.

Type:: bool

property n_msg_recv¶

The number of incoming messages in the connection.

Type:: int

no_serialization = True¶

open()[source]¶: Open the connection.

purge()[source]¶: Purge all messages from the comm.

send(*args, **kwargs)[source]¶: Send a message.

yggdrasil.communication.YAMLFileComm module¶

class yggdrasil.communication.YAMLFileComm.YAMLFileComm(*args, **kwargs)[source]¶

Bases: FileComm

Class for handling I/O from/to a YAML file on disk.

Parameters:

name (str) – The environment variable where file path is stored.
**kwargs – Additional keywords arguments are passed to parent class.

yggdrasil.communication.ZMQComm module¶

class yggdrasil.communication.ZMQComm.ZMQComm(*args, **kwargs)[source]¶

Bases: CommBase

Class for handling I/O using ZeroMQ sockets.

Parameters:

name (str) – The environment variable where the socket address is stored.
context (zmq.Context, optional) – ZeroMQ context that should be used. Defaults to None and the global context is used.
socket_type (str, optional) – The type of socket that should be created. Defaults to _default_socket_type. See zmq for all options.
socket_action (str, optional) – The action that the socket should perform. Defaults to action based on the direction (‘connect’ for ‘recv’, ‘bind’ for ‘send’.)
topic_filter (str, optional) – Message filter to use when subscribing. This is only used for ‘SUB’ socket types. Defaults to ‘’ which is all messages.
dealer_identity (str, optional) – Identity that should be used to route messages to a dealer socket. Defaults to ‘0’.
**kwargs – Additional keyword arguments are passed to :class:.CommBase.

context¶

ZeroMQ context that will be used.

Type:: zmq.Context

socket¶

ZeroMQ socket.

Type:: zmq.Socket

socket_type_name¶

The type of socket that should be created.

Type:: str

socket_type¶

ZeroMQ socket type.

Type:: int

socket_action¶

The action that the socket should perform.

Type:: str, optional

topic_filter¶

Message filter to use when subscribing.

Type:: str

dealer_identity¶

Identity that should be used to route messages to a dealer socket.

Type:: str

Developer Notes:: yggdrasil uses the tcp transport by default with a PAIR socket type. For every connection, yggdrasil establishes a second request/reply connection that is used to confirm messages passed between the primary PAIR of sockets. On the first send, the model should create a REP socket on an open tcp address and send that address in the header of the first message under the key ‘zmq_reply’. Receiving models should check message headers for this key and, on receipt, establish the partner REQ socket with the specified address (receiving comms can receive from more than one source so they can have more than one request addresses at at time for this purpose). Following every message, the sending model should wait for a message on the reply socket and, on receipt, return the message. Following every message, the receiving model should send the message ‘YGG_REPLY’ on the request socket and wait for a reply. When creating worker comms for sending large messages, the sending model should create the reply comm for the worker in advanced and send it in the header with the worker address under the key ‘zmq_reply_worker’.

address_description = 'A ZeroMQ endpoint of the form <transport>://<address>, where the format of address depends on the transport. Additional information can be found `here <http://api.zeromq.org/3-2:zmq-bind>`_.'¶

property address_param¶

Address parameters.

Type:: dict

bind()[source]¶: Bind to address, getting random port as necessary.

check_reply_socket_recv(msg)[source]¶

Check incoming message for reply address.

Parameters:: msg (str) – Incoming message to check.
Returns:: Messages with reply address removed if present.
Return type:: str

check_reply_socket_send(msg)[source]¶

Append reply socket address if it

Parameters:: msg (str) – Message that will be piggy backed on.
Returns:: Message with reply address if it has not been sent.
Return type:: str

classmethod close_registry_entry(value)[source]¶: Close a registry entry.

confirm_recv(noblock=False)[source]¶: Confirm that message was received.

confirm_send(noblock=False)[source]¶: Confirm that sent message was received.

connect()[source]¶: Connect to address.

property create_work_comm_kwargs¶

Keyword arguments for a new work comm.

Type:: dict

disconnect_socket(dont_close=False)[source]¶: Disconnect from address.

drain_server_signon_messages(**kwargs)[source]¶: Drain server signon messages. This should only be used for testing purposes.

get_status_message(nindent=0, **kwargs)[source]¶

Return lines composing a status message.

Parameters:

nindent (int, optional) – Number of tabs that should be used to indent each line. Defaults to 0.
**kwargs – Additional keyword arguments are passed to the parent class’s method.

Returns:

Lines composing the status message and the: prefix string used for the last message.

Return type:

tuple(list, prefix)

property get_work_comm_kwargs¶

Keyword arguments for an existing work comm.

Type:: dict

header2workcomm(header, **kwargs)[source]¶

Get a work comm based on header info.

Parameters:

header (dict) – Information that will be sent in the message header to the work comm.
**kwargs – Additional keyword arguments are passed to the parent method.

Returns:

class:.CommBase: Work comm.

property host¶

Host that socket is connected to.

Type:: str

property is_confirmed_recv¶

True if all received messages have been confirmed.

Type:: bool

is_message(flags)[source]¶

Poll the socket for a message.

Parameters:: flags (int) – ZMQ poll flags.
Returns:: True if there is a message matching the flags, False otherwise.
Return type:: bool

property is_open¶

True if the socket is open.

Type:: bool

property n_msg_recv¶

The number of incoming messages in the connection.

Type:: int

property n_msg_send¶

The number of outgoing messages in the connection.

Type:: int

classmethod new_comm_kwargs(name, protocol=None, host=None, port=None, **kwargs)[source]¶

Initialize communication with new queue.

Parameters:

name (str) – Name of new socket.
protocol (str, optional) – The protocol that should be used. Defaults to None and is set to _default_protocol. See zmq for details.
host (str, optional) – The host that should be used. Invalid for ‘inproc’ protocol. Defaults to ‘localhost’.
port (int, optional) – The port used. Invalid for ‘inproc’ protocol. Defaults to None and a random port is choosen.
**kwargs – Additional keywords arguments are returned as keyword arguments for the new comm.

Returns:

Arguments and keyword arguments for new socket.

Return type:

tuple(tuple, dict)

open()[source]¶: Open connection by binding/connect to the specified socket.

property opp_address¶

Address for opposite comm.

Type:: str

opp_comm_kwargs(for_yaml=False)[source]¶

Get keyword arguments to initialize communication with opposite comm object.

Parameters:: for_yaml (bool, optional) – If True, the returned dict will only contain values that can be specified in a YAML file. Defaults to False.
Returns:: Keyword arguments for opposite comm object.
Return type:: dict

property port¶

Port that socket is connected to.

Type:: str

prepare_header(header_kwargs)[source]¶: Prepare header kwargs for the communicator.

property protocol¶

Protocol that socket uses.

Type:: str

recv(*args, **kwargs)[source]¶: Receive a message.

property registry_key¶

String used to register the socket.

Type:: str

property reply_thread¶

Task that will handle sending or receiving backlogged messages.

Type:: tools.YggTask

send(*args, **kwargs)[source]¶: Send a message.

server_exists(srv_address)[source]¶

Determine if a server exists.

Parameters:

srv_address (str) – Address of server comm.

Returns:

True if a server with the provided address exists, False: otherwise.

Return type:

set_reply_socket_recv(address)[source]¶: Set the recv reply socket if the address dosn’t exist.

set_reply_socket_send()[source]¶: Set the send reply socket if it dosn’t exist.

unbind(dont_close=False)[source]¶: Unbind from address.

workcomm2header(work_comm, **kwargs)[source]¶

Get header information from a comm.

Parameters:

( (work_comm) – class:.CommBase): Work comm that header describes.
**kwargs – Additional keyword arguments are added to the header.

Returns:

Header information that will be sent with a message.

Return type:

class yggdrasil.communication.ZMQComm.ZMQProxy(*args, **kwargs)[source]¶

Bases: CommServer

Start a proxy in a new thread for a server address. A client-side address will be randomly generated.

Parameters:

srv_address (str) – Address that should face the server(s).
context (zmq.Context, optional) – ZeroMQ context that should be used. Defaults to None and the global context is used.
protocol (str, optional) – Protocol that should be used for the sockets. Defaults to None and is set to _default_protocol.
host (str, optional) – Host for socket address. Defaults to ‘localhost’.
retry_timeout (float, optional) – Time (in seconds) that should be waited before retrying to bind the sockets to the addresses. If negative, a retry will not be attempted and an error will be raised. Defaults to -1.
nretry (int, optional) – Number of times to try binding the sockets to the addresses. Defaults to 1.
**kwargs – Additional keyword arguments are passed to the parent class.

srv_address¶

Address that faces the server(s).

Type:: str

cli_address¶

Address that faces the client(s).

Type:: str

context¶

ZeroMQ context that will be used.

Type:: zmq.Context

srv_socket¶

Socket facing client(s).

Type:: zmq.Socket

cli_socket¶

Socket facing server(s).

Type:: zmq.Socket

cli_count¶

Number of clients that have connected to this proxy.

Type:: int

after_loop()[source]¶: Close sockets after the loop finishes.

cleanup()[source]¶: Clean up sockets on exit.

client_recv()[source]¶: Receive single message from the client.

client_signon_msg = b'ZMQ_CLIENT_SIGNED_ON::'¶

close_sockets()[source]¶: Close the sockets.

poll()[source]¶: Check if the process is finished and return the return code if it is.

run_loop()[source]¶: Forward messages from client to server.

server_send(msg)[source]¶: Send single message to the server.

server_signon_msg = b'ZMQ_SERVER_SIGNING_ON::'¶

yggdrasil.communication.ZMQComm.bind_socket(socket, address, retry_timeout=-1, nretry=1)[source]¶

Bind a socket to an address, getting a random port as necessary.

Parameters:

socket (zmq.Socket) – Socket that should be bound.
address (str) – Address that socket should be bound to.
retry_timeout (float, optional) – Time (in seconds) that should be waited before retrying to bind the socket to the address. If negative, a retry will not be attempted and an error will be raised. Defaults to -1.
nretry (int, optional) – Number of times to try binding the socket to the addresses. Defaults to 1.

Returns:

Address that socket was bound to, including random port if one: was used.

Return type:

yggdrasil.communication.ZMQComm.create_socket(context, socket_type)[source]¶

Create a socket w/ some default options to improve cleanup.

Parameters:

context (zmq.Context) – ZeroMQ context.
socket_type (int) – ZeroMQ socket type.

Returns:

New socket.

Return type:

zmq.Socket

yggdrasil.communication.ZMQComm.format_address(protocol, host, port=None)[source]¶

Format an address based on its parts.

Parameters:

protocol (str) – Communication protocol that should be used.
host (str) – Host that address should point to.
port (int, optional) – Port that address should point to. Defaults to None and is not added to the address.

Returns:

Complete address.

Return type:

Raises:

ValueError – If the protocol is not recognized.

yggdrasil.communication.ZMQComm.get_ipc_host()[source]¶

Get an IPC host using uuid.

Returns:: File path for IPC transport created using uuid.
Return type:: str

yggdrasil.communication.ZMQComm.get_socket_type_mate(t_in)[source]¶

Find the counterpart socket type.

Parameters:: t_in (str) – Socket type.
Returns:: Counterpart socket type.
Return type:: str
Raises:: ValueError – If t_in is not a recognized socket type.

yggdrasil.communication.ZMQComm.parse_address(address)[source]¶

Split an address into its parts.

Parameters:

address (str) – Address to be split.

Returns:

Parameters extracted from the address.

Return type: