Transformed I/O

In addition to I/O filters, yggdrasil also provides methods for specifying transformations that should be performed on input/output received/returned by a model. In addition to type specific operations (e.g. mapping field names, selecting columns), yggdrasil also supports generic transformations via arbitrary Python statements and/or Python functions.

Transformed Output

In the example below, the output from model A is passed to both model B and C, however model B and C expect slightly different forms of the varaible, which as output by model A, is a rate with units of g/s. In the case of model B, it expects an input value in the form of a rate multiplied by 10 to account for accumulation across 10 producers. In the case of Model C, it expects an input value in the form of a rate density, requiring the value returned by model A to be normalized by an area.

Model Code:

def modelA_function(in_val):
    out_val = in_val
    print("modelA_function(%s) = %s" % (in_val, out_val))
    return out_val

def modelB_function(in_val):
    out_val = 3 * in_val
    print("modelB_function(%s) = %s" % (in_val, out_val))
    return in_val, out_val

from yggdrasil import units


def modelC_function(in_val):
    out_val = 2 * in_val
    print("modelC_function(%s) = %s" % (in_val, out_val))
    return in_val, out_val


def transform_function(in_val):
    return (in_val / units.add_units(10, 'cm**2'))

(Example in other languages)

To instruct yggdrasil to transform output from model A into the forms expected by models B & C, the YAML should containg a connection to both models with the transform parameter added to both.

Model YAML:

models:
  - name: python_modelA
    language: python
    args: ./src/transformed_io_modelA.py
    function: modelA_function

  - name: python_modelB
    language: python
    args: ./src/transformed_io_modelB.py
    function: modelB_function

  - name: python_modelC
    language: python
    args: ./src/transformed_io_modelC.py
    function: modelC_function

connections:
  - input: ./Input/input.txt  # Connection between file and model A input
    output: python_modelA:input
    filetype: table
  - input: python_modelA:output  # Connection between model A output & model B input
    outputs:
      - name: python_modelB:input
        transform:
          statement: "%x%*5"
      - name: python_modelC:input
        transform:
          function: ./src/transformed_io_modelC.py:transform_function
  - input: python_modelB:output  # Connection between model B, function 1 output and file
    output: ./outputB.txt
    filetype: table
    field_names: InputMassRate,OuputMassRate
    field_units: g/s,g/s
    format_str: "%.1lf\t%.1lf\n"
  - input: python_modelC:output  # Connection between model C, function 1 output and file
    output: ./outputC.txt
    filetype: table
    field_names: InputMassRate,OuputMassRateDensity
    field_units: g/(s*(cm**2)),g/(s*(cm**2))
    format_str: "%.1lf\t%.1lf\n"

(Example in other languages)

transform values are maps with parameters specifying how the input/output should be transformed before being receieved/returned by a model. In the above example, the transform for model B uses a Python expression (the statement parameter) in terms of %x% which stands in for the variable being based by the comm. Alternately, the transform for model C uses a reference to a Python function. function values take the form <filename>:<function name> where filename is the full path to the location of a Python source file containing the desired function that should be used to transform the received data and function name is the name of the desired function. Functions used in such cases should take a single argument (the variable or tuple of variables being passed by the connection), and return the transformed value. The path to the file containing the function can be absolute or relative to the directory containing the yaml file.

Additional transform parameter options are provided below.

Transform Options

General Transform Options

Option	Type	Required	Description
original_datatype	schema		Datatype associated with expected messages. Defaults to None.
transformtype	string		Consolidate values into an array (Options described here)

Transform Types

Transformtype	Description
array	Consolidate values into an array
direct	Pass the message through unchanged
filter	Pass only some messages based on a filter
function	Transform messages using a function.
iterate	Split messages up into their iterable components
map	Convert an object into a dictionary.
map_fields	Change the names of fields in a message
pandas	Convert messages into Pandas data frames
select_fields	Select a subset of fields from a message
select_scalar	Select a single field from a message
statement	Transform messages according to a statement

Type Specific Transform Options

Option	Type	Valid For ‘Transformtype’ Of	Description
field_names	array	[‘map’, ‘array’, ‘pandas’]	Names of fields in the array.
filter	object	[‘filter’]	Schema for filter components.
function	function	[‘function’]	The handle for a callable Python object (e.g. function) that should be used to transform messages or a string of the form “<function file>:<function name>” identifying a function where “<function file>” is the module or Python file containing the function and “<function name>” is the name of the function. The function should take the message as input and return the transformed message.
index	[‘integer’, ‘string’]	[‘select_scalar’]	Array index or dictionary key to select. Defaults to selecting the first element in an array or the first key alphabetically.
map	object	[‘map_fields’]	A mapping from original field name to new field names.
original_order	array	[‘select_scalar’, ‘select_fields’]	The original order of fields that should be used for selecting from lists/tuples.
selected	array	[‘select_fields’]	A list of fields that should be selected.
single_as_scalar	boolean	[‘select_fields’]	If True and only a single field is selected, the transformed messages will be scalars rather than arrays with single elements. Defaults to False.
statement	string	[‘statement’]	Python statement in terms of the message as represented by the string “%x%” that should evaluate to the transformed message. The statement should only use a limited set of builtins and the math library (See yggdrasil.tools.safe_eval). If more complex relationships are required, use the FunctionTransform class.