Transformed I/O

In addition to I/O filters, yggdrasil also provides methods for specifying transformations that should be performed on input/output received/returned by a model. In addition to type specific operations (e.g. mapping field names, selecting columns), yggdrasil also supports generic transformations via arbitrary Python statements and/or Python functions.

Transformed Output

In the example below, the output from model A is passed to both model B and C, however model B and C expect slightly different forms of the varaible, which as output by model A, is a rate with units of g/s. In the case of model B, it expects an input value in the form of a rate multiplied by 10 to account for accumulation across 10 producers. In the case of Model C, it expects an input value in the form of a rate density, requiring the value returned by model A to be normalized by an area.

Model Code:

1def modelA_function(in_val):
2    out_val = in_val
3    print("modelA_function(%s) = %s" % (in_val, out_val))
4    return out_val
1def modelB_function(in_val):
2    out_val = 3 * in_val
3    print("modelB_function(%s) = %s" % (in_val, out_val))
4    return in_val, out_val
 1from yggdrasil import units
 2
 3
 4def modelC_function(in_val):
 5    out_val = 2 * in_val
 6    print("modelC_function(%s) = %s" % (in_val, out_val))
 7    return in_val, out_val
 8
 9
10def transform_function(in_val):
11    return (in_val / units.add_units(10, 'cm**2'))

(Example in other languages)

To instruct yggdrasil to transform output from model A into the forms expected by models B & C, the YAML should containg a connection to both models with the transform parameter added to both.

Model YAML:

 1models:
 2  - name: python_modelA
 3    language: python
 4    args: ./src/transformed_io_modelA.py
 5    function: modelA_function
 6
 7  - name: python_modelB
 8    language: python
 9    args: ./src/transformed_io_modelB.py
10    function: modelB_function
11
12  - name: python_modelC
13    language: python
14    args: ./src/transformed_io_modelC.py
15    function: modelC_function
16
17connections:
18  - input: ./Input/input.txt  # Connection between file and model A input
19    output: python_modelA:input
20    filetype: table
21  - input: python_modelA:output  # Connection between model A output & model B input
22    outputs:
23      - name: python_modelB:input
24        transform:
25          statement: "%x%*5"
26      - name: python_modelC:input
27        transform:
28          function: ./src/transformed_io_modelC.py:transform_function
29  - input: python_modelB:output  # Connection between model B, function 1 output and file
30    output: ./outputB.txt
31    filetype: table
32    field_names: InputMassRate,OuputMassRate
33    field_units: g/s,g/s
34    format_str: "%.1lf\t%.1lf\n"
35  - input: python_modelC:output  # Connection between model C, function 1 output and file
36    output: ./outputC.txt
37    filetype: table
38    field_names: InputMassRate,OuputMassRateDensity
39    field_units: g/(s*(cm**2)),g/(s*(cm**2))
40    format_str: "%.1lf\t%.1lf\n"

(Example in other languages)

transform values are maps with parameters specifying how the input/output should be transformed before being receieved/returned by a model. In the above example, the transform for model B uses a Python expression (the statement parameter) in terms of %x% which stands in for the variable being based by the comm. Alternately, the transform for model C uses a reference to a Python function. function values take the form <filename>:<function name> where filename is the full path to the location of a Python source file containing the desired function that should be used to transform the received data and function name is the name of the desired function. Functions used in such cases should take a single argument (the variable or tuple of variables being passed by the connection), and return the transformed value. The path to the file containing the function can be absolute or relative to the directory containing the yaml file.

Additional transform parameter options are provided below.

Transform Options

General Transform Options

Option

Type

Required

Description

initial_state

object

Dictionary of initial state variables that should be set when the transform is created.

original_datatype

schema

Datatype associated with expected messages. Defaults to None.

transformtype

(Options described here)

Transform Types

Transformtype

array

direct

filter

function

iterate

map_fields

pandas

select_fields

statement

Type Specific Transform Options

Option

Type

Description

Valid For ‘Transformtype’ Of

field_names

array

[‘pandas’, ‘array’]

filter

[‘filter’]

function

function

The handle for a callable Python object (e.g. function) that should be used to transform messages or a string of the form “<function file>:<function name>” identifying a function where “<function file>” is the module or Python file containing the function and “<function name>” is the name of the function. The function should take the message as input and return the transformed message.

[‘function’]

map

object

A mapping from original field name to new field names.

[‘map_fields’]

original_order

array

The original order of fields that should be used for selecting from lists/tuples.

[‘select_fields’]

selected

array

A list of fields that should be selected.

[‘select_fields’]

single_as_scalar

boolean

If True and only a single field is selected, the transformed messages will be scalars rather than arrays with single elements. Defaults to False.

[‘select_fields’]

statement

string

Python statement in terms of the message as represented by the string “%x%” that should evaluate to the transformed message. The statement should only use a limited set of builtins and the math library (See yggdrasil.tools.safe_eval). If more complex relationships are required, use the FunctionTransform class.

[‘statement’]