Formatted I/O¶
In addition to passing raw strings, the yggdrasil framework also has support for formatting/processing messages from/to language native objects.
Scalars¶
The format of messages containing scalar variables (strings, integers, and
floating point numbers) can be specified using a C-style format string
(See C-Style Format Strings for details). In
the example below, the format string "%6s\t%d\t%f\n"
indicates that each
message will contain a 6 character string, an integer, and a float. In addition
to providing a format string, the C API requires the use of different functions
for initializing channels and sending/receiving to/from them.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputA')
6out_channel = YggOutput('outputA')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, msg = in_channel.recv()
14 if not flag:
15 print("Model A: No more input.")
16 break
17
18 # Print received message
19 print('Model A: %s' % msg)
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(msg)
24 if not flag:
25 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputB')
6out_channel = YggOutput('outputB')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, msg = in_channel.recv()
14 if not flag:
15 print("Model B: No more input.")
16 break
17
18 # Print received message
19 print('Model B: %s' % msg)
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(msg)
24 if not flag:
25 raise RuntimeError("Model B: Error sending output.")
The same YAML can be used as was used for the example from
Getting started with the modification that
the files are now read/written line-by-line (filetype: ascii
)
rather than all at once:
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io1_modelA.py
5 inputs: inputA
6 outputs: outputA
7
8 - name: python_modelB
9 language: python
10 args: ./src/formatted_io1_modelB.py
11 inputs: inputB
12 outputs: outputB
13
14connections:
15 - input: outputA # Connection between model A output & model B input
16 output: inputB
17 - input: ./Input/input.txt # Connection between file and model A input
18 output: inputA
19 filetype: ascii
20 - input: outputB # Connection between model B output and file
21 output: ./output.txt
22 filetype: ascii
Tables by Row¶
Tables can also be passed in a similar manner. For input from a table, the format
string does not need to be provided and will be determined by the source of the
table. There are different API classes/functions for I/O from/to table channels
versus standard channels in each language. (e.g. YggInput
vs.
YggAsciiTableInput
in Python)
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggAsciiTableInput, YggAsciiTableOutput)
4
5# Initialize input/output channels
6in_channel = YggAsciiTableInput('inputA')
7out_channel = YggAsciiTableOutput('outputA', '%6s\t%d\t%f\n')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, msg = in_channel.recv()
15 if not flag:
16 print("Model A: No more input.")
17 break
18 name, count, size = msg
19
20 # Print received message
21 print('Model A: %s, %d, %f' % (name, count, size))
22
23 # Send output to output channel
24 # If there is an error, the flag will be False
25 flag = out_channel.send(name, count, size)
26 if not flag:
27 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggAsciiTableInput, YggAsciiTableOutput)
4
5# Initialize input/output channels
6in_channel = YggAsciiTableInput('inputB')
7out_channel = YggAsciiTableOutput('outputB', '%6s\t%d\t%f\n')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, msg = in_channel.recv()
15 if not flag:
16 print("Model B: No more input.")
17 break
18 name, count, size = msg
19
20 # Print received message
21 print('Model B: %s, %d, %f' % (name, count, size))
22
23 # Send output to output channel
24 # If there is an error, the flag will be False
25 flag = out_channel.send(name, count, size)
26 if not flag:
27 raise RuntimeError("Model B: Error sending output.")
The filetype: table
options in the YAML tell the
yggdrasil framework that the file should be read/written as a table
row-by-row including verification that each row conforms with the table.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io2_modelA.py
5 inputs: inputA
6 outputs: outputA
7
8 - name: python_modelB
9 language: python
10 args: ./src/formatted_io2_modelB.py
11 inputs: inputB
12 outputs: outputB
13
14connections:
15 - input: outputA # Connection between model A output & model B input
16 output: inputB
17 - input: ./Input/input.txt # Connection between file and model A input
18 output: inputA
19 filetype: table
20 - input: outputB # Connection between model B output and file
21 output: ./output.txt
22 filetype: table
Tables as Arrays¶
Tables can also be passed as arrays. In Python and Matlab, this is done using
separate classes/functions. In C and C++, this is done by passing 1 to the as_array
arguments of the table API classes/functions.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggAsciiArrayInput, YggAsciiArrayOutput)
4
5# Initialize input/output channels
6in_channel = YggAsciiArrayInput('inputA')
7out_channel = YggAsciiArrayOutput('outputA', '%6s\t%d\t%f\n')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, arr = in_channel.recv_array()
15 if not flag:
16 print("Model A: No more input.")
17 break
18
19 # Print received message
20 print('Model A: (%d rows)' % len(arr))
21 for i in range(len(arr)):
22 print(' %s, %d, %f' % tuple(arr[i]))
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send_array(arr)
27 if not flag:
28 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggAsciiArrayInput, YggAsciiArrayOutput)
4
5# Initialize input/output channels
6in_channel = YggAsciiArrayInput('inputB')
7out_channel = YggAsciiArrayOutput('outputB', '%6s\t%d\t%f\n')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, arr = in_channel.recv_array()
15 if not flag:
16 print("Model B: No more input.")
17 break
18
19 # Print received message
20 print('Model B: (%d rows)' % len(arr))
21 for i in range(len(arr)):
22 print(' %s, %d, %f' % tuple(arr[i]))
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send_array(arr)
27 if not flag:
28 raise RuntimeError("Model B: Error sending output.")
The YAML only differs in that as_array: True
for the connections to the files
to indicate that the files should be interpreted as tables and that the tables
should be read/written in their entirety as arrays.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io3_modelA.py
5 inputs: inputA
6 outputs:
7 name: outputA
8 field_names: name,count,size
9
10 - name: python_modelB
11 language: python
12 args: ./src/formatted_io3_modelB.py
13 inputs: inputB
14 outputs:
15 name: outputB
16 field_names: name,count,size
17
18connections:
19 - input: outputA # Connection between model A output & model B input
20 output: inputB
21 - input: ./Input/input.txt # Connection between file and model A input
22 output: inputA
23 as_array: True
24 filetype: table
25 - input: outputB # Connection between model B output and file
26 output: ./output.txt
27 as_array: True
28 filetype: table
Tables as Pandas Data Frames¶
In Python, tables can also be passed as Pandas data frames.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggPandasInput, YggPandasOutput)
4
5# Initialize input/output channels
6in_channel = YggPandasInput('inputA')
7out_channel = YggPandasOutput('outputA')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, frame = in_channel.recv()
15 if not flag:
16 print("Model A: No more input.")
17 break
18
19 # Print received message
20 nrows = len(frame.index)
21 print('Model A: (%d rows)' % len(frame.index))
22 print(frame)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(frame)
27 if not flag:
28 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggPandasInput, YggPandasOutput)
4
5# Initialize input/output channels
6in_channel = YggPandasInput('inputB')
7out_channel = YggPandasOutput('outputB')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, frame = in_channel.recv()
15 if not flag:
16 print("Model B: No more input.")
17 break
18
19 # Print received message
20 nrows = len(frame.index)
21 print('Model B: (%d rows)' % nrows)
22 print(frame)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(frame)
27 if not flag:
28 raise RuntimeError("Model B: Error sending output.")
The YAML specifies filetype: pandas
for the
connections to files to indicate that the files should be interpreted as CSV
tables using Pandas.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io4_modelA.py
5 inputs: inputA
6 outputs:
7 name: outputA
8 field_names: name,count,size
9
10 - name: python_modelB
11 language: python
12 args: ./src/formatted_io4_modelB.py
13 inputs: inputB
14 outputs:
15 name: outputB
16 field_names: name,count,size
17
18connections:
19 - input: outputA # Connection between model A output & model B input
20 output: inputB
21 - input: ./Input/input.txt # Connection between file and model A input
22 output: inputA
23 filetype: pandas
24 - input: outputB # Connection between model B output and file
25 output: ./output.txt
26 filetype: pandas
As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.
3D Structures as Ply/Obj¶
3D structures can be passed around in Ply or Obj formats.
Model Code:
1import pprint
2# Import classes for input/output channels
3from yggdrasil.interface.YggInterface import (
4 YggPlyInput, YggPlyOutput)
5
6# Initialize input/output channels
7in_channel = YggPlyInput('inputA')
8out_channel = YggPlyOutput('outputA')
9
10# Loop until there is no longer input or the queues are closed
11while True:
12
13 # Receive input from input channel
14 # If there is an error, the flag will be False
15 flag, ply = in_channel.recv()
16 if not flag:
17 print("Model A: No more input.")
18 break
19
20 # Print received message
21 print('Model A: (%d verts, %d faces)' % (ply.nvert, ply.nface))
22 pprint.pprint(ply)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(ply)
27 if not flag:
28 raise RuntimeError("Model A: Error sending output.")
1import pprint
2# Import classes for input/output channels
3from yggdrasil.interface.YggInterface import (
4 YggPlyInput, YggPlyOutput)
5
6# Initialize input/output channels
7in_channel = YggPlyInput('inputB')
8out_channel = YggPlyOutput('outputB')
9
10# Loop until there is no longer input or the queues are closed
11while True:
12
13 # Receive input from input channel
14 # If there is an error, the flag will be False
15 flag, ply = in_channel.recv()
16 if not flag:
17 print("Model B: No more input.")
18 break
19
20 # Print received message
21 print('Model B: (%d verts, %d faces)' % (ply.nvert, ply.nface))
22 pprint.pprint(ply)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(ply)
27 if not flag:
28 raise RuntimeError("Model B: Error sending output.")
The YAML specifies filetype: ply
(filetype: obj
for Obj) for the
connections to files to indicate that the files should be interpreted as
Ply/Obj file formats.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io5_modelA.py
5 inputs: inputA
6 outputs: outputA
7
8 - name: python_modelB
9 language: python
10 args: ./src/formatted_io5_modelB.py
11 inputs: inputB
12 outputs: outputB
13
14connections:
15 - input: outputA # Connection between model A output & model B input
16 output: inputB
17 - input: ./Input/input.ply # Connection between file and model A input
18 output: inputA
19 filetype: ply
20 - input: outputB # Connection between model B output and file
21 output: ./output.ply
22 filetype: ply
In Python the data is returned as a dictionary subclass
(yggdrasil.serialize.PlySerialize.PlyDict
or
yggdrasil.serialize.ObjSerialize.ObjDict
) while in
C/C++ it is returned as a structure (ply_t
or obj_t
).
Tables as Pandas Data Frames¶
In Python, tables can also be passed as Pandas data frames.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggPandasInput, YggPandasOutput)
4
5# Initialize input/output channels
6in_channel = YggPandasInput('inputA')
7out_channel = YggPandasOutput('outputA')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, frame = in_channel.recv()
15 if not flag:
16 print("Model A: No more input.")
17 break
18
19 # Print received message
20 nrows = len(frame.index)
21 print('Model A: (%d rows)' % len(frame.index))
22 print(frame)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(frame)
27 if not flag:
28 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import (
3 YggPandasInput, YggPandasOutput)
4
5# Initialize input/output channels
6in_channel = YggPandasInput('inputB')
7out_channel = YggPandasOutput('outputB')
8
9# Loop until there is no longer input or the queues are closed
10while True:
11
12 # Receive input from input channel
13 # If there is an error, the flag will be False
14 flag, frame = in_channel.recv()
15 if not flag:
16 print("Model B: No more input.")
17 break
18
19 # Print received message
20 nrows = len(frame.index)
21 print('Model B: (%d rows)' % nrows)
22 print(frame)
23
24 # Send output to output channel
25 # If there is an error, the flag will be False
26 flag = out_channel.send(frame)
27 if not flag:
28 raise RuntimeError("Model B: Error sending output.")
The YAML specifies filetype: pandas
for the
connections to files to indicate that the files should be interpreted as CSV
tables using Pandas.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io4_modelA.py
5 inputs: inputA
6 outputs:
7 name: outputA
8 field_names: name,count,size
9
10 - name: python_modelB
11 language: python
12 args: ./src/formatted_io4_modelB.py
13 inputs: inputB
14 outputs:
15 name: outputB
16 field_names: name,count,size
17
18connections:
19 - input: outputA # Connection between model A output & model B input
20 output: inputB
21 - input: ./Input/input.txt # Connection between file and model A input
22 output: inputA
23 filetype: pandas
24 - input: outputB # Connection between model B output and file
25 output: ./output.txt
26 filetype: pandas
As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.
Config style Mappings¶
Simple mapping (key/value) data can be passed around in a colon/tab delimited format.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputA')
6out_channel = YggOutput('outputA')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model A: No more input.")
16 break
17
18 # Print received message
19 print('Model A: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputB')
6out_channel = YggOutput('outputB')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model B: No more input.")
16 break
17
18 # Print received message
19 print('Model B: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model B: Error sending output.")
The YAML specifies filetype: map
for the
connections to files to indicate that the files should be interpreted as
config file formats with single, unique keys on each line followed by a
delimiter and then a value.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io7_modelA.py
5 inputs:
6 - name: inputA
7 type: object
8 outputs:
9 - name: outputA
10 type: object
11
12 - name: python_modelB
13 language: python
14 args: ./src/formatted_io7_modelB.py
15 inputs:
16 - name: inputB
17 type: object
18 outputs:
19 - name: outputB
20 type: object
21
22connections:
23 - input: outputA # Connection between model A output & model B input
24 output: inputB
25 - input: ./Input/input.txt # Connection between file and model A input
26 output: inputA
27 filetype: map
28 - input: outputB # Connection between model B output and file
29 output: ./output.txt
30 filetype: map
In Python the data is returned as a dictionary with one key/value for each line in the file.
JSON Files¶
More complex nested structures can be passed around using JSON serialization.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputA')
6out_channel = YggOutput('outputA')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model A: No more input.")
16 break
17
18 # Print received message
19 print('Model A: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputB')
6out_channel = YggOutput('outputB')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model B: No more input.")
16 break
17
18 # Print received message
19 print('Model B: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model B: Error sending output.")
The YAML specifies filetype: json
for the
connections to files to indicate that the files should be interpreted as
JSON files.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io8_modelA.py
5 inputs:
6 - name: inputA
7 type: object
8 outputs:
9 - name: outputA
10 type: object
11
12 - name: python_modelB
13 language: python
14 args: ./src/formatted_io8_modelB.py
15 inputs:
16 - name: inputB
17 type: object
18 outputs:
19 - name: outputB
20 type: object
21
22connections:
23 - input: outputA # Connection between model A output & model B input
24 output: inputB
25 - input: ./Input/input.txt # Connection between file and model A input
26 output: inputA
27 filetype: json
28 - input: outputB # Connection between model B output and file
29 output: ./output.txt
30 filetype: json
In Python, the data is returned in the type determined by the json package.
YAML Files¶
More complex nested structures can also be represented using the YAML syntax.
Model Code:
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputA')
6out_channel = YggOutput('outputA')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model A: No more input.")
16 break
17
18 # Print received message
19 print('Model A: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model A: Error sending output.")
1# Import classes for input/output channels
2from yggdrasil.interface.YggInterface import YggInput, YggOutput
3
4# Initialize input/output channels
5in_channel = YggInput('inputB')
6out_channel = YggOutput('outputB')
7
8# Loop until there is no longer input or the queues are closed
9while True:
10
11 # Receive input from input channel
12 # If there is an error, the flag will be False
13 flag, obj = in_channel.recv()
14 if not flag:
15 print("Model B: No more input.")
16 break
17
18 # Print received message
19 print('Model B: %s' % str(obj))
20
21 # Send output to output channel
22 # If there is an error, the flag will be False
23 flag = out_channel.send(obj)
24 if not flag:
25 raise RuntimeError("Model B: Error sending output.")
The YAML input file options specifies filetype: yaml
for the
connections to files to indicate that the files should be interpreted as
YAML files.
Model YAML:
1models:
2 - name: python_modelA
3 language: python
4 args: ./src/formatted_io9_modelA.py
5 inputs:
6 - name: inputA
7 type: any
8 outputs:
9 - name: outputA
10 type: any
11
12 - name: python_modelB
13 language: python
14 args: ./src/formatted_io9_modelB.py
15 inputs:
16 - name: inputB
17 type: any
18 outputs:
19 - name: outputB
20 type: any
21
22connections:
23 - input: outputA # Connection between model A output & model B input
24 output: inputB
25 - input: ./Input/input.txt # Connection between file and model A input
26 output: inputA
27 filetype: yaml
28 - input: outputB # Connection between model B output and file
29 output: ./output.txt
30 filetype: yaml
In Python, the data is returned in the type determined by the PyYAML package.