Formatted I/O¶
In addition to passing raw strings, the cis_interface framework also has support for formatting/processing messages from/to language native objects.
Scalars¶
The format of messages containing scalar variables (strings, integers, and
floating point numbers) can be specified using a C-style format string
(See C-Style Format Strings for details). In
the example below, the format string "%6s\t%d\t%f\n"
indicates that each
message will contain a 6 character string, an integer, and a float. In addition
to providing a format string, the C API requires the use of different functions
for initializing channels and sending/receiving to/from them.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputA')
out_channel = CisOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: %s' % msg)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(msg)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputB')
out_channel = CisOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: %s' % msg)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(msg)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The same YAML can be used as was used for the example from
Getting started with the modification that
the files are now read/written line-by-line (filetype: ascii
)
rather than all at once:
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | models:
- name: python_modelA
language: python
args: ./src/formatted_io1_modelA.py
inputs: inputA
outputs: outputA
- name: python_modelB
language: python
args: ./src/formatted_io1_modelB.py
inputs: inputB
outputs: outputB
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: ascii
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: ascii
|
Tables by Row¶
Tables can also be passed in a similar manner. For input from a table, the format
string does not need to be provided and will be determined by the source of the
table. There are different API classes/functions for I/O from/to table channels
versus standard channels in each language. (e.g. CisInput
vs.
CisAsciiTableInput
in Python)
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiTableInput, CisAsciiTableOutput)
# Initialize input/output channels
in_channel = CisAsciiTableInput('inputA')
out_channel = CisAsciiTableOutput('outputA', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
name, count, size = msg
# Print received message
print('Model A: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiTableInput, CisAsciiTableOutput)
# Initialize input/output channels
in_channel = CisAsciiTableInput('inputB')
out_channel = CisAsciiTableOutput('outputB', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
name, count, size = msg
# Print received message
print('Model B: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The filetype: table
options in the YAML tell the
cis_interface framework that the file should be read/written as a table
row-by-row including verification that each row conforms with the table.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | models:
- name: python_modelA
language: python
args: ./src/formatted_io2_modelA.py
inputs: inputA
outputs: outputA
- name: python_modelB
language: python
args: ./src/formatted_io2_modelB.py
inputs: inputB
outputs: outputB
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: table
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: table
|
Tables as Arrays¶
Tables can also be passed as arrays. In Python and Matlab, this is done using
separate classes/functions. In C and C++, this is done by passing 1 to the as_array
arguments of the table API classes/functions.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiArrayInput, CisAsciiArrayOutput)
# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputA')
out_channel = CisAsciiArrayOutput('outputA', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, arr = in_channel.recv_array()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: (%d rows)' % len(arr))
for i in range(len(arr)):
print(' %s, %d, %f' % tuple(arr[i]))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send_array(arr)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiArrayInput, CisAsciiArrayOutput)
# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputB')
out_channel = CisAsciiArrayOutput('outputB', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, arr = in_channel.recv_array()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: (%d rows)' % len(arr))
for i in range(len(arr)):
print(' %s, %d, %f' % tuple(arr[i]))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send_array(arr)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML only differs in that as_array: True
for the connections to the files
to indicate that the files should be interpreted as tables and that the tables
should be read/written in their entirety as arrays.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | models:
- name: python_modelA
language: python
args: ./src/formatted_io3_modelA.py
inputs: inputA
outputs:
name: outputA
field_names: name,count,size
- name: python_modelB
language: python
args: ./src/formatted_io3_modelB.py
inputs: inputB
outputs:
name: outputB
field_names: name,count,size
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
as_array: True
filetype: table
- input: outputB # Connection between model B output and file
output: ./output.txt
as_array: True
filetype: table
|
Tables as Pandas Data Frames¶
In Python, tables can also be passed as Pandas data frames.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputA')
out_channel = CisPandasOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model A: (%d rows)' % len(frame.index))
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputB')
out_channel = CisPandasOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model B: (%d rows)' % nrows)
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML specifies filetype: pandas
for the
connections to files to indicate that the files should be interpreted as CSV
tables using Pandas.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | models:
- name: python_modelA
language: python
args: ./src/formatted_io4_modelA.py
inputs: inputA
outputs:
name: outputA
field_names: name,count,size
- name: python_modelB
language: python
args: ./src/formatted_io4_modelB.py
inputs: inputB
outputs:
name: outputB
field_names: name,count,size
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: pandas
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: pandas
|
As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.
3D Structures as Ply/Obj¶
3D structures can be passed around in Ply or Obj formats.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | import pprint
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPlyInput, CisPlyOutput)
# Initialize input/output channels
in_channel = CisPlyInput('inputA')
out_channel = CisPlyOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, ply = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: (%d verts, %d faces)' % (ply.nvert, ply.nface))
pprint.pprint(ply)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(ply)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | import pprint
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPlyInput, CisPlyOutput)
# Initialize input/output channels
in_channel = CisPlyInput('inputB')
out_channel = CisPlyOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, ply = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: (%d verts, %d faces)' % (ply.nvert, ply.nface))
pprint.pprint(ply)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(ply)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML specifies filetype: ply
(filetype: obj
for Obj) for the
connections to files to indicate that the files should be interpreted as
Ply/Obj file formats.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | models:
- name: python_modelA
language: python
args: ./src/formatted_io5_modelA.py
inputs: inputA
outputs: outputA
- name: python_modelB
language: python
args: ./src/formatted_io5_modelB.py
inputs: inputB
outputs: outputB
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.ply # Connection between file and model A input
output: inputA
filetype: ply
- input: outputB # Connection between model B output and file
output: ./output.ply
filetype: ply
|
In Python the data is returned as a dictionary subclass
(cis_interface.serialize.PlySerialize.PlyDict
or
cis_interface.serialize.ObjSerialize.ObjDict
) while in
C/C++ it is returned as a structure (ply_t
or obj_t
).
Tables as Pandas Data Frames¶
In Python, tables can also be passed as Pandas data frames.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputA')
out_channel = CisPandasOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model A: (%d rows)' % len(frame.index))
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputB')
out_channel = CisPandasOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model B: (%d rows)' % nrows)
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML specifies filetype: pandas
for the
connections to files to indicate that the files should be interpreted as CSV
tables using Pandas.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | models:
- name: python_modelA
language: python
args: ./src/formatted_io4_modelA.py
inputs: inputA
outputs:
name: outputA
field_names: name,count,size
- name: python_modelB
language: python
args: ./src/formatted_io4_modelB.py
inputs: inputB
outputs:
name: outputB
field_names: name,count,size
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: pandas
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: pandas
|
As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.
Config style Mappings¶
Simple mapping (key/value) data can be passed around in a colon/tab delimited format.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputA')
out_channel = CisOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputB')
out_channel = CisOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML specifies filetype: map
for the
connections to files to indicate that the files should be interpreted as
config file formats with single, unique keys on each line followed by a
delimiter and then a value.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | models:
- name: python_modelA
language: python
args: ./src/formatted_io7_modelA.py
inputs:
- name: inputA
type: object
outputs:
- name: outputA
type: object
- name: python_modelB
language: python
args: ./src/formatted_io7_modelB.py
inputs:
- name: inputB
type: object
outputs:
- name: outputB
type: object
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: map
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: map
|
In Python the data is returned as a dictionary with one key/value for each line in the file.
JSON Files¶
More complex nested structures can be passed around using JSON serialization.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputA')
out_channel = CisOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputB')
out_channel = CisOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML specifies filetype: json
for the
connections to files to indicate that the files should be interpreted as
JSON files.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | models:
- name: python_modelA
language: python
args: ./src/formatted_io8_modelA.py
inputs:
- name: inputA
type: object
outputs:
- name: outputA
type: object
- name: python_modelB
language: python
args: ./src/formatted_io8_modelB.py
inputs:
- name: inputB
type: object
outputs:
- name: outputB
type: object
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: json
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: json
|
In Python, the data is returned in the type determined by the json package.
YAML Files¶
More complex nested structures can also be represented using the YAML syntax.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputA')
out_channel = CisOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model A: Error sending output.")
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputB')
out_channel = CisOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, obj = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: %s' % str(obj))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(obj)
if not flag:
raise RuntimeError("Model B: Error sending output.")
|
The YAML input file options specifies filetype: yaml
for the
connections to files to indicate that the files should be interpreted as
YAML files.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | models:
- name: python_modelA
language: python
args: ./src/formatted_io9_modelA.py
inputs:
- name: inputA
type: object
outputs:
- name: outputA
type: object
- name: python_modelB
language: python
args: ./src/formatted_io9_modelB.py
inputs:
- name: inputB
type: object
outputs:
- name: outputB
type: object
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
filetype: yaml
- input: outputB # Connection between model B output and file
output: ./output.txt
filetype: yaml
|
In Python, the data is returned in the type determined by the PyYAML package.