Main concept

If you already use IOLink, you know it allows handling of views from any origin. It can also allows to create in-memory views to work efficiently on memory images. That can be sufficient for mosts of your needs. But, with IOLink, view data cannot be extracted nor exported to/from current formats (bmp, jpeg, tiff, etc.) or proprietary formats.

IOLink is a set of interfaces with a few implementations for general needs. As these interfaces are generic and can theorically permit to handle any kind of Images, it is up to you to develop your own reader to handle a specific format, by implementing IOLink interfaces. These readers are called extractors. And thanks to this system of interfaces, any application based on IOLink will be able to handle your specific format.

No need to implement your own reader for any current formats though, as one implementation is sufficient, and it is often already done.

IOFormat is here to gather and provide readers and writers for various formats.

IOFormat is an aggregator of extractors/writers for any ordinary formats and even for your custom format.

Plugins loading

Extractor Plugins

Each extractor plugin declares the formats that it supports, and an extension mapping to these formats. For example, the plugin "Jpeg" tells that it supports the "JPEG" format and maps the "jpeg" and "jpg" extension to it. Some plugins support multiple formats, so this can become more complicated.

At the initialization of IOFormat, all the extractor plugins are loaded in an OS-specific order, usually the lexicographic order.

The default plugin order can be overriden via the YAML file located in plugins folder, and called: ioformat_config.yml. This file is optional for the IOFormat initialization.

It can contain a libraries section with a sequence of patterns, which allow to discriminate libraries (.dll, .so) you want to consider and load as plugins.

For example, if the yaml contains following section:

libraries:
    - dicom
    - png-lib

Only libraries which contain dicom or png-lib patterns will be opened and tested. If these are effective IOFormat plugins, they will be loaded.

It can also contain a plugin_order section with a sequence of plugins identified by their name to detail the order how plugins will be initialized. All non listed plugins will be initialized before the ones listed.

In this example, we configure IOFormat to try the UniversalPlugin first, the PngPlugin next, and finally the TiffPlugin:

plugin_order:
    - UniversalPlugin
    - PngPlugin
    - TiffPlugin

UniversalPlugin supports the TIFF, PNG and MRC formats.
PngPlugin only supports the PNG format.
TiffPlugin only supports the TIFF format.

Consequently:

when opening a TIFF file, IOFormat will try first the UniversalPlugin and then the TiffPlugin.
when opening a PNG file, IOFormat will try first the UniversalPlugin and then the PngPlugin.
when opening an MRC file, IOFormat will try first an MRC plugin if it exists, and then only the UniversalPlugin

The plugin order initialization, and the resulting format mapping can easily be viewed in the logs, at the LOG_INFO level. Here is an example of such a log:

<IOFormat> [14:21:52 +02:00] [info] [ioformat::ExtractorPluginSelector::initPlugins]: Initialization of plugin MrcPlugin v2.2.0
<IOFormat> [14:21:52 +02:00] [info] [ioformat::ExtractorPluginSelector::initPlugins]: Initialization of plugin PngPlugin v2.2.0
<IOFormat> [14:21:52 +02:00] [info] [ioformat::ExtractorPluginSelector::initPlugins]: Initialization of plugin TiffPlugin v2.3.0
...
<IOFormat> [14:21:52 +02:00] [info] [ioformat::WriterPluginSelector::WriterPluginSelector]: Format MRC writer plugins order: MrcPlugin

The last three lines indicate the order of the calls for plugins to open PNG, MRC and TIFF formats. The MrcPlugin didn't have to be listed in the configuration file because it is the only one to handle the MRC format.

It is possible to define an environment variable IOFORMAT_CONFIG_FILE which shall contain the yaml configuration with the same syntax (see above) than the file. If this variable is set, the yaml file content is ignored.

Writer Plugins

Writer plugins behave a lot like extractors, but they are not affected by the configuration file described above. There is currently no way to change the way each writer plugin is associated with a format.

Extractors

An extractor is an implementation of one member of the IOLink View family. This extractor has in charge of decoding data from a file or an accessor and provides decoded data through interface methods. Usually, IOFormat just instanciate these extractors and return them to the user.

Custom extractor

Any developer is free to implement an extractor as they wish for their particular needs. IOFormat handles extractors with a plugin system. Each available extractor in IOFormat is developed as a plugin.

To implement a new extractor, some rules should be respected to maintain coherency:

Extractors should read as little data as possible from their source at initialization. The view shall be created as fast as possible, and its content should not be loaded in memory.
For a format describing one image, it must implement the ImageView interface, the extractor should at least support the READ capability, enabling reading the data of the image. If the format enables editing the data of the image, the extractor could support the WRITE capability. And if the image's data are directly available in memory, it should support the MEMORY_ACCESS capability.
If the format stores an heterogeneous set of images, it must implement the MultiImageView interface, and at least support the READ capability.
If the format stores a collection of images representing different resolutions of the same source image, it must implement the LodImageView interface. The resolution of index 0 being the source image, positive indices returning images of lower resolution, and negative indices returning images of higher resolution.
If the format stores tabular data, it must implement the DataFrameView interface, and at least support the READ capability.

Writers

A writer is an implementation of IOLink the Writer interface for a specific set of formats. They are used when a View content must be encoded into an output format.

By design, a writer owns the view which is exported, because it can need to read all of its content in one step. Not all formats are not encoded line per line, or in tiles and thus each implementation can use its own efficient way to build its output data into the desired format from the view content.

Each writer is instantiated through its own factory, because each format has its own list of parameters, for example compression.