DataAccess and DataStorage

Table of Contents

• Introduction
• DataAccess: an Interface to Access Bytes
• StreamAccess: Stream-like Data Access
• RandomAccess: Array-like Data Access
• DataStorage: DataAccess Factory

Introduction

In IOLink we need interfaces to handle raw data, or byte arrays, in order to use them as underlying data of ImageView instances for example. We want to access that raw data in a way fitting the caller use, also a way to know where this data came from, and sometimes to access data stored relative to it. There are two concepts here:

• DataAccess: defines a way to access to a resource raw data. There are different families, that will be detailed after.
• DataStorage: acts like a factory of DataAccess instances, that also bear some information on the source of the data.

DataAccess: an Interface to Access Bytes

DataAccess instances provide various services in order to read or write their underlying raw data. There are two families of DataAccess:

• StreamAccess: behaves like a classical stream, with a cursor that will change everytime you read or write, so reading the same number of bytes in a row with the same function call will return different result.
• RandomAccess: works like an enhanced buffer, each call to read or write data will give the same result if the arguments are the same.

StreamAccess: Stream-like Data Access

This family of accessors mainly mimics the classic streams. It uses an internal cursor that will be the starting position of read and write operations, with the cursor moving during each of these. One of the properties of a stream is that it usually allows to extend its size when data is written at the end. The cursor is important, as it induces a state based behaviour: doing multiple read operations using the same arguments, will give different results. So a stream can be dangerous in a multithreaded environment.

StreamAccess defines various capabilities:

• READ: used to read data from the stream, usually reading the desired bytes from the current cursor position.
• WRITE: used to write data to the stream, usually writing the desired bytes at the current cursor position.
• SEEK: enable to move the stream's cursor position.
• MEMORY_ACCESS: enable to access the internal CPU memory buffer.

The StreamAccessFactory factory provides diverse ways to create StreamAccess instances. For example, you can create an in-memory extensible stream this way:

auto stream = StreamAccessFactory::allocate();

For a more complete example:

// open a stream access on a file
auto fileAccess = StreamAccessFactory::openFile("/home/user/folder/file.txt");
 
// check if we have all capabilities that we need
assert(fileAccess->support(StreamAccessCapability::READ_WRITE_SEEK));
 
// if the stream cursor is not at the beginning, we reset it.
if (fileAccess->tell() != 0)
{
  fileAccess->seek(0, SeekOrigin::BEGIN);
}
 
// all data is written in one call
std::vector<uint8_t> buffer = {1, 2, 3, 4, 5};
fileAccess->write(buffer.size(), buffer.data());
 
// as we reached the stream end by writing in it, we must reset the cursor.
fileAccess->seek(0, SeekOrigin::BEGIN);
 
// we read each byte, one by one
std::vector<uint8_t> extracted(buffer.size());
fileAccess->read(1, &extracted[0]);
fileAccess->read(1, &extracted[1]);
fileAccess->read(1, &extracted[2]);
fileAccess->read(1, &extracted[3]);
fileAccess->read(1, &extracted[4]);
 
// extracted now contains {1, 2, 3, 4, 5}

RandomAccess: Array-like Data Access

This family of interfaces behaves like a virtual buffer with a fixed size, from wich you can read or write some slice of data. Operations are stateless in this kind of objects and can be used in a multithreaded environment if the implementation allows it. By nature, a RandomAccess cannot be expanded automatically, and trying to write data over its end will cause an error. You can see these interfaces as fancy arrays.

RandomAccess support the following capabilities:

• READ: used to read data, usually reading a given number of bytes from a given offet.
• WRITE: used to write data, usually reading a given number of bytes from a given offet.
• RESIZE: enable to change the size of the accessor, as it can't be expanded by another way.
• MEMORY_ACCESS: enable to access the internal CPU memory buffer.

The RandomAccessFactory factory provides several ways to create RandomAccess instances. For example, you can create an in-memory RandomAccess instance this way:

auto accessor = RandomAccessFactory::allocate(1000);

For a more complete example:

// create an in-memory random access
auto memoryAccess = RandomAccessFactory::allocate(5);
 
// check if we have all needed capabilities (READ and WRITE in this example)
assert(memoryAccess->support(RandomAccessCapability::READ_WRITE));
 
std::vector<uint8_t> buffer = {1, 2, 3, 4, 5};
memoryAccess->write(0, 2, buffer.data()); // write the first two values
memoryAccess->write(2, 3, buffer.data()); // write the three last values
 
std::vector<uint8_t> extracted(buffer.size());
memoryAccess->read(0, extracted.size(), extracted.data()); // read the five values
 
// extracted now contains {1, 2, 3, 4, 5}

DataStorage: DataAccess Factory

DataStorage should be considered as a DataAccess factory/manager that will provide access to resources that are under that storage. It is an interface that will enable users to create StreamAccess or RandomAccess instances, that are relative to the storage instance. There is also a way to identify the storage's source, using the source method that will return a string describing the origin of the data, usually in an URI form. For example a file system storage will return a string of the form "file:/home/user/folder".

When requesting access to resources, you must give the resource id, and the capabilities you want: read, write, or both. If the storage can't return an accessor corresponding to these properties, it will raise an exception.

Current supported DataStorage variants:

• FileSystemStorage: usually having a source with a "file:" scheme, these storages only support opening of StreamAccess, but with all the access capacities.
• HttpStorage: described by a source using the "http:" or "https" schemes, supports RandomAccess opening only in read-only, and StreamAccess opening only in write-only.

An example, using LDM data:

auto headerStream = StreamAccessFactory::openFile("/home/user/folder/header.ldm");
auto storage = headerStream->storage();
auto dataStream = storage->openStreamAccess("data.dat", AccessCapabilities::READ_WRITE);