Skip to content

Draft: Change the internal HDF5 storage scheme of frames to avoid costly matrix copies

Ammar Nejati requested to merge improveFrameHDF5Storage into main

Each frame is stored in memory as a column-major Eigen (n_rows, n_columns)-matrix. The matrix data will be written to the HDF5 file with a (n_rows, n_columns)-DataSpace. HDF5 data storage is however row-major. That means that, in the current scheme, the stored data in the file does not correspond to the actual matrix structure.

For instance, the 2x3 matrix

{ {1, 2, 3}
  {4, 5, 6} }

will be stored in a column-major Eigen matrix as

{1, 4, 2, 5, 3, 6}

With the current scheme, the same matrix will be stored in the HDF5 as

{ {1, 4, 2}
  {5, 3, 6} }

However, the HDF5 storage scheme only matters internally, since the frame data cannot be accessed without using NSXTool. The outcome, i.e., the frame matrix, is the same regardless of the storage scheme. The major benefit of the current storage scheme is that it avoids costly copies of a frame performed to convert a row-matrix to a column matrix.

Edited by Ammar Nejati

Merge request reports