API

A general data handling toolkit with a powerful class for abstracting away work with multidimensional set. Store scientific data in HDF5 files, but with pleasant Python bindings!

Code by Jakub Dranczewski

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

dataset_suite.load(filename)[source]: Load a dataset-suite object saved as a pickle with dataset_suite.base_dataobject.save().

dataset_suite.load_h5(filename)[source]: Load a dataset-suite object saved as a HDF5 file with dataset_suite.base_dataobject.save_h5().

class dataset_suite.base_dataobject[source]

Bases: object

save(filename, compress=6)[source]

A dataset, datadict, or datalist can be saved as a compressed file to the drive. Use dataset_suite.load() to get an object from said file.

This method saves as a gzipped pickle, which is very flexible in terms of what objects and data can be saved, but requires the user to have the libraries for all pickled objects installed. If you pickle a datadict that includes a pytorch tensor, a person loading the file has to have pytorch installed. Mismatched package versions can also lead to trouble (most notably numpy <2.0 and >2.0).

Generally it is advisable to use dataset_suite.base_dataobject.save_h5() and dataset_suite.load_h5() instead.

Return type:: None

save_h5(filename, compress=6)[source]

A dataset, datadict, or datalist can be saved as a compressed HDF5 file to the drive. Use dataset_suite.load_h5() to get an object from said file.

HDF5 files are a standard format, and you can use viewers like https://myhdf5.hdfgroup.org/ to inspect them.

Parameters:

filename (str)
compress (int)

Return type:

None

classmethod from_h5(h5)[source]

Parameters:: h5 (File | Group)

class dataset_suite.dataset(data, cut=None, **axes)[source]

Bases: base_dataobject

A container for data arrays with labelled axes.

Provide an array of data, and keyword arguments defining the values of the dataset’s axes:

ds.dataset(spectrum, pixel=np.arange(spectrum.shape[0]), wavelength=wavelengths)

This way the axes of the data array are labelled, and information about what each row/column/… corresponds to is stored right with the data.

take(**i)[source]

Return a slice of the dataset at the specified index for a given axis (as a dataset).

For example:

data.take(power=2)

take_raw(**i)[source]

Return a slice of the dataset at the specified index for a given axis (as a numpy array).

For example:

data.take_raw(power=2)
# equivalent to:
data.take(power=2).raw

take_sum(axis)[source]

Sum this dataset along the named axis and return the resulting dataset.

For example:

data.take_sum("wavelength")

expand(new_axis, value)[source]

join(other, axis)[source]

axis(ax)[source]

astype(_type)[source]

property raw

property axes

property ax_dict

property cut

add_cut(key, value)[source]

classmethod from_h5(h5)[source]

Parameters:: h5 (File | Group)

class dataset_suite.datalist(axis, cut=None)[source]

Bases: base_dataobject

A container for data, like a list, but with an axis array that can contain values.

For example, this could store a dataset for each stage position in a scan (but generally you should consider if using a dataset directly may not be better).

axis is the name for the datalists one axis.

append(ds, value=nan)[source]

add_cut(key, value)[source]

property axis

property axes

property datasets

property cut

classmethod from_h5(h5)[source]

Parameters:: h5 (File | Group)

class dataset_suite.datadict(name, cut=None)[source]

Bases: base_dataobject, UserDict

A key: value store for data like a Python dictionary, but with the metadata, cut, and save mechanics from the dataset object.

name is the name of this datadict.

add_cut(key, value)[source]

property name

property dict

property cut

classmethod from_h5(h5)[source]

Parameters:: h5 (File | Group)

dataset_suite.print_dict(data, offset=0)[source]

Print a nicely formatted view of the data in a nested dictionary/datadict.

Parameters:: data (dict | datadict)

dataset_suite.to_datadict(name, data)[source]: Convert a dictionary-like object to a datadict for saving. Tries its best to make numpy arrays out of things.

dataset_suite.glob(path, yes=[], no=[])[source]: A small extension to the excellent glob library. ‘path’ is defined as for the usual glob: a Unix-style pathname with * as a wildcard. ‘yes’ and ‘no’ are lists/tuples or string that should/should not be in the full file name, so act as filters.

dataset_suite.folders(folder='', rel_current=True)[source]: Returns a list of folders within a specified directory. Current directory by default. Set rel_current to False to return just the folder names, instead of the full path relative to current directory.

dataset_suite.prefixes(folder, split, yes=[], no=[])[source]: Returns a list of file prefixes in a given folder that match some conditions. ‘prefix’ is defined as the part of the filename before the string given as ‘split’. ‘yes’ and ‘no’ are lists/tuples or string that should/should not be in the full file name, so act as filters.

dataset_suite.extract(string, *patterns)[source]: Given a string and an arbitrary number of patterns as arguments, this function extracts associated numbers from the string. For example, ‘_P_’ as a pattern will match ‘_P_123’ in a filename and return 123.

dataset_suite.extract_raw(string, *patterns)[source]: You can write regex? Good for you. Given a string and an arbitrary number of patterns as arguments, this function extracts associated things from the string. You have to include a capture group, otherwise errors ensue.

dataset_suite.extract_unique(files, pattern)[source]: Like dataset-suite.extract(), but returns a list of unique matched values in a list of filenames.

dataset_suite.sort_by(files, pattern)[source]

Given a list of strings (like file names or sth) and a pattern, the functions extracts an associated numeric parameter from each string and sorts the list according to that parameter.

For example, if your files contain power information as ‘_P_123’, pass ‘_P_’ as ‘pattern’. Sorting happens in-place in the passed list, so nothing is returned.

dataset_suite.colours(values, cmap=None, minmax=None)[source]

Return a list of colours that a matplotlib colourmap would assign to the given values. minmax is a tuple of v_min, v_max boundaries for the colourmap.

Parameters:: cmap (str | None)

dataset_suite.map_axes(data, **axes)[source]: Helper function for figuring out how the axes you have map to the data. data is the ndarray, and then you pass your axes in any order as keyword arguments.

dataset_suite.print_value(value, offset=0)[source]: Print a value with standard formatting. Used in dataset_suite.print_dict().