Load and save data

Load patterns

From a file

kikuchipy can read and write experimental EBSD patterns and EBSD master patterns from/to multiple formats (see Supported EBSD formats). To load patterns from file use the load() function. For example, to load the first scan from an EDAX TSL or Bruker Nano HDF5 file into memory:

>>> s = kp.load('patterns.h5')
>>> s
<EBSD, title: , dimensions: (10, 20|60, 60)>

Or, load the spherical projection of the northern hemisphere of an EBSD master pattern for beam energies from 10 to 20 keV from EMsoft’s master pattern file, returned from the their EMEBSDmaster.f90 program:

>>> s = kp.load("master_patterns.h5")
>>> s
<EBSDMasterPattern, title: , dimensions: (11|1001, 1001)>

All file readers support accessing the data without loading it into memory (with the Dask library), which can be useful when processing large data sets:

>>> s = kp.load('patterns.h5', lazy=True)

When loaded lazily, EBSD patterns are processed chunk by chunk, which in many cases leads to longer processing times, so processing should be done with some care. See the relevant HyperSpy user guide for information on how to do this.

To visualise the data:

>>> s.plot()

Upon loading, kikuchipy tries to read all scan information from the file and stores everything it can read in the original_metadata attribute. Also, some information may be stored in a standard location in the metadata attribute where it can be used by routines. The number of patterns in horizontal and vertical direction, pattern size in pixels, scan step size and detector pixel size is stored in the axes_manager attribute. To access this information:

>>> s.metadata
>>> s.original_metadata
>>> s.axes_manager

This information can be modified directly, and information in metadata and axes_manager can also be modified by the EBSD class methods set_experimental_parameters(), set_phase_parameters(), set_scan_calibration() and set_detector_calibration(), or the EBSDMasterPattern class methods set_simulation_parameters() or set_phase_parameters(). For example, to set or change the accelerating voltage, horizontal pattern centre coordinate and static background pattern (stored as a numpy.ndarray):

>>> s.set_experimental_parameters(
...     beam_energy=15, xpc=0.5073, static_bg=static_bg)

From a NumPy array

An EBSD or EBSDMasterPattern object can also be created directly from a numpy.ndarray. To create a data set of (60 x 60) pixel patterns in a (10 x 20) grid, i.e. 10 and 20 patterns in the horizontal and vertical scan directions respectively, of random intensities:

>>> import numpy as np
>>> import kikuchipy as kp
>>> s = kp.signals.EBSD(np.random.random((20, 10, 60, 60)))
>>> s
<EBSD, title: , dimensions: (10, 20|60, 60)>

From a Dask array

When processing large data sets, it is useful to load data lazily with the Dask library. This can be done upon reading patterns from a file by setting lazy=True when using load(), or directly from a dask.array.Array:

>>> import dask.array as da
>>> import kikuchipy as kp
>>> s = kp.signals.LazyEBSD(
...         da.random.random((20, 10, 60, 60), chunks=(2, 2, 60, 60)))
>>> s
<LazyEBSD, title: , dimensions: (10, 20|60, 60)>

From a HyperSpy signal

HyperSpy provides the method set_signal_type() to change between BaseSignal subclasses, of which EBSD, EBSDMasterPattern and VirtualBSEImage are three. To one of these objects from a Signal2D object:

>>> import numpy as np
>>> import hyperspy.api as hs
>>> import kikuchipy as kp
>>> s = hs.signals.Signal2D(np.random.random((20, 10, 60, 60)))
>>> s
<Signal2D, title: , dimensions: (10, 20|60, 60)>
>>> s.set_signal_type("EBSD")
>>> s
<EBSD, title: , dimensions: (10, 20|60, 60)>
>>> s.set_signal_type("EBSDMasterPattern")
>>> s
<EBSDMasterPattern, title: , dimensions: (10, 20|60, 60)>
>>> s.set_signal_type("VirtualBSEImage")
<VirtualBSEImage, title: , dimensions: (10, 20|60, 60)>

Save patterns

To save experimental EBSD patterns to file use the save() method. For example, to save an EBSD object s in an HDF5 file, with file name patterns.h5, in our default h5ebsd format:

>>> s.save('patterns')

Danger

If we want to overwrite an existing file:

>>> s.save('patterns.h5', overwrite=True)

If we want to save patterns in NORDIF’s binary .dat format instead:

>>> s.save('patterns.dat')

To save an EBSDMasterPattern object to an HDF5 file, use the save() method inherited from HyperSpy to write to their HDF5 specification:

>>> s
<EBSDMasterPattern, title: , dimensions: (10, 20|60, 60)>
>>> s.save("master_patterns.hspy")

These master patterns can then be read into an EBSDMasterPattern object again via HyperSpy’s load():

>>> s = hs.load("master_patterns.hspy", signal_type="EBSDMasterPattern")
<EBSDMasterPattern, title: , dimensions: (10, 20|60, 60)>

Note

To save results from statistical decomposition (machine learning) of patterns to file see the section Saving and loading results in HyperSpy’s user guide. Note that the file extension .hspy must be used upon saving, s.save('patterns.hspy'), as the default extension in kikuchipy, .h5, yields a kikuchipy h5ebsd file. The saved patterns can then be reloaded using HyperSpy’s load() function followed by set_signal_type('EBSD') as explained above.

Supported EBSD formats

Currently, kikuchipy has readers and writers for the following file formats:

Format	Read	Write
Bruker Nano h5ebsd	Yes	No
EDAX TSL h5ebsd	Yes	No
kikuchipy h5ebsd	Yes	Yes
NORDIF binary	Yes	Yes
EMsoft simulated EBSD HDF5	Yes	No
EMsoft EBSD master pattern HDF5	Yes	No

Note

If you want to process your patterns with kikuchipy, but use an unsupported EBSD vendor software, or if you want to write your processed patterns to a vendor format that does not support writing, please request this feature in our issue tracker.

h5ebsd

The h5ebsd format [Jackson2014] is based on the HDF5 open standard (Hierarchical Data Format version 5). HDF5 files can be read and edited using e.g. the HDF Group’s reader HDFView or the Python package used here, h5py. Upon loading an HDF5 file with extension .h5, .hdf5 or .h5ebsd, the correct reader is determined from the file. Supported h5ebsd formats are listed in the table above.

If an h5ebsd file contains multiple scans, as many scans as desirable can be read from the file. For example, if the file contains three scans with names Scan 1, Hello, C3PO! and The best patterns in that order:

>>> s1, s2, s3 = kp.load(
...     'patterns.h5',
...     scan_group_names=["Scan 1", "Hello, C3PO!", "The best patterns"]
... )

Here, the h5ebsd file_reader() is called. If only Hello, C3PO! is to be read, scan_group_names="Hello, C3PO!" can be passed. The scan_group_names parameter is unnecessary if only Scan 1 is to be read since reading the first scan in the file is the default behaviour.

So far, only saving patterns to kikuchipy’s own h5ebsd format is supported. It is possible to write a new scan with a scan name Scan x, where x is a number, to an existing, but closed, h5ebsd file in the kikuchipy format, e.g. one containing only Scan 1, by passing:

>>> s.save('patterns.h5', add_scan=True, scan_number=2)

Here, the h5ebsd file_writer() is called.

NORDIF binary

Patterns acquired using NORDIF’s acquisition software are stored in a binary file usually named Pattern.dat. Scan information is stored in a separate text file usually named Setting.txt, and both files usually reside in the same directory. If this is the case, the patterns can be loaded by passing the file name as the only parameter. If this is not the case, the setting file can be passed upon loading:

>>> s = kp.load('Pattern.dat', setting_file='/somewhere/Setting_new.txt')

Here, the NORDIF file_reader() is called. If the scan information, i.e. scan and pattern size, in the setting file is incorrect or the setting file is not available, patterns can be loaded by passing:

>>> s = kp.load('filename.dat', scan_size=(10, 20), pattern_size=(60, 60))

If a static background pattern named Background acquisition.bmp is stored in the same directory as the pattern file, this is stored in metadata upon loading.

Patterns can also be saved to a NORDIF binary file, upon which the NORDIF file_writer() is called. Note, however, that so far no new setting file, background pattern, or calibration patterns is created upon saving.

EMsoft simulated EBSD HDF5

Dynamically simulated EBSD patterns returned by EMsoft’s EMEBSD.f90 program as HDF5 files can be read into an EBSD object:

>>> s = kp.load("simulated_ebsd.h5")
>>> s
<EBSD, title: simulated_ebsd, dimensions: (29800|60, 60)>

Here, the EMsoft simulated EBSD file_reader() is called, which takes the optional argument scan_size. Passing scan_size=(149, 200) will reshape the pattern data shape from (29800, 60, 60) to (149, 200, 60, 60):

>>> s = kp.load("simulated_ebsd.h5", scan_size=(149, 200))
>>> s
<EBSD, title: simulated_ebsd, dimensions: (200, 149|60, 60)>

Simulated EBSD patterns can be written to the kikuchipy h5ebsd format, the NORDIF binary format or to HDF5 files using HyperSpy’s HDF5 specification as explained above.

EMsoft EBSD master pattern HDF5

Master patterns returned by EMsoft’s EMEBSDmaster.f90 program as HDF5 files can be read into an EBSDMasterPattern object:

>>> s = kp.load("master_patterns.h5")
>>> s
<EBSDMasterPattern, title: master_patterns, dimensions: (16|1001, 1001)>

Here, the EMsoft EBSD master pattern file_reader() is called, which takes the optional arguments projection, hemisphere and energy_range. The spherical projection is read by default. Passing projection="lambert" will read the square Lambert projection instead. The northern hemisphere is read by default. Passing hemisphere="south" or hemisphere="both" will read the southern hemisphere projection or both, respectively. Master patterns for all beam energies are read by default. Passing energy_range=(10, 20) will read the master patterns with beam energies from 10 to 20 keV.

>>> s = kp.load(
...     "master_patterns.h5",
...     projection="lambert",
...     hemisphere="both",
...     energy_range=(10, 20)
... )
>>> s
<EBSDMasterPattern, title: , dimensions: (2, 11|1001, 1001)>

Master patterns can be written to HDF5 files using HyperSpy’s HDF5 specification as explained above.

See [Jackson2019] for a hands-on tutorial explaining how to simulate these patterns with EMsoft, and [Callahan2013] for details of the underlying theory.

From kikuchipy into other software

Patterns saved in the h5ebsd format can be read by the dictionary indexing and related routines in EMsoft using the EMEBSD reader. Those routines in EMsoft also have a NORDIF reader.

Patterns saved in the h5ebsd format can of course be read in Python like any other HDF5 data set:

>>> import h5py
>>> with h5py.File('/path/to/patterns.h5', mode='r') as f:
...     patterns = f['Scan 1/EBSD/Data/patterns'][()]

Load and save virtual BSE image

One or more virtual backscatter electron (BSE) images in a VirtualBSEImage object can be read and written to file using one of HyperSpy’s many readers and writers. If they are only to be used internally in HyperSpy, they can be written to and read back from HyperSpy’s HDF5 specification as explained above for EBSD master patterns.

If we want to write the images to image files, HyperSpy also provides a series of image readers/writers, as explained in their user guide. If we wanted to write them as a stack of TIFF images:

>>> vbse
<VirtualBSEImage, title: , dimensions: (5, 5|200, 149)>
>>> vbse.rescale_intensity()  # Fill available data type range
>>> vbse.unfold_navigation_shape()  # 1D navigation space required for TIFF
>>> vbse
<VirtualBSEImage, title: , dimensions: (25|200, 149)>
>>> vbse.save("vbse.tif")  # Easilly read into e.g. ImageJ

We can also write them to e.g. png or bmp files with Matplotlib:

>>> import matplotlib.pyplot as plt
>>> nav_size = vbse.axes_manager.navigation_size
>>> _ = [
...     plt.imsave(
...         f"vbse{i}.png", vbse.inav[i].data) for i in range(nav_size))
... ]

Read the TIFF stack back into a VirtualBSEImage object:

>>> import hyperspy.api as hs
>>> vbse = hs.load("vbse.tif", signal_type="VirtualBSEImage")
>>> vbse
<VirtualBSEImage, title: , dimensions: (25|200, 149)>