Calibrating coordinate system alignment in a scanning transmission electron microscope using a digital twin

Abstract

In four-dimensional scanning transmission electron microscopy (4D STEM) a focused beam is scanned over a specimen and a diffraction pattern is recorded at each position using a pixelated detector. During the experiment, it must be ensured that the scan coordinate system of the beam is correctly calibrated relative to the detector coordinate system. Various simplified and approximate models are used implicitly and explicitly for understanding and analyzing the recorded data, requiring translation between the physical reality of the instrument and the abstractions used in data interpretation. Here, we introduce a calibration method where interactive live data processing in combination with a digital twin is used to match a set of models and their parameters with the action of a real-world instrument.

1 Introduction

For several 4D scanning transmission electron microscopy (STEM) data analysis methods such as differential phase contrast (DPC) (Hamilton & Sheppard, 1984; Lazić et al., 2016) or center of mass (CoM) (Lazić & Bosch, 2017), strain (Usuda et al., 2005; Ozdol et al., 2015) and orientation (Viladot et al., 2013; Rollett & Barmak, 2014) mapping, and ptychography (Hoppe, 1969; Yang et al., 2017) it is essential that the relative orientation between the scan coordinate system and the detector coordinate system is known so that directions on the detector can be put into correspondence with directions in the scan coordinate system (Ning et al., 2022). Figure 1 shows the impact of an incorrect rotation calibration on integrated CoM reconstruction for illustration. This calibration is surprisingly difficult and error-prone in practice, and it was often not required in STEM previously, since conventional detectors such as those used for bright field and annular dark field STEM are rotationally symmetric. Consequently, there has been substantial recent work in both theory and practical implementation to calibrate a microscope’s coordinate system as well as validate the calibration of a given 4D STEM dataset.

Figure 1: Impact of the rotation calibration on an integrated CoM (Lazić & Bosch, 2017) reconstruction of a 4D STEM dataset of $\text{SmB}_{6}$, recorded with a DECTRIS ARINA on a Thermo Fisher Spectra 200: a) The calibration of the scan rotation is off by 90°. b) Correct calibration of the scan rotation.

Three methods are established to perform this alignment. First, the deflection distribution around atom columns in atomic-resolution STEM can be analyzed. Since the deflection is, in a thin specimen approximation, proportional to the projected local field, a beam of negative electrons should be attracted to the positively charged atom columns (Shibata et al., 2012). Furthermore, an electrostatic field should be free of curl. The coordinate system alignment that minimizes curl and ensures a negative divergence at atom positions is assumed to be the correct one for this method (Savitzky et al., 2021).

Second, the self-consistency of the data in a ptychography reconstruction can be analyzed, such as described in (Ning et al., 2022).

In practice, however, real-world specimens are usually not thin enough to assume a simple direct dependence between displacement and field without multiple scattering. In particular, slight misalignments between the atom column axis and the beam axis as well as optical aberrations might lead to apparent distortions (Bürger et al., 2020). Furthermore, atom columns are only reliably resolved if the instrument aberrations are very low and a high spatial resolution is achieved. Not every scanning transmission electron microscope is capable of this, and it is not always desirable to tune and operate the microscope in this mode. Additionally, methods based on analyzing field distribution around atom columns are only applicable to thin crystalline specimens.

The third and more conventional approach relies on over- or underfocusing the beam so that the crossover above resp. below the specimen acts as a point source. In this configuration, a shadow image of the specimen is projected onto the detector. If this shadow image is compared with a STEM image, the transformation between STEM image coordinates and shadow image, i.e. detector coordinates, can be determined. Instead of a separate STEM image, a virtual detector image can be generated from a 4D STEM dataset using a very small virtual detector, for example a single detector pixel, so that it has a high depth of field. In that case both detector shadow image and STEM image are taken from the same 4D STEM dataset (Hu et al., 2023). Underfocus can be used instead of overfocus, which leads to an inverted detector image compared to overfocus.

This method relies on finding a field of view on the specimen without mirror or rotational symmetry and adjusting the microscope so that it is clearly recognizable, including exposing with sufficient dose. However, the matching procedure can be error-prone if performed manually without the help of a dedicated software stack. As an example, the direction of rotation is easily reversed if the procedure is performed manually, an additional handedness change could be missed, and different software or hardware uses different coordinate system conventions. If conventional STEM and 4D STEM use a different acquisition software and/or scan generator on the same microscope, which is often the case, the coordinate system of a conventional STEM image might differ from the coordinate system of a 4D STEM dataset.

Furthermore, different software might interpret and display the data differently. As an example, some software displays the first pixel of a detector frame in the lower left corner with the y axis pointing upwards, while other image processing software shows the first pixel in the upper left with the y axis pointing downwards. That may introduce inadvertent errors if data is inspected visually with one software to calibrate the alignment, but then processed numerically with another.

To eliminate such error sources, the calibration should be performed with the same acquisition system and software as the actual acquisition to ensure consistency. Furthermore, the calibration method and analysis methods should be validated together to interpret data and parameters in a consistent way.

Here we present a method that is derived from the defocus method. It uses automated data processing and a digital twin of the microscope to superimpose all shadow images in an over- or underfocused 4D STEM dataset. If the transformation by the digital twin matches the actual transformation by the microscope, a sharp image is obtained. In case of a mismatch between microscope and twin, the images are not superimposed correctly and the result is blurred. This blurring is independent of symmetries in the specimen, since the translation of the projected images on the detector caused by scanning the beam breaks any symmetry.

2 Methods

The shadow imaging in the overfocused state is modeled with a linear ray tracing approximation. All rays are assumed to originate from a focus point above the specimen, pass through the specimen, and then hit the detector plane. Underfocus, where the focus point is below the specimen, is represented with a negative overfocus value that leads to the correct result without separate adaptation of this model. Scanning the beam shifts the position of the focus point laterally relative to the specimen, but doesn’t shift the beam position on the detector or lead to a beam tilt in the specimen plane. The specimen is assumed to be thin and oriented orthogonal to the optical axis, i.e. not tilted or warped. All coordinate systems are right-handed by default.

Figure 2: Abstract model of the projection with overfocused STEM with the parameters and coordinate axes labeled.

Figure 3: Visualization of the ray paths in the digital twin. The figure is generated using TEMGYM’s visualization routine from the model used in the calculations with the actual dataset (Figures 5-9). The overfocus and the scan step are exaggerated by a factor of $10^{4}$ to make them visible. Note how this model is closer to the physical reality of a scanning transmission electron microscope than the abstract model in Figure 2. TEMGYM Basic translates the parameters of the abstract model into parameters for simulated optical elements such as deflectors and lenses to yield results equivalent to the abstract model.

With these assumptions in place, the electron-optical setup of this imaging mode is described with the following parameters:

• Overfocus Virtual vertical distance of the beam focus over the specimen plane under the assumption of straight propagation between focus point and specimen. Underfocus is specified as a negative overfocus value.

• Scan Pixel Size Lateral distance between scan points. The scan is assumed to be rectangular with uniform step size in both directions.

• Camera Length Virtual vertical distance between specimen and detector under the assumption of straight ray propagation between specimen and detector. In real microscopes the electrons pass through a set of projection lenses that allow adjusting this virtual distance.

• Detector Pixel Size Size resp. pitch of detector pixels. The detector is assumed to be rectangular with equidistant square pixels without gaps.

• Y and X Center Pixel position on the detector that is defined as the beam center, i.e. where the ray from the focus point straight down orthogonal through the specimen plane hits the detector.

• Scan Rotation Rotation between the scan coordinate system in scan pixel units and the detector pixel coordinate system in pixel units. Physically, this is composed of three components: Actual rotation of the scan, rotation imposed by the projection system, and rotation of the detector. In the simple model used here these three rotations are cumulative and can be described with a single parameter.

• Flip Y Invert the detector row index (y) axis. This encodes a handedness change between detector and scan. Any combination of axis inversion and rotation for both detector and scan can be described with a combination of Scan Rotation and Flip Y.

Figure 2 shows a schematic of this model of the experiment with labels for the parameters.

The parameters are chosen to correspond with the usual parameters that are available from a microscope’s user interface. However, it is not guaranteed that they are also interpreted in the same way. As an example, for Thermo Fisher microscopes the parameter “Scan Rotation” describes a clockwise rotation of the STEM image on the screen, while our implementation interprets it as a clockwise rotation of the scan coordinate system around the beam (Z) axis, following the convention (Clausen et al., 2019) in the LiberTEM software (Clausen et al., 2023b). That means the parameter “Scan Rotation” rotates in opposite direction between the Thermo Fisher software and LiberTEM.

“Flip Y” was chosen to describe a handedness difference since it makes it easy to account for an inverted Y axis of a detector.

The parameters Overfocus $o$, Scan Pixel Size $s$, Camera Length $l$ and Detector Pixel Size $d$ are not independent, but their ratios define a single scaling factor $M=\frac{sl}{do}$ between scan pixel coordinates and detector pixel coordinates. Since Scan Pixel Size, Camera Length and Detector Pixel Size are typically calibrated by other means or are fixed physical distances, Overfocus is a natural choice to adjust the scaling factor.

TEMGYM Basic (Landers et al., 2023c) was used to construct a digital twin for a STEM instrument (Figure 3) that can accept these parameters and interprets them in a consistent way to match the abstraction shown in Figure 2. This digital twin can calculate both the detector pixel position and the specimen pixel position of rays as a function of diffraction angle and scan position. With this information, the shadow image projected onto the detector can be transformed into images in scan pixel coordinates. This model assumes that changing the focus doesn’t affect rotation and handedness.

Figure 4: Illustration of the superposition of projected shadow images: a) Projection at first scan position. b) Projection at second scan position. c) Projected image from a). d) Projected image from b). e) Incorrect superposition of the projected images: The features don’t align. This will lead to blurring if many images from different scan positions are superimposed incorrectly. f) Correct superposition: The features align and the superposition of many images from different scan positions creates a sharp image of the object.

If the digital twin describes the action of the microscope correctly, superimposing the images from an overfocused 4D STEM experiment after transforming into scan pixel coordinates will form a sharp image of the specimen (Figure 4). This superimposed image will also correspond to a virtual STEM detector image formed from a single detector pixel, for example the central ray like described in (Hu et al., 2023). However, if the transformation of the digital twin differs from the actual microscope, the images will not be superimposed correctly, leading to a blurred result.

By adjusting the parameters of the digital twin while continuously performing the superposition operation, a user or algorithm can optimize the model parameters until the result is as sharp as possible. A possible metric for sharpness resp. blurriness was described by Crete et al. (2007). The process of optimizing sharpness is similar to conventional focusing and two-fold astigmatism adjustment, meaning it is intuitive for microscope users.

The data processing routine is implemented as a LiberTEM (Clausen et al., 2023b) user-defined function (UDF) called “OverfocusUDF” that performs the superposition and extracts the additional control images. This allows for fast offline and live processing together with live visualization and dynamic parameter updates.

For a smooth user interaction the superposition should be performed very quickly. TEMGYM Basic does trace rays at high speed, but it is still too slow to determine the target scan coordinates for each data point in a 4D STEM dataset in real time. For that reason a linear transformation from detector pixel position and scan position to specimen pixel position is derived from a set of sample rays using a least square optimization. This transformation is applied using an optimized kernel written in Numba.

In many cases the initial parameters will be far off their ideal values, resulting in a featureless superposition image. Since multiple parameters have to match at least approximately in order to form an image with recognizable features, two helper tranformations were implemented:

By selecting a single detector pixel and plotting its value as a function of scan position, a virtual detector image with very large depth of focus can be obtained, as described in (Hu et al., 2023) (Figures 5 - 8, label “OverfocusUDF: point”). As long as the dose and detector resolution is sufficient, this will be a recognizable image of the specimen in scan coordinates. Second, a single shadow image on the detector can be transformed to scan coordinates without superimposing it with shadow images from other scan positions (Figures 5 - 8 label “OverfocusUDF: selected”).

If the parameters are adjusted with sufficient accuracy (Figures 8 and 9), these two images will look the same. By visually comparing them, a user can roughly adjust the overfocus value by comparing the scale, and the rotation and handedness by observing a unique feature on the specimen (Figures 5 – 7). This coarse adjustment will usually be accurate enough to observe features in the superimposed image (Figure 7), allowing for further incremental optimization of all parameters following a protocol similar to optimizing focus and astigmatism.

The model currently doesn’t account for descan error. However, the presence of descan error is checked by summing up detector images without transformation. Without descan error, the beam should have the same position on the detector independent of the scan position. That means that a sum of all detector images should yield a sharp image of the aperture that forms the primary beam.

Additionally, a loss function suitable for numerical optimization can be derived from the calculation of the blurriness of the superimposed image as a function of Scan Rotation and Overfocus. A utility function is included with the “OverfocusUDF” that creates such a loss function by wrapping other parameters for executing it, such as LiberTEM Context, dataset and the remaining model parameters, in a closure (Sussman & Steele Jr., 1998). The returned function accepts only the Scan Rotation and Overfocus as a single vector of two values, and maps them from a range of [(-10, 10), (-10, 10)] to a sensible interval of $\pm 10\deg$ resp. $\pm 25\%$ around their starting value for conditioning. This ensures that the parameters are in a convenient range for numerical optimizers and have roughly the same impact on blurriness. That way, common optimizer implementations work with their default parameters. See section “Data and code availability” for a link to the source code and examples.

The test dataset presented in Figures 5 - 9 was obtained using a gold cross-grating replica sample. Data acquisition was conducted on a JEOL GRANDARM2 equipped with a MerlinEM 4R (quad-chip Medipix) system. The physical pixels on the detector chip are 55 µm wide. The scanning coils of the microscope are synchronized with the detector through TTL signals via a BNC cable. The scan array consists of 64 by 64 steps, with each step measuring 12.5 nm. The frames, each comprising 512 by 512 pixels, were each exposed for 1ms, culminating in a total acquisition time of approximately 4 seconds.

The Supplementary Material (Weber et al., 2023) contains a full screen capture video with explanation of the very first live adjustment using this method. The test specimen was a semiconductor heterostructure in the bump-bonding region of a PCB, where a random area was selected for the preparation of a TEM lamella using a focused ion beam (FIB) instrument. A prototype of the DECTRIS ARINA detector mounted on a probe corrected JEOL JEM-ARM200F transmission electron microscope was used to record the live data. The microscope was operated at 200kV acceleration voltage in STEM mode with a probe of 28.2 mrad semi-convergence angle and 116 pA beam current. The ARINA detector was operated at 50 kHz frame rate with (2x2) binning which resulted in 96x96 detector pixels (detector pixel size: 100 um). This was performed using a prototype implementation of the method described here. Most notably, it didn’t use TEMGYM and accelerated raytracing yet, but a preliminary manual ray tracing implementation of the model described above. The detector data was not recorded due to the ephemeral nature of live processing.

3 Results

A reliable manual alignment could be found in less than 15 min on the very first attempt with this approach using a prototype implementation. See the Supplementary Material at (Weber et al., 2023) for a screen capture video of this first attempt with explanations, and a detailed description. Note how live processing allowed adjustment of microscope parameters and specimen movement interactively. Observing specimen movement allowed rough calibration of scale, rotation and handedness even on a detector with a low number of pixels such as the DECTRIS ARINA, which can only show a small region of interest at coarse spatial resolution. We also note that the high frame rate of the ARINA detector gave a sufficient scan repetition rate to observe the effect of changes in near real time. The rotation and handedness parameters obtained with the digital twin were validated to correspond to the parameters of LiberTEM’s center of mass (CoM) analysis (Clausen et al., 2023a) and the single side band (SSB) implementation in the Ptychography 4.0 project (Weber et al., 2021). This enabled successful ptychographic reconstruction using this SSB implementation in unrelated experiments following the adjustment in the screen capture video. The results for Overfocus, Scan Pixel Size, Camera Length and Detector Pixel Size have not been validated experimentally yet.

The Figures 5-8 show the manual adjustment procedure with the test dataset described in the previous section using the most recent software version. The dataset is available at (Weber et al., 2023).

Figure 5: Starting condition with scale and alignment miscalibrated. Control interfaces for the calibration parameters are shown at the bottom. The field “Blur metric” shows the blurriness of the “OverfocusUDF: shifted_sum” image as calculated by the algorithm by Crete et al. (2007). The plot “OverfocusUDF: point” shows the trace of a single detector pixel as a function of scan position. “OverfocusUDF: shifted_sum” is the main adjustment plot that shows all detector images superimposed in specimen coordinates according to the transformation calculated by TEMGYM that is derived from the input parameters below the plots. If the transformation by TEMGYM matches the actual transformation by the microscope, this plot is a sharp image of the specimen. Note that it is blurred here and doesn’t show any specimen features, meaning the parameters are not correct yet. “SumUDF: intensity” shows a sum of all untransformed diffraction patterns. The plot “OverfocusUDF: selected” shows the diffraction pattern at the center of the scan grid rotated and scaled to scan coordinates according to the current settings. Structures of the specimen are only recognizable in “OverfocusUDF: point”, while “OverfocusUDF: selected” magnifies the frame so much with the current settings that only a few pixels are in the field of view. The sharp image of the beam-forming aperture shows that the pivot point is adjusted correctly since the aperture is at the same position for each scan point. “OverfocusUDF: sum” shows the sum of partially transformed diffraction patterns where only “Scan Rotation” and “Flip Y” are applied, in contrast to “SumUDF: intensity” that shows the sum of the untransformed diffraction patterns where the position of the beam on the detector can be judged.

Figure 6: Compared to Figure 5, the “Overfocus” parameter was adjusted so that “OverfocusUDF: point” and “OverfocusUDF: selected” have approximately the same scale.

Figure 7: Compared to Figure 6, the “Scan Rotation” and “Flip Y” parameters were adjusted so that “OverfocusUDF: point” and “OverfocusUDF: selected” show approximately the same image. Blurred specimen features are starting to appear in “OverfocusUDF: shifted_sum”.

Figure 8: Compared to Figure 7, the “Overfocus” and “Scan Rotation” parameters are fine-tuned so that “OverfocusUDF: shifted_sum” appears sharp. Note how a small change in “Overfocus” and “Scan Rotation” compared to Figure 7 causes a large difference in “OverfocusUDF: shifted_sum”, demonstrating the sensitivity of this method.

Figure 9: Compared to Figure 8, the “Overfocus” and “Scan Rotation” parameters are numerically optimized by minimizing the blur metric using global optimization within an interval around the values in Figure 8.

The processing takes 1.5 s for a single pass over the test file used in Figures 5-9, a MIB 4D STEM dataset of size 64 $\times$ 64 $\times$ 512 $\times$ 512 with 16 bits per pixel, corresponding to a data rate of about 1.4 GB/s or 2730 frames per second, on an AMD EPYC 7F72 with 24 physical cores and 256 GB RAM using LiberTEM 0.13 and the default Dask executor. Processing takes 1.38 s, corresponding to an uncompressed data rate of about 3.5 GB/s or 47,000 frames per second, for a file of size 256 $\times$ 256 $\times$ 192 $\times$ 192 with 16 bits per pixel in the compressed HDF5 format generated by the DECTRIS ARINA. This is fast enough to follow typical 4D STEM data rates live. Live processing allows adjustment of both microscope parameters, such as specimen region, magnification and focus, and model parameters. However, many repeated scans may degrade a specimen through radiation damage or contamination.

A loss function using the blurriness metric (Crete et al., 2007) as implemented in the Scikit-Image (van der Walt et al., 2014) function skimage.measure.blur_effect was created using the helper function described in the previous section. It was minimized numerically using the “simplicial homology global optimization” (SHGO) algorithm (Endres et al., 2018) with “constrained optimization by linear approximation” (COBYLA, Powell (1994)) for local search, as implemented in the SciPy (Virtanen et al., 2020) function scipy.optimize.shgo. The result is shown in Figure 9. The fast processing of LiberTEM and the OverfocusUDF allows numerical optimization within about 12 s from Figure 8 to Figure 9 on the system described above. The optimization converged reliably to near optimal values from various starting values, provided the optimum was within the optimization bounds around the starting value as described in the previous section. The optimization was implemented to update the plots live, which allows observing the convergence behavior and checking if the optimization result is sensible.

The following workflow turned out to be practical:

1. Select a specimen region, scan pixel size and other microscope parameters that show good contrast and recognizable features in the “OverfocusUDF: point” plot. The method only works at low or medium magnification, not at atomic resolution, since it doesn’t take diffraction into account.

2. Set the Scan Pixel Size, Camera Length and Detector Pixel Size to their correct values, as far as known.

3. Adjust the focus of the microscope so that specimen features are recognizable on the detector. Ideally, the features on the detector have roughly the same scale as in the “OverfocusUDF: point” plot.

4. Set the Overfocus value according to the overfocus given by the microscope. Specimen features should be recognizable in both the “OverfocusUDF: point” and “OverfocusUDF: selected” plots now.

5. If possible, adjust Overfocus so that features in “OverfocusUDF: point” and “OverfocusUDF: selected” have roughly the same scale. This works well on grids, for example.

6. Adjust Scan Rotation and Flip Y so that movements of the specimen or low-symmetry specimen features are consistent between “OverfocusUDF: point” and “OverfocusUDF: selected”. A blurred image of the specimen should appear in “OverfocusUDF: shifted_sum” now.

7. Confirm that the scale is approximately the same between “OverfocusUDF: point” and “OverfocusUDF: selected”.

8. Fine-tune Scan Rotation and Overfocus until “OverfocusUDF: shifted_sum” is sharp. Confirm that Flip Y is set correctly if the image remains blurred.

9. Optionally, refine with numerical optimization. This works once “OverfocusUDF: shifted_sum” appears only moderately blurred, meaning the starting values are already close enough.

4 Discussion

Live processing was helpful for finding a suitable specimen region and microscope parameters as well as coarse adjustment of Scan Rotation and Flip Y, since observing the effect of specimen movement interactively helps to orient the user. For fine adjustments it is advantageous to record a dataset after coarse adjustment of microscope and alignment parameters that show recognizable features in the superposition image. The user can then continue to adjust on such an offline dataset. LiberTEM allows using offline and live data interchangeably, which helps the implementation of such a workflow. This avoids excessive exposure of the specimen. Furthermore, recording such an overfocused calibration dataset before real measurements can allow later validation of the parameters for data interpretation.

Currently, the implementation uses TEMGYM Basic for the digital twin, i.e. a matrix optics approximation. However, the method is not limited to this: The ray tracing engine is exchangeable as long as it accepts the same parameters, meaning a more advanced model such as TEMGYM Advanced (Landers et al., 2023a, b) could be used. This software traces electron trajectories through electromagnetic fields and enables the inclusion of 3^rd order and higher aberration effects from the objective lens. If the internal interpolation for quick mapping of detector frames is adapted to allow non-linear relations between detector pixel, scan position and specimen pixel, this approach could also be extended to take aberrations into account. Furthermore, the digital twin could be extended to match each optical element of the electron microscope exactly, as already demonstrated in Landers et al. (2023c).

The method provides a clear validation for parameters that leaves little room for error or interpretation: If the resulting image is sharp, the parameters of microscope and digital twin are consistent. It can be performed as part of a 4D STEM acquisition workflow without switching detector or scan engine. Furthermore, it is sufficiently fast to perform many checks during a session, for example after a change of microscope parameters. In particular if a good initial guess is available, fine adjustment is quick and intuitive, and can even be performed automatically. Since the image used for fine calibration is composed of many detector frames, it has a better signal to noise ratio than individual detector frames or the plot of the central pixel, allowing quick fine calibration at a higher scan rate and low beam current if a good starting value is known.

The current user interface in a Jupyter notebook works as a proof of concept. In the future, a stand-alone tool could be developed or the method could be integrated into the microscope control software. In particular, the actual microscope parameters could be read out digitally and applied to the digital twin. If microscope calibration values instead of model parameters are optimized, it would allow the microscope to operate directly in units that correspond to the digital twin.

The approach is also extensible to other parameters of the microscope that change the detector image in a systematic way and allow superposition of transformed detector images to form a sharp image of an object in the beam, such as a specimen or aperture. As a disadvantage, it only works with 4D STEM detectors, not conventional or segmented STEM detectors.

It should be investigated if changing the focus strongly in real microscopes can affect the observed rotation or other parameters.

In this paper we already show how parameters can be transformed between three different models or digital twins that describe the projection performed by the microscope: TEMGYM Basic uses ray tracing through a simplified STEM model based on a linear approximation for an objective lens and double beam deflectors (Figure 3). The parameters exposed towards the user (controls in Figure 5-9) describe a simpler model where the beam is not deflected by double deflectors or focused by a lens, but where the specimen coordinate system is simply shifted and a focus position is specified (Figure 2). Internally, the “OverfocusUDF” uses a simple ray transfer matrix that directly performs a mapping from detector coordinates and scan position to specimen pixel coordinates for optimal performance.

Since each model is linear, and provides a good approximation of the actual instrument’s behaviour under the chosen conditions, these are equivalent descriptions of the same transformation that are specified in different ways to match the context in which they are used. It also allows matching them exactly using computational methods. The user interface aids the user in finding approximate parameters that match the actual projection by the instrument. Numerical optimization can then be used to fine-tune this result further. In effect, this determines the instrument parameters in a reliable and precise way.

Currently, the commonly used implicit models and parameters for metadata schemas and data analysis only approximate the action of the instrument in specific optical modes such as transmission electron microscopy (TEM), STEM, electron energy loss spectroscopy (EELS), etc. The model in Figure 2 can describe strongly defocused STEM, but can’t model the projection for in-focus STEM since it doesn’t consider diffraction, for example. If a more comprehensive or even full physical model is used to describe the projection by the microscope, the approach described here can allow transformation of physical parameters, such as objective lens current, into parameters for any simplified abstract model, such as the STEM model here. It also allows transformation of parameters between different models, for example to determine the parameters on a different microscope to perform an equivalent projection. A set of comprehensive model, parameters and calibration can describe the action of the instrument in every detail and in every state in a reproducible way, provided the model is sophisticated enough. At the same time, corresponding parameters for any simplified model can be derived using, for example, a numerical solver. That allows to use the same metadata schema and model framework for any analysis performed with any TEM. That means a comprehensive digital twin constitutes a potential foundation for universal metadata for electron microscopy.

5 Code and data availability

The source code for the adjustment code as well as the Jupyter notebook to run it on offline data can be found at https://github.com/LiberTEM/Microscope-Calibration/tree/37066766ab446153ec2ebb3370cb879905cc410b.

The TEMGYM Basic version used here is https://github.com/TemGym/TemGym/tree/bc28049225489b6ca75bc3377a4c64f53d77eb31.

The dataset used in the figures for offline calibration, a screen capture video and additional explanation, as well as a runnable Apptainer image with the complete software stack for reproducibility can be found at https://doi.org/10.5281/zenodo.10418769.

6 Acronyms

STEM

scanning transmission electron microscopy

DPC

differential phase contrast

CoM

center of mass

SSB

single side band

UDF

LiberTEM (Clausen et al., 2023b) user-defined function

7 Financial support

Swiss National Science Foundation is kindly acknowledged for co-funding the electron microscope (R’Equip Project 206021_177020) used herein.

We gratefully acknowledge support by the AIDAS project.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 823717 – ESTEEM3.

We gratefully acknowledge support by DECTRIS for providing access to a prototype of the ARINA detector at PSI.

8 Competing interests

The authors declare no competing interests.