Lecture Data Science for Electron Microscopy Winter 2024

Github Code

Studon Link

1 Lecture 1: Intro (25.10.2024)

• Introduction
• d2l Chapter 2: Preliminaries

2 Lecture 2: Regression and Sensor Fusion (8.11.2024)

• d2l Chapter 3: Regression
• Sensor Fusion Slides

3 Lecture 3: CNNs (15.11.2024)

• d2l Chapter 7: CNNs
• d2l Chapter 8: CNNs

4 Lecture 4: Classification, Segmentation, AutoEncoders (22.11.2024)

• d2l Chapter 4: Classification
• d2l Chapter 14.9: Segmentation
• Segmentation
• Dimensionality Reduction
  • PCA
  • Autoencoder
  • Variational Autoencoder

5 Miniproject (29.11. - 13.12.2024)

In the miniproject, you will test multiple deep neural network architectures on one of four microscopy-related tasks. You should summarize your results in a short presentation (5 minutes + 2 minutes discussion) and deliver a Jupyter Notebook with your code and results. The miniproject will be graded and will count as 40% towards your final grade.

1. Segmentation Task

   We will use the HRTEM dataset from “A robust synthetic data generation framework for machine learning in high-resolution transmission electron microscopy (HRTEM)” by Rangel DaCosta et al. (2024) to implement a segmentation model. The goal is to segment nanoparticles in HRTEM images.

   Please use the article “A robust synthetic data generation framework for machine learning in high-resolution transmission electron microscopy (HRTEM)” by Rangel DaCosta et al. (2024) as a starting point for your implementation.

   The datast contains pairs of HRTEM images and ground truth segmentations.

2. VAE & Dimensionality Reduction

   We will use the dataset from “Uncovering material deformations via machine learning combined with four-dimensional scanning transmission electron microscopy” by Shi et al. (2022) to implement a dimensionality reduction model and cluster 4DSTEM data.

   The goal is to learn a mapping from 4DSTEM data to a lower-dimensional embedding where you can perform clustering to identify different deformation modes.

   Please use the article “Uncovering material deformations via machine learning combined with four-dimensional scanning transmission electron microscopy” by Shi et al. (2022) as a starting point for your implementation.

3. Denoising

   We will use the dataset from “Unsupervised deep denoising for four-dimensional scanning transmission electron microscopy” by Sadri et al. (2024) to implement a denoising model for 4DSTEM data.

   The goal is to learn a mapping from noisy to clean 4DSTEM data.

   Please use the article “Unsupervised deep denoising for four-dimensional scanning transmission electron microscopy” by Sadri et al. (2024) as a starting point for your implementation.

   The article contains pytorch code for the model.

   Learn how to adapt it to your needs and try to replicate the results on the SrTiO3_High_mag_Low_dose.npy and SrTiO3_High_mag_High_dose.npy datasets.

4. Image-to-Image Translation

   We will use a simulated X-ray image dataset with pairs of projected thickness and phase contrast images to implement an Image to image translation model.

   The goal is to learn a mapping from phase contrast images to projected thickness images.

   This is usually a task that is solved with multiple measurements and a physical model of the imaging process.

   Here we will try to learn this mapping from simulated data. Please use the article “Multi-resolution convolutional neural networks for inverse problems” by Wang et al. (2020) as a starting point for your implementation.

6 Lecture 5: Mixed Bag (10.1.2025)

• Project presentation
• Generative Adversarial Networks
• Gaussian Processes 1

7 Lecture 6: Gaussian Processes Introduction (17.1.2025)

• Introduction to Gaussian Processes

8 Lecture 7: Gaussian Processes Applications (24.1.2025)

• Bayesian Optimization
• Active Learning
• Deep Kernel Learning

9 Lecture 8: Inverse Imaging Problems 1: Linear Problems (31.1.2025)

• Algorithms for linear inverse problems
• Tomography
• Deconvolution

10 Lecture 9: Inverse Imaging Problems 2: Nonlinear Problems (7.2.2025)

• Phase Contrast Imaging
• Superresolution Imaging
• Inverse Problems in Electron Microscopy

References

Rangel DaCosta, Luis, Katherine Sytwu, CK Groschner, and MC Scott. 2024. “A Robust Synthetic Data Generation Framework for Machine Learning in High-Resolution Transmission Electron Microscopy (HRTEM).” Npj Computational Materials 10 (1): 165.

Sadri, Alireza, Timothy C Petersen, Emmanuel WC Terzoudis-Lumsden, Bryan D Esser, Joanne Etheridge, and Scott D Findlay. 2024. “Unsupervised Deep Denoising for Four-Dimensional Scanning Transmission Electron Microscopy.” Npj Computational Materials 10 (1): 243.

Shi, Chuqiao, Michael C Cao, Sarah M Rehn, Sang-Hoon Bae, Jeehwan Kim, Matthew R Jones, David A Muller, and Yimo Han. 2022. “Uncovering Material Deformations via Machine Learning Combined with Four-Dimensional Scanning Transmission Electron Microscopy.” Npj Computational Materials 8 (1): 114.

Wang, Feng, Alberto Eljarrat, Johannes Müller, Trond R Henninen, Rolf Erni, and Christoph T Koch. 2020. “Multi-Resolution Convolutional Neural Networks for Inverse Problems.” Scientific Reports 10 (1): 5730.