Correlative image rendering

Prerequisites

Before starting this lesson, you should be familiar with:

Learning Objectives

After completing this lesson, learners should be able to:

  • Understand how heterogeneous image data can be mapped from voxel space into a global coordinate system.

  • Understand how an image viewer can render a plane from a global coordinate system.

Motivation

“In correlative microscopy the same specimen is imaged with multiple modalities. While this provides great opportunities for scientific discovery it poses some visualisation challenges. The various images may have different voxel sizes and different dimensionality and may be translated and rotated with respect to another. To tackle this challenge appropriate image viewer software and image data file formats must be chosen. To make those hoices it is important to understand the basic concepts of correlative image rendering, as well as know some concrete implementations.”

Concept map

graph TD DC1("Image data 1") -->|"transform (dgt_1)"| GC("Global (physical) coordinates") DC2("Image data 2") -->|"transform (dgt_2)"| GC GC -->|"transform (gvt)"| VC("Viewer (screen) coordinates")



Figure


Depiction of how the data (array) spaces of two hetero-dimensional images are mapped onto a computer screen. Note that even though the first image is 2-D, it is depicted 3-D in data space by means of adding a singleton dimension. In practice, the computer can loop through all screen pixels (viewer space) and use the given formula to fetch the corresponding values from the data spaces. If there are several values (in this example there are two), then a blending and coloring scheme must be applied to produce the final RGB value that is displayed on the computer screen (this will be discussed in other teaching modules). NOTE: probably it is better to speak of array space rather than data space, because the actual data may live below the array in some other space, like a series of time-tagged measurements in a point-scanning confocal. Then there also is the storage space, which is probably linear, e.g. on a hard-disk, and may be chunked.



Activities


Show activity for:  

MoBIE

Requirements: Fiji with MoBIE update site

  • Open a MoBIE project containing correlative data
    • [ Plugins › MoBIE › Open › Open MoBIE Project... ]
      • Project Location https://github.com/mobie/clem-example-project/
  • Explore the transformation from viewer to global space:
    • Log the current dgt and gvt
      • BDV context menu: Log Source Transforms & Log Current Location
    • Zoom in
      • BDV: arrow up
    • Log dgt and gvt again and appreciate that only the gvt has changed.
      • Appreciate that the first component of the gvt is bigger, corresponding to the higher zoom level.
    • Add another image:
      • MoBIE UI: From the tomogram drop-down choose tomo_37_hm and click [ view ]
    • Appreciate that navigation (i.e. finding the tomo_37_hm image) in large correlative data sets is challenging.
    • Focus on the tomo_37_hm image:
      • MoBIE UI: tomo_37_hm click [ F ]
  • Explore heterogeneous voxel sizes
    • Toggle off interpolation
      • BDV: i (it should say “nearest neighbor interpolation”)
    • Zoom in a bit more
      • BDV: arrow up
    • Move to the edge of the tomo_37_hm
      • BDV: Mouse right button drag
    • Alternatively copy below gvt transform into the location field and click [ move ]
      • {"normalizedAffine":[0.4963397000728218,0.0,0.0,-108.59369157880992,0.0,0.4963397000728218,0.0,-152.83843100300004,0.0,0.0,0.4963397000728218,3.865989923867204E-4],"timepoint":0}
    • Appreciate that the voxels of the em-overview are visible (and are larger than the voxels of tomo_37_hm)
    • This information is encoded in the transformations from data space to global space (dgt):
  • Explore heterodimensional (2D & 3D) rendering
    • Look at the data from the side
      • BDV: shift y
    • Zoom out until you see the edges of em-overview along the z-axis
      • BDV: arrow down
    • Alternatively copy below gvt transform into the location field and click [ move ]
      • {"normalizedAffine":[-0.0020015317086036193,-0.01932460609735281,-5.114194702548178E-19,6.3944917447396445,-5.114194702548184E-19,-7.789826789933008E-19,0.019427983168573904,-0.0015838942648179167,-0.01932460609735281,0.0020015317086036193,-7.789826789933008E-19,3.638083567375694],"timepoint":0}
    • Appreciate that the voxels of em-overview are rendered 300 nm along the z-axis.
    • This is specified here
  • Change the dgt (“registration”) of one image:
    • Focus on tomo_37_hm
      • MoBIE UI: tomo_37_hm click [ F ]
    • Log the dgt
      • BDV context menu: Log Source Transformations
    • Change the dgt of tomo_37_hm:
      • BDV context menu: Registration - Manual Transform
      • Follow the instructions to change the location of tomo_37_hm
    • Log the dgt again (s.a.) and appreciate that it has changed.






Assessment

Fill in the blanks

  1. TODO ___ .
  2. TODO ___ .

Solution

  1. TODO
  2. TODO

Explanations




Follow-up material

Recommended follow-up modules:

Learn more: