Deep learning instance segmentation

Prerequisites

Before starting this lesson, you should be familiar with:

Learning Objectives

After completing this lesson, learners should be able to:

  • Run a deep learning model for instance segmentation

  • Visually assess the quality of the segmentation

  • Appreciate that, in order to work well, the model has to match the input data

  • Appreciate that even deep learning models may have parameters that need to be tuned

  • Appreciate that subtle differences in the image data may cause the model to fail

Motivation

Deep learning based nuclei and cell segmentation can be much faster and more accurate than conventional segmentation methods. In addition, there may be less parameter tuning required than for conventional methods. However, one can still make mistakes, such as applying the wrong deep learning model for the given input data. Both the usefulness and potential pitfalls make it very important to learn how to properly use and judge deep learning based image segmentation.

Concept map

graph TD I("Intensity image") --> DL("Deep learning segmentation tool") DL --> L("Label mask image") M("Trained model") --- DL P("Parameters") --- DL



Figure


From left to right - _input intensity image_ (could be multichannel, multidimensional), _UNET architecture_ that is mostly used a base for modern deep learning methods (such as cellpose/stardist/mesmer) for bioimage segmentation, _network output_ varies based on the model used i.e., xy-gradients/binary mask in case of cellpose and distances to object boundaries/object probabilities in case of stardist, _label mask_ generated using different methods depending upon preceding steps.



Activities


Show activity for:  

CellPose GUI

  • Download this two-channel image.
  • Open the image in Fiji and measure the typical diameter of one cell.
  • Open the image in Cellpose
    • cell diameter: 60
    • chan to segment: red (membranes)
    • chan2 (optional): green (nuclei)
    • Run the cyto model
      • Appreciate that the cell segmentation works well
    • Decrease the cell diameter to 1/5 of its correct value
    • Run the cyto model again
      • Appreciate that the segmentation does not work well
    • Put the cell diameter back to 60
    • Run the nuclei model
      • Appreciate that also this does not work well
    • Appreciate that only the correct parameters and model yield a good result

Debug a tricky case (optional)

  • Download a similar two-channel image.
  • Try to segment it in CellPose
  • Appreciate that some cells are not segmented
  • Try to find out why that could be the case

Explore additional parameters (optional)

Explore the flow_threshold, cell_threshold and stitch_threshold parameters

Exercises

Show exercise/solution for:

CellPose GUI

  • Download this two-channel image.
  • Open the image in Cellpose
    • cell diameter: 60
    • chan to segment: 1 red (membranes)
    • chan2 (optional): 0 none
    • Run the cyto model

Do you have a different opinion on the segmentation, if yes, can you locate regions where you think cellpose may have segmented differently?

Solution

Hint: Use MASKS ON [x] and outlines on [Z] to view the problematic areas in much intuitive way. Use arrow keys up and down to switch between channels and different display modes. Increase contrast if necessary. The problematic segmented areas are shown here: problematic areas

Play with Cell diameter, chan to segment and chan2(optional) and find parameter values to improve the segmentation from the preceding step

Solution

One way to do this is to use nuclear channel information by setting chan2 (optional): 2 green and run cyto model again. Another way to make results better is to use a lower Cell diameter = 40




Assessment

Fill in the blanks

  1. The typical output of a (deep learning based) instance segmentation is a ___ .
  2. If the segmentation is faulty you could consider to __ or __ .
  3. If you wonder which deep learning tool to use to segment your data you could _ or __.
  4. To run a deep learning model efficiently you should have a computer with ___ .

Solution

  1. label mask image
  2. use a different model or change the tool’s parameters
  3. ask on the forum or check the BioImage Model Zoo
  4. a GPU

Explanations




Follow-up material

Recommended follow-up modules:

  • Train a model (TODO)

Learn more: