ShuttleTracker

A software for semi-automated cell tracking

ShuttleTracker can detect stained cell nuclei, generate corresponding annular perinuclear extensions, measure geometric and photometric properties of detected objects, and perform frame-to-frame nuclei tracking.  Parameters of the underlying image processing algorithms can be changed manually and the effects of these changes can be seen nearly immediately overlaid on currently displayed microscopic images.  A key capability of ShuttleTracker is that nuclear contours and tracks can be generated automatically and then corrected manually.

Complete manual

ShuttleTracker – User's manual

Overview

Overview: Capabilities

ShuttleTracker (permalink to homepage: http://pmbm.ippt.pan.pl/software/shuttletracker) can detect stained cell nuclei, generate corresponding annular perinuclear extensions, measure properties of detected objects, and perform frame-to-frame nuclei tracking. Parameters of the underlying image processing algorithms can be changed manually and the effects of these changes can be seen nearly immediately overlaid on currently displayed microscopic images. A key capability of ShuttleTracker is that nuclear contours and tracks can be generated automatically and then corrected manually.

Detailed quantitative analysis of properties of tracked nuclei and tracks is out of the scope of the program. Quantified geometric and photometric properties of nuclei and tracks are exported to plain-text files that can be easily analyzed using external tools. A package of Python scripts and example IPython Notebooks are distributed together with this software to demonstrate exemplary approaches to the analysis of features quantified in and exported from ShuttleTracker.

Launching

Launching with GUI

Initially, ShuttleTracker displays an "empty" window. A directory with images to be analyzed can be selected after clicking menu File → Open directory... (or by pressing Ctrl + Shift + O). Image files in the selected directory are expected to be properly named, see <a href="#h2_input_images_names">naming guidelines</a>.

Launching from terminal

When ShuttleTracker is executed from the command line without additional arguments, it displays an "empty" GUI window. Optionally, as the argument one may give a path to a directory with images; ShuttleTracker will then open GUI and load images. If one would like to execute a ShuttleTracker script immediately after loading images, path to the ShuttleTracker script file (*.stscript) should be given just after the path to the directory. After successful script execution, ShuttleTracker will quit.

This software has been created with the intent to enable user to interactively control behavior of image processing algorithms, but it is also possible to run ShuttleTracker without displaying GUI (in a "headless" mode) and in this way perform batch processing (e.g., on a cluster). To this end, one should use ShuttleTracker scripts and invoke the program through a virtual frame buffer such as Xvfb:

  xvfb-run ShuttleTracker PATH_TO_DIRECTORY PATH_TO_SCRIPT

Input images

Input images: File names

The following regular expressions are used to fish for images in the selected directory:

• For a single time-frame:  ^(.+)(?:_ch([0-9]+))\\.(jpe?g|tiff?|png)$
• For multiple time-frames (i.e., a time-lapse serries):  ^(.+)(?:_t([0-9]+))(?:_ch([0-9]+))\\.(jpe?g|tiff?|png)$

In other words, as an example, the following list of files will be understood

correctly:

  MyExperiment_t000_ch0.tif
  MyExperiment_t000_ch1.tif
  MyExperiment_t000_ch2.tif
  MyExperiment_t001_ch0.tif
  MyExperiment_t001_ch1.tif
  MyExperiment_t001_ch2.tif

(and so on). Please note that the index of the time point should be padded with zeros (in the example it is assumed that more than one hundred but not more than one thousand of time points is available). Both the index of a time point and the channel number are 0-based.

If names of your files do not conform the neither of these patterns, please rename your files. Often, proper naming of files can be assured during conversion: for example, assume that you have a two-channel movie saved as two multi-layer TIFF files. In the command line under Linux or Mac, having bash shell and imagemagick installed, you can use the following method to generate a series of

properly named input images:

  export NFRAMES=100
  for i in $(seq -w 0 $NFRAMES); do
    convert MyMovieChannel1.tif[${i}] -depth 8 MyMMovie_t${i}_ch0.tif
    convert MyMovieChannel2.tif[${i}] -depth 8 MyMMovie_t${i}_ch1.tif
  done

To split multi-frame TIFFs and simultaneously perform controlled bit depth conversions you may use a python script tiff_expander.py that comes with ShuttleTracker. Its advantages are described in a subsection on bit depth <a href="#h2_input_images_bit_depth">below</a>.

Input images: Image formats

ShuttleTracker can read only gray-scale images in the following graphics formats: TIF (*.tif, *.tiff), PNG (*.png), and JPEG (*.jpg, *.jpeg; use of JPEGs in microscopic image analysis is strongly discouraged because they are compressed using a lossy algorithm that distorts originally recorded pixel intensities).

Input images: Bit depth

Bit depth of images (i.e., maximum-intensity gray-scale value) depends, i.a., on the scanner or camera used to record the images. ShuttleTracker reads bit depth from the metadata associated with the given image set (see below). If an image has more than 8-bit gray-scale resolution, its depth is scaled linearly to 8-bit resolution. Please note that all image processing and analyses (especially, quantifications) are performed using 8-bit images.

You can perform bit depth conversions with the above-mentioned tiff_expander.py. This python script allows for converting images of any over-8-bit depth to 8-bit depth with or without "clipping" of the over-the-range values (clipping can be advantageous, for example, when a 14-bit image, that has very few pixels of intensity higher than 2^12, is converted to an 8-bit image by treating it as a 12-bit image, because then more "resolution" in lower-value pixels will be preserved than in the case of 14-to-8 conversion). Invoke the script with argument -h or look at the docstring in file header to learn how to use it. The script is usually placed in INSTALL_PREFIX/shared/shuttletracker/support and requires python3-pil and python3-numpy.

Input images: Metadata

Metadata are used to:

• establish the mapping between channel number and pseudocolor/LUT,
• declare the bit depth (i.e., greyscale resolution) of the scanner or camera used to record the images,
• get the number of time frames and their relative times (in seconds).

Metadata can be read from the MetaData directory which is exported by Leica LAS software. Information expected to be found in metadata can be also provided manually – by creating a file hints_for_shuttletracker.txt in the directory that contains images. In an example hints file of the following contents:

  channel 0 Protein1-GFP     green 8
  channel 1 Protein2-mCherry red   8
  channel 2 H2B              blue  8
  time_interval 120.3

first three lines define the mapping from channel number (and mini-description) to channel LUT (color), and specify bit depth of images. Color assignments are used only to display images in specific colors in GUI and may be relevant for generating legible channel overlays (still, within ShuttleTracker all images can be viewed in greyscale). In the last line, time interval between frames is provided (in seconds). This is a free-format text file: the exact number of white-spaces does not matter; lines can appear in any order.

Viewing

Navigation

After images are loaded, each channel is displayed in a separate pane. Channels are referred to using their assigned pseudocolors. One may zoom into/out of a region of an image using mouse scroll and move around when the left mouse button is pressed. Fields of view in all channel panes are synchronized to assure that the corresponding image region is displayed in all panes.

Using menu View one can hide each pane or black out contained image. Blacking-out is useful when one would like to show only image annotations (called markings). When multiple markings occlude microscopic image in the background, one may also, conversely, hide all markings to display only the microscopic image (it's convenient to use the spacebar key instead of selecting hide/unhide in the menu).

Enhanced view

Microscopic images can be displayed (i) as originally provided, (ii) with pixel intensities normalized, or weakly (iii) or (iv) strongly auto-leveled. Weak (strong) auto-leveling discards 2% (8%) of brightest pixels and performs normalization of the remaining pixels. Enhancements (i)–(iv) are available in the saturated (pseudocolor) as well as desaturated (grayscale) mode.

Switching to the normalized view enables assessment whether input images have a reasonable dynamic range. Auto-leveled mode helps to visually discern darker portions of the image (after some time you may find yourself falling into the habit pressing of F7 or F8 just after loading input images). Desaturated images appear brighter on the screen than their saturated counterparts and thus may be more convenient to work on. The enhancements affect only how the images are displayed, and in particular do not impact image segmentation nor quantification.

Toolboxen

All available toolboxen are available from the Toolbox Menu and, additionally, from a dockable tool-bar that can be displayed when selected in a pop-up menu that appears after right-clicking on the main menu bar (under Linux and Windows). You may find it handy to use keyboard shortcuts Ctrl+1, Ctrl+2, and so on, to switch between the toolboxen, especially in the full-screen mode (press F11 or use Window menu to enter/exit).

Visibility or "selectability" of objects overlaid over microscopic image depends crucially on the currently active toolbox.

Toolbox: Image masking

Selected areas of an analyzed image can be masked to get rid of the regions in the image that should not be analyzed nor quantified (because, for example, they contain contaminations that would skew analysis). Masked areas are excluded from segmentation (within Nuclei Detection toolbox) and whole-image quantification (within Quantification toolbox).

If one wants to mask some portion of the image, one should activate the Image Masking toolbox, click with right mouse button over the microscopic image to enter the marking mode and while pressing left mouse button mark the area to be masked. Several (non-overlapping) areas can be marked in this way. To clear all markings, one can use button Reset in the toolbox. If all desired markings are ready, one can click toolbox button Mask to create the mask. Masked areas are blackened out. Masking contours for a current image can be stored and restored using suitable buttons at the bottom of the toolbox pane, next to Mask ↔ file. To create masks for a series of images, each image has to be treated separately.

Toolbox: Nuclei detection

Image preprocessing

Before nuclear contours are detected, the input image has to be preprocessed. Image preprocessing pipeline consists of the series of filters:

1. Normalization linearly stretches the range of intensities of all 8-bit image pixels so that they fill the full available intensity range (0...255).
2. Denoising removes salt-and-pepper noise using non-local means denoising. Associated parameter, strength, influences jointly several filter properties: size of the scanning window, range of a local scan, and the strength of filter application. This filter is CPU-intensive for larger values of strength that imply more non-local scan.
3. Smoothing performs bilateral noise filtering. Neighborhood of each pixel is defined based on the filter only parameter, radius, and the weights of the photometric and geometric distances are set as linearly proportional to radius. This filter may be used as a faster replacement, or as a complement, of the previous denoising step.
4. Thresholding binarizes input image using local adaptive thresholding: for each pixel, a mean intensity of pixels in its neighborhood (of radius controlled by parameter block size) is set as the binarization threshold. The threshold can be futher adjusted using parameter base-line. Local adaptive thresholding handles well images with uneven illumination. If a single, global intensity threshold is desired, one should just increase block size to be twice the larger dimension of the image. Optimal values of both the parameters can be found automatically via a naïve grid search (radio button Manual → Auto), with resultant nuclear contours solidity being the optimization objective. For some parameter values the algorithm for nuclei detection and splitting may have long completion times, hence it's possible to impose a time-out for testing a single set of parameters.
5. Opening regularizes silhouettes of nuclei after thresholding and removes small debris by performing morphological erosion followed by dilation. A number of consecutive opening rounds can be performed; the sizes of structuring elements for erosion and dilation are controlled jointly to prevent having nuclear contours that would be overtly round or too tight/loose.

All these stages (filters) in the pipeline can be individually turned on and off. If the preview stages checkbox has been checked, previews of the effects of the filters will be displayed in separate windows. The filters and their parameters are also described briefly in tooltips that pop up when hovering the mouse cursor over filter names or their parameter labels.

Image preprocessing does not impact image quantification nor the way in which the image is displayed.

Nuclei detection and assessment

The binarized image resulting from the preprocessing pipeline is used to find contours. Median nuclear contour area serves as the reference contour area and two user-defined parameters, min area and max area, expressed as fractions of the median area, are used to classify all contours as: (i) too small to be a nucleus, (ii) too large to be a single nucleus, or (iii) having area close enough to the reference nucleus and thus being single-nucleus contours. Too small contours are called debris. Contours that are too large can be split depending on their solidity – defined as the surface area of the contour divided by the surface area of its convex hull. If a too large contour has solidity higher that user-defined min solidity, it is deemed non-splittable. However, when solidity of a too large contour is lower, then it will be subjected to splitting based on convexity defects called indents. Indents that can be used for splitting should be sufficiently large, which is decided based on the user-defined min indent parameter. A split can be attempted only when a contour has two or more indents. Splitting works recursively as long as there are indents that can be used to obtain contours of nuclei of areas close to the reference area. Contours that result from splitting but according to the min area parameter are too small are called split orphans. Descriptions of parameters involved in nuclei assessment are shown in tooltips that pop up when hovering the mouse pointer over parameter labels.

Nuclear contours are drawn over microscopic images in colors associated with their origin in the above-described segmentation algorithm: (1) typically sized contours that do not result from splitting ("solo" contours)

   are white,

(2) typically sized contours resulting from splitting – green, (3) large non-splittable – red, (4) split orphans (relatively rare) – yellow, (5) debris – violet. The obtained set of contours is scored according to the variance of their solidities: the lower is variability, the higher is the score. Scoring is used currently for automated optimization of thresholding parameters at the image preprocessing stage.

If the checkbox auto-click has been checked, then one can adjust image preprocessing and nuclei assessment parameters and nearly instantaneously observe their impact on nuclei detection (without clicking Detect button after each parameter adjustment).

Both the image preprocessing parameters and nuclei assessment parameters may be saved to/loaded from a text file using buttons labeled Protocol ↔ file.

Toolbox: Nuclei editing

It is possible to correct automatically detected nuclei one-by-one by deleting existing nuclear contours and adding new ones manually:

• To remove a nucleus: select the unwanted nucleus by a left-click and then right-click on it.
• To add a nucleus: after right-clicking in a free space, press left mouse button and use mouse pointer to draw the contour of a new nucleus;

All nuclei are numbered sequentially starting from 1. Use checkbox numbers in box Annotations to show/hide the numbers. After a nucleus of a given number is removed, all higher nuclei numbers are decreased by one. After a new nuclear contour has been drawn, it is assigned the smallest available number (equal to the current number of all nuclear contours). A contour can be redrawn, that is, replaced without changing its number, only in the tracks editing toolbox.

Double-left-click on a contour can be used to manually toggle a binary state of a nucleus called completeness. This state can be used at a later stage of analysis. One can toggle completeness status of nuclei that lie on the image border using a suitable button in the box Editing by location. Using other buttons in this box one can remove such border touching contours as well as contours classified as debris or split orphans.

Coordinates of nuclear contours can be save to/loaded from a text file using button labeled Nuclei ↔ file. The order of contours in the file corresponds to the numbering displayed on the screen.

When all nuclei in a frame are removed (which takes place also upon nuclei contours file loading), all tracks are removed as well.

Toolbox: Perinuclei derivation

(to be written)

Toolbox: Regions editing

(to be written)

Toolbox: Quantification

Areas enclosed in nuclear and region contours can be quantified w.r. to: surface area, intensity sum, median, mean, min, and max, in each channel separately. The same values can be calculated for the whole image. Quantification results for the current frame can be exported to a text file and analyzed with external tools.

Toolbox: Tracking

Tracking is local in time and space, i.e., corresponding nuclei are matched between adjacent time frames using a "greedy" matching algorithm.

Tracks are created only for the nuclei which have a contour. The "completeness" status of nuclei (displayed by using either solid or dashed contour line) is not taken into account.

Nuclei matching is performed in three phases:

1. Nuclei similarity scoring. Similarity in all nuclei pairs, in which one nucleus comes from to the previous time frame and the other nucleus belongs to the current time frame, is estimated. Similarity scores form a matrix, where the number of rows and the number of columns are equal to the number of nuclei in the previous and the current time frame. (At this phase, the distance between nuclei in each pair is not directly considered.)
2. Internuclear distance-dependent similarity scaling. Each estimated similarity score is scaled depending on the internuclear distance.
3. Tracks elongation based on max similarity. A distances-scaled similarity score of maximum value is sought in the similarity matrix. When found, two corresponding nuclei are considered as matched. The procedure of finding maximum similarity is repeated for the still unmatched nuclei. When two nuclei are matched but their distance is larger than an allowed inter-nuclear spacing, the nuclei matching stops.

Parameters:

1. Nuclei similarity score (phase 1) is computed as the linear combination of six features, which are all order statistics. Weights used for calculating a linear combination of these features are following:

   • proximity – order statistics is calculated based on the distances of nuclear positions,
   • surface area,
   • eccentricity – order statistics is calculated based on how far is the nucleus shape from the circle,
   • orientation – order statistics is calculated using the angle of the first principal axis
   • sum of pixel intensities,
   • distribution of pixel intensities – order statistics is based on the Kolmogorov-Smirnov distance between two distributions of nuclear intensities.

2. Distance-dependent similarity scaling (phase 2) is parametrized by Δ_xy scaling. The value of this parameter is used as the exponent for scaling inter-nuclear distances (in feature space).

3. Tracks are elongated (phase 3) if the currently matched nuclei are not more than than Δ_xy cutoff median inter-nuclear distances apart.

Currently, tracking must be gapless, i.e., the outline of the nucleus must be marked in every track frame (although in some frames it's status can be set to "incomplete" so that, by convention, it is possible to omit them in quantifications).

Toolbox: Tracks editing

When the correspondence between nucleus' appearances in different time frames has been found, its centers of the mass are connected with straight lines (called segments). Unmatched nuclei are marked with small squares (called stubs).

Tracks are listed in a tabular form, sorted by their lengths. Tracks of the same length are ordered by a descending confidence score. It is possible to remove a track by right-clicking on the corresponding row of the tabular tracks view.

It is possible to correct erroneous tracks manually: segments can be added and removed with mouse:

• To add a segment, select one segment (or a stub) using mouse left-click, press Ctrl key, and select another segment (or a stub) with the left-click. As tracks are assumed to be gapless, only the tracks which end/begin at consecutive time frames can be joined.
• To remove a segment, select it with the mouse left-click and then right-click on the selection.

To redraw a nucleus within a track, first: select it with the left-click, second: while pressing right mouse button make the left-click, third: draw the new contour (by pressing the left mouse button) or left-click in any empty space to abandon the replacing mode.

To add a new nucleus (while tracks are already present), switch to the Nuclei Editing toolbox, draw a new contour, then switch back to the Tracks Editing toolbox and click with mouse middle button on the new contour to create a new stub. Stubs can be merged into segments, as described previously.

Tracks are saved to files in a simple textual format. First column is the frame number (0-based), and the second column is the number of nucleus (consistent with the nuclei numbering in the frame nuclei contours file and in the nuclei quantification file; nuclei numbering begins with 1).

Note: When all nuclei in a frame are removed (which takes place also upon nuclei contours file loading), tracks are removed as well to prevent the program from being internally inconsistent.

Scripting

ShuttleTracker ships an embedded JavaScript interpreter to enable automation of repeatable, daunting mouse clicks. The interpreter has been fitted with syntactic extensions which facilitate, i.a., looping over all the frames. To see the available functions please open the editor window (Menu Script) and click on the Show API button. A simple debugging capabilities are provided: error message and erratic script line number are to be displayed upon script failure.

Scripts are an integral part of ShuttleTracker functionalities, for example are used to play the sequence of images as a movie. As such, the scripts can be launched directly from the Scripts menu (also via keyboard shortcuts; shortcuts are assigned after ordering script file names in the lexicographic order). Newly added scripts become immediately available in the menu.

On Linux, scripts are installed in INSTALL_PREFIX/share/shuttletracker/scripts. On Windows, scripts are usually stored as plain text files in user's Document/STScripts directory. On macOS, scripts are stored in ShuttleTracker.app/Contents/Resources/shuttletracker/scripts. (On all systems, the exact location can be checked at the bottom of the Help → About window.)

Preferences

The following preferences can be set or unset in the Preferences menu:

• Display buffers: Cache images in memory for fast display. This preference affects the way images are prepared just after they are loaded. Cached images occupy consume memory but are displayed faster.
• Over-8-bit images: normalize to camera bit depth (if available). This preference affects the way images are loaded, and has effect virtually only on 16-bit images.
• Bright-field images: Skip when loading. This preference affects the way images are loaded. If BF images are not to be used, skipping them will save memory.
• Bright-field images: Hide even when available. Auto-hide BF panel even when BF images were loaded.
• Bright-field images: Exclude from overlays. BF images overlaid on fluorescence images attenuate (pseudo)colors, thus it may be preferred to not use BF images for overlays. It may be actively overridden by manually selecting overlay components in menu View.

When exiting, ShuttleTracker stores these preferences and some other settings to restore them when launched again. Stored settings include: widths of contours, ranges of parameters for performing nuclei detection parameters scan, nuclei detection scoring criterion, window geometry, the location of the last opened directory, exported quantifications, etc. On Linux, preferences and settings are saved to an INI-file (which is placed in INSTALL_PREFIX/shared/shuttletracker/config). On macOS, settings.ini is located in ShuttleTracker.app/Contents/Resources/shuttletracker/config. Under Windows, preferences are saved to the system registry (don't worry: upon deinstallation, the registry is purged of all entries created by ShuttleTracker).

The number of threads used by several image processing algorithms is controlled by the environmental variable SHUTTLETRACKER_OPENCV_NUM_THREADS. Since the images are usually not that large, it is advised to set this number low. In Linux terminal, the variable can be set in the bash shell with:

  export SHUTTLETRACKER_OPENCV_NUM_THREADS=2

Further analysis

Quantification results and tracks generated with ShuttleTracker as saved to plain-text files that should be readily parseable by external data analysis scripts and tools. Python module "shuttletracker" located in INSTALL_PREFIX/shared/analysis shows how to "zip" tracks and respective quantifications.