deepcell.applications

Application

class deepcell.applications.application.Application(model, model_image_shape=(128, 128, 1), model_mpp=0.65, preprocessing_fn=None, postprocessing_fn=None, format_model_output_fn=None, dataset_metadata=None, model_metadata=None)

Bases: object

Application object that takes a model with weights and manages predictions

Parameters

• model (tensorflow.keras.Model) – tf.keras.Model with loaded weights.

• model_image_shape (tuple) – Shape of input expected by model.

• dataset_metadata (str or dict) – Metadata for the data that model was trained on.

• model_metadata (str or dict) – Training metadata for model.

• model_mpp (float) – Microns per pixel resolution of the training data used for model.

• preprocessing_fn (function) – Pre-processing function to apply to data prior to prediction.

• postprocessing_fn (function) – Post-processing function to apply to data after prediction. Must accept an input of a list of arrays and then return a single array.

• format_model_output_fn (function) – Convert model output from a list of matrices to a dictionary with keys for each semantic head.

Raises

• ValueError – preprocessing_fn must be a callable function

• ValueError – postprocessing_fn must be a callable function

• ValueError – model_output_fn must be a callable function

_batch_predict(tiles, batch_size)

Batch process tiles to generate model predictions.

The built-in keras.predict function has support for batching, but loads the entire image stack into GPU memory, which is prohibitive for large images. This function uses similar code to the underlying model.predict function without soaking up GPU memory.

Parameters

• tiles (numpy.array) – Tiled data which will be fed to model

• batch_size (int) – Number of images to predict on per batch

Returns

Model outputs

Return type

list

_format_model_output(output_images)

Applies formatting function the output from the model if one was provided. Otherwise, returns the unmodified model output.

Parameters

output_images – stack of untiled images to be reformatted

Returns

reformatted images stored as a dict, or input images stored as list if no formatting function is specified.

Return type

dict or list

_postprocess(image, **kwargs)

Applies postprocessing function to image if one has been defined. Otherwise returns unmodified image.

Parameters

image (numpy.array or list) – Input to postprocessing function either an numpy.array or list of numpy.arrays.

Returns

labeled image

Return type

numpy.array

_predict_segmentation(image, batch_size=4, image_mpp=None, pad_mode='constant', preprocess_kwargs={}, postprocess_kwargs={})

Generates a labeled image of the input running prediction with appropriate pre and post processing functions.

Input images are required to have 4 dimensions [batch, x, y, channel]. Additional empty dimensions can be added using np.expand_dims.

Parameters

• image (numpy.array) – Input image with shape [batch, x, y, channel].

• batch_size (int) – Number of images to predict on per batch.

• image_mpp (float) – Microns per pixel for image.

• pad_mode (str) – The padding mode, one of “constant” or “reflect”.

• preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.

• postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.

Raises

• ValueError – Input data must match required rank, calculated as one dimension more (batch dimension) than expected by the model.

• ValueError – Input data must match required number of channels.

Returns

Labeled image

Return type

numpy.array

_preprocess(image, **kwargs)

Preprocess image if preprocessing_fn is defined. Otherwise return image unmodified.

Parameters

• image (numpy.array) – 4D stack of images

• kwargs (dict) – Keyword arguments for preprocessing_fn.

Returns

The pre-processed image.

Return type

numpy.array

_resize_input(image, image_mpp)

Checks if there is a difference between image and model resolution and resizes if they are different. Otherwise returns the unmodified image.

Parameters

• image (numpy.array) – Input image to resize.

• image_mpp (float) – Microns per pixel for the image.

Returns

Input image resized if necessary to match model_mpp

Return type

numpy.array

_resize_output(image, original_shape)

Rescales input if the shape does not match the original shape excluding the batch and channel dimensions.

Parameters

• image (numpy.array) – Image to be rescaled to original shape

• original_shape (tuple) – Shape of the original input image

Returns

Rescaled image

Return type

numpy.array

_run_model(image, batch_size=4, pad_mode='constant', preprocess_kwargs={})

Run the model to generate output probabilities on the data.

Parameters

• image (numpy.array) – Image with shape [batch, x, y, channel]

• batch_size (int) – Number of images to predict on per batch.

• pad_mode (str) – The padding mode, one of “constant” or “reflect”.

• preprocess_kwargs (dict) – Keyword arguments to pass to the preprocessing function.

Returns

Model outputs

Return type

numpy.array

_tile_input(image, pad_mode='constant')

Tile the input image to match shape expected by model using the deepcell_toolbox function.

Only supports 4D images.

Parameters

• image (numpy.array) – Input image to tile

• pad_mode (str) – The padding mode, one of “constant” or “reflect”.

Raises

ValueError – Input images must have only 4 dimensions

Returns

Tuple of tiled image and dict of tiling information.

Return type

(numpy.array, dict)

_untile_output(output_tiles, tiles_info)

Untiles either a single array or a list of arrays according to a dictionary of tiling specs

Parameters

• output_tiles (numpy.array or list) – Array or list of arrays.

• tiles_info (dict) – Tiling specs output by the tiling function.

Returns

Array or list according to input with untiled images

Return type

numpy.array or list

predict(x)

CytoplasmSegmentation

class deepcell.applications.cytoplasm_segmentation.CytoplasmSegmentation(model=None, preprocessing_fn=deepcell_toolbox.processing.histogram_normalization, postprocessing_fn=deepcell_toolbox.deep_watershed.deep_watershed)

Bases: Application

Loads a deepcell.model_zoo.panopticnet.PanopticNet model for cytoplasm segmentation with pretrained weights.

The predict method handles prep and post processing steps to return a labeled image.

Example:

from skimage.io import imread
from deepcell.applications import CytoplasmSegmentation

# Load the image
im = imread('HeLa_cytoplasm.png')

# Expand image dimensions to rank 4
im = np.expand_dims(im, axis=-1)
im = np.expand_dims(im, axis=0)

# Create the application
app = CytoplasmSegmentation()

# create the lab
labeled_image = app.predict(image)

Parameters

model (tf.keras.Model) – The model to load. If None, a pre-trained model will be downloaded.

dataset_metadata = {'name': 'general_cyto', 'other': 'Pooled phase and fluorescent cytoplasm data - computationally curated'}

Metadata for the dataset used to train the model

model_metadata = {'batch_size': 16, 'lr': 0.0001, 'lr_decay': 0.9, 'n_epochs': 8, 'training_seed': 0, 'training_steps_per_epoch': 3949, 'validation_steps_per_epoch': 986}

Metadata for the model and training process

predict(image, batch_size=4, image_mpp=None, pad_mode='reflect', preprocess_kwargs=None, postprocess_kwargs=None)

Generates a labeled image of the input running prediction with appropriate pre and post processing functions.

Input images are required to have 4 dimensions [batch, x, y, channel].

Additional empty dimensions can be added using np.expand_dims.

Parameters

• image (numpy.array) – Input image with shape [batch, x, y, channel].

• batch_size (int) – Number of images to predict on per batch.

• image_mpp (float) – Microns per pixel for image.

• pad_mode (str) – The padding mode, one of “constant” or “reflect”.

• preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.

• postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.

Raises

• ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.

• ValueError – Input data must match required number of channels.

Returns

Labeled image

Return type

numpy.array

NuclearSegmentation

class deepcell.applications.nuclear_segmentation.NuclearSegmentation(model=None, preprocessing_fn=deepcell_toolbox.processing.histogram_normalization, postprocessing_fn=deepcell_toolbox.deep_watershed.deep_watershed)

Bases: Application

Loads a deepcell.model_zoo.panopticnet.PanopticNet model for nuclear segmentation with pretrained weights.

The predict method handles prep and post processing steps to return a labeled image.

Example:

from skimage.io import imread
from deepcell.applications import NuclearSegmentation

# Load the image
im = imread('HeLa_nuclear.png')

# Expand image dimensions to rank 4
im = np.expand_dims(im, axis=-1)
im = np.expand_dims(im, axis=0)

# Create the application
app = NuclearSegmentation()

# create the lab
labeled_image = app.predict(image)

Parameters

model (tf.keras.Model) – The model to load. If None, a pre-trained model will be downloaded.

dataset_metadata = {'name': 'general_nuclear_train_large', 'other': 'Pooled nuclear data from HEK293, HeLa-S3, NIH-3T3, and RAW264.7 cells.'}

Metadata for the dataset used to train the model

model_metadata = {'backbone': 'efficientnetv2bl', 'batch_size': 16, 'crop_size': 256, 'epochs': 16, 'location': True, 'lr': 0.0001, 'min_objects': 1, 'pyramid_levels': 'P1-P2-P3-P4-P5-P6-P7', 'zoom_min': 0.75}

Metadata for the model and training process

predict(image, batch_size=4, image_mpp=None, pad_mode='reflect', preprocess_kwargs=None, postprocess_kwargs=None)

Generates a labeled image of the input running prediction with appropriate pre and post processing functions.

Input images are required to have 4 dimensions [batch, x, y, channel].

Additional empty dimensions can be added using np.expand_dims.

Parameters

• image (numpy.array) – Input image with shape [batch, x, y, channel].

• batch_size (int) – Number of images to predict on per batch.

• image_mpp (float) – Microns per pixel for image.

• pad_mode (str) – The padding mode, one of “constant” or “reflect”.

• preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.

• postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.

Raises

• ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.

• ValueError – Input data must match required number of channels.

Returns

Labeled image

Return type

numpy.array

Mesmer

class deepcell.applications.mesmer.Mesmer(model=None)

Bases: Application

Loads a deepcell.model_zoo.panopticnet.PanopticNet model for tissue segmentation with pretrained weights.

The predict method handles prep and post processing steps to return a labeled image.

Example:

from skimage.io import imread
from deepcell.applications import Mesmer

# Load the images
im1 = imread('TNBC_DNA.tiff')
im2 = imread('TNBC_Membrane.tiff')

# Combined together and expand to 4D
im = np.stack((im1, im2), axis=-1)
im = np.expand_dims(im,0)

# Create the application
app = Mesmer()

# create the lab
labeled_image = app.predict(image)

Parameters

model (tf.keras.Model) – The model to load. If None, a pre-trained model will be downloaded.

dataset_metadata = {'name': '20200315_IF_Training_6.npz', 'other': 'Pooled whole-cell data across tissue types'}

Metadata for the dataset used to train the model

model_metadata = {'batch_size': 1, 'lr': 1e-05, 'lr_decay': 0.99, 'n_epochs': 30, 'training_seed': 0, 'training_steps_per_epoch': 1739, 'validation_steps_per_epoch': 193}

Metadata for the model and training process

predict(image, batch_size=4, image_mpp=None, preprocess_kwargs={}, compartment='whole-cell', pad_mode='constant', postprocess_kwargs_whole_cell={}, postprocess_kwargs_nuclear={})

Generates a labeled image of the input running prediction with appropriate pre and post processing functions.

Input images are required to have 4 dimensions [batch, x, y, channel]. Additional empty dimensions can be added using np.expand_dims.

Parameters

• image (numpy.array) – Input image with shape [batch, x, y, channel].

• batch_size (int) – Number of images to predict on per batch.

• image_mpp (float) – Microns per pixel for image.

• compartment (str) – Specify type of segmentation to predict. Must be one of "whole-cell", "nuclear", "both".

• preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.

• postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.

Raises

• ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.

• ValueError – Input data must match required number of channels.

Returns

Instance segmentation mask.

Return type

numpy.array

CellTracking

class deepcell.applications.cell_tracking.CellTracking(model=None, neighborhood_encoder=None, distance_threshold=64, appearance_dim=32, birth=0.99, death=0.99, division=0.01, track_length=8, embedding_axis=0, crop_mode='resize', norm=True)

Bases: Application

Loads a deepcell.model_zoo.tracking.GNNTrackingModel model for object tracking with pretrained weights using a simple predict interface.

Parameters

• model (tf.keras.model) – Tracking inference model, defaults to latest published model

• neighborhood_encoder (tf.keras.model) – Tracking neighborhood encoder, defaults to latest published model

• distance_threshold (int) – Maximum distance between two cells to be considered adjacent

• appearance_dim (int) – Length of appearance dimension

• birth (float) – Cost of new cell in linear assignment matrix.

• death (float) – Cost of cell death in linear assignment matrix.

• division (float) – Cost of cell division in linear assignment matrix.

• track_length (int) – Number of frames per track

• crop_mode (str) – Type of cropping around each cell

• norm (str) – Type of normalization layer

dataset_metadata = {'name': 'tracked_nuclear_train_large', 'other': 'Pooled tracked nuclear data from HEK293, HeLa-S3, NIH-3T3, and RAW264.7 cells.'}

Metadata for the dataset used to train the model

model_metadata = {'appearance_dim': 32, 'batch_size': 8, 'buffer_size': 128, 'crop_mode': 'resize', 'data_fraction': 1, 'distance_threshold': 64, 'embedding_dim': 64, 'encoder_dim': 64, 'epochs': 50, 'graph_layer': 'gat', 'lr': 0.001, 'n_filters': 64, 'n_layers': 1, 'norm_layer': 'batch', 'rotation_range': 180, 'steps_per_epoch': 1000, 'translation_range': 512, 'validation_steps': 200}

Metadata for the model and training process

predict(image, labels, **kwargs)

Using both raw image data and segmentation masks, track objects across all frames.

Parameters

• image (numpy.array) – Raw image data.

• labels (numpy.array) – Labels for image, integer masks.

Returns

Tracked labels and lineage information.

Return type

dict

track(image, labels, **kwargs)

Wrapper around predict() for convenience.

LabelDetectionModel

deepcell.applications.label_detection.LabelDetectionModel(input_shape=(None, None, 1), inputs=None, backbone='mobilenetv2', num_classes=3)

Classify a microscopy image as Nuclear, Cytoplasm, or Phase.

This can be helpful in determining the type of data (nuclear, cytoplasm, etc.) so that this data can be forwared to the correct segmenation model.

Based on a standard backbone with an intiial ImageNormalization2D and final AveragePooling2D, TensorProduct, and Softmax layers.

Parameters

• input_shape (tuple) – a 3-length tuple of the input data shape.

• inputs (tensorflow.keras.Layer) – Optional input layer of the model. If not provided, creates a Layer based on input_shape.

• backbone (str) – name of the backbone to use for the model.

• num_classes (int) – The number of labels to detect.

ScaleDetectionModel

deepcell.applications.scale_detection.ScaleDetectionModel(input_shape=(None, None, 1), inputs=None, backbone='mobilenetv2')

Create a ScaleDetectionModel for detecting scales of input data.

This enables data to be scaled appropriately for other segmentation models which may not be resolution tolerant.

Based on a standard backbone with an intiial ImageNormalization2D and final AveragePooling2D and TensorProduct layers.

Parameters

• input_shape (tuple) – a 3-length tuple of the input data shape.

• inputs (tensorflow.keras.Layer) – Optional input layer of the model. If not provided, creates a Layer based on input_shape.

• backbone (str) – name of the backbone to use for the model.