deepcell.applications
Application
- class deepcell.applications.application.Application(model, model_image_shape=(128, 128, 1), model_mpp=0.65, preprocessing_fn=None, postprocessing_fn=None, format_model_output_fn=None, dataset_metadata=None, model_metadata=None)[source]
Bases:
object
Application object that takes a model with weights and manages predictions
- Parameters:
model (tensorflow.keras.Model) –
tf.keras.Model
with loaded weights.model_image_shape (tuple) – Shape of input expected by
model
.dataset_metadata (str or dict) – Metadata for the data that
model
was trained on.model_mpp (float) – Microns per pixel resolution of the training data used for
model
.preprocessing_fn (function) – Pre-processing function to apply to data prior to prediction.
postprocessing_fn (function) – Post-processing function to apply to data after prediction. Must accept an input of a list of arrays and then return a single array.
format_model_output_fn (function) – Convert model output from a list of matrices to a dictionary with keys for each semantic head.
- Raises:
ValueError –
preprocessing_fn
must be a callable functionValueError –
postprocessing_fn
must be a callable functionValueError –
model_output_fn
must be a callable function
- _batch_predict(tiles, batch_size)[source]
Batch process tiles to generate model predictions.
The built-in keras.predict function has support for batching, but loads the entire image stack into GPU memory, which is prohibitive for large images. This function uses similar code to the underlying model.predict function without soaking up GPU memory.
- _format_model_output(output_images)[source]
Applies formatting function the output from the model if one was provided. Otherwise, returns the unmodified model output.
- _postprocess(image, **kwargs)[source]
Applies postprocessing function to image if one has been defined. Otherwise returns unmodified image.
- Parameters:
image (numpy.array or list) – Input to postprocessing function either an
numpy.array
or list ofnumpy.arrays
.- Returns:
labeled image
- Return type:
numpy.array
- _predict_segmentation(image, batch_size=4, image_mpp=None, pad_mode='constant', preprocess_kwargs={}, postprocess_kwargs={})[source]
Generates a labeled image of the input running prediction with appropriate pre and post processing functions.
Input images are required to have 4 dimensions
[batch, x, y, channel]
. Additional empty dimensions can be added usingnp.expand_dims
.- Parameters:
image (numpy.array) – Input image with shape
[batch, x, y, channel]
.batch_size (int) – Number of images to predict on per batch.
image_mpp (float) – Microns per pixel for
image
.pad_mode (str) – The padding mode, one of “constant” or “reflect”.
preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.
postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.
- Raises:
ValueError – Input data must match required rank, calculated as one dimension more (batch dimension) than expected by the model.
ValueError – Input data must match required number of channels.
- Returns:
Labeled image
- Return type:
numpy.array
- _preprocess(image, **kwargs)[source]
Preprocess
image
ifpreprocessing_fn
is defined. Otherwise returnimage
unmodified.- Parameters:
image (numpy.array) – 4D stack of images
kwargs (dict) – Keyword arguments for
preprocessing_fn
.
- Returns:
The pre-processed
image
.- Return type:
numpy.array
- _resize_input(image, image_mpp)[source]
Checks if there is a difference between image and model resolution and resizes if they are different. Otherwise returns the unmodified image.
- Parameters:
image (numpy.array) – Input image to resize.
image_mpp (float) – Microns per pixel for the
image
.
- Returns:
Input image resized if necessary to match
model_mpp
- Return type:
numpy.array
- _resize_output(image, original_shape)[source]
Rescales input if the shape does not match the original shape excluding the batch and channel dimensions.
- Parameters:
image (numpy.array) – Image to be rescaled to original shape
original_shape (tuple) – Shape of the original input image
- Returns:
Rescaled image
- Return type:
numpy.array
- _run_model(image, batch_size=4, pad_mode='constant', preprocess_kwargs={})[source]
Run the model to generate output probabilities on the data.
- Parameters:
- Returns:
Model outputs
- Return type:
numpy.array
- _tile_input(image, pad_mode='constant')[source]
Tile the input image to match shape expected by model using the
deepcell_toolbox
function.Only supports 4D images.
- Parameters:
image (numpy.array) – Input image to tile
pad_mode (str) – The padding mode, one of “constant” or “reflect”.
- Raises:
ValueError – Input images must have only 4 dimensions
- Returns:
Tuple of tiled image and dict of tiling information.
- Return type:
(numpy.array, dict)
CytoplasmSegmentation
- class deepcell.applications.cytoplasm_segmentation.CytoplasmSegmentation(model=None, preprocessing_fn=deepcell_toolbox.processing.histogram_normalization, postprocessing_fn=deepcell_toolbox.deep_watershed.deep_watershed)[source]
Bases:
Application
Loads a
deepcell.model_zoo.panopticnet.PanopticNet
model for cytoplasm segmentation with pretrained weights.The
predict
method handles prep and post processing steps to return a labeled image.Example:
from skimage.io import imread from deepcell.applications import CytoplasmSegmentation # Load the image im = imread('HeLa_cytoplasm.png') # Expand image dimensions to rank 4 im = np.expand_dims(im, axis=-1) im = np.expand_dims(im, axis=0) # Create the application app = CytoplasmSegmentation() # create the lab labeled_image = app.predict(image)
- Parameters:
model (tf.keras.Model) – The model to load. If
None
, a pre-trained model will be downloaded.
- dataset_metadata = {'name': 'general_cyto', 'other': 'Pooled phase and fluorescent cytoplasm data - computationally curated'}
Metadata for the dataset used to train the model
- model_metadata = {'batch_size': 16, 'lr': 0.0001, 'lr_decay': 0.9, 'n_epochs': 8, 'training_seed': 0, 'training_steps_per_epoch': 3949, 'validation_steps_per_epoch': 986}
Metadata for the model and training process
- predict(image, batch_size=4, image_mpp=None, pad_mode='reflect', preprocess_kwargs=None, postprocess_kwargs=None)[source]
Generates a labeled image of the input running prediction with appropriate pre and post processing functions.
Input images are required to have 4 dimensions
[batch, x, y, channel]
.Additional empty dimensions can be added using
np.expand_dims
.- Parameters:
image (numpy.array) – Input image with shape
[batch, x, y, channel]
.batch_size (int) – Number of images to predict on per batch.
image_mpp (float) – Microns per pixel for
image
.pad_mode (str) – The padding mode, one of “constant” or “reflect”.
preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.
postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.
- Raises:
ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.
ValueError – Input data must match required number of channels.
- Returns:
Labeled image
- Return type:
numpy.array
NuclearSegmentation
- class deepcell.applications.nuclear_segmentation.NuclearSegmentation(model=None, preprocessing_fn=deepcell_toolbox.processing.histogram_normalization, postprocessing_fn=deepcell_toolbox.deep_watershed.deep_watershed)[source]
Bases:
Application
Loads a
deepcell.model_zoo.panopticnet.PanopticNet
model for nuclear segmentation with pretrained weights.The
predict
method handles prep and post processing steps to return a labeled image.Example:
from skimage.io import imread from deepcell.applications import NuclearSegmentation # Load the image im = imread('HeLa_nuclear.png') # Expand image dimensions to rank 4 im = np.expand_dims(im, axis=-1) im = np.expand_dims(im, axis=0) # Create the application app = NuclearSegmentation() # create the lab labeled_image = app.predict(image)
- Parameters:
model (tf.keras.Model) – The model to load. If
None
, a pre-trained model will be downloaded.
- dataset_metadata = {'name': 'general_nuclear_train_large', 'other': 'Pooled nuclear data from HEK293, HeLa-S3, NIH-3T3, and RAW264.7 cells.'}
Metadata for the dataset used to train the model
- model_metadata = {'backbone': 'efficientnetv2bl', 'batch_size': 16, 'crop_size': 256, 'epochs': 16, 'location': True, 'lr': 0.0001, 'min_objects': 1, 'pyramid_levels': 'P1-P2-P3-P4-P5-P6-P7', 'zoom_min': 0.75}
Metadata for the model and training process
- predict(image, batch_size=4, image_mpp=None, pad_mode='reflect', preprocess_kwargs=None, postprocess_kwargs=None)[source]
Generates a labeled image of the input running prediction with appropriate pre and post processing functions.
Input images are required to have 4 dimensions
[batch, x, y, channel]
.Additional empty dimensions can be added using
np.expand_dims
.- Parameters:
image (numpy.array) – Input image with shape
[batch, x, y, channel]
.batch_size (int) – Number of images to predict on per batch.
image_mpp (float) – Microns per pixel for
image
.pad_mode (str) – The padding mode, one of “constant” or “reflect”.
preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.
postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.
- Raises:
ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.
ValueError – Input data must match required number of channels.
- Returns:
Labeled image
- Return type:
numpy.array
Mesmer
- class deepcell.applications.mesmer.Mesmer(model=None)[source]
Bases:
Application
Loads a
deepcell.model_zoo.panopticnet.PanopticNet
model for tissue segmentation with pretrained weights.The
predict
method handles prep and post processing steps to return a labeled image.Example:
from skimage.io import imread from deepcell.applications import Mesmer # Load the images im1 = imread('TNBC_DNA.tiff') im2 = imread('TNBC_Membrane.tiff') # Combined together and expand to 4D im = np.stack((im1, im2), axis=-1) im = np.expand_dims(im,0) # Create the application app = Mesmer() # create the lab labeled_image = app.predict(image)
- Parameters:
model (tf.keras.Model) – The model to load. If
None
, a pre-trained model will be downloaded.
- dataset_metadata = {'name': '20200315_IF_Training_6.npz', 'other': 'Pooled whole-cell data across tissue types'}
Metadata for the dataset used to train the model
- model_metadata = {'batch_size': 1, 'lr': 1e-05, 'lr_decay': 0.99, 'n_epochs': 30, 'training_seed': 0, 'training_steps_per_epoch': 1739, 'validation_steps_per_epoch': 193}
Metadata for the model and training process
- predict(image, batch_size=4, image_mpp=None, preprocess_kwargs={}, compartment='whole-cell', pad_mode='constant', postprocess_kwargs_whole_cell={}, postprocess_kwargs_nuclear={})[source]
Generates a labeled image of the input running prediction with appropriate pre and post processing functions.
Input images are required to have 4 dimensions
[batch, x, y, channel]
. Additional empty dimensions can be added usingnp.expand_dims
.- Parameters:
image (numpy.array) – Input image with shape
[batch, x, y, channel]
.batch_size (int) – Number of images to predict on per batch.
image_mpp (float) – Microns per pixel for
image
.compartment (str) – Specify type of segmentation to predict. Must be one of
"whole-cell"
,"nuclear"
,"both"
.preprocess_kwargs (dict) – Keyword arguments to pass to the pre-processing function.
postprocess_kwargs (dict) – Keyword arguments to pass to the post-processing function.
- Raises:
ValueError – Input data must match required rank of the application, calculated as one dimension more (batch dimension) than expected by the model.
ValueError – Input data must match required number of channels.
- Returns:
Instance segmentation mask.
- Return type:
numpy.array
CellTracking
- class deepcell.applications.cell_tracking.CellTracking(model=None, neighborhood_encoder=None, distance_threshold=64, appearance_dim=32, birth=0.99, death=0.99, division=0.01, track_length=8, embedding_axis=0, crop_mode='resize', norm=True)[source]
Bases:
Application
Loads a
deepcell.model_zoo.tracking.GNNTrackingModel
model for object tracking with pretrained weights using a simplepredict
interface.- Parameters:
model (
tf.keras.model
) – Tracking inference model, defaults to latest published modelneighborhood_encoder (
tf.keras.model
) – Tracking neighborhood encoder, defaults to latest published modeldistance_threshold (int) – Maximum distance between two cells to be considered adjacent
appearance_dim (int) – Length of appearance dimension
birth (float) – Cost of new cell in linear assignment matrix.
death (float) – Cost of cell death in linear assignment matrix.
division (float) – Cost of cell division in linear assignment matrix.
track_length (int) – Number of frames per track
crop_mode (str) – Type of cropping around each cell
norm (str) – Type of normalization layer
- dataset_metadata = {'name': 'tracked_nuclear_train_large', 'other': 'Pooled tracked nuclear data from HEK293, HeLa-S3, NIH-3T3, and RAW264.7 cells.'}
Metadata for the dataset used to train the model
- model_metadata = {'appearance_dim': 32, 'batch_size': 8, 'buffer_size': 128, 'crop_mode': 'resize', 'data_fraction': 1, 'distance_threshold': 64, 'embedding_dim': 64, 'encoder_dim': 64, 'epochs': 50, 'graph_layer': 'gat', 'lr': 0.001, 'n_filters': 64, 'n_layers': 1, 'norm_layer': 'batch', 'rotation_range': 180, 'steps_per_epoch': 1000, 'translation_range': 512, 'validation_steps': 200}
Metadata for the model and training process
LabelDetectionModel
- deepcell.applications.label_detection.LabelDetectionModel(input_shape=(None, None, 1), inputs=None, backbone='mobilenetv2', num_classes=3)[source]
Classify a microscopy image as Nuclear, Cytoplasm, or Phase.
This can be helpful in determining the type of data (nuclear, cytoplasm, etc.) so that this data can be forwared to the correct segmenation model.
Based on a standard backbone with an intiial
ImageNormalization2D
and finalAveragePooling2D
,TensorProduct
, andSoftmax
layers.- Parameters:
ScaleDetectionModel
- deepcell.applications.scale_detection.ScaleDetectionModel(input_shape=(None, None, 1), inputs=None, backbone='mobilenetv2')[source]
Create a
ScaleDetectionModel
for detecting scales of input data.This enables data to be scaled appropriately for other segmentation models which may not be resolution tolerant.
Based on a standard backbone with an intiial
ImageNormalization2D
and finalAveragePooling2D
andTensorProduct
layers.