Mesmer: Tissue Segmentation

Mesmer can be accessed using deepcell.applications with a DeepCell API key.

For more information about using a DeepCell API key, please see DeepCell API Key.

from matplotlib import pyplot as plt

from deepcell.datasets import TissueNetSample
from deepcell.utils.plot_utils import create_rgb_image, make_outline_overlay

# Download multiplex data
X, y, _ = TissueNetSample().load_data()

create rgb overlay of image data for visualization

rgb_images = create_rgb_image(X, channel_colors=['green', 'blue'])

plot the data

fig, ax = plt.subplots(1, 3, figsize=(15, 5))
ax[0].imshow(X[0, ..., 0], cmap='Greys_r')
ax[1].imshow(X[0, ..., 1], cmap='Greys_r')
ax[2].imshow(rgb_images[0, ...])

ax[0].set_title('Nuclear channel')
ax[1].set_title('Membrane channel')
ax[2].set_title('Overlay')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-input.png')

The application will download pretrained weights for tissue segmentation. For more information about application objects, please see our documentation.

from deepcell.applications import Mesmer
app = Mesmer()

Whole Cell Segmentation

Typically, neural networks perform best on test data that is similar to the training data. In the realm of biological imaging, the most common difference between datasets is the resolution of the data measured in microns per pixel. The training resolution of the model can be identified using app.model_mpp.

print('Training Resolution:', app.model_mpp, 'microns per pixel')

The resolution of the input data can be specified in app.predict using the image_mpp option. The Application will rescale the input data to match the training resolution and then rescale to the original size before returning the labeled image.

segmentation_predictions = app.predict(X, image_mpp=0.5)

create overlay of predictions

overlay_data = make_outline_overlay(rgb_data=rgb_images, predictions=segmentation_predictions)

select index for displaying

idx = 0

# plot the data
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(rgb_images[idx, ...])
ax[1].imshow(overlay_data[idx, ...])

ax[0].set_title('Raw data')
ax[1].set_title('Predictions')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-wc.png')

Nuclear Segmentation

In addition to predicting whole-cell segmentation, Mesmer can also be used for nuclear predictions

segmentation_predictions_nuc = app.predict(X, image_mpp=0.5, compartment='nuclear')

overlay_data_nuc = make_outline_overlay(
    rgb_data=rgb_images,
    predictions=segmentation_predictions_nuc)

select index for displaying

idx = 0

# plot the data
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(rgb_images[idx, ...])
ax[1].imshow(overlay_data_nuc[idx, ...])

ax[0].set_title('Raw data')
ax[1].set_title('Nuclear Predictions')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-nuc.png')

Fine-tuning the model output

In most cases, we find that the default settings for the model work quite well across a range of tissues. However, if you notice specific, consistent errors in your data, there are a few things you can change.

The first is the interior_threshold parameter. This controls how conservative the model is in estimating what is a cell vs what is background. Lower values of interior_threshold will result in larger cells, whereas higher values will result in smaller cells.

The second is the maxima_threshold parameter. This controls what the model considers a unique cell. Lower values will result in more separate cells being predicted, whereas higher values will result in fewer cells.

To demonstrate the effect of interior_threshold, we’ll compare the default with a much more stringent setting

segmentation_predictions_interior = app.predict(
    X,
    image_mpp=0.5,
    postprocess_kwargs_whole_cell={'interior_threshold': 0.5})
overlay_data_interior = make_outline_overlay(
    rgb_data=rgb_images,
    predictions=segmentation_predictions_interior)

select index for displaying

idx = 0

# plot the data
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(overlay_data[idx, ...])
ax[1].imshow(overlay_data_interior[idx, ...])

ax[0].set_title('Default settings')
ax[1].set_title('More restrictive interior threshold')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-interior-threshold.png')

To demonstrate the effect of maxima_threshold, we’ll compare the default with a much more stringent setting

segmentation_predictions_maxima = app.predict(
    X,
    image_mpp=0.5,
    postprocess_kwargs_whole_cell={'maxima_threshold': 0.8})
overlay_data_maxima = make_outline_overlay(
    rgb_data=rgb_images,
    predictions=segmentation_predictions_maxima)

select index for displaying

idx = 0

# plot the data
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(overlay_data[idx, ...])
ax[1].imshow(overlay_data_maxima[idx, ...])

ax[0].set_title('Default settings')
ax[1].set_title('More stringent maxima threshold')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-maxima-threshold.png')

Finally, if your data doesn’t include in a strong membrane marker, the model will default to just predicting the nuclear segmentation, even for whole-cell mode. If you’d like to add a manual pixel expansion after segmentation, you can do that using the pixel_expansion argument. This will universally apply an expansion after segmentation to each cell

To demonstrate the effect of pixel_expansion, we’ll compare the nuclear output with expanded output

segmentation_predictions_expansion = app.predict(
    X,
    image_mpp=0.5,
    compartment='nuclear',
    postprocess_kwargs_nuclear={'pixel_expansion': 5}
)
overlay_data_expansion = make_outline_overlay(
    rgb_data=rgb_images,
    predictions=segmentation_predictions_expansion
)

select index for displaying

idx = 0

# plot the data
fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(overlay_data_nuc[idx, ...])
ax[1].imshow(overlay_data_expansion[idx, ...])

ax[0].set_title('Default nuclear segmentation')
ax[1].set_title('Nuclear segmentation with an expansion')

for a in ax:
    a.axis('off')

plt.show()
fig.savefig('mesmer-nuc-expansion.png')

There’s a separate dictionary passed to the model that controls the post-processing for whole-cell and nuclear predictions. You can modify them independently to fine-tune the output. The current defaults the model is using can be found here