84e732da08
* Add `spring-` prefix to function names Fixes: #9 This commit renames each sub-module in the common, consumer, function and supplier groups with a prefix of `spring-`. * Update README.adoc links to new prefixed names
105 lines
5.2 KiB
Plaintext
105 lines
5.2 KiB
Plaintext
:images-asciidoc: https://raw.githubusercontent.com/spring-cloud/stream-applications/master/functions/function/semantic-segmentation-function/src/main/resources/images/
|
|
# Semantic Segmentation
|
|
|
|
[.lead]
|
|
Image Semantic Segmentation based on the state-of-art https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab] Tensorflow model.
|
|
|
|
[cols="1,2", frame=none, grid=none]
|
|
|===
|
|
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
|
|Semantic Segmentation is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car).
|
|
Unlike the `Instance Segmentation`, which produces instance-aware region masks, the `Semantic Segmentation` produces class-aware masks.
|
|
For implementing `Instance Segmentation` consult the https://github.com/spring-cloud/stream-applications/tree/master/functions/function/object-detection-function[Object Detection Service] instead.
|
|
|===
|
|
|
|
The https://github.com/spring-cloud/stream-applications/blob/master/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
|
|
converting the `List<ObjectDetection>` into JSON objects, and the
|
|
https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
|
|
allow to augment the input image with the detected bounding boxes and segmentation masks.
|
|
|
|
## Usage
|
|
|
|
Add the `semantic-segmentation` dependency to your pom (_use the latest version available_):
|
|
|
|
[source,xml]
|
|
----
|
|
<dependency>
|
|
<groupId>org.springframework.cloud.fn</groupId>
|
|
<artifactId>semantic-segmentation-function</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.springframework.cloud.fn</groupId>
|
|
<artifactId>object-detection-function</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
----
|
|
|
|
Following snippet demos how to use the PASCAL VOC model to apply mask to an input image
|
|
|
|
[source,java,linenums]
|
|
----
|
|
|
|
SemanticSegmentation segmentationService = new SemanticSegmentation(
|
|
"https://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", // <1>
|
|
true); // <2>
|
|
|
|
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); // <3>
|
|
|
|
byte[] imageMask = segmentationService.masksAsImage(inputImage); // <4>
|
|
BufferedImage bi = ImageIO.read(new ByteArrayInputStream(imageMask));
|
|
ImageIO.write(bi, "png", new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_masks.png"));
|
|
|
|
byte[] augmentedImage = segmentationService.augment(inputImage); // <5>
|
|
IOUtils.write(augmentedImage, new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_augmented.jpg"));
|
|
----
|
|
<1> Download the PASCAL 2012 trained model directly from the web. The `frozen_inference_graph.pb` is the name of the model
|
|
file inside the `tar.gz` archive.
|
|
<2> Cache the downloaded model locally
|
|
<3> Load the input image as byte array
|
|
<4> Read get the segmentation mask as separate image
|
|
<5> Blend the segmentation mask on top of the original image
|
|
|
|
## Models
|
|
|
|
Based on the training datasets, three groups of pre-trained models provided:
|
|
|
|
[cols="1,2", frame=none, grid=none]
|
|
|===
|
|
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-pascal-voc-2012[DeepLab models trained on PASCAL VOC 2012]
|
|
|
|
| image:{images-asciidoc}/cityscape-all-small.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-cityscapes[DeepLab models trained on Cityscapes]
|
|
|
|
| image:{images-asciidoc}/ADE20K-all-small.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-ade20k[DeepLab models trained on ADE20K]
|
|
|===
|
|
|
|
Select the model you want to use, copy its archive download Url and add a `#frozen_inference_graph.pb` fragment to it.
|
|
Later fragment is the frozen model's file name inside the archive
|
|
|
|
TIP: Download the archive and uncompress the `frozen_inference_graph.pb` for required model. Then use the `file://<local-file-name>` URI schema.
|
|
|
|
Also, convenience there are a couple of models, extracted from the archive and uploaded to bintray:
|
|
|
|
[cols=2*,, frame=none, grid=none]
|
|
|===
|
|
|PASCAL VOC 2012 (default)
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_pascal_train_aug_frozen_inference_graph.pb
|
|
|
|
|CITYSCAPE
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_cityscapes_train_2018_02_05_frozen_inference_graph.pb
|
|
|
|
|ADE20K
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_xception_ade20k_train_2018_05_29_frozen_inference_graph.pb
|
|
|===
|
|
|
|
## References:
|
|
[.small]
|
|
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html[Semantic Image Segmentation with DeepLab in TensorFlow]
|
|
* https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab Project]
|
|
* https://medium.freecodecamp.org/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b[How to re-train DeepLab Segmentation models using Transfer Learning]
|
|
|