724c4ace5f
This is so they can be updated in a minimal amount of places and leverage the maven hierarchy to propagate them where appropriate. While working with SNAPSHOT releases of Spring Boot, Spring Integration, and Spring Cloud, it required touching every pom.xml file in order to reflect the use of the SNAPSHOT. After this change, it will only be required to change the property in the top functions pom.xml and the versions will reflect to all submodules. Maven introduced the ability use a key property, revision to propagate this information to the <parent> declarations in each pom. This was enabled in Maven 3.5+ https://maven.apache.org/maven-ci-friendly.html
105 lines
5.2 KiB
Plaintext
105 lines
5.2 KiB
Plaintext
:images-asciidoc: https://raw.githubusercontent.com/spring-cloud/stream-applications/master/functions/function/semantic-segmentation-function/src/main/resources/images/
|
|
# Semantic Segmentation
|
|
|
|
[.lead]
|
|
Image Semantic Segmentation based on the state-of-art https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab] Tensorflow model.
|
|
|
|
[cols="1,2", frame=none, grid=none]
|
|
|===
|
|
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
|
|Semantic Segmentation is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car).
|
|
Unlike the `Instance Segmentation`, which produces instance-aware region masks, the `Semantic Segmentation` produces class-aware masks.
|
|
For implementing `Instance Segmentation` consult the https://github.com/spring-cloud/stream-applications/tree/master/functions/function/object-detection-function[Object Detection Service] instead.
|
|
|===
|
|
|
|
The https://github.com/spring-cloud/stream-applications/blob/master/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
|
|
converting the `List<ObjectDetection>` into JSON objects, and the
|
|
https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
|
|
allow to augment the input image with the detected bounding boxes and segmentation masks.
|
|
|
|
## Usage
|
|
|
|
Add the `semantic-segmentation` dependency to your pom (_use the latest version available_):
|
|
|
|
[source,xml]
|
|
----
|
|
<dependency>
|
|
<groupId>org.springframework.cloud.fn</groupId>
|
|
<artifactId>semantic-segmentation-function</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.springframework.cloud.fn</groupId>
|
|
<artifactId>object-detection-function</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
----
|
|
|
|
Following snippet demos how to use the PASCAL VOC model to apply mask to an input image
|
|
|
|
[source,java,linenums]
|
|
----
|
|
|
|
SemanticSegmentation segmentationService = new SemanticSegmentation(
|
|
"https://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", // <1>
|
|
true); // <2>
|
|
|
|
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); // <3>
|
|
|
|
byte[] imageMask = segmentationService.masksAsImage(inputImage); // <4>
|
|
BufferedImage bi = ImageIO.read(new ByteArrayInputStream(imageMask));
|
|
ImageIO.write(bi, "png", new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_masks.png"));
|
|
|
|
byte[] augmentedImage = segmentationService.augment(inputImage); // <5>
|
|
IOUtils.write(augmentedImage, new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_augmented.jpg"));
|
|
----
|
|
<1> Download the PASCAL 2012 trained model directly from the web. The `frozen_inference_graph.pb` is the name of the model
|
|
file inside the `tar.gz` archive.
|
|
<2> Cache the downloaded model locally
|
|
<3> Load the input image as byte array
|
|
<4> Read get the segmentation mask as separate image
|
|
<5> Blend the segmentation mask on top of the original image
|
|
|
|
## Models
|
|
|
|
Based on the training datasets, three groups of pre-trained models provided:
|
|
|
|
[cols="1,2", frame=none, grid=none]
|
|
|===
|
|
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-pascal-voc-2012[DeepLab models trained on PASCAL VOC 2012]
|
|
|
|
| image:{images-asciidoc}/cityscape-all-small.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-cityscapes[DeepLab models trained on Cityscapes]
|
|
|
|
| image:{images-asciidoc}/ADE20K-all-small.png[width=100%]
|
|
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-ade20k[DeepLab models trained on ADE20K]
|
|
|===
|
|
|
|
Select the model you want to use, copy its archive download Url and add a `#frozen_inference_graph.pb` fragment to it.
|
|
Later fragment is the frozen model's file name inside the archive
|
|
|
|
TIP: Download the archive and uncompress the `frozen_inference_graph.pb` for required model. Then use the `file://<local-file-name>` URI schema.
|
|
|
|
Also, convenience there are a couple of models, extracted from the archive and uploaded to bintray:
|
|
|
|
[cols=2*,, frame=none, grid=none]
|
|
|===
|
|
|PASCAL VOC 2012 (default)
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_pascal_train_aug_frozen_inference_graph.pb
|
|
|
|
|CITYSCAPE
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_cityscapes_train_2018_02_05_frozen_inference_graph.pb
|
|
|
|
|ADE20K
|
|
|https://dl.bintray.com/big-data/generic/deeplabv3_xception_ade20k_train_2018_05_29_frozen_inference_graph.pb
|
|
|===
|
|
|
|
## References:
|
|
[.small]
|
|
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html[Semantic Image Segmentation with DeepLab in TensorFlow]
|
|
* https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab Project]
|
|
* https://medium.freecodecamp.org/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b[How to re-train DeepLab Segmentation models using Transfer Learning]
|
|
|