Tensorflow functions and applications
* initial step
* Tensorflow models functional model redesign
-- Based on https://tzolov.github.io/mind-model-services
-- Resolves #5
* Add object detection processor README
* Add image recognition processor README
* Initial Tensorflow commonn README
* Initial Tensorflow commonn README
* Tensorflow common diagram
* Tensorflow docs code
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Add semantic segmentation function. add object detecteion function readme
* oo images
* Furether oo readme improvments
* Final obj detection readme fixes
* Add image recognition readme
* Add image recognition readme 2
* Semantic segmentation readme
* Segmentation readme
* Semantic segmentation readme 3
* Fix image recognition and object detcion app starter dependecies
* Add metadata for Tensorflow apps
This commit is contained in:
Christian Tzolov
committed by
Soby Chacko
Soby Chacko
parent
3bb9e066b9
commit
dffb467da4
104
function/semantic-segmentation-function/README.adoc
Normal file
104
function/semantic-segmentation-function/README.adoc
Normal file
@@ -0,0 +1,104 @@
|
||||
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/function/semantic-segmentation-function/src/main/resources/images/
|
||||
# Semantic Segmentation
|
||||
|
||||
[.lead]
|
||||
Image Semantic Segmentation based on the state-of-art https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab] Tensorflow model.
|
||||
|
||||
[cols="1,2", frame=none, grid=none]
|
||||
|===
|
||||
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
||||
|Semantic Segmentation is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car).
|
||||
Unlike the `Instance Segmentation`, which produces instance-aware region masks, the `Semantic Segmentation` produces class-aware masks.
|
||||
For implementing `Instance Segmentation` consult the https://github.com/tzolov/stream-applications/tree/tensorflow-redesign/functions/function/object-detection-function[Object Detection Service] instead.
|
||||
|===
|
||||
|
||||
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
|
||||
converting the `List<ObjectDetection>` into JSON objects, and the
|
||||
https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
|
||||
allow to augment the input image with the detected bounding boxes and segmentation masks.
|
||||
|
||||
## Usage
|
||||
|
||||
Add the `semantic-segmentation` dependency to your pom (_use the latest version available_):
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>semantic-segmentation-function</artifactId>
|
||||
<version>${spring-cloud-fn.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>object-detection-function</artifactId>
|
||||
<version>${spring-cloud-fn.version}</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
Following snippet demos how to use the PASCAL VOC model to apply mask to an input image
|
||||
|
||||
[source,java,linenums]
|
||||
----
|
||||
|
||||
SemanticSegmentation segmentationService = new SemanticSegmentation(
|
||||
"http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", // <1>
|
||||
true); // <2>
|
||||
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); // <3>
|
||||
|
||||
byte[] imageMask = segmentationService.masksAsImage(inputImage); // <4>
|
||||
BufferedImage bi = ImageIO.read(new ByteArrayInputStream(imageMask));
|
||||
ImageIO.write(bi, "png", new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_masks.png"));
|
||||
|
||||
byte[] augmentedImage = segmentationService.augment(inputImage); // <5>
|
||||
IOUtils.write(augmentedImage, new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_augmented.jpg"));
|
||||
----
|
||||
<1> Download the PASCAL 2012 trained model directly from the web. The `frozen_inference_graph.pb` is the name of the model
|
||||
file inside the `tar.gz` archive.
|
||||
<2> Cache the downloaded model locally
|
||||
<3> Load the input image as byte array
|
||||
<4> Read get the segmentation mask as separate image
|
||||
<5> Blend the segmentation mask on top of the original image
|
||||
|
||||
## Models
|
||||
|
||||
Based on the training datasets, three groups of pre-trained models provided:
|
||||
|
||||
[cols="1,2", frame=none, grid=none]
|
||||
|===
|
||||
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|
||||
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-pascal-voc-2012[DeepLab models trained on PASCAL VOC 2012]
|
||||
|
||||
| image:{images-asciidoc}/cityscape-all-small.png[width=100%]
|
||||
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-cityscapes[DeepLab models trained on Cityscapes]
|
||||
|
||||
| image:{images-asciidoc}/ADE20K-all-small.png[width=100%]
|
||||
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-ade20k[DeepLab models trained on ADE20K]
|
||||
|===
|
||||
|
||||
Select the model you want to use, copy its archive download Url and add a `#frozen_inference_graph.pb` fragment to it.
|
||||
Later fragment is the frozen model's file name inside the archive
|
||||
|
||||
TIP: Download the archive and uncompress the `frozen_inference_graph.pb` for required model. Then use the `file://<local-file-name>` URI schema.
|
||||
|
||||
Also, convenience there are a couple of models, extracted from the archive and uploaded to bintray:
|
||||
|
||||
[cols=2*,, frame=none, grid=none]
|
||||
|===
|
||||
|PASCAL VOC 2012 (default)
|
||||
|http://dl.bintray.com/big-data/generic/deeplabv3_mnv2_pascal_train_aug_frozen_inference_graph.pb
|
||||
|
||||
|CITYSCAPE
|
||||
|http://dl.bintray.com/big-data/generic/deeplabv3_mnv2_cityscapes_train_2018_02_05_frozen_inference_graph.pb
|
||||
|
||||
|ADE20K
|
||||
|http://dl.bintray.com/big-data/generic/deeplabv3_xception_ade20k_train_2018_05_29_frozen_inference_graph.pb
|
||||
|===
|
||||
|
||||
## References:
|
||||
[.small]
|
||||
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html[Semantic Image Segmentation with DeepLab in TensorFlow]
|
||||
* https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab Project]
|
||||
* https://medium.freecodecamp.org/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b[How to re-train DeepLab Segmentation models using Transfer Learning]
|
||||
|
||||
38
function/semantic-segmentation-function/pom.xml
Normal file
38
function/semantic-segmentation-function/pom.xml
Normal file
@@ -0,0 +1,38 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>semantic-segmentation-function</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>semantic-segmentation-function</name>
|
||||
<description>Spring Native Function for Tensorflow semantic-segmentation integration</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<properties>
|
||||
<commons-io.version>1.3.2</commons-io.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>tensorflow-common</artifactId>
|
||||
<version>${spring-cloud-fn.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<id>tensorflow-snapshots</id>
|
||||
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
</repository>
|
||||
</repositories>
|
||||
</project>
|
||||
@@ -0,0 +1,107 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.semantic.segmentation;
|
||||
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.tensorflow.Operand;
|
||||
import org.tensorflow.op.Ops;
|
||||
import org.tensorflow.op.core.Concat;
|
||||
import org.tensorflow.op.core.ExpandDims;
|
||||
import org.tensorflow.op.core.Gather;
|
||||
import org.tensorflow.op.core.Range;
|
||||
import org.tensorflow.op.core.ReduceMax;
|
||||
import org.tensorflow.op.core.Tile;
|
||||
import org.tensorflow.op.math.Add;
|
||||
import org.tensorflow.op.math.Mul;
|
||||
import org.tensorflow.op.math.Sub;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class NativeImageUtils {
|
||||
|
||||
private NativeImageUtils() {
|
||||
}
|
||||
|
||||
/**
|
||||
* grayscaleToRgb.
|
||||
* https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/ops/image_ops_impl.py#L1536
|
||||
*/
|
||||
public static <T> Operand<T> grayscaleToRgb(Ops tf, Operand<T> images) {
|
||||
ExpandDims<Integer> rank_1 = tf.expandDims(
|
||||
tf.math.sub(tf.rank(images), tf.constant(1)),
|
||||
tf.constant(0));
|
||||
// Create once 1D vector of the shape defined by the rank_1.
|
||||
// E.g. for rank [2] will produce matrix [1, 1]. For [3] rank will produce a cube [1, 1, 1]
|
||||
Add<Integer> ones = tf.math.add(tf.zeros(rank_1, Integer.class), tf.constant(1));
|
||||
// Convert scalar 3 into 1D array [3]
|
||||
ExpandDims<Integer> channelsAs1D = tf.expandDims(tf.constant(3), tf.constant(0));
|
||||
Concat<Integer> shapeList = tf.concat(Arrays.asList(ones, channelsAs1D), tf.constant(0));
|
||||
Tile<T> tile = tf.withName("grayscaleToRgb").tile(images, shapeList);
|
||||
return tile;
|
||||
}
|
||||
|
||||
public static Operand<Float> normalizeMask(Ops tf, Operand<Float> mask, float newValue) {
|
||||
// generate array representing the axis indexes.
|
||||
// For example of tensor of rank K the axisRange is {0, 1, 2 ...K}
|
||||
Range<Integer> axisRange = tf.range(tf.constant(0), // from
|
||||
tf.dtypes.cast(tf.rank(mask), Integer.class), // to
|
||||
tf.constant(1)); // step
|
||||
|
||||
ReduceMax<Float> max = tf.reduceMax(mask, axisRange);
|
||||
//Mul<Float> input2Float1 = tf.math.mul(tf.math.div(input2Float, max), tf.constant(1f));
|
||||
Mul<Float> normalizedMask = tf.math.mul(tf.math.div(mask, max), tf.constant(newValue));
|
||||
|
||||
return normalizedMask;
|
||||
}
|
||||
|
||||
/**
|
||||
* Alpha Blending .
|
||||
* https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending
|
||||
*/
|
||||
public static Operand<Float> alphaBlending(Ops tf, Operand<Float> srcRgb, Operand<Float> dstRgb, Operand<Float> srcAlpha) {
|
||||
Sub<Float> alpha = tf.math.sub(tf.onesLike(srcRgb), srcAlpha);
|
||||
Mul<Float> src = tf.math.mul(srcRgb, alpha);
|
||||
Mul<Float> dst = tf.math.mul(dstRgb, tf.math.sub(tf.constant(1.0f), alpha));
|
||||
Add<Float> out = tf.math.add(dst, src);
|
||||
|
||||
//Mul<Float> out = tf.math.mul(srcRgbNormalized, dstRgb);
|
||||
//Squeeze<Float> squeeze = tf.withName("squeeze").squeeze(out, Squeeze.axis(Arrays.asList(0L)));
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The mask can contain label values larger than the list of colors provided in the color map.
|
||||
* To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
|
||||
* @param tf - tensorflow
|
||||
* @param colorTable Color map of shape [n, 3]. n is the count of label entries and 3 is the RGB color assigned
|
||||
* to that label.
|
||||
* @param mask Mask of shape [h, w] containing label vales.
|
||||
* @return Mask of shape [h, w] fromMemory values normalized between [0, n]
|
||||
*/
|
||||
public static Operand<Long> normalizeMaskLabels(Ops tf, Operand<Integer> colorTable, Operand<Long> mask) {
|
||||
// The mask can contain label values larger than the list of colors provided in the color map.
|
||||
// To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
|
||||
Sub<Long> colorTableShape = tf.math.sub(tf.shape(colorTable, Long.class), tf.constant(1L));
|
||||
// Color tables have shape [N, 3], where N is the count of label entries. Therefore the max label id is (N - 1).
|
||||
Gather<Long> colorTableSize = tf.gather(colorTableShape, tf.constant(new int[] { 0 }), tf.constant(0));
|
||||
// Normalize the label values in the mask so they don't exceed the max value in the color map.
|
||||
return tf.math.mod(mask, colorTableSize);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.semantic.segmentation;
|
||||
|
||||
/**
|
||||
*
|
||||
* Visualizes the segmentation results via specified color map.
|
||||
* Color maps helping to visualize the semantic segmentation results for the different datasets.
|
||||
*
|
||||
* Supported colormaps are:
|
||||
* - ADE20K (http://groups.csail.mit.edu/vision/datasets/ADE20K/).
|
||||
* - Cityscapes dataset (https://www.cityscapes-dataset.com).
|
||||
* - Mapillary Vistas (https://research.mapillary.com).
|
||||
* - PASCAL VOC 2012 (http://host.robots.ox.ac.uk/pascal/VOC/).
|
||||
*
|
||||
* Based on: https://github.com/tensorflow/models/blob/master/research/deeplab/utils/get_dataset_colormap.py
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class SegmentationColorMap {
|
||||
|
||||
private SegmentationColorMap() {
|
||||
|
||||
}
|
||||
|
||||
/** MAPILLARY_COLORMAP . */
|
||||
public static final int[][] MAPILLARY_COLORMAP = new int[][] {
|
||||
{ 165, 42, 42 },
|
||||
{ 0, 192, 0 },
|
||||
{ 196, 196, 196 },
|
||||
{ 190, 153, 153 },
|
||||
{ 180, 165, 180 },
|
||||
{ 102, 102, 156 },
|
||||
{ 102, 102, 156 },
|
||||
{ 128, 64, 255 },
|
||||
{ 140, 140, 200 },
|
||||
{ 170, 170, 170 },
|
||||
{ 250, 170, 160 },
|
||||
{ 96, 96, 96 },
|
||||
{ 230, 150, 140 },
|
||||
{ 128, 64, 128 },
|
||||
{ 110, 110, 110 },
|
||||
{ 244, 35, 232 },
|
||||
{ 150, 100, 100 },
|
||||
{ 70, 70, 70 },
|
||||
{ 150, 120, 90 },
|
||||
{ 220, 20, 60 },
|
||||
{ 255, 0, 0 },
|
||||
{ 255, 0, 0 },
|
||||
{ 255, 0, 0 },
|
||||
{ 200, 128, 128 },
|
||||
{ 255, 255, 255 },
|
||||
{ 64, 170, 64 },
|
||||
{ 128, 64, 64 },
|
||||
{ 70, 130, 180 },
|
||||
{ 255, 255, 255 },
|
||||
{ 152, 251, 152 },
|
||||
{ 107, 142, 35 },
|
||||
{ 0, 170, 30 },
|
||||
{ 255, 255, 128 },
|
||||
{ 250, 0, 30 },
|
||||
{ 0, 0, 0 },
|
||||
{ 220, 220, 220 },
|
||||
{ 170, 170, 170 },
|
||||
{ 222, 40, 40 },
|
||||
{ 100, 170, 30 },
|
||||
{ 40, 40, 40 },
|
||||
{ 33, 33, 33 },
|
||||
{ 170, 170, 170 },
|
||||
{ 0, 0, 142 },
|
||||
{ 170, 170, 170 },
|
||||
{ 210, 170, 100 },
|
||||
{ 153, 153, 153 },
|
||||
{ 128, 128, 128 },
|
||||
{ 0, 0, 142 },
|
||||
{ 250, 170, 30 },
|
||||
{ 192, 192, 192 },
|
||||
{ 220, 220, 0 },
|
||||
{ 180, 165, 180 },
|
||||
{ 119, 11, 32 },
|
||||
{ 0, 0, 142 },
|
||||
{ 0, 60, 100 },
|
||||
{ 0, 0, 142 },
|
||||
{ 0, 0, 90 },
|
||||
{ 0, 0, 230 },
|
||||
{ 0, 80, 100 },
|
||||
{ 128, 64, 64 },
|
||||
{ 0, 0, 110 },
|
||||
{ 0, 0, 70 },
|
||||
{ 0, 0, 192 },
|
||||
{ 32, 32, 32 },
|
||||
{ 0, 0, 0 },
|
||||
{ 0, 0, 0 },
|
||||
};
|
||||
|
||||
/**
|
||||
* Label colormap used in ADE20K segmentation benchmark.
|
||||
*/
|
||||
public static final int[][] ADE20K_COLORMAP = new int[][] {
|
||||
{ 0, 0, 0 },
|
||||
{ 120, 120, 120 },
|
||||
{ 180, 120, 120 },
|
||||
{ 6, 230, 230 },
|
||||
{ 80, 50, 50 },
|
||||
{ 4, 200, 3 },
|
||||
{ 120, 120, 80 },
|
||||
{ 140, 140, 140 },
|
||||
{ 204, 5, 255 },
|
||||
{ 230, 230, 230 },
|
||||
{ 4, 250, 7 },
|
||||
{ 224, 5, 255 },
|
||||
{ 235, 255, 7 },
|
||||
{ 150, 5, 61 },
|
||||
{ 120, 120, 70 },
|
||||
{ 8, 255, 51 },
|
||||
{ 255, 6, 82 },
|
||||
{ 143, 255, 140 },
|
||||
{ 204, 255, 4 },
|
||||
{ 255, 51, 7 },
|
||||
{ 204, 70, 3 },
|
||||
{ 0, 102, 200 },
|
||||
{ 61, 230, 250 },
|
||||
{ 255, 6, 51 },
|
||||
{ 11, 102, 255 },
|
||||
{ 255, 7, 71 },
|
||||
{ 255, 9, 224 },
|
||||
{ 9, 7, 230 },
|
||||
{ 220, 220, 220 },
|
||||
{ 255, 9, 92 },
|
||||
{ 112, 9, 255 },
|
||||
{ 8, 255, 214 },
|
||||
{ 7, 255, 224 },
|
||||
{ 255, 184, 6 },
|
||||
{ 10, 255, 71 },
|
||||
{ 255, 41, 10 },
|
||||
{ 7, 255, 255 },
|
||||
{ 224, 255, 8 },
|
||||
{ 102, 8, 255 },
|
||||
{ 255, 61, 6 },
|
||||
{ 255, 194, 7 },
|
||||
{ 255, 122, 8 },
|
||||
{ 0, 255, 20 },
|
||||
{ 255, 8, 41 },
|
||||
{ 255, 5, 153 },
|
||||
{ 6, 51, 255 },
|
||||
{ 235, 12, 255 },
|
||||
{ 160, 150, 20 },
|
||||
{ 0, 163, 255 },
|
||||
{ 140, 140, 140 },
|
||||
{ 250, 10, 15 },
|
||||
{ 20, 255, 0 },
|
||||
{ 31, 255, 0 },
|
||||
{ 255, 31, 0 },
|
||||
{ 255, 224, 0 },
|
||||
{ 153, 255, 0 },
|
||||
{ 0, 0, 255 },
|
||||
{ 255, 71, 0 },
|
||||
{ 0, 235, 255 },
|
||||
{ 0, 173, 255 },
|
||||
{ 31, 0, 255 },
|
||||
{ 11, 200, 200 },
|
||||
{ 255, 82, 0 },
|
||||
{ 0, 255, 245 },
|
||||
{ 0, 61, 255 },
|
||||
{ 0, 255, 112 },
|
||||
{ 0, 255, 133 },
|
||||
{ 255, 0, 0 },
|
||||
{ 255, 163, 0 },
|
||||
{ 255, 102, 0 },
|
||||
{ 194, 255, 0 },
|
||||
{ 0, 143, 255 },
|
||||
{ 51, 255, 0 },
|
||||
{ 0, 82, 255 },
|
||||
{ 0, 255, 41 },
|
||||
{ 0, 255, 173 },
|
||||
{ 10, 0, 255 },
|
||||
{ 173, 255, 0 },
|
||||
{ 0, 255, 153 },
|
||||
{ 255, 92, 0 },
|
||||
{ 255, 0, 255 },
|
||||
{ 255, 0, 245 },
|
||||
{ 255, 0, 102 },
|
||||
{ 255, 173, 0 },
|
||||
{ 255, 0, 20 },
|
||||
{ 255, 184, 184 },
|
||||
{ 0, 31, 255 },
|
||||
{ 0, 255, 61 },
|
||||
{ 0, 71, 255 },
|
||||
{ 255, 0, 204 },
|
||||
{ 0, 255, 194 },
|
||||
{ 0, 255, 82 },
|
||||
{ 0, 10, 255 },
|
||||
{ 0, 112, 255 },
|
||||
{ 51, 0, 255 },
|
||||
{ 0, 194, 255 },
|
||||
{ 0, 122, 255 },
|
||||
{ 0, 255, 163 },
|
||||
{ 255, 153, 0 },
|
||||
{ 0, 255, 10 },
|
||||
{ 255, 112, 0 },
|
||||
{ 143, 255, 0 },
|
||||
{ 82, 0, 255 },
|
||||
{ 163, 255, 0 },
|
||||
{ 255, 235, 0 },
|
||||
{ 8, 184, 170 },
|
||||
{ 133, 0, 255 },
|
||||
{ 0, 255, 92 },
|
||||
{ 184, 0, 255 },
|
||||
{ 255, 0, 31 },
|
||||
{ 0, 184, 255 },
|
||||
{ 0, 214, 255 },
|
||||
{ 255, 0, 112 },
|
||||
{ 92, 255, 0 },
|
||||
{ 0, 224, 255 },
|
||||
{ 112, 224, 255 },
|
||||
{ 70, 184, 160 },
|
||||
{ 163, 0, 255 },
|
||||
{ 153, 0, 255 },
|
||||
{ 71, 255, 0 },
|
||||
{ 255, 0, 163 },
|
||||
{ 255, 204, 0 },
|
||||
{ 255, 0, 143 },
|
||||
{ 0, 255, 235 },
|
||||
{ 133, 255, 0 },
|
||||
{ 255, 0, 235 },
|
||||
{ 245, 0, 255 },
|
||||
{ 255, 0, 122 },
|
||||
{ 255, 245, 0 },
|
||||
{ 10, 190, 212 },
|
||||
{ 214, 255, 0 },
|
||||
{ 0, 204, 255 },
|
||||
{ 20, 0, 255 },
|
||||
{ 255, 255, 0 },
|
||||
{ 0, 153, 255 },
|
||||
{ 0, 41, 255 },
|
||||
{ 0, 255, 204 },
|
||||
{ 41, 0, 255 },
|
||||
{ 41, 255, 0 },
|
||||
{ 173, 0, 255 },
|
||||
{ 0, 245, 255 },
|
||||
{ 71, 0, 255 },
|
||||
{ 122, 0, 255 },
|
||||
{ 0, 255, 184 },
|
||||
{ 0, 92, 255 },
|
||||
{ 184, 255, 0 },
|
||||
{ 0, 133, 255 },
|
||||
{ 255, 214, 0 },
|
||||
{ 25, 194, 194 },
|
||||
{ 102, 255, 0 },
|
||||
{ 92, 0, 255 },
|
||||
};
|
||||
|
||||
/** BLACK_WHITE_COLORMAP . */
|
||||
public static int[][] BLACK_WHITE_COLORMAP = new int[][] {
|
||||
{ 0, 0, 0 },
|
||||
{ 127, 127, 127 },
|
||||
{ 255, 255, 255 },
|
||||
};
|
||||
|
||||
/** CITYMAP_COLORMAP . */
|
||||
public static final int[][] CITYMAP_COLORMAP = new int[255][3];
|
||||
|
||||
static {
|
||||
|
||||
// Initialize citymap
|
||||
int[][] _CITYMAP_COLORMAP = new int[][] {
|
||||
{ 128, 64, 128 },
|
||||
{ 244, 35, 232 },
|
||||
{ 70, 70, 70 },
|
||||
{ 102, 102, 156 },
|
||||
{ 190, 153, 153 },
|
||||
{ 153, 153, 153 },
|
||||
{ 250, 170, 30 },
|
||||
{ 220, 220, 0 },
|
||||
{ 107, 142, 35 },
|
||||
{ 152, 251, 152 },
|
||||
{ 70, 130, 180 },
|
||||
{ 220, 20, 60 },
|
||||
{ 255, 0, 0 },
|
||||
{ 0, 0, 142 },
|
||||
{ 0, 0, 70 },
|
||||
{ 0, 60, 100 },
|
||||
{ 0, 80, 100 },
|
||||
{ 0, 0, 230 },
|
||||
{ 119, 11, 32 }
|
||||
};
|
||||
|
||||
for (int i = 0; i < _CITYMAP_COLORMAP.length; i++) {
|
||||
System.arraycopy(_CITYMAP_COLORMAP[i], 0, CITYMAP_COLORMAP[i], 0, _CITYMAP_COLORMAP[i].length);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.semantic.segmentation;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.Map;
|
||||
|
||||
import javax.imageio.ImageIO;
|
||||
|
||||
import org.tensorflow.Operand;
|
||||
import org.tensorflow.Tensor;
|
||||
import org.tensorflow.op.Ops;
|
||||
import org.tensorflow.op.core.Gather;
|
||||
import org.tensorflow.op.core.Placeholder;
|
||||
import org.tensorflow.op.core.Squeeze;
|
||||
import org.tensorflow.op.core.ZerosLike;
|
||||
import org.tensorflow.op.dtypes.Cast;
|
||||
import org.tensorflow.op.image.DecodeJpeg;
|
||||
import org.tensorflow.op.image.ExtractJpegShape;
|
||||
import org.tensorflow.op.math.Add;
|
||||
import org.tensorflow.op.math.Div;
|
||||
import org.tensorflow.op.math.Equal;
|
||||
import org.tensorflow.types.UInt8;
|
||||
|
||||
import org.springframework.cloud.fn.common.tensorflow.Functions;
|
||||
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
|
||||
import org.springframework.cloud.fn.common.tensorflow.GraphRunnerMemory;
|
||||
import org.springframework.cloud.fn.common.tensorflow.ProtoBufGraphDefinition;
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class SemanticSegmentation implements AutoCloseable {
|
||||
|
||||
private static final long CHANNELS = 3;
|
||||
private static final float REQUIRED_INPUT_IMAGE_SIZE = 513f;
|
||||
private final GraphRunner imageNormalization;
|
||||
private final GraphRunner semanticSegmentation;
|
||||
private final GraphRunner maskImageEncoding;
|
||||
private final GraphRunner alphaBlending;
|
||||
private final Tensor<Integer> colorMapTensor;
|
||||
private final Tensor<Float> maskTransparencyTensor;
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
this.imageNormalization.close();
|
||||
this.semanticSegmentation.close();
|
||||
this.maskImageEncoding.close();
|
||||
this.alphaBlending.close();
|
||||
|
||||
this.colorMapTensor.close();
|
||||
this.maskTransparencyTensor.close();
|
||||
}
|
||||
|
||||
public SemanticSegmentation(String modelUrl, int[][] colorMap, long[] labelFilter, float maskTransparency) {
|
||||
|
||||
this.imageNormalization = new GraphRunner("input_image", "resized_image")
|
||||
.withGraphDefinition(tf -> {
|
||||
Placeholder<String> input = tf.withName("input_image").placeholder(String.class);
|
||||
ExtractJpegShape<Integer> imageShapeAndChannel = tf.image.extractJpegShape(input);
|
||||
Gather<Integer> imageShape = tf.gather(imageShapeAndChannel, tf.constant(new int[] { 0, 1 }), tf.constant(0));
|
||||
|
||||
Cast<Float> maxSize = tf.dtypes.cast(tf.max(imageShape, tf.constant(0)), Float.class);
|
||||
Div<Float> scale = tf.math.div(tf.constant(REQUIRED_INPUT_IMAGE_SIZE), maxSize);
|
||||
Cast<Integer> newSize = tf.dtypes.cast(tf.math.mul(scale, tf.dtypes.cast(imageShape, Float.class)), Integer.class);
|
||||
|
||||
final Operand<Float> decodedImage =
|
||||
tf.dtypes.cast(tf.image.decodeJpeg(input, DecodeJpeg.channels(CHANNELS)), Float.class);
|
||||
|
||||
final Operand<Float> resizedImageFloat =
|
||||
tf.image.resizeBilinear(tf.expandDims(decodedImage, tf.constant(0)), newSize);
|
||||
|
||||
tf.withName("resized_image").dtypes.cast(resizedImageFloat, UInt8.class);
|
||||
});
|
||||
|
||||
this.semanticSegmentation = new GraphRunner("ImageTensor:0", "SemanticPredictions:0")
|
||||
.withGraphDefinition(new ProtoBufGraphDefinition(new DefaultResourceLoader().getResource(modelUrl), true));
|
||||
|
||||
this.colorMapTensor = Tensor.create(colorMap).expect(Integer.class);
|
||||
|
||||
this.maskImageEncoding = new GraphRunner(Arrays.asList("color_map", "mask_pixels"), Arrays.asList("mask_png", "mask_rgb"))
|
||||
.withGraphDefinition(tf -> {
|
||||
Placeholder<Integer> colorTable = tf.withName("color_map").placeholder(Integer.class);
|
||||
|
||||
Placeholder<Long> batchedMask = tf.withName("mask_pixels").placeholder(Long.class);
|
||||
// Remove batch dimension
|
||||
Squeeze<Long> mask = tf.squeeze(batchedMask, Squeeze.axis(Arrays.asList(0L)));
|
||||
|
||||
Operand<Long> filteredMask = labelFilter(tf, mask, labelFilter);
|
||||
|
||||
// The mask can contain label values larger than the list of colors provided in the color map.
|
||||
// To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
|
||||
Operand<Long> mask3 = NativeImageUtils.normalizeMaskLabels(tf, colorTable, filteredMask);
|
||||
|
||||
Gather<Integer> maskRgb = tf.withName("mask_rgb").gather(colorTable, mask3, tf.constant(0));
|
||||
|
||||
Operand<String> png = tf.withName("mask_png").image.encodePng(tf.dtypes.cast(maskRgb, UInt8.class));
|
||||
|
||||
});
|
||||
|
||||
this.maskTransparencyTensor = Tensor.create(maskTransparency).expect(Float.class);
|
||||
|
||||
this.alphaBlending = new GraphRunner(
|
||||
Arrays.asList("input_image", "mask_image", "mask_transparency"), Arrays.asList("blended_png"))
|
||||
.withGraphDefinition(tf -> {
|
||||
// Input image [B, H, W, 3]
|
||||
Cast<Float> inputImageRgb = tf.dtypes.cast(tf.withName("input_image").placeholder(UInt8.class), Float.class);
|
||||
|
||||
Placeholder<Integer> a = tf.withName("mask_image").placeholder(Integer.class);
|
||||
Cast<Float> maskRgb = tf.dtypes.cast(a, Float.class);
|
||||
|
||||
Squeeze<Float> inputImageRgb2 = tf.squeeze(inputImageRgb, Squeeze.axis(Arrays.asList(0L)));
|
||||
|
||||
Placeholder<Float> maskTransparencyHolder = tf.withName("mask_transparency").placeholder(Float.class);
|
||||
|
||||
// Blend the transparent maskImage on top of the input image.
|
||||
Operand<Float> blended = NativeImageUtils.alphaBlending(tf, maskRgb, inputImageRgb2, maskTransparencyHolder);
|
||||
|
||||
// Cut
|
||||
//Operand<Boolean> condition = tf.math.equal(a, tf.zerosLike(a));
|
||||
//Operand<Float> blended = tf.where3(condition, tf.zerosLike(maskRgb), inputImageRgb2);
|
||||
|
||||
// Encode PNG
|
||||
tf.withName("blended_png").image.encodePng(tf.dtypes.cast(blended, UInt8.class));
|
||||
|
||||
});
|
||||
}
|
||||
|
||||
public byte[] blendMask(byte[] image) {
|
||||
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
|
||||
|
||||
Map<String, Tensor<?>> blendedTensors =
|
||||
this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
|
||||
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
|
||||
.andThen(Functions.rename("SemanticPredictions:0", "mask_pixels")) // (SemanticPredictions:0) -> (mask_pixels)
|
||||
.andThen(Functions.enrichWith("color_map", this.colorMapTensor)) // (mask_pixels) -> (mask_pixels, color_map)
|
||||
.andThen(this.maskImageEncoding).andThen(memory) // (color_map, mask_pixels) -> (mask_png, mask_rgb) and memorize (mask_png, mask_rgb)
|
||||
.andThen(Functions.enrichFromMemory(
|
||||
memory, "resized_image")) // (mask_png, mask_rgb) -> (mask_png, mask_rgb, resized_image), e.g. join the normalizedImageTensor
|
||||
.andThen(Functions.rename(
|
||||
"resized_image", "input_image",
|
||||
"mask_rgb", "mask_image")) // (mask_png, mask_rgb, resized_image) -> (mask_image, input_image)
|
||||
.andThen(Functions.enrichWith("mask_transparency", this.maskTransparencyTensor)) // (mask_image, input_image) -> (mask_image, input_image, mask_transparency)
|
||||
.andThen(this.alphaBlending).andThen(memory) // (mask_image, input_image, mask_transparency) -> (blended_png)
|
||||
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
|
||||
|
||||
byte[] blendedImage = blendedTensors.get("blended_png").bytesValue();
|
||||
|
||||
memory.getTensorMap().entrySet().stream().forEach(e -> System.out.println(e));
|
||||
|
||||
return blendedImage;
|
||||
}
|
||||
}
|
||||
|
||||
public long[][] maskPixels(byte[] image) {
|
||||
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
|
||||
|
||||
return this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
|
||||
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
|
||||
.andThen(tensorMap -> {
|
||||
Tensor<?> maskTensor = tensorMap.get("SemanticPredictions:0");
|
||||
int width = (int) maskTensor.shape()[1];
|
||||
int height = (int) maskTensor.shape()[2];
|
||||
return maskTensor.copyTo(new long[1][width][height])[0]; // 1 == batch size
|
||||
})
|
||||
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
|
||||
}
|
||||
}
|
||||
|
||||
public byte[] maskImage(byte[] image) {
|
||||
|
||||
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
|
||||
|
||||
return this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
|
||||
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
|
||||
.andThen(Functions.rename("SemanticPredictions:0", "mask_pixels")) // (SemanticPredictions:0) -> (mask_pixels)
|
||||
.andThen(Functions.enrichWith("color_map", this.colorMapTensor)) // (mask_pixels) -> (mask_pixels, color_map)
|
||||
.andThen(this.maskImageEncoding).andThen(memory) // (color_map, mask_pixels) -> (mask_png, mask_rgb) and memorize (mask_png, mask_rgb)
|
||||
.andThen(tensorMap -> tensorMap.get("mask_png").bytesValue())
|
||||
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
|
||||
}
|
||||
}
|
||||
|
||||
private Operand<Long> labelFilter(Ops tf, Operand<Long> mask, long[] labels) {
|
||||
|
||||
if (labels == null || labels.length == 0) {
|
||||
return mask;
|
||||
}
|
||||
|
||||
ZerosLike<Long> zeroMask = tf.zerosLike(mask);
|
||||
Operand<Long> result = zeroMask;
|
||||
for (long label : labels) {
|
||||
Add<Long> labelMask = tf.math.add(tf.zerosLike(mask), tf.constant(label));
|
||||
Equal condition = tf.math.equal(mask, labelMask);
|
||||
result = tf.math.add(result, tf.where3(condition, labelMask, zeroMask));
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws IOException {
|
||||
|
||||
//String inputImageUri = "file:/Users/ctzolov/Dev/projects/mindmodel/mind-model-services/semantic-segmentation/src/test/resources/images/VikiMaxiAdi.jpg";
|
||||
String outputBlendedImagePath = "./semantic-segmentation/target/blendedImage.png";
|
||||
String outputMaskImagePath = "./semantic-segmentation/target/maskImage.png";
|
||||
|
||||
|
||||
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
|
||||
"http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb",
|
||||
SegmentationColorMap.CITYMAP_COLORMAP, null, 0.45f)
|
||||
) {
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/amsterdam-cityscape1.jpg");
|
||||
|
||||
// 1. Mask pixels
|
||||
long[][] maskPixels = segmentationService.maskPixels(inputImage);
|
||||
|
||||
// 2. Alpha Blending
|
||||
byte[] blended = segmentationService.blendMask(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", new File(outputBlendedImagePath));
|
||||
|
||||
// 3. Mask Image
|
||||
byte[] maskImage = segmentationService.maskImage(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", new File(outputMaskImagePath));
|
||||
}
|
||||
|
||||
|
||||
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
|
||||
"http://download.tensorflow.org/models/deeplabv3_xception_ade20k_train_2018_05_29.tar.gz#frozen_inference_graph.pb",
|
||||
SegmentationColorMap.ADE20K_COLORMAP, null, 0.45f)
|
||||
) {
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/interior.jpg");
|
||||
|
||||
// 1. Mask pixels
|
||||
long[][] maskPixels = segmentationService.maskPixels(inputImage);
|
||||
|
||||
// 2. Alpha Blending
|
||||
byte[] blended = segmentationService.blendMask(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png",
|
||||
new File("./semantic-segmentation/target/inventory-blendedImage.png"));
|
||||
|
||||
// 3. Mask Image
|
||||
byte[] maskImage = segmentationService.maskImage(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png",
|
||||
new File("./semantic-segmentation/target/inventory-MaskImage.png"));
|
||||
}
|
||||
|
||||
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
|
||||
"http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb",
|
||||
SegmentationColorMap.BLACK_WHITE_COLORMAP, null, 0.45f)
|
||||
) {
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg");
|
||||
|
||||
// 1. Mask pixels
|
||||
long[][] maskPixels = segmentationService.maskPixels(inputImage);
|
||||
|
||||
// 2. Alpha Blending
|
||||
byte[] blended = segmentationService.blendMask(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png",
|
||||
new File("./semantic-segmentation/target/pascal-blendedImage.png"));
|
||||
|
||||
// 3. Mask Image
|
||||
byte[] maskImage = segmentationService.maskImage(inputImage);
|
||||
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png",
|
||||
new File("./semantic-segmentation/target/pascal-MaskImage.png"));
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.semantic.segmentation.attic;
|
||||
|
||||
import java.awt.Color;
|
||||
import java.awt.Graphics2D;
|
||||
import java.awt.Image;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.awt.image.DataBufferByte;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.util.Arrays;
|
||||
import java.util.Base64;
|
||||
import java.util.Collections;
|
||||
import java.util.Map;
|
||||
|
||||
import javax.imageio.ImageIO;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.tensorflow.Tensor;
|
||||
import org.tensorflow.types.UInt8;
|
||||
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.TensorFlowService;
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
|
||||
import static java.awt.image.BufferedImage.TYPE_3BYTE_BGR;
|
||||
|
||||
/**
|
||||
*
|
||||
* Semantic image segmentation - the task of assigning a semantic label, such as “road”, “sky”, “person”, “dog”, to
|
||||
* every pixel in an image.
|
||||
*
|
||||
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html
|
||||
* https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
|
||||
* https://github.com/tensorflow/models/tree/master/research/deeplab
|
||||
* https://github.com/tensorflow/models/blob/master/research/deeplab/deeplab_demo.ipynb
|
||||
* http://presentations.cocodataset.org/Places17-GMRI.pdf
|
||||
*
|
||||
* http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html
|
||||
* https://www.cityscapes-dataset.com/dataset-overview/#class-definitions
|
||||
* http://groups.csail.mit.edu/vision/datasets/ADE20K/
|
||||
*
|
||||
* https://github.com/mapillary/inplace_abn
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class SemanticSegmentationUtils {
|
||||
|
||||
/** INPUT_TENSOR_NAME . */
|
||||
public static final String INPUT_TENSOR_NAME = "ImageTensor:0";
|
||||
/** OUTPUT_TENSOR_NAME . */
|
||||
public static final String OUTPUT_TENSOR_NAME = "SemanticPredictions:0";
|
||||
|
||||
private static final int BATCH_SIZE = 1;
|
||||
private static final long CHANNELS = 3;
|
||||
private static final int REQUIRED_INPUT_IMAGE_SIZE = 513;
|
||||
|
||||
public static BufferedImage scaledImage(String imagePath) {
|
||||
try {
|
||||
return scaledImage(ImageIO.read(new DefaultResourceLoader().getResource(imagePath).getInputStream()));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new IllegalStateException("Failed to load Image from: " + imagePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
public static BufferedImage scaledImage(byte[] image) {
|
||||
try {
|
||||
return scaledImage(ImageIO.read(new ByteArrayInputStream(image)));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new IllegalStateException("Failed to load Image from byte array", e);
|
||||
}
|
||||
}
|
||||
|
||||
public static BufferedImage scaledImage(BufferedImage image) {
|
||||
double scaleRatio = 1.0 * REQUIRED_INPUT_IMAGE_SIZE / Math.max(image.getWidth(), image.getHeight());
|
||||
return scale(image, scaleRatio);
|
||||
}
|
||||
|
||||
private static BufferedImage scale(BufferedImage originalImage, double scale) {
|
||||
int newWidth = (int) (originalImage.getWidth() * scale);
|
||||
int newHeight = (int) (originalImage.getHeight() * scale);
|
||||
|
||||
Image tmpImage = originalImage.getScaledInstance(newWidth, newHeight, Image.SCALE_DEFAULT);
|
||||
//BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, TYPE_INT_BGR);
|
||||
BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, TYPE_3BYTE_BGR);
|
||||
//BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, originalImage.getType());
|
||||
|
||||
Graphics2D g2d = resizedImage.createGraphics();
|
||||
g2d.drawImage(tmpImage, 0, 0, null);
|
||||
g2d.dispose();
|
||||
|
||||
return resizedImage;
|
||||
}
|
||||
|
||||
public static BufferedImage blendMask(BufferedImage mask, BufferedImage background) {
|
||||
GraphicsUtils.overlayImages(background, mask, 0, 0);
|
||||
return background;
|
||||
}
|
||||
|
||||
public static Tensor<UInt8> createInputTensor(BufferedImage scaledImage) {
|
||||
if (scaledImage.getType() != TYPE_3BYTE_BGR) {
|
||||
throw new IllegalArgumentException(
|
||||
String.format("Expected 3-byte BGR encoding in BufferedImage, found %d", scaledImage.getType()));
|
||||
}
|
||||
|
||||
// ImageIO.read produces BGR-encoded images, while the model expects RGB.
|
||||
byte[] data = bgrToRgb(toBytes(scaledImage));
|
||||
|
||||
// Expand dimensions since the model expects images to have shape: [1, None, None, 3]
|
||||
long[] shape = new long[] { BATCH_SIZE, scaledImage.getHeight(), scaledImage.getWidth(), CHANNELS };
|
||||
|
||||
return Tensor.create(UInt8.class, shape, ByteBuffer.wrap(data));
|
||||
}
|
||||
|
||||
private static byte[] bgrToRgb(byte[] brgImage) {
|
||||
byte[] rgbImage = new byte[brgImage.length];
|
||||
for (int i = 0; i < brgImage.length; i += 3) {
|
||||
rgbImage[i] = brgImage[i + 2];
|
||||
rgbImage[i + 1] = brgImage[i + 1];
|
||||
rgbImage[i + 2] = brgImage[i];
|
||||
}
|
||||
return rgbImage;
|
||||
}
|
||||
|
||||
private static byte[] toBytes(BufferedImage bufferedImage) {
|
||||
return ((DataBufferByte) bufferedImage.getRaster().getDataBuffer()).getData();
|
||||
}
|
||||
|
||||
public static BufferedImage createMaskImage(int[][] maskPixels, int width, int height, double transparency) {
|
||||
|
||||
maskPixels = rotate(maskPixels);
|
||||
|
||||
int maskWidth = maskPixels.length;
|
||||
int maskHeight = maskPixels[0].length;
|
||||
|
||||
int[] maskArray = new int[maskWidth * maskHeight];
|
||||
int k = 0;
|
||||
for (int i = 0; i < maskHeight; i++) {
|
||||
for (int j = 0; j < maskWidth; j++) {
|
||||
Color c = (maskPixels[j][i] == 0) ? Color.BLACK : GraphicsUtils.getClassColor(maskPixels[j][i]);
|
||||
int t = (int) (255 * (1 - transparency));
|
||||
maskArray[k++] = new Color(c.getRed(), c.getGreen(), c.getBlue(), t).getRGB();
|
||||
}
|
||||
}
|
||||
|
||||
// Turn the pixel array into image;
|
||||
BufferedImage maskImage = new BufferedImage(maskWidth, maskHeight, BufferedImage.TYPE_INT_ARGB);
|
||||
maskImage.setRGB(0, 0, maskWidth, maskHeight, maskArray, 0, maskWidth);
|
||||
|
||||
// Stretch the image to fit the target box width and height!
|
||||
return GraphicsUtils.toBufferedImage(maskImage.getScaledInstance(width, height, Image.SCALE_SMOOTH));
|
||||
}
|
||||
|
||||
/**
|
||||
* rotate clockwise in 90 degree.
|
||||
* @param input The 2D matrix to be rotated
|
||||
* @return The input matrix rotated clockwise in 90 degrees
|
||||
*/
|
||||
private static int[][] rotate(int[][] input) {
|
||||
|
||||
int w = input.length;
|
||||
int h = input[0].length;
|
||||
|
||||
int[][] output = new int[h][w];
|
||||
for (int y = 0; y < h; y++) {
|
||||
for (int x = w - 1; x >= 0; x--) {
|
||||
output[y][x] = input[x][y];
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
public static int[][] toIntArray(long[][] longArray) {
|
||||
int[][] intArray = new int[longArray.length][longArray[0].length];
|
||||
for (int i = 0; i < longArray.length; i++) {
|
||||
for (int j = 0; j < longArray[0].length; j++) {
|
||||
intArray[i][j] = (int) longArray[i][j];
|
||||
}
|
||||
}
|
||||
return intArray;
|
||||
}
|
||||
|
||||
public String serializeToJson(int[][] pixels) {
|
||||
String masksBase64 = Base64.getEncoder().encodeToString(toBytes(pixels));
|
||||
return String.format("{ \"columns\":%d, \"rows\":%d, \"masks\":\"%s\"}", pixels.length, pixels[0].length, masksBase64);
|
||||
}
|
||||
|
||||
public int[][] deserializeToMasks(String json) throws IOException {
|
||||
Map<String, Object> map = new ObjectMapper().readValue(json, Map.class);
|
||||
int cols = (int) map.get("columns");
|
||||
int rows = (int) map.get("rows");
|
||||
String masksBase64 = (String) map.get("masks");
|
||||
byte[] masks = Base64.getDecoder().decode(masksBase64);
|
||||
return toInts(masks, cols, rows);
|
||||
}
|
||||
|
||||
private byte[] toBytes(int[][] pixels) {
|
||||
byte[] b = new byte[pixels.length * pixels[0].length * 4];
|
||||
int bi = 0;
|
||||
for (int i = 0; i < pixels.length; i++) {
|
||||
for (int j = 0; j < pixels[0].length; j++) {
|
||||
b[bi + 0] = (byte) (i >> 24);
|
||||
b[bi + 1] = (byte) (i >> 16);
|
||||
b[bi + 2] = (byte) (i >> 8);
|
||||
b[bi + 3] = (byte) (i /*>> 0*/);
|
||||
bi = bi + 4;
|
||||
}
|
||||
}
|
||||
return b;
|
||||
}
|
||||
|
||||
private int[][] toInts(byte[] b, int ic, int jc) {
|
||||
int[][] intResult = new int[ic][jc];
|
||||
int bi = 0;
|
||||
for (int i = 0; i < ic; i++) {
|
||||
for (int j = 0; j < jc; j++) {
|
||||
intResult[i][j] = (b[bi] << 24) +
|
||||
(b[bi + 1] << 16) +
|
||||
(b[bi + 2] << 8) +
|
||||
b[bi + 3];
|
||||
bi = bi + 4;
|
||||
}
|
||||
}
|
||||
return intResult;
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws IOException {
|
||||
|
||||
// PASCAL VOC 2012
|
||||
//String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_mnv2_pascal_train_aug/frozen_inference_graph.pb";
|
||||
//String imagePath = "classpath:/images/VikiMaxiAdi.jpg";
|
||||
|
||||
// CITYSCAPE
|
||||
//String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_mnv2_cityscapes_train/frozen_inference_graph.pb";
|
||||
//String imagePath = "classpath:/images/amsterdam-cityscape1.jpg";
|
||||
//String imagePath = "classpath:/images/amsterdam-channel.jpg";
|
||||
//String imagePath = "classpath:/images/landsmeer.png";
|
||||
|
||||
// ADE20K
|
||||
String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_xception_ade20k_train/frozen_inference_graph.pb";
|
||||
String imagePath = "classpath:/images/interior.jpg";
|
||||
|
||||
BufferedImage inputImage = ImageIO.read(new DefaultResourceLoader().getResource(imagePath).getInputStream());
|
||||
|
||||
TensorFlowService tf = new TensorFlowService(new DefaultResourceLoader().getResource(tensorflowModelLocation), Arrays.asList(OUTPUT_TENSOR_NAME));
|
||||
|
||||
SemanticSegmentationUtils segmentationService = new SemanticSegmentationUtils();
|
||||
|
||||
BufferedImage scaledImage = segmentationService.scaledImage(inputImage);
|
||||
|
||||
Tensor<UInt8> inTensor = segmentationService.createInputTensor(scaledImage);
|
||||
|
||||
Map<String, Tensor<?>> output = tf.apply(Collections.singletonMap(INPUT_TENSOR_NAME, inTensor));
|
||||
|
||||
Tensor<?> maskPixelsTensor = output.get(OUTPUT_TENSOR_NAME);
|
||||
|
||||
int height = (int) maskPixelsTensor.shape()[1];
|
||||
int width = (int) maskPixelsTensor.shape()[2];
|
||||
long[][] maskPixels = maskPixelsTensor.copyTo(new long[BATCH_SIZE][height][width])[0]; // take 0 because the batch size is 1.
|
||||
|
||||
int[][] maskPixelsInt = segmentationService.toIntArray(maskPixels);
|
||||
|
||||
BufferedImage maskImage = segmentationService.createMaskImage(maskPixelsInt, scaledImage.getWidth(), scaledImage.getHeight(), 0.35);
|
||||
|
||||
BufferedImage blended = segmentationService.blendMask(maskImage, scaledImage);
|
||||
|
||||
ImageIO.write(maskImage, "png", new File("./semantic-segmentation/target/java2Dmask.jpg"));
|
||||
ImageIO.write(blended, "png", new File("./semantic-segmentation/target/java2Dblended.jpg"));
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 184 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 243 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 236 KiB |
Reference in New Issue
Block a user