SpringCloud/spring-functions-catalog
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Tensorflow functions and applications

Browse Source 
* initial step
 * Tensorflow models functional model redesign

      -- Based on https://tzolov.github.io/mind-model-services
      -- Resolves #5

 * Add object detection processor README
 * Add image recognition processor README
 * Initial Tensorflow commonn README
 * Initial Tensorflow commonn README
 * Tensorflow common diagram
 * Tensorflow docs code
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Add semantic segmentation function. add object detecteion function readme
 * oo images
 * Furether oo readme improvments
 * Final obj detection readme fixes
 * Add image recognition readme
 * Add image recognition readme 2
 * Semantic segmentation readme
 * Segmentation readme
 * Semantic segmentation readme 3
 * Fix image recognition and object detcion app starter dependecies

 * Add metadata for Tensorflow apps
This commit is contained in:
Christian Tzolov
2020-06-11 17:07:44 +02:00
committed by Soby Chacko
parent 3bb9e066b9
commit dffb467da4
66 changed files with 10300 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
function/image-recognition-function/README.adoc Normal file
View File

@@ -0,0 +1,103 @@
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/function/image-recognition-function/src/main/resources/images/
# Image Recognition
[.lead]
Java model inference library for the https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] image recognition architectures.
Provides real-time recognition of the https://dl.bintray.com/big-data/generic/imagenet_comp_graph_label_strings.txt[LSVRC-2012-CLS categories] in the input images.
[cols="1,2",frame=none,grid=none]
|===
| image:{images-asciidoc}/image-augmented.jpg[alt=Inception 1,width=100%]
|The https://github.com/tzolov/stream-applications/tree/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition[ImageRecognition] takes an image and outputs a list of probable categories the image contains. The response is represented by https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/RecognitionResponse.java[RecognitionResponse] class.
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
converting the `RecognitionResponse` into JSON objects and the
https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionAugmenter.java[ImageRecognitionAugmenter] can augment the input image with the detected categories (as shown in pic. 1).
|===
## Usage
Add the `image-recognition` dependency to the pom (use the latest version available):
[source,xml]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>image-recognition-function</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
----
#### Example 1: Image Recognition
The following snippet demonstrates how to use the `ImageRecognition` for detecting the categories present in an input image.
It also shows how to convert the result into JSON format and augment the input image with the detected category labels.
[source,java,linenums]
----
ImageRecognition recognitionService = ImageRecognition.mobilenetModeV2(
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb", //<1>
224, //<2>
5, //<3>
true); //<4>
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg"); //<5>
List<RecognitionResponse> recognizedObjects = recognitionService.recognize(inputImage); //<6>
----
<1> Downloads and loads a pre-trained `mobilenet_v2_1.4_224_frozen.pb` model.
Mind that on first attempt it will download few hundreds of MBs.
The consecutive runs will use the cached copy (5) instead.
The category labels for the MobileNetV2 are resolved from `src/main/resources/labels/mobilenet_labels.txt`.
<2> The wxh sieze of the input nomralized image.
<3> Top K result to return.
<4> Cache the model on the local file system.
<5> Load the image to recognise.
<6> Return a map of the top-k most probable category names and their probabilities.
The `ImageRecognition.mobilenetModeV1` and `ImageRecognition.inception` factory methods help to load and configure pretrained mobilenetModeV1 and and Inception models.
Next you can convert the result in JSON format.
[source,java,linenums]
----
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
----
.Sample Image Recognition JSON representation
[source,json]
----
[{"label":"giant panda","probability":0.9946687817573547},{"label":"Arctic fox","probability":0.0036631098482757807},{"label":"ice bear","probability":3.3782739774324E-4},{"label":"American black bear","probability":2.3452856112271547E-4},{"label":"skunk","probability":1.6454080468975008E-4}]
----
Use the `ImageRecognitionAugmenter` to draw the recognise categories on top of the input image.
[source,java,linenums]
----
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects); //<1>
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented.jpg"));//<2>
----
<1> Augment the image with the recognized categories (uses Java2D internally).
<2> Stores the augmented image as `image-augmented.jpg` image file.
.Augmented image-augmented.jpg file
image:{images-asciidoc}/image-recognition-panda-augmented.jpg[alt=Augmented,width=30%]
## Models
This implementation supports all pre-trained https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] models.
Following URI notation can be used to download any of the models directly from the zoo.
----
http://<zoo model tar.gz url>#<frozen inference graph name.pb>
----
The `<frozen inference graph name.pb>` is the frozen model file name within the archive.
TIP: To speedup the bootstrap performance you may consider extracting the model and caching it locally.
Then you can use the `file://path-to-my-local-copy` URI schema to access it.
NOTE: It is important to use the labels that correspond to the model being used!
Table below highlights this mapping.

38
function/image-recognition-function/pom.xml Normal file
View File

@@ -0,0 +1,38 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>image-recognition-function</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>image-recognition-function</name>
<description>Spring Native Function for Tensorflow Integration</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<properties>
<commons-io.version>1.3.2</commons-io.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>tensorflow-common</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>tensorflow-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>

272
function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognition.java Normal file
View File

@@ -0,0 +1,272 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import org.tensorflow.Operand;
import org.tensorflow.Tensor;
import org.tensorflow.op.core.Placeholder;
import org.tensorflow.op.image.DecodeJpeg;
import org.tensorflow.op.nn.TopK;
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
import org.springframework.cloud.fn.common.tensorflow.GraphRunnerMemory;
import org.springframework.cloud.fn.common.tensorflow.ProtoBufGraphDefinition;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.util.StreamUtils;
/**
* @author Christian Tzolov
*/
public class ImageRecognition implements AutoCloseable {
private final List<String> labels;
private final GraphRunner imageNormalization;
private final GraphRunner imageRecognition;
private final GraphRunner maxProbability;
private final GraphRunner topKProbabilities;
/**
* Instead of creating the {@link ImageRecognition} service explicitly via the constructor,
* you should consider the convenience factory methods below. E.g.
*
* {@link #inception(String, int, int, boolean)}
* {@link #mobileNetV1(String, int, int, boolean)}
* {@link #mobileNetV2(String, int, int, boolean)}
*
* @param modelUri location of the pre-trained model to use.
* @param labelsUri location of the list fromMemory pre-trained categories used by the model.
* @param imageRecognitionGraphInputName name of the Model's input node to send the input image to.
* @param imageRecognitionGraphOutputName name of the Model's output node to retrieve the predictions from.
* @param imageHeight normalized image height.
* @param imageWidth normalized image width.
* @param mean mean value to normalize the input image.
* @param scale scale to normalize the input image.
* @param responseSize Max number of predictions per recognize.
* @param cacheModel if true the pre-trained model is cached on the local file system.
*/
public ImageRecognition(String modelUri, String labelsUri, int imageHeight, int imageWidth, float mean, float scale,
String imageRecognitionGraphInputName, String imageRecognitionGraphOutputName, int responseSize, boolean cacheModel) {
this.labels = labels(labelsUri);
/**
* Normalizes the raw input image into format expected by the pre-trained Inception/MobileNetV1/MobileNetV2 models.
* Typically the model is trained fromMemory images scaled to certain size. Usually it is 224x224 pixels, but can be
* also 192x192, 160x160, 128128, 92x92. Use the (imageHeight, imageWidth) to set the desired size.
* The colors, represented as R, G, B in 1-byte each were converted to float using (Value - Mean)/Scale.
*
* imageHeight normalized image height.
* imageWidth normalized image width.
* mean mean value to normalize the input image.
* scale scale to normalize the input image.
*/
this.imageNormalization = new GraphRunner("raw_image", "normalized_image")
.withGraphDefinition(tf -> {
Placeholder<String> input = tf.withName("raw_image").placeholder(String.class);
final Operand<Float> decodedImage =
tf.dtypes.cast(tf.image.decodeJpeg(input, DecodeJpeg.channels(3L)), Float.class);
final Operand<Float> resizedImage = tf.image.resizeBilinear(
tf.expandDims(decodedImage, tf.constant(0)),
tf.constant(new int[] { imageHeight, imageWidth }));
tf.withName("normalized_image").math.div(tf.math.sub(resizedImage, tf.constant(mean)), tf.constant(scale));
});
this.imageRecognition = new GraphRunner(imageRecognitionGraphInputName, imageRecognitionGraphOutputName)
.withGraphDefinition(new ProtoBufGraphDefinition(toResource(modelUri), cacheModel));
this.maxProbability = new GraphRunner(Arrays.asList("recognition_result"), Arrays.asList("category", "probability"))
.withGraphDefinition(tf -> {
Placeholder<Float> input = tf.withName("recognition_result").placeholder(Float.class);
tf.withName("category").math.argMax(input, tf.constant(1));
tf.withName("probability").max(input, tf.constant(1));
});
this.topKProbabilities = new GraphRunner("recognition_result", "topK")
.withGraphDefinition(tf -> {
Placeholder<Float> input = tf.withName("recognition_result").placeholder(Float.class);
tf.withName("topK").nn.topK(input, tf.constant(responseSize), TopK.sorted(true));
});
}
/**
* Takes an byte encoded image and returns the most probable category recognized in the image along fromMemory its probability.
* @param inputImage Byte array encoded image to recognize.
* @return Returns a single map entry containing the names of the recognized categories as key and the confidence as value.
*/
public Map<String, Double> recognizeMax(byte[] inputImage) {
try (Tensor inputTensor = Tensor.create(inputImage); GraphRunnerMemory memorize = new GraphRunnerMemory()) {
Map<String, Tensor<?>> max = this.imageNormalization.andThen(memorize)
.andThen(this.imageRecognition).andThen(memorize)
.andThen(this.maxProbability).andThen(memorize)
.apply(Collections.singletonMap("raw_image", inputTensor));
long[] category = new long[1];
max.get("category").copyTo(category);
float[] probability = new float[1];
max.get("probability").copyTo(probability);
return Collections.singletonMap(labels.get((int) category[0]), Double.valueOf(probability[0]));
}
}
/**
* Takes an byte encoded input image and returns the top K most probable categories recognized in the image
* along fromMemory their probabilities.
*
* @param inputImage Byte array encoded image to recognize.
* @return Returns a list of key-value pairs. Every key-value pair represents a single category recognized.
* The key stands for the name(s) of the category while the value states the confidence that there is an
* object of this category. The entries in the Map are ordered from the higher to the lower confidences.
*/
public Map<String, Double> recognizeTopK(byte[] inputImage) {
try (Tensor inputTensor = Tensor.create(inputImage); GraphRunnerMemory memorize = new GraphRunnerMemory()) {
Map<String, Tensor<?>> topKResults =
this.imageNormalization.andThen(memorize)
.andThen(this.imageRecognition).andThen(memorize)
.andThen(this.topKProbabilities).andThen(memorize)
.apply(Collections.singletonMap("raw_image", inputTensor));
Tensor recognizedImagesTensor = memorize.getTensorMap().get(this.imageRecognition.getSingleFetchName());
float[][] results = new float[(int) recognizedImagesTensor.shape()[0]][(int) recognizedImagesTensor.shape()[1]];
recognizedImagesTensor.copyTo(results);
Tensor<Float> topKTensor = topKResults.get("topK").expect(Float.class);
float[][] topK = new float[(int) topKTensor.shape()[0]][(int) topKTensor.shape()[1]];
topKTensor.copyTo(topK);
float min = topK[0][topK[0].length - 1];
Map<Float, Integer> valueToIndex = new HashMap<>();
for (int i = 0; i < results[0].length; i++) {
if (results[0][i] >= min) {
valueToIndex.put(results[0][i], i);
}
}
Map<String, Double> map = new LinkedHashMap<>();
for (float tk : topK[0]) {
map.put(labels.get(valueToIndex.get(tk)), (double) tk);
}
return map;
}
}
private Resource toResource(String uri) {
return new DefaultResourceLoader().getResource(uri);
}
/**
* Converts a labels resources into string list.
* @return Returns string lists. One line per different category.
*/
private List<String> labels(String labelsUri) {
try (InputStream is = toResource(labelsUri).getInputStream()) {
return Arrays.asList(StreamUtils.copyToString(is, Charset.forName("UTF-8")).split("\n"));
}
catch (IOException e) {
throw new RuntimeException("Failed to initialize the Vocabulary", e);
}
}
/**
*
* The Inception graph uses "input" as input and "output" as output.
*
*/
public static ImageRecognition inception(String inceptionModelUri,
int normalizedImageSize, int responseSize, boolean cacheModel) {
return new ImageRecognition(inceptionModelUri, "classpath:/labels/inception_labels.txt",
normalizedImageSize, normalizedImageSize, 117f, 1f,
"input", "output",
responseSize, cacheModel);
}
/**
* Convenience for MobileNetV2 pre-trained models:
* https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models
*
* The normalized image size is always square (e.g. H=W)
*
* The MobileNetV2 graph uses "input" as input and "MobilenetV2/Predictions/Reshape_1" as output.
*
* @param mobileNetV2ModelUri model uri
* @param normalizedImageSize Depends on the pre-trained model used. Usually 224px is used.
* @param responseSize Number of responses fot topK requests.
* @param cacheModel cache model
* @return ImageRecognition instance configured fromMemory a MobileNetV2 pre-trained model.
*/
public static ImageRecognition mobileNetV2(String mobileNetV2ModelUri,
int normalizedImageSize, int responseSize, boolean cacheModel) {
return new ImageRecognition(mobileNetV2ModelUri, "classpath:/labels/mobilenet_labels.txt",
normalizedImageSize, normalizedImageSize, 0f, 127f,
"input", "MobilenetV2/Predictions/Reshape_1",
responseSize, cacheModel);
}
/**
* Convenience for MobileNetV1 pre-trained models:
* https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models
*
* The MobileNetV1 graph uses "input" as input and "MobilenetV1/Predictions/Reshape_1" as output.
*
*/
public static ImageRecognition mobileNetV1(String mobileNetV1ModelUri,
int normalizedImageSize, int responseSize, boolean cacheModel) {
return new ImageRecognition(mobileNetV1ModelUri, "classpath:/labels/mobilenet_labels.txt",
normalizedImageSize, normalizedImageSize,
0f, 127f,
"input", "MobilenetV1/Predictions/Reshape_1",
responseSize, cacheModel);
}
/**
* Convert image recognition results into {@link RecognitionResponse} domain list.
* @param recognitionMap map containing the category mames and its probability. Returned by the
* {@link ImageRecognition#recognizeMax(byte[])} and the ImageRecognition{@link #recognizeTopK(byte[])} methods
* @return List of {@link RecognitionResponse} objects representing the name-to-probability pairs in the input map.
*/
public static List<RecognitionResponse> toRecognitionResponse(Map<String, Double> recognitionMap) {
return recognitionMap.entrySet().stream()
.map(nameProbabilityPair -> new RecognitionResponse(nameProbabilityPair.getKey(), nameProbabilityPair.getValue()))
.collect(Collectors.toList());
}
@Override
public void close() {
this.imageNormalization.close();
this.imageRecognition.close();
this.maxProbability.close();
this.topKProbabilities.close();
}
}

104
function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionAugmenter.java Normal file
View File

@@ -0,0 +1,104 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
import java.awt.Color;
import java.awt.FontMetrics;
import java.awt.Graphics2D;
import java.awt.RenderingHints;
import java.awt.geom.Rectangle2D;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.List;
import java.util.function.BiFunction;
import javax.imageio.ImageIO;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* Ability to to augment the input image fromMemory the recognized labels.
*
* @author Christian Tzolov
*/
public class ImageRecognitionAugmenter implements BiFunction<byte[], List<RecognitionResponse>, byte[]> {
private static final Log logger = LogFactory.getLog(ImageRecognitionAugmenter.class);
/** IMAGE_FORMAT. */
public static final String IMAGE_FORMAT = "jpg";
private final Color textColor = Color.BLACK;
private final Color bgColor = new Color(167, 252, 0);
public ImageRecognitionAugmenter() {
}
/**
* Augment the input image by adding the recognized classes.
*
* @param imageBytes input image as byte array
* @param result computed recognition labels
* @return the image augmented fromMemory recognized labels.
*/
@Override
public byte[] apply(byte[] imageBytes, List<RecognitionResponse> result) {
try {
if (result != null) {
BufferedImage originalImage = ImageIO.read(new ByteArrayInputStream(imageBytes));
Graphics2D g = originalImage.createGraphics();
g.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
FontMetrics fm = g.getFontMetrics();
int x = 1;
int y = 1;
for (RecognitionResponse r : result) {
String labelName = r.getLabel();
int probability = (int) (100 * r.getProbability());
String title = labelName + ": " + probability + "%";
Rectangle2D rect = fm.getStringBounds(title, g);
g.setColor(bgColor);
g.fillRect(x, y, (int) rect.getWidth() + 6, (int) rect.getHeight());
g.setColor(textColor);
g.drawString(title, x + 3, (int) (y + rect.getHeight() - 3));
y = (int) (y + rect.getHeight() + 1);
}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(originalImage, IMAGE_FORMAT, baos);
baos.flush();
imageBytes = baos.toByteArray();
baos.close();
}
}
catch (IOException e) {
logger.error("Failed to draw labels in the input image", e);
}
return imageBytes;
}
}

54
function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/RecognitionResponse.java Normal file
View File

@@ -0,0 +1,54 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
/**
* @author Christian Tzolov
*/
public class RecognitionResponse {
private String label;
private Double probability;
public RecognitionResponse() {
}
public RecognitionResponse(String label, Double probability) {
this.label = label;
this.probability = probability;
}
public String getLabel() {
return label;
}
public void setLabel(String label) {
this.label = label;
}
public Double getProbability() {
return probability;
}
public void setProbability(Double probability) {
this.probability = probability;
}
@Override
public String toString() {
return "{label='" + label + ", probability=" + probability + '}';
}
}

105
function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/util/ImageNetReadableNamesWriter.java Normal file
View File

@@ -0,0 +1,105 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition.util;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.commons.io.FileUtils;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.util.Assert;
import org.springframework.util.StreamUtils;
/**
* Create a text file mapping label id to human readable string.
*
* Produces a text file where every line represents single category. The line number represents the category id, while
* the line text is human-readable names for the categories fromMemory this imagenet id.
*
* Based on https://github.com/tensorflow/models/blob/master/research/slim/datasets/imagenet.py#L66
*
* We retrieve a synset file, which contains a list of valid synset labels used
* by ILSVRC competition. There is one synset one per line, eg.
* # n01440764
* # n01443537
* We also retrieve a synset_to_human_file, which contains a mapping from synsets
* to human-readable names for every synset in Imagenet. These are stored in a
* tsv format, as follows:
* # n02119247 black fox
* # n02119359 silver fox
* We assign each synset (in alphabetical order) an integer, starting from 1
* (since 0 is reserved for the background class)
*
* @author Christian Tzolov
*/
public final class ImageNetReadableNamesWriter {
private ImageNetReadableNamesWriter() {
}
/** BASE_URL. */
public final static String BASE_URL = "https://raw.githubusercontent.com/tensorflow/models/master/research/inception/inception/data/";
/** SYNSET_URI. */
public final static String SYNSET_URI = BASE_URL + "imagenet_lsvrc_2015_synsets.txt";
/** SYNSET_TO_HUMAN_URI. */
public final static String SYNSET_TO_HUMAN_URI = BASE_URL + "imagenet_metadata.txt";
public static void main(String[] args) {
Charset utf8 = Charset.forName("UTF-8");
try (InputStream synsetIs = toResource(SYNSET_URI).getInputStream();
InputStream synsetToHumanIs = toResource(SYNSET_TO_HUMAN_URI).getInputStream()) {
List<String> synsetList = Arrays.asList(StreamUtils.copyToString(synsetIs, utf8)
.split("\n")).stream().map(l -> l.trim()).collect(Collectors.toList());
Assert.notNull(synsetList, "Failed to initialize the labels list");
Assert.isTrue(synsetList.size() == 1000, "Labels list is expected to be of " +
"size 1000 but was:" + synsetList.size());
Map<String, String> synsetToHuman = Arrays.asList(StreamUtils.copyToString(synsetToHumanIs, utf8)
.split("\n")).stream().map(s2h -> s2h.split("\t")).collect(Collectors.toMap(s -> s[0], s -> s[1]));
Assert.notNull(synsetToHuman, "Failed to initialize the synsetToHuman");
Assert.isTrue(synsetToHuman.size() == 21842, "synsetToHuman is expected to be of " +
"size 21842 but was:" + synsetToHuman.size());
List<String> l = synsetList.stream().map(id -> synsetToHuman.get(id)).collect(Collectors.toList());
List<String> ll = new ArrayList<>();
ll.add("dummy");
ll.addAll(l);
System.out.println(ll.get(389));
FileUtils.writeLines(new File("labels.txt"), ll);
}
catch (IOException e) {
throw new RuntimeException("Failed to initialize the Vocabulary", e);
}
}
public static Resource toResource(String uri) {
return new DefaultResourceLoader().getResource(uri);
}
}

BIN
function/image-recognition-function/src/main/resources/images/image-augmented.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
function/image-recognition-function/src/main/resources/images/image-recognition-panda-augmented.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 101 KiB

1001
function/image-recognition-function/src/main/resources/labels/inception_labels.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1001
function/image-recognition-function/src/main/resources/labels/mobilenet_labels.txt Normal file
View File

File diff suppressed because it is too large Load Diff

103
function/image-recognition-function/src/test/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionExample.java Normal file
View File

@@ -0,0 +1,103 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.List;
import org.apache.commons.io.IOUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction;
/**
* @author Christian Tzolov
*/
public final class ImageRecognitionExample {
private ImageRecognitionExample() {
}
public static void main(String[] args) throws IOException {
// You can use file:, http: or classpath: to provide the path to the input image.
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg");
// MmobileNetV2 models
// https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models
String mobilenet_v2_modelUri = "https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb";
//String mobilenet_v2_modelUri = "https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_96.tgz#mobilenet_v2_0.35_96_frozen.pb";
try (ImageRecognition imageRecognition = ImageRecognition.mobileNetV2(
mobilenet_v2_modelUri,
224,
5,
true)) {
List<RecognitionResponse> recognizedObjects =
ImageRecognition.toRecognitionResponse(imageRecognition.recognizeTopK(inputImage));
// Draw the predicted labels on top of the input image.
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-mobilnetV2.jpg"));
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
System.out.println("mobilnetV2 result:" + jsonRecognizedObjects);
}
String mobilenet_v1_modelUri = "https://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz#mobilenet_v1_1.0_224_frozen.pb";
try (ImageRecognition recognitionService = ImageRecognition.mobileNetV1(
mobilenet_v1_modelUri,
224,
5,
true)) {
List<RecognitionResponse> recognizedObjects =
ImageRecognition.toRecognitionResponse(recognitionService.recognizeTopK(inputImage));
// Draw the predicted labels on top of the input image.
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-mobilnetV1.jpg"));
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
System.out.println("mobilnetV1 result:" + jsonRecognizedObjects);
}
String inception_modelUri = "https://storage.googleapis.com/scdf-tensorflow-models/image-recognition/tensorflow_inception_graph.pb";
try (ImageRecognition recognitionService = ImageRecognition.inception(
inception_modelUri,
224,
5,
true)) {
List<RecognitionResponse> recognizedObjects =
ImageRecognition.toRecognitionResponse(recognitionService.recognizeTopK(inputImage));
// Draw the predicted labels on top of the input image.
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-inception.jpg"));
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
System.out.println("inception result:" + jsonRecognizedObjects);
}
}
}

70
function/image-recognition-function/src/test/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionExample2.java Normal file
View File

@@ -0,0 +1,70 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.commons.io.IOUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
/**
* @author Christian Tzolov
*/
public final class ImageRecognitionExample2 {
private ImageRecognitionExample2() {
}
public static void main(String[] args) throws IOException {
ImageRecognitionAugmenter augmenter = new ImageRecognitionAugmenter();
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg");
ImageRecognition inceptions = ImageRecognition.inception(
"https://storage.googleapis.com/scdf-tensorflow-models/image-recognition/tensorflow_inception_graph.pb",
224, 10, true);
System.out.println(inceptions.recognizeMax(inputImage));
System.out.println(inceptions.recognizeTopK(inputImage));
System.out.println(ImageRecognition.toRecognitionResponse(inceptions.recognizeTopK(inputImage)));
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(inceptions.recognizeTopK(inputImage))),
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-inceptions.jpg"));
inceptions.close();
ImageRecognition mobileNetV2 = ImageRecognition.mobileNetV2(
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb",
224, 10, true);
System.out.println(mobileNetV2.recognizeMax(inputImage));
System.out.println(mobileNetV2.recognizeTopK(inputImage));
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(mobileNetV2.recognizeTopK(inputImage))),
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-mobilnetV2.jpg"));
mobileNetV2.close();
ImageRecognition mobileNetV1 = ImageRecognition.mobileNetV1(
"https://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz#mobilenet_v1_1.0_224_frozen.pb",
224, 10, true);
System.out.println(mobileNetV1.recognizeMax(inputImage));
System.out.println(mobileNetV1.recognizeTopK(inputImage));
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(mobileNetV1.recognizeTopK(inputImage))),
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-mobilnetV1.jpg"));
mobileNetV1.close();
}
}

63
function/image-recognition-function/src/test/java/org/springframework/cloud/fn/image/recognition/SavedModelTest.java Normal file
View File

@@ -0,0 +1,63 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.image.recognition;
import java.util.Map;
import com.google.protobuf.InvalidProtocolBufferException;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.framework.MetaGraphDef;
import org.tensorflow.framework.SignatureDef;
/**
* @author Christian Tzolov
*/
public final class SavedModelTest {
private SavedModelTest() {
}
/**
* https://medium.com/@jsflo.dev/saving-and-loading-a-tensorflow-model-using-the-savedmodel-api-17645576527
*
* https://www.tensorflow.org/alpha/guide/saved_model
*
*/
public static void main(String[] args) throws InvalidProtocolBufferException {
SavedModelBundle savedModelBundle =
SavedModelBundle.load("/Users/ctzolov/Downloads/ssd_mobilenet_v1_coco_2017_11_17/saved_model", "serve");
//SavedModelBundle.load("/Users/ctzolov/Downloads/aiy_vision_classifier_plants_V1_1/", "serve");
//SavedModelBundle savedModelBundle =
// SavedModelBundle.load("/Users/ctzolov/Downloads/mnasnet-a1/saved_model", "serve");
MetaGraphDef meta = MetaGraphDef.parseFrom(savedModelBundle.metaGraphDef());
Map<String, SignatureDef> signatures = meta.getSignatureDefMap();
System.out.println(signatures);
savedModelBundle.session();
//Iterator<Operation> itr = savedModelBundle.graph().operations();
//
//while (itr.hasNext()) {
// System.out.println("Operation: " + itr.next());
//}
}
}

BIN
function/image-recognition-function/src/test/resources/images/giant_panda_in_beijing_zoo_1.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

BIN
function/image-recognition-function/src/test/resources/images/panda.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

189
function/object-detection-function/README.adoc Normal file
View File

@@ -0,0 +1,189 @@
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/function/object-detection-function/src/main/resources/images/
# Object Detection Function
Java model inference library for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. Allows real-time localization and identification of multiple objects in a single or batch of images. Works with all https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[pre-trained zoo models] and ttps://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels].
[cols="1,2", frame=none, grid=none]
|===
| image:{images-asciidoc}/object_detection_1.jpg[alt=Object Detection 1, width=100%]
|The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionService.java[ObjectDetectionService]
takes an image or a batch of images and outputs a list of predicted objects bounding boxes
represented by https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/domain/ObjectDetection.java[ObjectDetection].
For the models supporting https://github.com/tensorflow/models/tree/master/research/object_detection#february-9-2018[Instance Segmentation],
the `ObjectDetectionService` can predict the instance segmentation `masks` in addition to object bounding boxes.
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
converting the `List<ObjectDetection>` into JSON objects and the
https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
allow to augment the input image with the detected bounding boxes and segmentation masks.
|===
## Usage
Add the `object-detection` dependency to the pom (use the latest version available):
[source,xml]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>object-detection-function</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
----
#### Example 1: Object Detection
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleObjectDetection.java[ExampleObjectDetection.java]
sample demonstrates how to use the `ObjectDetectionService` for detecting objects in input images. It also shows how to
convert the result into JSON format and augment the input image with the detected object bounding boxes.
[source,java,linenums]
----
ObjectDetectionService detectionService = new ObjectDetectionService(
"http://download.tensorflow.org/models/object_detection/faster_rcnn_nas_coco_2018_01_28.tar.gz#frozen_inference_graph.pb", //<1>
"https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt", //<2>
0.4f, //<3>
false, //<4>
true); //<5>
byte[] image = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg"); //<6>
List<ObjectDetection> detectedObjects = detectionService.detect(image); //<7>
----
<1> Downloads and loads a pre-trained `frozen_inference_graph.pb` model directly from the `faster_rcnn_nas_coco.tar.gz` archive in the
Tensorflow model zoo. Mind that on first attempt it will download few hundreds of MBs. The consecutive runs will use the
cached copy (5) instead.
<2> Object category labels (e.g. names) for the model
<3> Confidence threshold - Only object with estimate above the threshold are returned
<4> Indicate that this is not a `mask` (e.g. not an instance segmentation) model type
<5> Cache the model on the local file system.
<6> Load the input image to evaluate
<7> Detect the objects in the image and represent the result as a list of ObjectDetection instances.
Next you can convert the result in JSON format.
[source,java,linenums]
----
String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects);
System.out.println(jsonObjectDetections);
----
.Sample Object Detection JSON representation
[source,json]
----
[{"name":"person","estimate":0.998,"x1":0.160,"y1":0.774,"x2":0.201,"y2":0.946,"cid":1},
{"name":"kite","estimate":0.998,"x1":0.437,"y1":0.089,"x2":0.495,"y2":0.169,"cid":38},
{"name":"person","estimate":0.997,"x1":0.084,"y1":0.681,"x2":0.121,"y2":0.848,"cid":1},
{"name":"kite","estimate":0.988,"x1":0.206,"y1":0.263,"x2":0.225,"y2":0.314,"cid":38}]]
----
Use the https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
to draw the detected objects on top of the input image.
[source,java,linenums]
----
byte[] annotatedImage = new ObjectDetectionImageAugmenter().apply(image, detectedObjects); // <1>
IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-augmented.jpg")); //<2>
----
<1> Augment the image with the detected object bounding boxes (Uses Java2D internally).
<2> Stores the augmented image as `object-detection-augmented.jpg` image file.
.Augmented object-detection-augmented.jpg file
image:{images-asciidoc}/object-detection-augmented.jpg[alt=Object Detection, width=60%]
TIP: Set the `ObjectDetectionImageAugmenter#agnosticColors` property to `true` to use a monochrome color schema.
#### Example 2: Instance Segmentation
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleInstanceSegmentation.java[ExampleInstanceSegmentation.java]
sample shows how to use the `ObjectDetectionService` for `Instance Segmentation`.
NOTE: It requires a trained model that supports `Masks` as well as setting the instance segmentation (e.g. `useMasks`) flag to `true`.
[source,java,linenums]
----
ObjectDetectionService detectionService = new ObjectDetectionService(
"http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz#frozen_inference_graph.pb", // <1>
"https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt", // <2>
0.4f, // <3>
true, // <4>
true); // <5>
byte[] image = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg");
List<ObjectDetection> detectedObjects = detectionService.detect(image); // <6>
String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects); // <7>
System.out.println(jsonObjectDetections);
byte[] annotatedImage = new ObjectDetectionImageAugmenter(true) // <8>
.apply(image, detectedObjects);
IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-segmentation-augmented.jpg"));
----
<1> Uses one of the 4 MASK pre-trained models
<2> Object category labels (e.g. names) for the model
<3> Confidence threshold - Only object with estimate above the threshold are returned.
<4> Use masks output - For the pre-trained models instruct to use the extended fetch names that include instance segmentation masks as well.
<5> Cache model - Create a local copy of the model to speed up consecutive runs.
<6> Evaluate the model to predict the object in the input image.
<7> Convert the detected object in to JSON array. NOTE: that with mask there is an additional field: `mask`
<8> Draw the detected object on top of the input image. Mind the `true` constructor parameter stands for draw detected masks.
If false only the bounding boxes will be shown.
.Result augmented object-detection-segmentation-augmented.jpg file
image:{images-asciidoc}/object-detection-segmentation-augmented.jpg[alt=Object Detection Augmented, width=60%]
## Models
All pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[detection_model_zoo.md]
models are supported. Following URI notation can be used to download any of the models directly from the zoo.
----
http://<zoo model tar.gz url>#frozen_inference_graph.pb
----
The `frozen_inference_graph.pb` is the frozen model file name within the archive.
NOTE: For some models this name may differ. You have to download and open the archive to find the real name.
TIP: To speedup the bootstrap performance you may consider extracting the `frozen_inference_graph.pb` and caching it
locally. Then you can use the `file://path-to-my-local-copy` URI schema to access it.
Following models can be used for `Instance Segmentation` as well:
[frame=none, grid=none]
|===
| http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz[mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz]
| http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz[mask_rcnn_inception_v2_coco_2018_01_28.tar.gz]
| http://download.tensorflow.org/models/object_detection/mask_rcnn_resnet101_atrous_coco_2018_01_28.tar.gz[mask_rcnn_resnet101_atrous_coco_2018_01_28.tar.gz]
| http://download.tensorflow.org/models/object_detection/mask_rcnn_resnet50_atrous_coco_2018_01_28.tar.gz[mask_rcnn_resnet50_atrous_coco_2018_01_28.tar.gz]
|===
In addition to the model, the `ObjectDetectionService` requires a list of labels that correspond to the categories detectable by the selected model.
All labels files are available in the https://github.com/tensorflow/models/tree/master/research/object_detection/data[object_detection/data] folder.
NOTE: It is important to use the labels that correspond to the model being used! Table below highlights this mapping.
.Relationsip between trained model types and category labels
[%header, cols="1,2", frame=none, grid=none]
|===
| Model
| Labels
| https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#coco-trained-models[coco]
| https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt[mscoco_label_map.pbtxt]
| https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#kitti-trained-models[kitti]
| https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/kitti_label_map.pbtxt[kitti_label_map.pbtxt]
| https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#open-images-trained-models[open-images]
| https://github.com/tensorflow/models/blob/master/research/object_detection/data/oid_bbox_trainable_label_map.pbtxt[oid_bbox_trainable_label_map.pbtxt]
| https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#inaturalist-species-trained-models[inaturalist-species]
| https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/fgvc_2854_classes_label_map.pbtxt[fgvc_2854_classes_label_map.pbtxt]
| https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#ava-v21-trained-models[ava]
| https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/ava_label_map_v2.1.pbtxt[ava_label_map_v2.1.pbtxt]
|===
TIP: For performance reasons you may consider downloading the required label files to the local file system.

72
function/object-detection-function/pom.xml Normal file
View File

@@ -0,0 +1,72 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>object-detection-function</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>object-detection-function</name>
<description>Spring Native Function for Tensorflow Integration</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>tensorflow-common</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.tensorflow</groupId>-->
<!-- <artifactId>tensorflow-hadoop</artifactId>-->
<!-- <version>1.15.0</version>-->
<!-- </dependency>-->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<!-- NOTE upgrading with versions above 3.8.0 generates incompatible for protobuf-java:3.5.1 comming with TF 1.15.0 -->
<version>3.7.0</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<!-- <includeDirectories> <include>src/main/protobuf</include> </includeDirectories> -->
<inputDirectories>
<include>src/main/proto</include>
</inputDirectories>
<!-- Create java files. And put them in the src/main/java directory. -->
<outputTargets>
<outputTarget>
<type>java</type>
<outputDirectory>src/main/java</outputDirectory>
</outputTarget>
</outputTargets>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<!-- <repositories>-->
<!-- <repository>-->
<!-- <id>tensorflow-snapshots</id>-->
<!-- <url>https://oss.sonatype.org/content/repositories/snapshots/</url>-->
<!-- <snapshots>-->
<!-- <enabled>true</enabled>-->
<!-- </snapshots>-->
<!-- </repository>-->
<!-- </repositories>-->
</project>

119
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java Normal file
View File

@@ -0,0 +1,119 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.awt.Color;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.List;
import java.util.function.BiFunction;
import javax.imageio.ImageIO;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.util.CollectionUtils;
/**
* Augment the input image fromMemory detected object bounding boxes and categories.
* For mask models and withMask set to true it draws the instance segmentation image as well.
*
* @author Christian Tzolov
*/
public class ObjectDetectionImageAugmenter implements BiFunction<byte[], List<ObjectDetection>, byte[]> {
private static final Log logger = LogFactory.getLog(ObjectDetectionImageAugmenter.class);
/** Make checkstyle happy. **/
public static final String DEFAULT_IMAGE_FORMAT = "jpg";
private String imageFormat = DEFAULT_IMAGE_FORMAT;
private final boolean withMask;
private boolean agnosticColors = false;
public ObjectDetectionImageAugmenter() {
this(false);
}
public ObjectDetectionImageAugmenter(boolean withMask) {
this.withMask = withMask;
}
public boolean isAgnosticColors() {
return agnosticColors;
}
public void setAgnosticColors(boolean agnosticColors) {
this.agnosticColors = agnosticColors;
}
public String getImageFormat() {
return imageFormat;
}
public void setImageFormat(String imageFormat) {
this.imageFormat = imageFormat;
}
@Override
public byte[] apply(byte[] imageBytes, List<ObjectDetection> objectDetections) {
if (!CollectionUtils.isEmpty(objectDetections)) {
try {
BufferedImage bufferedImage = ImageIO.read(new ByteArrayInputStream(imageBytes));
for (ObjectDetection od : objectDetections) {
int y1 = (int) (od.getY1() * (float) bufferedImage.getHeight());
int x1 = (int) (od.getX1() * (float) bufferedImage.getWidth());
int y2 = (int) (od.getY2() * (float) bufferedImage.getHeight());
int x2 = (int) (od.getX2() * (float) bufferedImage.getWidth());
int cid = od.getCid();
String labelName = od.getName();
int probability = (int) (100 * od.getConfidence());
String title = labelName + ": " + probability + "%";
GraphicsUtils.drawBoundingBox(bufferedImage, cid, title, x1, y1, x2, y2, this.agnosticColors);
if (this.withMask && od.getMask() != null) {
float[][] mask = od.getMask();
if (mask != null) {
Color maskColor = this.agnosticColors ? null : GraphicsUtils.getClassColor(cid);
BufferedImage maskImage = GraphicsUtils.createMaskImage(
mask, x2 - x1, y2 - y1, maskColor);
GraphicsUtils.overlayImages(bufferedImage, maskImage, x1, y1);
}
}
}
imageBytes = GraphicsUtils.toImageByteArray(bufferedImage, this.getImageFormat());
}
catch (IOException e) {
logger.error(e);
}
}
// Null mend that QR image is found and not output message will be send.
return imageBytes;
}
}

76
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionInputAdapter.java Normal file
View File

@@ -0,0 +1,76 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.util.Collections;
import java.util.Map;
import java.util.function.Function;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.tensorflow.Operand;
import org.tensorflow.Tensor;
import org.tensorflow.op.core.Placeholder;
import org.tensorflow.op.image.DecodeJpeg;
import org.tensorflow.types.UInt8;
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
/**
* Converts byte array image into a input Tensor for the Object Detection API.
*
* @author Christian Tzolov
*/
public class ObjectDetectionInputAdapter implements Function<byte[], Map<String, Tensor<?>>>, AutoCloseable {
private static final Log logger = LogFactory.getLog(ObjectDetectionInputAdapter.class);
/** Make checkstyle happy. **/
public static final String RAW_IMAGE = "raw_image";
/** Make checkstyle happy. **/
public static final String NORMALIZED_IMAGE = "normalized_image";
/** Make checkstyle happy. **/
public static final long CHANNELS = 3;
private final GraphRunner imageLoaderGraph;
public ObjectDetectionInputAdapter() {
this.imageLoaderGraph = new GraphRunner(RAW_IMAGE, NORMALIZED_IMAGE)
.withGraphDefinition(tf -> {
Placeholder<String> rawImage = tf.withName(RAW_IMAGE).placeholder(String.class);
Operand<UInt8> decodedImage = tf.dtypes.cast(
tf.image.decodeJpeg(rawImage, DecodeJpeg.channels(CHANNELS)), UInt8.class);
// Expand dimensions since the model expects images to have shape: [1, H, W, 3]
tf.withName(NORMALIZED_IMAGE).expandDims(decodedImage, tf.constant(0));
});
}
@Override
public Map<String, Tensor<?>> apply(byte[] inputImage) {
try (Tensor inputTensor = Tensor.create(inputImage)) {
return this.imageLoaderGraph.apply(Collections.singletonMap(RAW_IMAGE, inputTensor));
}
}
@Override
public void close() {
if (this.imageLoaderGraph != null) {
this.imageLoaderGraph.close();
}
}
}

98
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionInputConverter.java Normal file
View File

@@ -0,0 +1,98 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.Map;
import java.util.function.Function;
import javax.imageio.ImageIO;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.tensorflow.Tensor;
import org.tensorflow.types.UInt8;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
/**
* Converts byte array image into a input Tensor for the Object Detection API. The computed image tensors uses the
* 'image_tensor' model placeholder.
*
* @author Christian Tzolov
*/
public class ObjectDetectionInputConverter implements Function<byte[][], Map<String, Tensor<?>>> {
private static final Log logger = LogFactory.getLog(ObjectDetectionInputConverter.class);
private static final long CHANNELS = 3;
/** Make checkstyle happy. **/
public static final String IMAGE_TENSOR_FEED_NAME = "image_tensor";
@Override
public Map<String, Tensor<?>> apply(byte[][] input) {
return Collections.singletonMap(IMAGE_TENSOR_FEED_NAME, makeImageTensor(input));
}
private static Tensor<UInt8> makeImageTensor(byte[][] imageBytesArray) {
try {
int batchSize = imageBytesArray.length;
ByteBuffer byteBuffer = null;
long[] shape = null;
for (int batchIndex = 0; batchIndex < batchSize; batchIndex++) {
byte[] imageBytes = imageBytesArray[batchIndex];
ByteArrayInputStream is = new ByteArrayInputStream(imageBytes);
BufferedImage img = ImageIO.read(is);
if (img.getType() != BufferedImage.TYPE_3BYTE_BGR) {
img = GraphicsUtils.toBufferedImageType(img, BufferedImage.TYPE_3BYTE_BGR);
}
if (byteBuffer == null) {
byteBuffer = ByteBuffer.allocate((int) (batchSize * img.getHeight() * img.getWidth() * CHANNELS));
shape = new long[] { batchSize, img.getHeight(), img.getWidth(), CHANNELS };
}
byte[] data = ((DataBufferByte) img.getData().getDataBuffer()).getData();
// ImageIO.read produces BGR-encoded images, while the model expects RGB.
bgrToRgb(data);
byteBuffer.put(data);
}
byteBuffer.flip();
return Tensor.create(UInt8.class, shape, byteBuffer);
}
catch (IOException e) {
throw new IllegalArgumentException("Incorrect image format", e);
}
}
private static void bgrToRgb(byte[] data) {
for (int i = 0; i < data.length; i += 3) {
byte tmp = data[i];
data[i] = data[i + 2];
data[i + 2] = tmp;
}
}
}

187
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionOutputConverter.java Normal file
View File

@@ -0,0 +1,187 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import com.google.protobuf.TextFormat;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.tensorflow.Tensor;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.cloud.fn.object.detection.protos.StringIntLabelMapOuterClass;
import org.springframework.core.io.Resource;
import org.springframework.util.Assert;
import org.springframework.util.StreamUtils;
import org.springframework.util.StringUtils;
/**
* Converts the Tensorflow Object Detection result into {@link ObjectDetection} list.
* The pre-trained Object Detection models (http://bit.ly/2osxMAY) produce 3 tensor outputs:
* (1) detection_classes - containing the ids of detected objects, (2) detection_scores - confidence probabilities of the
* detected object and (3) detection_boxes - the object bounding boxes withing the images.
*
* The MASK based models provide to 2 additional tensors: (4) num_detections and (5) detection_masks.
*
* All outputs tensors are float arrays, having:
* - 1 as the first dimension
* - maxObjects as the second dimension
* While boxesT will have 4 as the third dimension (2 sets of (x, y) coordinates).
* This can be verified by looking at scoresT.shape() etc.
*
* The format detected classes (e.g. labels) names is defined by the 'string_int_labels_map.proto'. The input list
* is available at: https://github.com/tensorflow/models/tree/master/research/object_detection/data
*
* @author Christian Tzolov
*/
public class ObjectDetectionOutputConverter implements Function<Map<String, Tensor<?>>, List<List<ObjectDetection>>> {
private static final Log logger = LogFactory.getLog(ObjectDetectionOutputConverter.class);
/** DETECTION_CLASSES. */
public static final String DETECTION_CLASSES = "detection_classes";
/** DETECTION_SCORES. */
public static final String DETECTION_SCORES = "detection_scores";
/** DETECTION_BOXES. */
public static final String DETECTION_BOXES = "detection_boxes";
/** DETECTION_MASKS. */
public static final String DETECTION_MASKS = "detection_masks";
/** NUM_DETECTIONS. */
public static final String NUM_DETECTIONS = "num_detections";
private final String[] labels;
private float confidence;
private List<String> modelFetch;
public ObjectDetectionOutputConverter(Resource labelsResource, float confidence, List<String> modelFetch) {
this.confidence = confidence;
this.modelFetch = modelFetch;
try {
this.labels = loadLabels(labelsResource);
Assert.notNull(this.labels, String.format("Failed to initialize object labels [%s].", labelsResource));
}
catch (Exception e) {
throw new RuntimeException(String.format("Failed to initialize object labels [%s].", labelsResource), e);
}
logger.info(String.format("Object labels [%s] loaded.", labelsResource));
}
/**
* Loads object labels in the string_int_label_map.proto.
* @param labelsResource location of the labels as a resource
* @return
*/
private static String[] loadLabels(Resource labelsResource) throws Exception {
try (InputStream is = labelsResource.getInputStream()) {
String text = StreamUtils.copyToString(is, Charset.forName("UTF-8"));
StringIntLabelMapOuterClass.StringIntLabelMap.Builder builder =
StringIntLabelMapOuterClass.StringIntLabelMap.newBuilder();
TextFormat.merge(text, builder);
StringIntLabelMapOuterClass.StringIntLabelMap proto = builder.build();
int maxLabelId = proto.getItemList().stream()
.map(StringIntLabelMapOuterClass.StringIntLabelMapItem::getId)
.max(Comparator.comparing(i -> i))
.orElse(-1);
String[] labelIdToNameMap = new String[maxLabelId + 1];
for (StringIntLabelMapOuterClass.StringIntLabelMapItem item : proto.getItemList()) {
if (!StringUtils.isEmpty(item.getDisplayName())) {
labelIdToNameMap[item.getId()] = item.getDisplayName();
}
else {
// Common practice is to set the name to a MID or Synsets Id. Synset is a set of synonyms that
// share a common meaning: https://en.wikipedia.org/wiki/WordNet
labelIdToNameMap[item.getId()] = item.getName();
}
}
return labelIdToNameMap;
}
}
@Override
public List<List<ObjectDetection>> apply(Map<String, Tensor<?>> tensorMap) {
try (Tensor<Float> scoresTensor = tensorMap.get(DETECTION_SCORES).expect(Float.class);
Tensor<Float> classesTensor = tensorMap.get(DETECTION_CLASSES).expect(Float.class);
Tensor<Float> boxesTensor = tensorMap.get(DETECTION_BOXES).expect(Float.class)
) {
// All these tensors have:
// - 1 as the first dimension
// - maxObjects as the second dimension
// While boxesT will have 4 as the third dimension (2 sets of (x, y) coordinates).
// This can be verified by looking at scoresT.shape() etc.
int batchSize = (int) scoresTensor.shape()[0];
int maxObjects = (int) scoresTensor.shape()[1];
float[][] scores = scoresTensor.copyTo(new float[batchSize][maxObjects]);
float[][] classes = classesTensor.copyTo(new float[batchSize][maxObjects]);
float[][][] boxes = boxesTensor.copyTo(new float[batchSize][maxObjects][4]);
List<List<ObjectDetection>> batchObjectDetections = new ArrayList<>();
for (int batchIndex = 0; batchIndex < batchSize; batchIndex++) {
List<ObjectDetection> objectDetections = new ArrayList<>();
// Collect only the objects whose scores are at above the configured confidence threshold.
for (int i = 0; i < scores[batchIndex].length; ++i) {
if (scores[batchIndex][i] >= confidence) {
String category = labels[(int) classes[batchIndex][i]];
float score = scores[batchIndex][i];
ObjectDetection od = new ObjectDetection();
od.setName(category);
od.setConfidence(score);
od.setX1(boxes[batchIndex][i][1]);
od.setY1(boxes[batchIndex][i][0]);
od.setX2(boxes[batchIndex][i][3]);
od.setY2(boxes[batchIndex][i][2]);
od.setCid((int) classes[batchIndex][i]);
// Mask allows image-segmentation
if (modelFetch.contains(DETECTION_MASKS) && modelFetch.contains(NUM_DETECTIONS)) {
Tensor<Float> masksTensor = tensorMap.get(DETECTION_MASKS).expect(Float.class);
Tensor<Float> numDetections = tensorMap.get(NUM_DETECTIONS).expect(Float.class);
float nd = numDetections.copyTo(new float[batchSize])[0];
if (masksTensor != null) {
long[] shape = masksTensor.shape();
float[][][][] masks = masksTensor.copyTo(new float[(int) shape[0]][(int) shape[1]][(int) shape[2]][(int) shape[3]]);
od.setMask(masks[batchIndex][i]);
logger.debug(String.format("Num detections: %s, Masks: %s", nd, masks));
}
}
objectDetections.add(od);
}
}
batchObjectDetections.add(objectDetections);
}
return batchObjectDetections;
}
}
}

147
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionService.java Normal file
View File

@@ -0,0 +1,147 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.List;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.TensorFlowService;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.util.StreamUtils;
/**
* Convenience class that leverages the the {@link ObjectDetectionInputConverter}, {@link ObjectDetectionOutputConverter} and {@link TensorFlowService}
* in combination fromMemory the Tensorflow Object Detection API (https://github.com/tensorflow/models/tree/master/research/object_detection)
* models for detection objects in input images.
*
* All pre-trained models (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) and labels are supported.
*
* You can download pre-trained models directly from the zoo: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
* Just use the URI notation: (zoo model tar.gz url)#(name of the frozen model file name). To speedup the bootstrap
* performance you should consider downloading the models locally and use the file:/"path to my model" URI instead!
*
* The object category labels for the pre-trained models are available at: https://github.com/tensorflow/models/tree/master/research/object_detection/data
* Use the labels applicable for the model. Also, for performance reasons you may consider to download the labels
* and load them from file: instead.
*
* @author Christian Tzolov
*/
public class ObjectDetectionService {
/** Default list of fetch names for Box models. */
public static List<String> FETCH_NAMES = Arrays.asList(
ObjectDetectionOutputConverter.DETECTION_SCORES, ObjectDetectionOutputConverter.DETECTION_CLASSES,
ObjectDetectionOutputConverter.DETECTION_BOXES, ObjectDetectionOutputConverter.NUM_DETECTIONS);
/** Default list of fetch names for mask supporting models. */
public static List<String> FETCH_NAMES_WITH_MASKS = Arrays.asList(
ObjectDetectionOutputConverter.DETECTION_SCORES, ObjectDetectionOutputConverter.DETECTION_CLASSES,
ObjectDetectionOutputConverter.DETECTION_BOXES, ObjectDetectionOutputConverter.DETECTION_MASKS,
ObjectDetectionOutputConverter.NUM_DETECTIONS);
private final ObjectDetectionInputConverter inputConverter;
private final ObjectDetectionOutputConverter outputConverter;
private final TensorFlowService tensorFlowService;
public ObjectDetectionService() {
this("https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb",
"https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt",
0.4f, false, true);
}
/**
* Convenience constructor that would initialize all necessary internal components.
* @param modelUri URI of the pre-trained, frozen Tensorflow model.
* @param labelsUri URI of the pre-trained category labels.
* @param confidence Confidence threshold. Only objects detected wth confidence above this threshold will be returned.
* @param withMasks If a Mask model is selected then you can use this flag to extract the instance segmentation masks as well.
*/
public ObjectDetectionService(String modelUri, String labelsUri,
float confidence, boolean withMasks, boolean cacheModel) {
this.inputConverter = new ObjectDetectionInputConverter();
List<String> fetchNames = withMasks ? FETCH_NAMES_WITH_MASKS : FETCH_NAMES;
this.outputConverter = new ObjectDetectionOutputConverter(
new DefaultResourceLoader().getResource(labelsUri), confidence, fetchNames);
this.tensorFlowService = new TensorFlowService(
new DefaultResourceLoader().getResource(modelUri), fetchNames, cacheModel);
}
/**
* Generic constructor thea allow the converter to be pre-configured before passed to the service.
* @param inputConverter Converter from byte array to object detection input image tensor
* @param outputConverter Covets the object detection output tensors into {@link ObjectDetection } list
* @param tensorFlowService Java tensorflow runner instance
*/
public ObjectDetectionService(ObjectDetectionInputConverter inputConverter,
ObjectDetectionOutputConverter outputConverter, TensorFlowService tensorFlowService) {
this.inputConverter = inputConverter;
this.outputConverter = outputConverter;
this.tensorFlowService = tensorFlowService;
}
/**
* Detects objects in a single input image identified by its URI.
*
* @param imageUri input image's URI
* @return Returns a list of {@link ObjectDetection} domain objects representing detected objects
*/
public List<ObjectDetection> detect(String imageUri) {
try (InputStream is = new DefaultResourceLoader().getResource(imageUri).getInputStream()) {
return this.detect(StreamUtils.copyToByteArray(is));
}
catch (IOException e) {
e.printStackTrace();
throw new IllegalStateException("Failed to detect the image:" + imageUri, e);
}
}
/**
* Detects objects in a single {@link BufferedImage}.
*
* @param image Input image to detect objects from.
* @param format Image format (e.g. jpg, png ...) to use when converting the buffer into byte array.
* @return Returns a list of {@link ObjectDetection} domain objects representing detected objects in the input image
*/
public List<ObjectDetection> detect(BufferedImage image, String format) {
return this.detect(GraphicsUtils.toImageByteArray(image, format));
}
/**
* Detects objects from a single input image encoded as byte array.
*
* @param image Input image encoded as byte array
* @return Returns a list of {@link ObjectDetection} domain objects representing detected objects in the input image
*/
public List<ObjectDetection> detect(byte[] image) {
return this.inputConverter.andThen(this.tensorFlowService).andThen(this.outputConverter).apply(new byte[][] { image }).get(0);
}
/**
* Uses detects objects from a batch of input images encoded as byte array.
*
* @param batchedImages Batch of input images encoded as byte arrays. First dimension is the batch size and second the image bytes.
* @return Returns list of lists. For every input image in the batch a list of {@link ObjectDetection} domain objects representing detected objects in the input image.
*/
public List<List<ObjectDetection>> detect(byte[][] batchedImages) {
return this.inputConverter.andThen(this.tensorFlowService).andThen(this.outputConverter).apply(batchedImages);
}
}

113
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionService2.java Normal file
View File

@@ -0,0 +1,113 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import org.tensorflow.Operand;
import org.tensorflow.Tensor;
import org.tensorflow.op.core.Placeholder;
import org.tensorflow.op.image.DecodeJpeg;
import org.tensorflow.types.UInt8;
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
import org.springframework.cloud.fn.common.tensorflow.GraphRunnerMemory;
import org.springframework.cloud.fn.common.tensorflow.ProtoBufGraphDefinition;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.core.io.DefaultResourceLoader;
/**
* @author Christian Tzolov
*/
public class ObjectDetectionService2 implements AutoCloseable {
/** Default Box models fetch names. */
public static List<String> FETCH_NAMES = Arrays.asList(
ObjectDetectionOutputConverter.DETECTION_SCORES, ObjectDetectionOutputConverter.DETECTION_CLASSES,
ObjectDetectionOutputConverter.DETECTION_BOXES, ObjectDetectionOutputConverter.NUM_DETECTIONS);
/** Default Models models fetch names. */
public static List<String> FETCH_NAMES_WITH_MASKS = Arrays.asList(
ObjectDetectionOutputConverter.DETECTION_SCORES, ObjectDetectionOutputConverter.DETECTION_CLASSES,
ObjectDetectionOutputConverter.DETECTION_BOXES, ObjectDetectionOutputConverter.DETECTION_MASKS,
ObjectDetectionOutputConverter.NUM_DETECTIONS);
private final GraphRunner imageNormalization;
private final GraphRunner objectDetection;
private final ObjectDetectionOutputConverter outputConverter;
public ObjectDetectionService2(String modelUri, ObjectDetectionOutputConverter outputConverter) {
this.imageNormalization = new GraphRunner("raw_image", "normalized_image")
.withGraphDefinition(tf -> {
Placeholder<String> rawImage = tf.withName("raw_image").placeholder(String.class);
Operand<UInt8> decodedImage = tf.dtypes.cast(
tf.image.decodeJpeg(rawImage, DecodeJpeg.channels(3L)), UInt8.class);
// Expand dimensions since the model expects images to have shape: [1, H, W, 3]
tf.withName("normalized_image").expandDims(decodedImage, tf.constant(0));
});
this.objectDetection = new GraphRunner(Arrays.asList("image_tensor"), FETCH_NAMES)
.withGraphDefinition(new ProtoBufGraphDefinition(
new DefaultResourceLoader().getResource(modelUri), true));
this.outputConverter = outputConverter;
}
public List<ObjectDetection> detect(byte[] image) {
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memorize = new GraphRunnerMemory()) {
List<List<ObjectDetection>> out = this.imageNormalization.andThen(memorize)
.andThen(this.objectDetection).andThen(memorize)
.andThen(outputConverter)
.apply(Collections.singletonMap("raw_image", inputTensor));
return out.get(0);
}
}
@Override
public void close() {
this.imageNormalization.close();
this.objectDetection.close();
//this.outputConverter.close();
}
public static void main(String[] args) throws IOException {
String modelUri = "http://dl.bintray.com/big-data/generic/ssdlite_mobilenet_v2_coco_2018_05_09_frozen_inference_graph.pb";
String labelUri = "http://dl.bintray.com/big-data/generic/mscoco_label_map.pbtxt";
ObjectDetectionOutputConverter outputAdapter = new ObjectDetectionOutputConverter(
new DefaultResourceLoader().getResource(labelUri), 0.4f, FETCH_NAMES);
//byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg");
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/wild-animals-15.jpg");
try (ObjectDetectionService2 objectDetectionService2 = new ObjectDetectionService2(modelUri, outputAdapter)) {
List<ObjectDetection> boza = objectDetectionService2.detect(inputImage);
System.out.println(boza);
}
}
}

116
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/domain/ObjectDetection.java Normal file
View File

@@ -0,0 +1,116 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection.domain;
import java.util.Arrays;
import com.fasterxml.jackson.annotation.JsonInclude;
/**
* @author Christian Tzolov
*/
@JsonInclude(JsonInclude.Include.NON_NULL)
public class ObjectDetection {
private String name;
private float confidence;
private float x1;
private float y1;
private float x2;
private float y2;
private float[][] mask;
private int cid;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public float getConfidence() {
return confidence;
}
public void setConfidence(float confidence) {
this.confidence = confidence;
}
public float getX1() {
return x1;
}
public void setX1(float x1) {
this.x1 = x1;
}
public float getY1() {
return y1;
}
public void setY1(float y1) {
this.y1 = y1;
}
public float getX2() {
return x2;
}
public void setX2(float x2) {
this.x2 = x2;
}
public float getY2() {
return y2;
}
public void setY2(float y2) {
this.y2 = y2;
}
public int getCid() {
return cid;
}
public void setCid(int cid) {
this.cid = cid;
}
public float[][] getMask() {
return mask;
}
public void setMask(float[][] mask) {
this.mask = mask;
}
@Override
public String toString() {
return "ObjectDetection{" +
"name='" + name + '\'' +
", confidence=" + confidence +
", x1=" + x1 +
", y1=" + y1 +
", x2=" + x2 +
", y2=" + y2 +
", mask=" + Arrays.toString(mask) +
", cid=" + cid +
'}';
}
}

1845
function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/protos/StringIntLabelMapOuterClass.java Normal file
View File

File diff suppressed because it is too large Load Diff

301
function/object-detection-function/src/main/proto/example.proto__ Normal file
View File

@@ -0,0 +1,301 @@
// Protocol messages for describing input data Examples for machine learning
// model training or inference.
syntax = "proto3";
import "feature.proto";
option cc_enable_arenas = true;
option java_outer_classname = "ExampleProtos";
option java_multiple_files = true;
option java_package = "org.tensorflow.example";
option go_package = "github.com/tensorflow/tensorflow/tensorflow/go/core/example";
package tensorflow;
// An Example is a mostly-normalized data format for storing data for
// training and inference. It contains a key-value store (features); where
// each key (string) maps to a Feature message (which is oneof packed BytesList,
// FloatList, or Int64List). This flexible and compact format allows the
// storage of large amounts of typed data, but requires that the data shape
// and use be determined by the configuration files and parsers that are used to
// read and write this format. That is, the Example is mostly *not* a
// self-describing format. In TensorFlow, Examples are read in row-major
// format, so any configuration that describes data with rank-2 or above
// should keep this in mind. For example, to store an M x N matrix of Bytes,
// the BytesList must contain M*N bytes, with M rows of N contiguous values
// each. That is, the BytesList value must store the matrix as:
// .... row 0 .... .... row 1 .... // ........... // ... row M-1 ....
//
// An Example for a movie recommendation application:
// features {
// feature {
// key: "age"
// value { float_list {
// value: 29.0
// }}
// }
// feature {
// key: "movie"
// value { bytes_list {
// value: "The Shawshank Redemption"
// value: "Fight Club"
// }}
// }
// feature {
// key: "movie_ratings"
// value { float_list {
// value: 9.0
// value: 9.7
// }}
// }
// feature {
// key: "suggestion"
// value { bytes_list {
// value: "Inception"
// }}
// }
// # Note that this feature exists to be used as a label in training.
// # E.g., if training a logistic regression model to predict purchase
// # probability in our learning tool we would set the label feature to
// # "suggestion_purchased".
// feature {
// key: "suggestion_purchased"
// value { float_list {
// value: 1.0
// }}
// }
// # Similar to "suggestion_purchased" above this feature exists to be used
// # as a label in training.
// # E.g., if training a linear regression model to predict purchase
// # price in our learning tool we would set the label feature to
// # "purchase_price".
// feature {
// key: "purchase_price"
// value { float_list {
// value: 9.99
// }}
// }
// }
//
// A conformant Example data set obeys the following conventions:
// - If a Feature K exists in one example with data type T, it must be of
// type T in all other examples when present. It may be omitted.
// - The number of instances of Feature K list data may vary across examples,
// depending on the requirements of the model.
// - If a Feature K doesn't exist in an example, a K-specific default will be
// used, if configured.
// - If a Feature K exists in an example but contains no items, the intent
// is considered to be an empty tensor and no default will be used.
message Example {
Features features = 1;
};
// A SequenceExample is an Example representing one or more sequences, and
// some context. The context contains features which apply to the entire
// example. The feature_lists contain a key, value map where each key is
// associated with a repeated set of Features (a FeatureList).
// A FeatureList thus represents the values of a feature identified by its key
// over time / frames.
//
// Below is a SequenceExample for a movie recommendation application recording a
// sequence of ratings by a user. The time-independent features ("locale",
// "age", "favorites") describing the user are part of the context. The sequence
// of movies the user rated are part of the feature_lists. For each movie in the
// sequence we have information on its name and actors and the user's rating.
// This information is recorded in three separate feature_list(s).
// In the example below there are only two movies. All three feature_list(s),
// namely "movie_ratings", "movie_names", and "actors" have a feature value for
// both movies. Note, that "actors" is itself a bytes_list with multiple
// strings per movie.
//
// context: {
// feature: {
// key : "locale"
// value: {
// bytes_list: {
// value: [ "pt_BR" ]
// }
// }
// }
// feature: {
// key : "age"
// value: {
// float_list: {
// value: [ 19.0 ]
// }
// }
// }
// feature: {
// key : "favorites"
// value: {
// bytes_list: {
// value: [ "Majesty Rose", "Savannah Outen", "One Direction" ]
// }
// }
// }
// }
// feature_lists: {
// feature_list: {
// key : "movie_ratings"
// value: {
// feature: {
// float_list: {
// value: [ 4.5 ]
// }
// }
// feature: {
// float_list: {
// value: [ 5.0 ]
// }
// }
// }
// }
// feature_list: {
// key : "movie_names"
// value: {
// feature: {
// bytes_list: {
// value: [ "The Shawshank Redemption" ]
// }
// }
// feature: {
// bytes_list: {
// value: [ "Fight Club" ]
// }
// }
// }
// }
// feature_list: {
// key : "actors"
// value: {
// feature: {
// bytes_list: {
// value: [ "Tim Robbins", "Morgan Freeman" ]
// }
// }
// feature: {
// bytes_list: {
// value: [ "Brad Pitt", "Edward Norton", "Helena Bonham Carter" ]
// }
// }
// }
// }
// }
//
// A conformant SequenceExample data set obeys the following conventions:
//
// Context:
// - All conformant context features K must obey the same conventions as
// a conformant Example's features (see above).
// Feature lists:
// - A FeatureList L may be missing in an example; it is up to the
// parser configuration to determine if this is allowed or considered
// an empty list (zero length).
// - If a FeatureList L exists, it may be empty (zero length).
// - If a FeatureList L is non-empty, all features within the FeatureList
// must have the same data type T. Even across SequenceExamples, the type T
// of the FeatureList identified by the same key must be the same. An entry
// without any values may serve as an empty feature.
// - If a FeatureList L is non-empty, it is up to the parser configuration
// to determine if all features within the FeatureList must
// have the same size. The same holds for this FeatureList across multiple
// examples.
// - For sequence modeling, e.g.:
// http://colah.github.io/posts/2015-08-Understanding-LSTMs/
// https://github.com/tensorflow/nmt
// the feature lists represent a sequence of frames.
// In this scenario, all FeatureLists in a SequenceExample have the same
// number of Feature messages, so that the ith element in each FeatureList
// is part of the ith frame (or time step).
// Examples of conformant and non-conformant examples' FeatureLists:
//
// Conformant FeatureLists:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
//
// Non-conformant FeatureLists (mismatched types):
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { int64_list: { value: [ 5 ] } } }
// } }
//
// Conditionally conformant FeatureLists, the parser configuration determines
// if the feature sizes must match:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0, 6.0 ] } } }
// } }
//
// Conformant pair of SequenceExample
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
// and:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } }
// feature: { float_list: { value: [ 2.0 ] } } }
// } }
//
// Conformant pair of SequenceExample
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
// and:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { }
// } }
//
// Conditionally conformant pair of SequenceExample, the parser configuration
// determines if the second feature_lists is consistent (zero-length) or
// invalid (missing "movie_ratings"):
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
// and:
// feature_lists: { }
//
// Non-conformant pair of SequenceExample (mismatched types)
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
// and:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { int64_list: { value: [ 4 ] } }
// feature: { int64_list: { value: [ 5 ] } }
// feature: { int64_list: { value: [ 2 ] } } }
// } }
//
// Conditionally conformant pair of SequenceExample; the parser configuration
// determines if the feature sizes must match:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.5 ] } }
// feature: { float_list: { value: [ 5.0 ] } } }
// } }
// and:
// feature_lists: { feature_list: {
// key: "movie_ratings"
// value: { feature: { float_list: { value: [ 4.0 ] } }
// feature: { float_list: { value: [ 5.0, 3.0 ] } }
// } }
message SequenceExample {
Features context = 1;
FeatureLists feature_lists = 2;
};

105
function/object-detection-function/src/main/proto/feature.proto__ Normal file
View File

@@ -0,0 +1,105 @@
// Protocol messages for describing features for machine learning model
// training or inference.
//
// There are three base Feature types:
// - bytes
// - float
// - int64
//
// A Feature contains Lists which may hold zero or more values. These
// lists are the base values BytesList, FloatList, Int64List.
//
// Features are organized into categories by name. The Features message
// contains the mapping from name to Feature.
//
// Example Features for a movie recommendation application:
// feature {
// key: "age"
// value { float_list {
// value: 29.0
// }}
// }
// feature {
// key: "movie"
// value { bytes_list {
// value: "The Shawshank Redemption"
// value: "Fight Club"
// }}
// }
// feature {
// key: "movie_ratings"
// value { float_list {
// value: 9.0
// value: 9.7
// }}
// }
// feature {
// key: "suggestion"
// value { bytes_list {
// value: "Inception"
// }}
// }
// feature {
// key: "suggestion_purchased"
// value { int64_list {
// value: 1
// }}
// }
// feature {
// key: "purchase_price"
// value { float_list {
// value: 9.99
// }}
// }
//
syntax = "proto3";
option cc_enable_arenas = true;
option java_outer_classname = "FeatureProtos";
option java_multiple_files = true;
option java_package = "org.tensorflow.example";
option go_package = "github.com/tensorflow/tensorflow/tensorflow/go/core/example";
package tensorflow;
// Containers to hold repeated fundamental values.
message BytesList {
repeated bytes value = 1;
}
message FloatList {
repeated float value = 1 [packed = true];
}
message Int64List {
repeated int64 value = 1 [packed = true];
}
// Containers for non-sequential data.
message Feature {
// Each feature can be exactly one kind.
oneof kind {
BytesList bytes_list = 1;
FloatList float_list = 2;
Int64List int64_list = 3;
}
};
message Features {
// Map from feature name to feature.
map<string, Feature> feature = 1;
};
// Containers for sequential data.
//
// A FeatureList contains lists of Features. These may hold zero or more
// Feature values.
//
// FeatureLists are organized into categories by name. The FeatureLists message
// contains the mapping from name to FeatureList.
//
message FeatureList {
repeated Feature feature = 1;
};
message FeatureLists {
// Map from feature name to feature list.
map<string, FeatureList> feature_list = 1;
};

25
function/object-detection-function/src/main/proto/string_int_label_map.proto Normal file
View File

@@ -0,0 +1,25 @@
// Message to store the mapping from class label strings to class id. Datasets
// use string labels to represent classes while the object detection framework
// works fromMemory class ids. This message maps them so they can be converted back
// and forth as needed.
syntax = "proto2";
package org.springframework.cloud.fn.object.detection.protos;
message StringIntLabelMapItem {
// String name. The most common practice is to set this to a MID or synsets
// id. Synset: a set of synonyms that share a common meaning.
// https://en.wikipedia.org/wiki/WordNet
optional string name = 1;
// Integer id that maps to the string name above. Label ids should start
// from 1.
optional int32 id = 2;
// Human readable string label.
optional string display_name = 3;
};
message StringIntLabelMap {
repeated StringIntLabelMapItem item = 1;
};

BIN
function/object-detection-function/src/main/resources/images/object-detection-augmented.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

BIN
function/object-detection-function/src/main/resources/images/object-detection-segmentation-augmented.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 140 KiB

BIN
function/object-detection-function/src/main/resources/images/object-detection.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 MiB

BIN
function/object-detection-function/src/main/resources/images/object_detection_1.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB

89
function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleInstanceSegmentation.java Normal file
View File

@@ -0,0 +1,89 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection.examples;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.List;
import org.apache.commons.io.IOUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction;
import org.springframework.cloud.fn.object.detection.ObjectDetectionImageAugmenter;
import org.springframework.cloud.fn.object.detection.ObjectDetectionService;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.ResourceLoader;
/**
* 4 of the pre-trained model in the model zoo (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md)
* can also compute the masks of the detected objects, providing instance segmentation.
*
* Here are the models that can be used for instance segmentation.
*
* mask_rcnn_inception_resnet_v2_atrous_coco 771 36 Masks
* mask_rcnn_inception_v2_coco 79 25 Masks
* mask_rcnn_resnet101_atrous_coco 470 33 Masks
* mask_rcnn_resnet50_atrous_coco 343 29 Masks
*
* @author Christian Tzolov
*/
public class ExampleInstanceSegmentation {
public static void main(String[] args) throws IOException {
ResourceLoader resourceLoader = new DefaultResourceLoader();
// You can download pre-trained models directly from the zoo: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
// Just use the notation <zoo model tar.gz url>#<name of the frozen model file name>
// For performance reasons you may consider downloading the model locally and use the file:/<path to my model> URI instead!
String model = "http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz#frozen_inference_graph.pb";
// All labels for the pre-trained models are available at:
// https://github.com/tensorflow/models/tree/master/research/object_detection/data
// Use the labels applicable for the model.
// Also, for performance reasons you may consider to download the labels and load them from file: instead.
String labels = "https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt";
// You can cache the TF model on the local file system to improve the bootstrap performance on consecutive runs!
boolean CACHE_TF_MODEL = true;
// For the pre-trained models fromMemory mask you can set the INSTANCE_SEGMENTATION to enable object instance segmentation as well
boolean INSTANCE_SEGMENTATION = true;
// Only object fromMemory confidence above the threshold are returned
float CONFIDENCE_THRESHOLD = 0.4f;
ObjectDetectionService detectionService =
new ObjectDetectionService(model, labels, CONFIDENCE_THRESHOLD, INSTANCE_SEGMENTATION, CACHE_TF_MODEL);
// You can use file:, http: or classpath: to provide the path to the input image.
byte[] image = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg");
// Returns a list ObjectDetection domain classes to allow programmatic accesses to the detected objects's metadata
List<ObjectDetection> detectedObjects = detectionService.detect(image);
// Get JSON representation of the detected objects
String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects);
System.out.println(jsonObjectDetections);
// Draw the detected object metadata on top of the original image and store the result
byte[] annotatedImage = new ObjectDetectionImageAugmenter(INSTANCE_SEGMENTATION).apply(image, detectedObjects);
IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-segmentation-augmented.jpg"));
}
}

84
function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleObjectDetection.java Normal file
View File

@@ -0,0 +1,84 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection.examples;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import org.apache.commons.io.IOUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction;
import org.springframework.cloud.fn.object.detection.ObjectDetectionImageAugmenter;
import org.springframework.cloud.fn.object.detection.ObjectDetectionService;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.util.StreamUtils;
/**
* @author Christian Tzolov
*/
public class ExampleObjectDetection {
public static void main(String[] args) throws IOException {
// You can download pre-trained models directly from the zoo: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
// Just use the notation <zoo model tar.gz url>#<name of the frozen model file name>
// For performance reasons you may consider downloading the model locally and use the file:/<path to my model> URI instead!
String model = "http://download.tensorflow.org/models/object_detection/faster_rcnn_nas_coco_2018_01_28.tar.gz#frozen_inference_graph.pb";
//Resource model = resourceLoader.getResource("http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_fgvc_2018_07_19.tar.gz#frozen_inference_graph.pb");
//Resource model = resourceLoader.getResource("http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet50_fgvc_2018_07_19.tar.gz#frozen_inference_graph.pb");
// All labels for the pre-trained models are available at:
// https://github.com/tensorflow/models/tree/master/research/object_detection/data
// Use the labels applicable for the model.
// Also, for performance reasons you may consider to download the labels and load them from file: instead.
String labels = "https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt";
//Resource labels = resourceLoader.getResource("https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/fgvc_2854_classes_label_map.pbtxt");
// You can cache the TF model on the local file system to improve the bootstrap performance on consecutive runs!
boolean CACHE_TF_MODEL = true;
// For the pre-trained models fromMemory mask you can set the INSTANCE_SEGMENTATION to enable object instance segmentation as well
boolean NO_INSTANCE_SEGMENTATION = false;
// Only object fromMemory confidence above the threshold are returned
float CONFIDENCE_THRESHOLD = 0.4f;
ObjectDetectionService detectionService =
new ObjectDetectionService(model, labels, CONFIDENCE_THRESHOLD, NO_INSTANCE_SEGMENTATION, CACHE_TF_MODEL);
// You can use file:, http: or classpath: to provide the path to the input image.
String inputImageUri = "classpath:/images/object-detection.jpg";
try (InputStream is = new DefaultResourceLoader().getResource(inputImageUri).getInputStream()) {
byte[] image = StreamUtils.copyToByteArray(is);
// Returns a list ObjectDetection domain classes to allow programmatic accesses to the detected objects's metadata
List<ObjectDetection> detectedObjects = detectionService.detect(image);
// Get JSON representation of the detected objects
String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects);
System.out.println(jsonObjectDetections);
// Draw the detected object metadata on top of the original image and store the result
byte[] annotatedImage = new ObjectDetectionImageAugmenter(NO_INSTANCE_SEGMENTATION).apply(image, detectedObjects);
IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-augmented.jpg"));
}
}
}

47
function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/SimpleExample.java Normal file
View File

@@ -0,0 +1,47 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.object.detection.examples;
import java.util.List;
import org.springframework.cloud.fn.object.detection.ObjectDetectionService;
import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
/**
* @author Christian Tzolov
*/
public class SimpleExample {
public static void main(String[] args) {
// Select a pre-trained model from the model zoo: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
// Just use the notation <model zoo url>#<name of the frozen model file in the zoo's tar.gz>
String model = "http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_ppn_shared_box_predictor_300x300_coco14_sync_2018_07_03.tar.gz#frozen_inference_graph.pb";
// All labels for the pre-trained models are available at: https://github.com/tensorflow/models/tree/master/research/object_detection/data
String labels = "https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt";
ObjectDetectionService detectionService = new ObjectDetectionService(model, labels,
0.4f, // Only object fromMemory confidence above the threshold are returned. Confidence range is [0, 1].
false, // No instance segmentation
true); // cache the TF model locally
// You can use file:, http: or classpath: to provide the path to the input image.
List<ObjectDetection> detectedObjects = detectionService.detect("classpath:/images/object-detection.jpg");
detectedObjects.stream().map(o -> o.toString()).forEach(System.out::println);
}
}

104
function/semantic-segmentation-function/README.adoc Normal file
View File

@@ -0,0 +1,104 @@
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/function/semantic-segmentation-function/src/main/resources/images/
# Semantic Segmentation
[.lead]
Image Semantic Segmentation based on the state-of-art https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab] Tensorflow model.
[cols="1,2", frame=none, grid=none]
|===
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
|Semantic Segmentation is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car).
Unlike the `Instance Segmentation`, which produces instance-aware region masks, the `Semantic Segmentation` produces class-aware masks.
For implementing `Instance Segmentation` consult the https://github.com/tzolov/stream-applications/tree/tensorflow-redesign/functions/function/object-detection-function[Object Detection Service] instead.
|===
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
converting the `List<ObjectDetection>` into JSON objects, and the
https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter]
allow to augment the input image with the detected bounding boxes and segmentation masks.
## Usage
Add the `semantic-segmentation` dependency to your pom (_use the latest version available_):
[source,xml]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>semantic-segmentation-function</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>object-detection-function</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
----
Following snippet demos how to use the PASCAL VOC model to apply mask to an input image
[source,java,linenums]
----
SemanticSegmentation segmentationService = new SemanticSegmentation(
"http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", // <1>
true); // <2>
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); // <3>
byte[] imageMask = segmentationService.masksAsImage(inputImage); // <4>
BufferedImage bi = ImageIO.read(new ByteArrayInputStream(imageMask));
ImageIO.write(bi, "png", new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_masks.png"));
byte[] augmentedImage = segmentationService.augment(inputImage); // <5>
IOUtils.write(augmentedImage, new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_augmented.jpg"));
----
<1> Download the PASCAL 2012 trained model directly from the web. The `frozen_inference_graph.pb` is the name of the model
file inside the `tar.gz` archive.
<2> Cache the downloaded model locally
<3> Load the input image as byte array
<4> Read get the segmentation mask as separate image
<5> Blend the segmentation mask on top of the original image
## Models
Based on the training datasets, three groups of pre-trained models provided:
[cols="1,2", frame=none, grid=none]
|===
| image:{images-asciidoc}/VikiMaxiAdi-all.png[width=100%]
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-pascal-voc-2012[DeepLab models trained on PASCAL VOC 2012]
| image:{images-asciidoc}/cityscape-all-small.png[width=100%]
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-cityscapes[DeepLab models trained on Cityscapes]
| image:{images-asciidoc}/ADE20K-all-small.png[width=100%]
| https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md#deeplab-models-trained-on-ade20k[DeepLab models trained on ADE20K]
|===
Select the model you want to use, copy its archive download Url and add a `#frozen_inference_graph.pb` fragment to it.
Later fragment is the frozen model's file name inside the archive
TIP: Download the archive and uncompress the `frozen_inference_graph.pb` for required model. Then use the `file://<local-file-name>` URI schema.
Also, convenience there are a couple of models, extracted from the archive and uploaded to bintray:
[cols=2*,, frame=none, grid=none]
|===
|PASCAL VOC 2012 (default)
|http://dl.bintray.com/big-data/generic/deeplabv3_mnv2_pascal_train_aug_frozen_inference_graph.pb
|CITYSCAPE
|http://dl.bintray.com/big-data/generic/deeplabv3_mnv2_cityscapes_train_2018_02_05_frozen_inference_graph.pb
|ADE20K
|http://dl.bintray.com/big-data/generic/deeplabv3_xception_ade20k_train_2018_05_29_frozen_inference_graph.pb
|===
## References:
[.small]
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html[Semantic Image Segmentation with DeepLab in TensorFlow]
* https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab Project]
* https://medium.freecodecamp.org/how-to-use-deeplab-in-tensorflow-for-object-segmentation-using-deep-learning-a5777290ab6b[How to re-train DeepLab Segmentation models using Transfer Learning]

38
function/semantic-segmentation-function/pom.xml Normal file
View File

@@ -0,0 +1,38 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>semantic-segmentation-function</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>semantic-segmentation-function</name>
<description>Spring Native Function for Tensorflow semantic-segmentation integration</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<properties>
<commons-io.version>1.3.2</commons-io.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>tensorflow-common</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>tensorflow-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>

107
function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/NativeImageUtils.java Normal file
View File

@@ -0,0 +1,107 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.semantic.segmentation;
import java.util.Arrays;
import org.tensorflow.Operand;
import org.tensorflow.op.Ops;
import org.tensorflow.op.core.Concat;
import org.tensorflow.op.core.ExpandDims;
import org.tensorflow.op.core.Gather;
import org.tensorflow.op.core.Range;
import org.tensorflow.op.core.ReduceMax;
import org.tensorflow.op.core.Tile;
import org.tensorflow.op.math.Add;
import org.tensorflow.op.math.Mul;
import org.tensorflow.op.math.Sub;
/**
* @author Christian Tzolov
*/
public final class NativeImageUtils {
private NativeImageUtils() {
}
/**
* grayscaleToRgb.
* https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/ops/image_ops_impl.py#L1536
*/
public static <T> Operand<T> grayscaleToRgb(Ops tf, Operand<T> images) {
ExpandDims<Integer> rank_1 = tf.expandDims(
tf.math.sub(tf.rank(images), tf.constant(1)),
tf.constant(0));
// Create once 1D vector of the shape defined by the rank_1.
// E.g. for rank [2] will produce matrix [1, 1]. For [3] rank will produce a cube [1, 1, 1]
Add<Integer> ones = tf.math.add(tf.zeros(rank_1, Integer.class), tf.constant(1));
// Convert scalar 3 into 1D array [3]
ExpandDims<Integer> channelsAs1D = tf.expandDims(tf.constant(3), tf.constant(0));
Concat<Integer> shapeList = tf.concat(Arrays.asList(ones, channelsAs1D), tf.constant(0));
Tile<T> tile = tf.withName("grayscaleToRgb").tile(images, shapeList);
return tile;
}
public static Operand<Float> normalizeMask(Ops tf, Operand<Float> mask, float newValue) {
// generate array representing the axis indexes.
// For example of tensor of rank K the axisRange is {0, 1, 2 ...K}
Range<Integer> axisRange = tf.range(tf.constant(0), // from
tf.dtypes.cast(tf.rank(mask), Integer.class), // to
tf.constant(1)); // step
ReduceMax<Float> max = tf.reduceMax(mask, axisRange);
//Mul<Float> input2Float1 = tf.math.mul(tf.math.div(input2Float, max), tf.constant(1f));
Mul<Float> normalizedMask = tf.math.mul(tf.math.div(mask, max), tf.constant(newValue));
return normalizedMask;
}
/**
* Alpha Blending .
* https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending
*/
public static Operand<Float> alphaBlending(Ops tf, Operand<Float> srcRgb, Operand<Float> dstRgb, Operand<Float> srcAlpha) {
Sub<Float> alpha = tf.math.sub(tf.onesLike(srcRgb), srcAlpha);
Mul<Float> src = tf.math.mul(srcRgb, alpha);
Mul<Float> dst = tf.math.mul(dstRgb, tf.math.sub(tf.constant(1.0f), alpha));
Add<Float> out = tf.math.add(dst, src);
//Mul<Float> out = tf.math.mul(srcRgbNormalized, dstRgb);
//Squeeze<Float> squeeze = tf.withName("squeeze").squeeze(out, Squeeze.axis(Arrays.asList(0L)));
return out;
}
/**
* The mask can contain label values larger than the list of colors provided in the color map.
* To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
* @param tf - tensorflow
* @param colorTable Color map of shape [n, 3]. n is the count of label entries and 3 is the RGB color assigned
* to that label.
* @param mask Mask of shape [h, w] containing label vales.
* @return Mask of shape [h, w] fromMemory values normalized between [0, n]
*/
public static Operand<Long> normalizeMaskLabels(Ops tf, Operand<Integer> colorTable, Operand<Long> mask) {
// The mask can contain label values larger than the list of colors provided in the color map.
// To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
Sub<Long> colorTableShape = tf.math.sub(tf.shape(colorTable, Long.class), tf.constant(1L));
// Color tables have shape [N, 3], where N is the count of label entries. Therefore the max label id is (N - 1).
Gather<Long> colorTableSize = tf.gather(colorTableShape, tf.constant(new int[] { 0 }), tf.constant(0));
// Normalize the label values in the mask so they don't exceed the max value in the color map.
return tf.math.mod(mask, colorTableSize);
}
}

306
function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SegmentationColorMap.java Normal file
View File

@@ -0,0 +1,306 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.semantic.segmentation;
/**
*
* Visualizes the segmentation results via specified color map.
* Color maps helping to visualize the semantic segmentation results for the different datasets.
*
* Supported colormaps are:
* - ADE20K (http://groups.csail.mit.edu/vision/datasets/ADE20K/).
* - Cityscapes dataset (https://www.cityscapes-dataset.com).
* - Mapillary Vistas (https://research.mapillary.com).
* - PASCAL VOC 2012 (http://host.robots.ox.ac.uk/pascal/VOC/).
*
* Based on: https://github.com/tensorflow/models/blob/master/research/deeplab/utils/get_dataset_colormap.py
*
* @author Christian Tzolov
*/
public final class SegmentationColorMap {
private SegmentationColorMap() {
}
/** MAPILLARY_COLORMAP . */
public static final int[][] MAPILLARY_COLORMAP = new int[][] {
{ 165, 42, 42 },
{ 0, 192, 0 },
{ 196, 196, 196 },
{ 190, 153, 153 },
{ 180, 165, 180 },
{ 102, 102, 156 },
{ 102, 102, 156 },
{ 128, 64, 255 },
{ 140, 140, 200 },
{ 170, 170, 170 },
{ 250, 170, 160 },
{ 96, 96, 96 },
{ 230, 150, 140 },
{ 128, 64, 128 },
{ 110, 110, 110 },
{ 244, 35, 232 },
{ 150, 100, 100 },
{ 70, 70, 70 },
{ 150, 120, 90 },
{ 220, 20, 60 },
{ 255, 0, 0 },
{ 255, 0, 0 },
{ 255, 0, 0 },
{ 200, 128, 128 },
{ 255, 255, 255 },
{ 64, 170, 64 },
{ 128, 64, 64 },
{ 70, 130, 180 },
{ 255, 255, 255 },
{ 152, 251, 152 },
{ 107, 142, 35 },
{ 0, 170, 30 },
{ 255, 255, 128 },
{ 250, 0, 30 },
{ 0, 0, 0 },
{ 220, 220, 220 },
{ 170, 170, 170 },
{ 222, 40, 40 },
{ 100, 170, 30 },
{ 40, 40, 40 },
{ 33, 33, 33 },
{ 170, 170, 170 },
{ 0, 0, 142 },
{ 170, 170, 170 },
{ 210, 170, 100 },
{ 153, 153, 153 },
{ 128, 128, 128 },
{ 0, 0, 142 },
{ 250, 170, 30 },
{ 192, 192, 192 },
{ 220, 220, 0 },
{ 180, 165, 180 },
{ 119, 11, 32 },
{ 0, 0, 142 },
{ 0, 60, 100 },
{ 0, 0, 142 },
{ 0, 0, 90 },
{ 0, 0, 230 },
{ 0, 80, 100 },
{ 128, 64, 64 },
{ 0, 0, 110 },
{ 0, 0, 70 },
{ 0, 0, 192 },
{ 32, 32, 32 },
{ 0, 0, 0 },
{ 0, 0, 0 },
};
/**
* Label colormap used in ADE20K segmentation benchmark.
*/
public static final int[][] ADE20K_COLORMAP = new int[][] {
{ 0, 0, 0 },
{ 120, 120, 120 },
{ 180, 120, 120 },
{ 6, 230, 230 },
{ 80, 50, 50 },
{ 4, 200, 3 },
{ 120, 120, 80 },
{ 140, 140, 140 },
{ 204, 5, 255 },
{ 230, 230, 230 },
{ 4, 250, 7 },
{ 224, 5, 255 },
{ 235, 255, 7 },
{ 150, 5, 61 },
{ 120, 120, 70 },
{ 8, 255, 51 },
{ 255, 6, 82 },
{ 143, 255, 140 },
{ 204, 255, 4 },
{ 255, 51, 7 },
{ 204, 70, 3 },
{ 0, 102, 200 },
{ 61, 230, 250 },
{ 255, 6, 51 },
{ 11, 102, 255 },
{ 255, 7, 71 },
{ 255, 9, 224 },
{ 9, 7, 230 },
{ 220, 220, 220 },
{ 255, 9, 92 },
{ 112, 9, 255 },
{ 8, 255, 214 },
{ 7, 255, 224 },
{ 255, 184, 6 },
{ 10, 255, 71 },
{ 255, 41, 10 },
{ 7, 255, 255 },
{ 224, 255, 8 },
{ 102, 8, 255 },
{ 255, 61, 6 },
{ 255, 194, 7 },
{ 255, 122, 8 },
{ 0, 255, 20 },
{ 255, 8, 41 },
{ 255, 5, 153 },
{ 6, 51, 255 },
{ 235, 12, 255 },
{ 160, 150, 20 },
{ 0, 163, 255 },
{ 140, 140, 140 },
{ 250, 10, 15 },
{ 20, 255, 0 },
{ 31, 255, 0 },
{ 255, 31, 0 },
{ 255, 224, 0 },
{ 153, 255, 0 },
{ 0, 0, 255 },
{ 255, 71, 0 },
{ 0, 235, 255 },
{ 0, 173, 255 },
{ 31, 0, 255 },
{ 11, 200, 200 },
{ 255, 82, 0 },
{ 0, 255, 245 },
{ 0, 61, 255 },
{ 0, 255, 112 },
{ 0, 255, 133 },
{ 255, 0, 0 },
{ 255, 163, 0 },
{ 255, 102, 0 },
{ 194, 255, 0 },
{ 0, 143, 255 },
{ 51, 255, 0 },
{ 0, 82, 255 },
{ 0, 255, 41 },
{ 0, 255, 173 },
{ 10, 0, 255 },
{ 173, 255, 0 },
{ 0, 255, 153 },
{ 255, 92, 0 },
{ 255, 0, 255 },
{ 255, 0, 245 },
{ 255, 0, 102 },
{ 255, 173, 0 },
{ 255, 0, 20 },
{ 255, 184, 184 },
{ 0, 31, 255 },
{ 0, 255, 61 },
{ 0, 71, 255 },
{ 255, 0, 204 },
{ 0, 255, 194 },
{ 0, 255, 82 },
{ 0, 10, 255 },
{ 0, 112, 255 },
{ 51, 0, 255 },
{ 0, 194, 255 },
{ 0, 122, 255 },
{ 0, 255, 163 },
{ 255, 153, 0 },
{ 0, 255, 10 },
{ 255, 112, 0 },
{ 143, 255, 0 },
{ 82, 0, 255 },
{ 163, 255, 0 },
{ 255, 235, 0 },
{ 8, 184, 170 },
{ 133, 0, 255 },
{ 0, 255, 92 },
{ 184, 0, 255 },
{ 255, 0, 31 },
{ 0, 184, 255 },
{ 0, 214, 255 },
{ 255, 0, 112 },
{ 92, 255, 0 },
{ 0, 224, 255 },
{ 112, 224, 255 },
{ 70, 184, 160 },
{ 163, 0, 255 },
{ 153, 0, 255 },
{ 71, 255, 0 },
{ 255, 0, 163 },
{ 255, 204, 0 },
{ 255, 0, 143 },
{ 0, 255, 235 },
{ 133, 255, 0 },
{ 255, 0, 235 },
{ 245, 0, 255 },
{ 255, 0, 122 },
{ 255, 245, 0 },
{ 10, 190, 212 },
{ 214, 255, 0 },
{ 0, 204, 255 },
{ 20, 0, 255 },
{ 255, 255, 0 },
{ 0, 153, 255 },
{ 0, 41, 255 },
{ 0, 255, 204 },
{ 41, 0, 255 },
{ 41, 255, 0 },
{ 173, 0, 255 },
{ 0, 245, 255 },
{ 71, 0, 255 },
{ 122, 0, 255 },
{ 0, 255, 184 },
{ 0, 92, 255 },
{ 184, 255, 0 },
{ 0, 133, 255 },
{ 255, 214, 0 },
{ 25, 194, 194 },
{ 102, 255, 0 },
{ 92, 0, 255 },
};
/** BLACK_WHITE_COLORMAP . */
public static int[][] BLACK_WHITE_COLORMAP = new int[][] {
{ 0, 0, 0 },
{ 127, 127, 127 },
{ 255, 255, 255 },
};
/** CITYMAP_COLORMAP . */
public static final int[][] CITYMAP_COLORMAP = new int[255][3];
static {
// Initialize citymap
int[][] _CITYMAP_COLORMAP = new int[][] {
{ 128, 64, 128 },
{ 244, 35, 232 },
{ 70, 70, 70 },
{ 102, 102, 156 },
{ 190, 153, 153 },
{ 153, 153, 153 },
{ 250, 170, 30 },
{ 220, 220, 0 },
{ 107, 142, 35 },
{ 152, 251, 152 },
{ 70, 130, 180 },
{ 220, 20, 60 },
{ 255, 0, 0 },
{ 0, 0, 142 },
{ 0, 0, 70 },
{ 0, 60, 100 },
{ 0, 80, 100 },
{ 0, 0, 230 },
{ 119, 11, 32 }
};
for (int i = 0; i < _CITYMAP_COLORMAP.length; i++) {
System.arraycopy(_CITYMAP_COLORMAP[i], 0, CITYMAP_COLORMAP[i], 0, _CITYMAP_COLORMAP[i].length);
}
}
}

287
function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SemanticSegmentation.java Normal file
View File

@@ -0,0 +1,287 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.semantic.segmentation;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import javax.imageio.ImageIO;
import org.tensorflow.Operand;
import org.tensorflow.Tensor;
import org.tensorflow.op.Ops;
import org.tensorflow.op.core.Gather;
import org.tensorflow.op.core.Placeholder;
import org.tensorflow.op.core.Squeeze;
import org.tensorflow.op.core.ZerosLike;
import org.tensorflow.op.dtypes.Cast;
import org.tensorflow.op.image.DecodeJpeg;
import org.tensorflow.op.image.ExtractJpegShape;
import org.tensorflow.op.math.Add;
import org.tensorflow.op.math.Div;
import org.tensorflow.op.math.Equal;
import org.tensorflow.types.UInt8;
import org.springframework.cloud.fn.common.tensorflow.Functions;
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
import org.springframework.cloud.fn.common.tensorflow.GraphRunnerMemory;
import org.springframework.cloud.fn.common.tensorflow.ProtoBufGraphDefinition;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.core.io.DefaultResourceLoader;
/**
* @author Christian Tzolov
*/
public class SemanticSegmentation implements AutoCloseable {
private static final long CHANNELS = 3;
private static final float REQUIRED_INPUT_IMAGE_SIZE = 513f;
private final GraphRunner imageNormalization;
private final GraphRunner semanticSegmentation;
private final GraphRunner maskImageEncoding;
private final GraphRunner alphaBlending;
private final Tensor<Integer> colorMapTensor;
private final Tensor<Float> maskTransparencyTensor;
@Override
public void close() {
this.imageNormalization.close();
this.semanticSegmentation.close();
this.maskImageEncoding.close();
this.alphaBlending.close();
this.colorMapTensor.close();
this.maskTransparencyTensor.close();
}
public SemanticSegmentation(String modelUrl, int[][] colorMap, long[] labelFilter, float maskTransparency) {
this.imageNormalization = new GraphRunner("input_image", "resized_image")
.withGraphDefinition(tf -> {
Placeholder<String> input = tf.withName("input_image").placeholder(String.class);
ExtractJpegShape<Integer> imageShapeAndChannel = tf.image.extractJpegShape(input);
Gather<Integer> imageShape = tf.gather(imageShapeAndChannel, tf.constant(new int[] { 0, 1 }), tf.constant(0));
Cast<Float> maxSize = tf.dtypes.cast(tf.max(imageShape, tf.constant(0)), Float.class);
Div<Float> scale = tf.math.div(tf.constant(REQUIRED_INPUT_IMAGE_SIZE), maxSize);
Cast<Integer> newSize = tf.dtypes.cast(tf.math.mul(scale, tf.dtypes.cast(imageShape, Float.class)), Integer.class);
final Operand<Float> decodedImage =
tf.dtypes.cast(tf.image.decodeJpeg(input, DecodeJpeg.channels(CHANNELS)), Float.class);
final Operand<Float> resizedImageFloat =
tf.image.resizeBilinear(tf.expandDims(decodedImage, tf.constant(0)), newSize);
tf.withName("resized_image").dtypes.cast(resizedImageFloat, UInt8.class);
});
this.semanticSegmentation = new GraphRunner("ImageTensor:0", "SemanticPredictions:0")
.withGraphDefinition(new ProtoBufGraphDefinition(new DefaultResourceLoader().getResource(modelUrl), true));
this.colorMapTensor = Tensor.create(colorMap).expect(Integer.class);
this.maskImageEncoding = new GraphRunner(Arrays.asList("color_map", "mask_pixels"), Arrays.asList("mask_png", "mask_rgb"))
.withGraphDefinition(tf -> {
Placeholder<Integer> colorTable = tf.withName("color_map").placeholder(Integer.class);
Placeholder<Long> batchedMask = tf.withName("mask_pixels").placeholder(Long.class);
// Remove batch dimension
Squeeze<Long> mask = tf.squeeze(batchedMask, Squeeze.axis(Arrays.asList(0L)));
Operand<Long> filteredMask = labelFilter(tf, mask, labelFilter);
// The mask can contain label values larger than the list of colors provided in the color map.
// To avoid out-of-index errors we will "normalize" the label values in the mask to MOD max-color-table-value.
Operand<Long> mask3 = NativeImageUtils.normalizeMaskLabels(tf, colorTable, filteredMask);
Gather<Integer> maskRgb = tf.withName("mask_rgb").gather(colorTable, mask3, tf.constant(0));
Operand<String> png = tf.withName("mask_png").image.encodePng(tf.dtypes.cast(maskRgb, UInt8.class));
});
this.maskTransparencyTensor = Tensor.create(maskTransparency).expect(Float.class);
this.alphaBlending = new GraphRunner(
Arrays.asList("input_image", "mask_image", "mask_transparency"), Arrays.asList("blended_png"))
.withGraphDefinition(tf -> {
// Input image [B, H, W, 3]
Cast<Float> inputImageRgb = tf.dtypes.cast(tf.withName("input_image").placeholder(UInt8.class), Float.class);
Placeholder<Integer> a = tf.withName("mask_image").placeholder(Integer.class);
Cast<Float> maskRgb = tf.dtypes.cast(a, Float.class);
Squeeze<Float> inputImageRgb2 = tf.squeeze(inputImageRgb, Squeeze.axis(Arrays.asList(0L)));
Placeholder<Float> maskTransparencyHolder = tf.withName("mask_transparency").placeholder(Float.class);
// Blend the transparent maskImage on top of the input image.
Operand<Float> blended = NativeImageUtils.alphaBlending(tf, maskRgb, inputImageRgb2, maskTransparencyHolder);
// Cut
//Operand<Boolean> condition = tf.math.equal(a, tf.zerosLike(a));
//Operand<Float> blended = tf.where3(condition, tf.zerosLike(maskRgb), inputImageRgb2);
// Encode PNG
tf.withName("blended_png").image.encodePng(tf.dtypes.cast(blended, UInt8.class));
});
}
public byte[] blendMask(byte[] image) {
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
Map<String, Tensor<?>> blendedTensors =
this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
.andThen(Functions.rename("SemanticPredictions:0", "mask_pixels")) // (SemanticPredictions:0) -> (mask_pixels)
.andThen(Functions.enrichWith("color_map", this.colorMapTensor)) // (mask_pixels) -> (mask_pixels, color_map)
.andThen(this.maskImageEncoding).andThen(memory) // (color_map, mask_pixels) -> (mask_png, mask_rgb) and memorize (mask_png, mask_rgb)
.andThen(Functions.enrichFromMemory(
memory, "resized_image")) // (mask_png, mask_rgb) -> (mask_png, mask_rgb, resized_image), e.g. join the normalizedImageTensor
.andThen(Functions.rename(
"resized_image", "input_image",
"mask_rgb", "mask_image")) // (mask_png, mask_rgb, resized_image) -> (mask_image, input_image)
.andThen(Functions.enrichWith("mask_transparency", this.maskTransparencyTensor)) // (mask_image, input_image) -> (mask_image, input_image, mask_transparency)
.andThen(this.alphaBlending).andThen(memory) // (mask_image, input_image, mask_transparency) -> (blended_png)
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
byte[] blendedImage = blendedTensors.get("blended_png").bytesValue();
memory.getTensorMap().entrySet().stream().forEach(e -> System.out.println(e));
return blendedImage;
}
}
public long[][] maskPixels(byte[] image) {
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
return this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
.andThen(tensorMap -> {
Tensor<?> maskTensor = tensorMap.get("SemanticPredictions:0");
int width = (int) maskTensor.shape()[1];
int height = (int) maskTensor.shape()[2];
return maskTensor.copyTo(new long[1][width][height])[0]; // 1 == batch size
})
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
}
}
public byte[] maskImage(byte[] image) {
try (Tensor inputTensor = Tensor.create(image); GraphRunnerMemory memory = new GraphRunnerMemory()) {
return this.imageNormalization.andThen(memory) // (input_image) -> (resized_image) and memorize (resized_image)
.andThen(this.semanticSegmentation).andThen(memory) // (ImageTensor:0) -> (SemanticPredictions:0) and memorize (SemanticPredictions:0)
.andThen(Functions.rename("SemanticPredictions:0", "mask_pixels")) // (SemanticPredictions:0) -> (mask_pixels)
.andThen(Functions.enrichWith("color_map", this.colorMapTensor)) // (mask_pixels) -> (mask_pixels, color_map)
.andThen(this.maskImageEncoding).andThen(memory) // (color_map, mask_pixels) -> (mask_png, mask_rgb) and memorize (mask_png, mask_rgb)
.andThen(tensorMap -> tensorMap.get("mask_png").bytesValue())
.apply(Collections.singletonMap("input_image", inputTensor)); // () -> (input_image)
}
}
private Operand<Long> labelFilter(Ops tf, Operand<Long> mask, long[] labels) {
if (labels == null || labels.length == 0) {
return mask;
}
ZerosLike<Long> zeroMask = tf.zerosLike(mask);
Operand<Long> result = zeroMask;
for (long label : labels) {
Add<Long> labelMask = tf.math.add(tf.zerosLike(mask), tf.constant(label));
Equal condition = tf.math.equal(mask, labelMask);
result = tf.math.add(result, tf.where3(condition, labelMask, zeroMask));
}
return result;
}
public static void main(String[] args) throws IOException {
//String inputImageUri = "file:/Users/ctzolov/Dev/projects/mindmodel/mind-model-services/semantic-segmentation/src/test/resources/images/VikiMaxiAdi.jpg";
String outputBlendedImagePath = "./semantic-segmentation/target/blendedImage.png";
String outputMaskImagePath = "./semantic-segmentation/target/maskImage.png";
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
"http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb",
SegmentationColorMap.CITYMAP_COLORMAP, null, 0.45f)
) {
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/amsterdam-cityscape1.jpg");
// 1. Mask pixels
long[][] maskPixels = segmentationService.maskPixels(inputImage);
// 2. Alpha Blending
byte[] blended = segmentationService.blendMask(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", new File(outputBlendedImagePath));
// 3. Mask Image
byte[] maskImage = segmentationService.maskImage(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", new File(outputMaskImagePath));
}
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
"http://download.tensorflow.org/models/deeplabv3_xception_ade20k_train_2018_05_29.tar.gz#frozen_inference_graph.pb",
SegmentationColorMap.ADE20K_COLORMAP, null, 0.45f)
) {
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/interior.jpg");
// 1. Mask pixels
long[][] maskPixels = segmentationService.maskPixels(inputImage);
// 2. Alpha Blending
byte[] blended = segmentationService.blendMask(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png",
new File("./semantic-segmentation/target/inventory-blendedImage.png"));
// 3. Mask Image
byte[] maskImage = segmentationService.maskImage(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png",
new File("./semantic-segmentation/target/inventory-MaskImage.png"));
}
try (SemanticSegmentation segmentationService = new SemanticSegmentation(
"http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb",
SegmentationColorMap.BLACK_WHITE_COLORMAP, null, 0.45f)
) {
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg");
// 1. Mask pixels
long[][] maskPixels = segmentationService.maskPixels(inputImage);
// 2. Alpha Blending
byte[] blended = segmentationService.blendMask(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png",
new File("./semantic-segmentation/target/pascal-blendedImage.png"));
// 3. Mask Image
byte[] maskImage = segmentationService.maskImage(inputImage);
ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png",
new File("./semantic-segmentation/target/pascal-MaskImage.png"));
}
}
}

289
function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/attic/SemanticSegmentationUtils.java Normal file
View File

@@ -0,0 +1,289 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.semantic.segmentation.attic;
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.Image;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Base64;
import java.util.Collections;
import java.util.Map;
import javax.imageio.ImageIO;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.tensorflow.Tensor;
import org.tensorflow.types.UInt8;
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
import org.springframework.cloud.fn.common.tensorflow.deprecated.TensorFlowService;
import org.springframework.core.io.DefaultResourceLoader;
import static java.awt.image.BufferedImage.TYPE_3BYTE_BGR;
/**
*
* Semantic image segmentation - the task of assigning a semantic label, such as “road”, “sky”, “person”, “dog”, to
* every pixel in an image.
*
* https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html
* https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
* https://github.com/tensorflow/models/tree/master/research/deeplab
* https://github.com/tensorflow/models/blob/master/research/deeplab/deeplab_demo.ipynb
* http://presentations.cocodataset.org/Places17-GMRI.pdf
*
* http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html
* https://www.cityscapes-dataset.com/dataset-overview/#class-definitions
* http://groups.csail.mit.edu/vision/datasets/ADE20K/
*
* https://github.com/mapillary/inplace_abn
*
* @author Christian Tzolov
*/
public class SemanticSegmentationUtils {
/** INPUT_TENSOR_NAME . */
public static final String INPUT_TENSOR_NAME = "ImageTensor:0";
/** OUTPUT_TENSOR_NAME . */
public static final String OUTPUT_TENSOR_NAME = "SemanticPredictions:0";
private static final int BATCH_SIZE = 1;
private static final long CHANNELS = 3;
private static final int REQUIRED_INPUT_IMAGE_SIZE = 513;
public static BufferedImage scaledImage(String imagePath) {
try {
return scaledImage(ImageIO.read(new DefaultResourceLoader().getResource(imagePath).getInputStream()));
}
catch (IOException e) {
throw new IllegalStateException("Failed to load Image from: " + imagePath, e);
}
}
public static BufferedImage scaledImage(byte[] image) {
try {
return scaledImage(ImageIO.read(new ByteArrayInputStream(image)));
}
catch (IOException e) {
throw new IllegalStateException("Failed to load Image from byte array", e);
}
}
public static BufferedImage scaledImage(BufferedImage image) {
double scaleRatio = 1.0 * REQUIRED_INPUT_IMAGE_SIZE / Math.max(image.getWidth(), image.getHeight());
return scale(image, scaleRatio);
}
private static BufferedImage scale(BufferedImage originalImage, double scale) {
int newWidth = (int) (originalImage.getWidth() * scale);
int newHeight = (int) (originalImage.getHeight() * scale);
Image tmpImage = originalImage.getScaledInstance(newWidth, newHeight, Image.SCALE_DEFAULT);
//BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, TYPE_INT_BGR);
BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, TYPE_3BYTE_BGR);
//BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, originalImage.getType());
Graphics2D g2d = resizedImage.createGraphics();
g2d.drawImage(tmpImage, 0, 0, null);
g2d.dispose();
return resizedImage;
}
public static BufferedImage blendMask(BufferedImage mask, BufferedImage background) {
GraphicsUtils.overlayImages(background, mask, 0, 0);
return background;
}
public static Tensor<UInt8> createInputTensor(BufferedImage scaledImage) {
if (scaledImage.getType() != TYPE_3BYTE_BGR) {
throw new IllegalArgumentException(
String.format("Expected 3-byte BGR encoding in BufferedImage, found %d", scaledImage.getType()));
}
// ImageIO.read produces BGR-encoded images, while the model expects RGB.
byte[] data = bgrToRgb(toBytes(scaledImage));
// Expand dimensions since the model expects images to have shape: [1, None, None, 3]
long[] shape = new long[] { BATCH_SIZE, scaledImage.getHeight(), scaledImage.getWidth(), CHANNELS };
return Tensor.create(UInt8.class, shape, ByteBuffer.wrap(data));
}
private static byte[] bgrToRgb(byte[] brgImage) {
byte[] rgbImage = new byte[brgImage.length];
for (int i = 0; i < brgImage.length; i += 3) {
rgbImage[i] = brgImage[i + 2];
rgbImage[i + 1] = brgImage[i + 1];
rgbImage[i + 2] = brgImage[i];
}
return rgbImage;
}
private static byte[] toBytes(BufferedImage bufferedImage) {
return ((DataBufferByte) bufferedImage.getRaster().getDataBuffer()).getData();
}
public static BufferedImage createMaskImage(int[][] maskPixels, int width, int height, double transparency) {
maskPixels = rotate(maskPixels);
int maskWidth = maskPixels.length;
int maskHeight = maskPixels[0].length;
int[] maskArray = new int[maskWidth * maskHeight];
int k = 0;
for (int i = 0; i < maskHeight; i++) {
for (int j = 0; j < maskWidth; j++) {
Color c = (maskPixels[j][i] == 0) ? Color.BLACK : GraphicsUtils.getClassColor(maskPixels[j][i]);
int t = (int) (255 * (1 - transparency));
maskArray[k++] = new Color(c.getRed(), c.getGreen(), c.getBlue(), t).getRGB();
}
}
// Turn the pixel array into image;
BufferedImage maskImage = new BufferedImage(maskWidth, maskHeight, BufferedImage.TYPE_INT_ARGB);
maskImage.setRGB(0, 0, maskWidth, maskHeight, maskArray, 0, maskWidth);
// Stretch the image to fit the target box width and height!
return GraphicsUtils.toBufferedImage(maskImage.getScaledInstance(width, height, Image.SCALE_SMOOTH));
}
/**
* rotate clockwise in 90 degree.
* @param input The 2D matrix to be rotated
* @return The input matrix rotated clockwise in 90 degrees
*/
private static int[][] rotate(int[][] input) {
int w = input.length;
int h = input[0].length;
int[][] output = new int[h][w];
for (int y = 0; y < h; y++) {
for (int x = w - 1; x >= 0; x--) {
output[y][x] = input[x][y];
}
}
return output;
}
public static int[][] toIntArray(long[][] longArray) {
int[][] intArray = new int[longArray.length][longArray[0].length];
for (int i = 0; i < longArray.length; i++) {
for (int j = 0; j < longArray[0].length; j++) {
intArray[i][j] = (int) longArray[i][j];
}
}
return intArray;
}
public String serializeToJson(int[][] pixels) {
String masksBase64 = Base64.getEncoder().encodeToString(toBytes(pixels));
return String.format("{ \"columns\":%d, \"rows\":%d, \"masks\":\"%s\"}", pixels.length, pixels[0].length, masksBase64);
}
public int[][] deserializeToMasks(String json) throws IOException {
Map<String, Object> map = new ObjectMapper().readValue(json, Map.class);
int cols = (int) map.get("columns");
int rows = (int) map.get("rows");
String masksBase64 = (String) map.get("masks");
byte[] masks = Base64.getDecoder().decode(masksBase64);
return toInts(masks, cols, rows);
}
private byte[] toBytes(int[][] pixels) {
byte[] b = new byte[pixels.length * pixels[0].length * 4];
int bi = 0;
for (int i = 0; i < pixels.length; i++) {
for (int j = 0; j < pixels[0].length; j++) {
b[bi + 0] = (byte) (i >> 24);
b[bi + 1] = (byte) (i >> 16);
b[bi + 2] = (byte) (i >> 8);
b[bi + 3] = (byte) (i /*>> 0*/);
bi = bi + 4;
}
}
return b;
}
private int[][] toInts(byte[] b, int ic, int jc) {
int[][] intResult = new int[ic][jc];
int bi = 0;
for (int i = 0; i < ic; i++) {
for (int j = 0; j < jc; j++) {
intResult[i][j] = (b[bi] << 24) +
(b[bi + 1] << 16) +
(b[bi + 2] << 8) +
b[bi + 3];
bi = bi + 4;
}
}
return intResult;
}
public static void main(String[] args) throws IOException {
// PASCAL VOC 2012
//String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_mnv2_pascal_train_aug/frozen_inference_graph.pb";
//String imagePath = "classpath:/images/VikiMaxiAdi.jpg";
// CITYSCAPE
//String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_mnv2_cityscapes_train/frozen_inference_graph.pb";
//String imagePath = "classpath:/images/amsterdam-cityscape1.jpg";
//String imagePath = "classpath:/images/amsterdam-channel.jpg";
//String imagePath = "classpath:/images/landsmeer.png";
// ADE20K
String tensorflowModelLocation = "file:/Users/ctzolov/Downloads/deeplabv3_xception_ade20k_train/frozen_inference_graph.pb";
String imagePath = "classpath:/images/interior.jpg";
BufferedImage inputImage = ImageIO.read(new DefaultResourceLoader().getResource(imagePath).getInputStream());
TensorFlowService tf = new TensorFlowService(new DefaultResourceLoader().getResource(tensorflowModelLocation), Arrays.asList(OUTPUT_TENSOR_NAME));
SemanticSegmentationUtils segmentationService = new SemanticSegmentationUtils();
BufferedImage scaledImage = segmentationService.scaledImage(inputImage);
Tensor<UInt8> inTensor = segmentationService.createInputTensor(scaledImage);
Map<String, Tensor<?>> output = tf.apply(Collections.singletonMap(INPUT_TENSOR_NAME, inTensor));
Tensor<?> maskPixelsTensor = output.get(OUTPUT_TENSOR_NAME);
int height = (int) maskPixelsTensor.shape()[1];
int width = (int) maskPixelsTensor.shape()[2];
long[][] maskPixels = maskPixelsTensor.copyTo(new long[BATCH_SIZE][height][width])[0]; // take 0 because the batch size is 1.
int[][] maskPixelsInt = segmentationService.toIntArray(maskPixels);
BufferedImage maskImage = segmentationService.createMaskImage(maskPixelsInt, scaledImage.getWidth(), scaledImage.getHeight(), 0.35);
BufferedImage blended = segmentationService.blendMask(maskImage, scaledImage);
ImageIO.write(maskImage, "png", new File("./semantic-segmentation/target/java2Dmask.jpg"));
ImageIO.write(blended, "png", new File("./semantic-segmentation/target/java2Dblended.jpg"));
}
}

BIN
function/semantic-segmentation-function/src/main/resources/images/ADE20K-all-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 184 KiB

BIN
function/semantic-segmentation-function/src/main/resources/images/VikiMaxiAdi-all.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 243 KiB

BIN
function/semantic-segmentation-function/src/main/resources/images/cityscape-all-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 236 KiB