Tensorflow functions and applications
* initial step
* Tensorflow models functional model redesign
-- Based on https://tzolov.github.io/mind-model-services
-- Resolves #5
* Add object detection processor README
* Add image recognition processor README
* Initial Tensorflow commonn README
* Initial Tensorflow commonn README
* Tensorflow common diagram
* Tensorflow docs code
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Tensorflow docs code snippets improve
* Add semantic segmentation function. add object detecteion function readme
* oo images
* Furether oo readme improvments
* Final obj detection readme fixes
* Add image recognition readme
* Add image recognition readme 2
* Semantic segmentation readme
* Segmentation readme
* Semantic segmentation readme 3
* Fix image recognition and object detcion app starter dependecies
* Add metadata for Tensorflow apps
This commit is contained in:
Christian Tzolov
committed by
Soby Chacko
Soby Chacko
parent
3bb9e066b9
commit
dffb467da4
103
function/image-recognition-function/README.adoc
Normal file
103
function/image-recognition-function/README.adoc
Normal file
@@ -0,0 +1,103 @@
|
||||
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/function/image-recognition-function/src/main/resources/images/
|
||||
|
||||
# Image Recognition
|
||||
|
||||
[.lead]
|
||||
Java model inference library for the https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] image recognition architectures.
|
||||
Provides real-time recognition of the https://dl.bintray.com/big-data/generic/imagenet_comp_graph_label_strings.txt[LSVRC-2012-CLS categories] in the input images.
|
||||
|
||||
[cols="1,2",frame=none,grid=none]
|
||||
|===
|
||||
| image:{images-asciidoc}/image-augmented.jpg[alt=Inception 1,width=100%]
|
||||
|The https://github.com/tzolov/stream-applications/tree/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition[ImageRecognition] takes an image and outputs a list of probable categories the image contains. The response is represented by https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/RecognitionResponse.java[RecognitionResponse] class.
|
||||
|
||||
The https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
|
||||
converting the `RecognitionResponse` into JSON objects and the
|
||||
https://github.com/tzolov/stream-applications/blob/tensorflow-redesign/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionAugmenter.java[ImageRecognitionAugmenter] can augment the input image with the detected categories (as shown in pic. 1).
|
||||
|===
|
||||
|
||||
## Usage
|
||||
|
||||
Add the `image-recognition` dependency to the pom (use the latest version available):
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>image-recognition-function</artifactId>
|
||||
<version>${spring-cloud-fn.version}</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
#### Example 1: Image Recognition
|
||||
|
||||
The following snippet demonstrates how to use the `ImageRecognition` for detecting the categories present in an input image.
|
||||
It also shows how to convert the result into JSON format and augment the input image with the detected category labels.
|
||||
|
||||
[source,java,linenums]
|
||||
----
|
||||
ImageRecognition recognitionService = ImageRecognition.mobilenetModeV2(
|
||||
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb", //<1>
|
||||
224, //<2>
|
||||
5, //<3>
|
||||
true); //<4>
|
||||
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg"); //<5>
|
||||
|
||||
List<RecognitionResponse> recognizedObjects = recognitionService.recognize(inputImage); //<6>
|
||||
----
|
||||
<1> Downloads and loads a pre-trained `mobilenet_v2_1.4_224_frozen.pb` model.
|
||||
Mind that on first attempt it will download few hundreds of MBs.
|
||||
The consecutive runs will use the cached copy (5) instead.
|
||||
The category labels for the MobileNetV2 are resolved from `src/main/resources/labels/mobilenet_labels.txt`.
|
||||
<2> The wxh sieze of the input nomralized image.
|
||||
<3> Top K result to return.
|
||||
<4> Cache the model on the local file system.
|
||||
<5> Load the image to recognise.
|
||||
<6> Return a map of the top-k most probable category names and their probabilities.
|
||||
|
||||
The `ImageRecognition.mobilenetModeV1` and `ImageRecognition.inception` factory methods help to load and configure pretrained mobilenetModeV1 and and Inception models.
|
||||
|
||||
Next you can convert the result in JSON format.
|
||||
|
||||
[source,java,linenums]
|
||||
----
|
||||
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
|
||||
----
|
||||
|
||||
.Sample Image Recognition JSON representation
|
||||
[source,json]
|
||||
----
|
||||
[{"label":"giant panda","probability":0.9946687817573547},{"label":"Arctic fox","probability":0.0036631098482757807},{"label":"ice bear","probability":3.3782739774324E-4},{"label":"American black bear","probability":2.3452856112271547E-4},{"label":"skunk","probability":1.6454080468975008E-4}]
|
||||
----
|
||||
|
||||
Use the `ImageRecognitionAugmenter` to draw the recognise categories on top of the input image.
|
||||
|
||||
[source,java,linenums]
|
||||
----
|
||||
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects); //<1>
|
||||
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented.jpg"));//<2>
|
||||
----
|
||||
<1> Augment the image with the recognized categories (uses Java2D internally).
|
||||
<2> Stores the augmented image as `image-augmented.jpg` image file.
|
||||
|
||||
.Augmented image-augmented.jpg file
|
||||
image:{images-asciidoc}/image-recognition-panda-augmented.jpg[alt=Augmented,width=30%]
|
||||
|
||||
## Models
|
||||
|
||||
This implementation supports all pre-trained https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] models.
|
||||
Following URI notation can be used to download any of the models directly from the zoo.
|
||||
|
||||
----
|
||||
http://<zoo model tar.gz url>#<frozen inference graph name.pb>
|
||||
----
|
||||
|
||||
The `<frozen inference graph name.pb>` is the frozen model file name within the archive.
|
||||
|
||||
|
||||
TIP: To speedup the bootstrap performance you may consider extracting the model and caching it locally.
|
||||
Then you can use the `file://path-to-my-local-copy` URI schema to access it.
|
||||
|
||||
NOTE: It is important to use the labels that correspond to the model being used!
|
||||
Table below highlights this mapping.
|
||||
38
function/image-recognition-function/pom.xml
Normal file
38
function/image-recognition-function/pom.xml
Normal file
@@ -0,0 +1,38 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>image-recognition-function</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>image-recognition-function</name>
|
||||
<description>Spring Native Function for Tensorflow Integration</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<properties>
|
||||
<commons-io.version>1.3.2</commons-io.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>tensorflow-common</artifactId>
|
||||
<version>${spring-cloud-fn.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<id>tensorflow-snapshots</id>
|
||||
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
</repository>
|
||||
</repositories>
|
||||
</project>
|
||||
@@ -0,0 +1,272 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.tensorflow.Operand;
|
||||
import org.tensorflow.Tensor;
|
||||
import org.tensorflow.op.core.Placeholder;
|
||||
import org.tensorflow.op.image.DecodeJpeg;
|
||||
import org.tensorflow.op.nn.TopK;
|
||||
|
||||
import org.springframework.cloud.fn.common.tensorflow.GraphRunner;
|
||||
import org.springframework.cloud.fn.common.tensorflow.GraphRunnerMemory;
|
||||
import org.springframework.cloud.fn.common.tensorflow.ProtoBufGraphDefinition;
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class ImageRecognition implements AutoCloseable {
|
||||
|
||||
private final List<String> labels;
|
||||
private final GraphRunner imageNormalization;
|
||||
private final GraphRunner imageRecognition;
|
||||
private final GraphRunner maxProbability;
|
||||
private final GraphRunner topKProbabilities;
|
||||
|
||||
/**
|
||||
* Instead of creating the {@link ImageRecognition} service explicitly via the constructor,
|
||||
* you should consider the convenience factory methods below. E.g.
|
||||
*
|
||||
* {@link #inception(String, int, int, boolean)}
|
||||
* {@link #mobileNetV1(String, int, int, boolean)}
|
||||
* {@link #mobileNetV2(String, int, int, boolean)}
|
||||
*
|
||||
* @param modelUri location of the pre-trained model to use.
|
||||
* @param labelsUri location of the list fromMemory pre-trained categories used by the model.
|
||||
* @param imageRecognitionGraphInputName name of the Model's input node to send the input image to.
|
||||
* @param imageRecognitionGraphOutputName name of the Model's output node to retrieve the predictions from.
|
||||
* @param imageHeight normalized image height.
|
||||
* @param imageWidth normalized image width.
|
||||
* @param mean mean value to normalize the input image.
|
||||
* @param scale scale to normalize the input image.
|
||||
* @param responseSize Max number of predictions per recognize.
|
||||
* @param cacheModel if true the pre-trained model is cached on the local file system.
|
||||
*/
|
||||
public ImageRecognition(String modelUri, String labelsUri, int imageHeight, int imageWidth, float mean, float scale,
|
||||
String imageRecognitionGraphInputName, String imageRecognitionGraphOutputName, int responseSize, boolean cacheModel) {
|
||||
|
||||
this.labels = labels(labelsUri);
|
||||
|
||||
/**
|
||||
* Normalizes the raw input image into format expected by the pre-trained Inception/MobileNetV1/MobileNetV2 models.
|
||||
* Typically the model is trained fromMemory images scaled to certain size. Usually it is 224x224 pixels, but can be
|
||||
* also 192x192, 160x160, 128128, 92x92. Use the (imageHeight, imageWidth) to set the desired size.
|
||||
* The colors, represented as R, G, B in 1-byte each were converted to float using (Value - Mean)/Scale.
|
||||
*
|
||||
* imageHeight normalized image height.
|
||||
* imageWidth normalized image width.
|
||||
* mean mean value to normalize the input image.
|
||||
* scale scale to normalize the input image.
|
||||
*/
|
||||
this.imageNormalization = new GraphRunner("raw_image", "normalized_image")
|
||||
.withGraphDefinition(tf -> {
|
||||
Placeholder<String> input = tf.withName("raw_image").placeholder(String.class);
|
||||
final Operand<Float> decodedImage =
|
||||
tf.dtypes.cast(tf.image.decodeJpeg(input, DecodeJpeg.channels(3L)), Float.class);
|
||||
final Operand<Float> resizedImage = tf.image.resizeBilinear(
|
||||
tf.expandDims(decodedImage, tf.constant(0)),
|
||||
tf.constant(new int[] { imageHeight, imageWidth }));
|
||||
tf.withName("normalized_image").math.div(tf.math.sub(resizedImage, tf.constant(mean)), tf.constant(scale));
|
||||
});
|
||||
|
||||
this.imageRecognition = new GraphRunner(imageRecognitionGraphInputName, imageRecognitionGraphOutputName)
|
||||
.withGraphDefinition(new ProtoBufGraphDefinition(toResource(modelUri), cacheModel));
|
||||
|
||||
this.maxProbability = new GraphRunner(Arrays.asList("recognition_result"), Arrays.asList("category", "probability"))
|
||||
.withGraphDefinition(tf -> {
|
||||
Placeholder<Float> input = tf.withName("recognition_result").placeholder(Float.class);
|
||||
tf.withName("category").math.argMax(input, tf.constant(1));
|
||||
tf.withName("probability").max(input, tf.constant(1));
|
||||
});
|
||||
|
||||
this.topKProbabilities = new GraphRunner("recognition_result", "topK")
|
||||
.withGraphDefinition(tf -> {
|
||||
Placeholder<Float> input = tf.withName("recognition_result").placeholder(Float.class);
|
||||
tf.withName("topK").nn.topK(input, tf.constant(responseSize), TopK.sorted(true));
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Takes an byte encoded image and returns the most probable category recognized in the image along fromMemory its probability.
|
||||
* @param inputImage Byte array encoded image to recognize.
|
||||
* @return Returns a single map entry containing the names of the recognized categories as key and the confidence as value.
|
||||
*/
|
||||
public Map<String, Double> recognizeMax(byte[] inputImage) {
|
||||
|
||||
try (Tensor inputTensor = Tensor.create(inputImage); GraphRunnerMemory memorize = new GraphRunnerMemory()) {
|
||||
|
||||
Map<String, Tensor<?>> max = this.imageNormalization.andThen(memorize)
|
||||
.andThen(this.imageRecognition).andThen(memorize)
|
||||
.andThen(this.maxProbability).andThen(memorize)
|
||||
.apply(Collections.singletonMap("raw_image", inputTensor));
|
||||
|
||||
long[] category = new long[1];
|
||||
max.get("category").copyTo(category);
|
||||
float[] probability = new float[1];
|
||||
max.get("probability").copyTo(probability);
|
||||
|
||||
return Collections.singletonMap(labels.get((int) category[0]), Double.valueOf(probability[0]));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Takes an byte encoded input image and returns the top K most probable categories recognized in the image
|
||||
* along fromMemory their probabilities.
|
||||
*
|
||||
* @param inputImage Byte array encoded image to recognize.
|
||||
* @return Returns a list of key-value pairs. Every key-value pair represents a single category recognized.
|
||||
* The key stands for the name(s) of the category while the value states the confidence that there is an
|
||||
* object of this category. The entries in the Map are ordered from the higher to the lower confidences.
|
||||
*/
|
||||
public Map<String, Double> recognizeTopK(byte[] inputImage) {
|
||||
|
||||
try (Tensor inputTensor = Tensor.create(inputImage); GraphRunnerMemory memorize = new GraphRunnerMemory()) {
|
||||
|
||||
Map<String, Tensor<?>> topKResults =
|
||||
this.imageNormalization.andThen(memorize)
|
||||
.andThen(this.imageRecognition).andThen(memorize)
|
||||
.andThen(this.topKProbabilities).andThen(memorize)
|
||||
.apply(Collections.singletonMap("raw_image", inputTensor));
|
||||
|
||||
Tensor recognizedImagesTensor = memorize.getTensorMap().get(this.imageRecognition.getSingleFetchName());
|
||||
float[][] results = new float[(int) recognizedImagesTensor.shape()[0]][(int) recognizedImagesTensor.shape()[1]];
|
||||
recognizedImagesTensor.copyTo(results);
|
||||
|
||||
Tensor<Float> topKTensor = topKResults.get("topK").expect(Float.class);
|
||||
float[][] topK = new float[(int) topKTensor.shape()[0]][(int) topKTensor.shape()[1]];
|
||||
topKTensor.copyTo(topK);
|
||||
|
||||
float min = topK[0][topK[0].length - 1];
|
||||
|
||||
Map<Float, Integer> valueToIndex = new HashMap<>();
|
||||
for (int i = 0; i < results[0].length; i++) {
|
||||
if (results[0][i] >= min) {
|
||||
valueToIndex.put(results[0][i], i);
|
||||
}
|
||||
}
|
||||
|
||||
Map<String, Double> map = new LinkedHashMap<>();
|
||||
for (float tk : topK[0]) {
|
||||
map.put(labels.get(valueToIndex.get(tk)), (double) tk);
|
||||
}
|
||||
|
||||
return map;
|
||||
}
|
||||
}
|
||||
|
||||
private Resource toResource(String uri) {
|
||||
return new DefaultResourceLoader().getResource(uri);
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a labels resources into string list.
|
||||
* @return Returns string lists. One line per different category.
|
||||
*/
|
||||
private List<String> labels(String labelsUri) {
|
||||
try (InputStream is = toResource(labelsUri).getInputStream()) {
|
||||
return Arrays.asList(StreamUtils.copyToString(is, Charset.forName("UTF-8")).split("\n"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Failed to initialize the Vocabulary", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* The Inception graph uses "input" as input and "output" as output.
|
||||
*
|
||||
*/
|
||||
public static ImageRecognition inception(String inceptionModelUri,
|
||||
int normalizedImageSize, int responseSize, boolean cacheModel) {
|
||||
return new ImageRecognition(inceptionModelUri, "classpath:/labels/inception_labels.txt",
|
||||
normalizedImageSize, normalizedImageSize, 117f, 1f,
|
||||
"input", "output",
|
||||
responseSize, cacheModel);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience for MobileNetV2 pre-trained models:
|
||||
* https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models
|
||||
*
|
||||
* The normalized image size is always square (e.g. H=W)
|
||||
*
|
||||
* The MobileNetV2 graph uses "input" as input and "MobilenetV2/Predictions/Reshape_1" as output.
|
||||
*
|
||||
* @param mobileNetV2ModelUri model uri
|
||||
* @param normalizedImageSize Depends on the pre-trained model used. Usually 224px is used.
|
||||
* @param responseSize Number of responses fot topK requests.
|
||||
* @param cacheModel cache model
|
||||
* @return ImageRecognition instance configured fromMemory a MobileNetV2 pre-trained model.
|
||||
*/
|
||||
public static ImageRecognition mobileNetV2(String mobileNetV2ModelUri,
|
||||
int normalizedImageSize, int responseSize, boolean cacheModel) {
|
||||
return new ImageRecognition(mobileNetV2ModelUri, "classpath:/labels/mobilenet_labels.txt",
|
||||
normalizedImageSize, normalizedImageSize, 0f, 127f,
|
||||
"input", "MobilenetV2/Predictions/Reshape_1",
|
||||
responseSize, cacheModel);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience for MobileNetV1 pre-trained models:
|
||||
* https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models
|
||||
*
|
||||
* The MobileNetV1 graph uses "input" as input and "MobilenetV1/Predictions/Reshape_1" as output.
|
||||
*
|
||||
*/
|
||||
public static ImageRecognition mobileNetV1(String mobileNetV1ModelUri,
|
||||
int normalizedImageSize, int responseSize, boolean cacheModel) {
|
||||
return new ImageRecognition(mobileNetV1ModelUri, "classpath:/labels/mobilenet_labels.txt",
|
||||
normalizedImageSize, normalizedImageSize,
|
||||
0f, 127f,
|
||||
"input", "MobilenetV1/Predictions/Reshape_1",
|
||||
responseSize, cacheModel);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert image recognition results into {@link RecognitionResponse} domain list.
|
||||
* @param recognitionMap map containing the category mames and its probability. Returned by the
|
||||
* {@link ImageRecognition#recognizeMax(byte[])} and the ImageRecognition{@link #recognizeTopK(byte[])} methods
|
||||
* @return List of {@link RecognitionResponse} objects representing the name-to-probability pairs in the input map.
|
||||
*/
|
||||
public static List<RecognitionResponse> toRecognitionResponse(Map<String, Double> recognitionMap) {
|
||||
return recognitionMap.entrySet().stream()
|
||||
.map(nameProbabilityPair -> new RecognitionResponse(nameProbabilityPair.getKey(), nameProbabilityPair.getValue()))
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
this.imageNormalization.close();
|
||||
this.imageRecognition.close();
|
||||
this.maxProbability.close();
|
||||
this.topKProbabilities.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
import java.awt.Color;
|
||||
import java.awt.FontMetrics;
|
||||
import java.awt.Graphics2D;
|
||||
import java.awt.RenderingHints;
|
||||
import java.awt.geom.Rectangle2D;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
import java.util.function.BiFunction;
|
||||
|
||||
import javax.imageio.ImageIO;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
|
||||
|
||||
/**
|
||||
* Ability to to augment the input image fromMemory the recognized labels.
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class ImageRecognitionAugmenter implements BiFunction<byte[], List<RecognitionResponse>, byte[]> {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(ImageRecognitionAugmenter.class);
|
||||
|
||||
/** IMAGE_FORMAT. */
|
||||
public static final String IMAGE_FORMAT = "jpg";
|
||||
|
||||
private final Color textColor = Color.BLACK;
|
||||
private final Color bgColor = new Color(167, 252, 0);
|
||||
|
||||
public ImageRecognitionAugmenter() {
|
||||
}
|
||||
|
||||
/**
|
||||
* Augment the input image by adding the recognized classes.
|
||||
*
|
||||
* @param imageBytes input image as byte array
|
||||
* @param result computed recognition labels
|
||||
* @return the image augmented fromMemory recognized labels.
|
||||
*/
|
||||
@Override
|
||||
public byte[] apply(byte[] imageBytes, List<RecognitionResponse> result) {
|
||||
try {
|
||||
if (result != null) {
|
||||
BufferedImage originalImage = ImageIO.read(new ByteArrayInputStream(imageBytes));
|
||||
|
||||
Graphics2D g = originalImage.createGraphics();
|
||||
g.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
|
||||
FontMetrics fm = g.getFontMetrics();
|
||||
|
||||
int x = 1;
|
||||
int y = 1;
|
||||
for (RecognitionResponse r : result) {
|
||||
|
||||
String labelName = r.getLabel();
|
||||
int probability = (int) (100 * r.getProbability());
|
||||
String title = labelName + ": " + probability + "%";
|
||||
|
||||
Rectangle2D rect = fm.getStringBounds(title, g);
|
||||
|
||||
g.setColor(bgColor);
|
||||
g.fillRect(x, y, (int) rect.getWidth() + 6, (int) rect.getHeight());
|
||||
|
||||
g.setColor(textColor);
|
||||
g.drawString(title, x + 3, (int) (y + rect.getHeight() - 3));
|
||||
y = (int) (y + rect.getHeight() + 1);
|
||||
}
|
||||
|
||||
ByteArrayOutputStream baos = new ByteArrayOutputStream();
|
||||
ImageIO.write(originalImage, IMAGE_FORMAT, baos);
|
||||
baos.flush();
|
||||
imageBytes = baos.toByteArray();
|
||||
baos.close();
|
||||
}
|
||||
}
|
||||
catch (IOException e) {
|
||||
logger.error("Failed to draw labels in the input image", e);
|
||||
}
|
||||
|
||||
return imageBytes;
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class RecognitionResponse {
|
||||
private String label;
|
||||
private Double probability;
|
||||
|
||||
public RecognitionResponse() {
|
||||
}
|
||||
|
||||
public RecognitionResponse(String label, Double probability) {
|
||||
this.label = label;
|
||||
this.probability = probability;
|
||||
}
|
||||
|
||||
public String getLabel() {
|
||||
return label;
|
||||
}
|
||||
|
||||
public void setLabel(String label) {
|
||||
this.label = label;
|
||||
}
|
||||
|
||||
public Double getProbability() {
|
||||
return probability;
|
||||
}
|
||||
|
||||
public void setProbability(Double probability) {
|
||||
this.probability = probability;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "{label='" + label + ", probability=" + probability + '}';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition.util;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.apache.commons.io.FileUtils;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.util.Assert;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* Create a text file mapping label id to human readable string.
|
||||
*
|
||||
* Produces a text file where every line represents single category. The line number represents the category id, while
|
||||
* the line text is human-readable names for the categories fromMemory this imagenet id.
|
||||
*
|
||||
* Based on https://github.com/tensorflow/models/blob/master/research/slim/datasets/imagenet.py#L66
|
||||
*
|
||||
* We retrieve a synset file, which contains a list of valid synset labels used
|
||||
* by ILSVRC competition. There is one synset one per line, eg.
|
||||
* # n01440764
|
||||
* # n01443537
|
||||
* We also retrieve a synset_to_human_file, which contains a mapping from synsets
|
||||
* to human-readable names for every synset in Imagenet. These are stored in a
|
||||
* tsv format, as follows:
|
||||
* # n02119247 black fox
|
||||
* # n02119359 silver fox
|
||||
* We assign each synset (in alphabetical order) an integer, starting from 1
|
||||
* (since 0 is reserved for the background class)
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class ImageNetReadableNamesWriter {
|
||||
|
||||
private ImageNetReadableNamesWriter() {
|
||||
}
|
||||
|
||||
/** BASE_URL. */
|
||||
public final static String BASE_URL = "https://raw.githubusercontent.com/tensorflow/models/master/research/inception/inception/data/";
|
||||
/** SYNSET_URI. */
|
||||
public final static String SYNSET_URI = BASE_URL + "imagenet_lsvrc_2015_synsets.txt";
|
||||
/** SYNSET_TO_HUMAN_URI. */
|
||||
public final static String SYNSET_TO_HUMAN_URI = BASE_URL + "imagenet_metadata.txt";
|
||||
|
||||
public static void main(String[] args) {
|
||||
Charset utf8 = Charset.forName("UTF-8");
|
||||
|
||||
try (InputStream synsetIs = toResource(SYNSET_URI).getInputStream();
|
||||
InputStream synsetToHumanIs = toResource(SYNSET_TO_HUMAN_URI).getInputStream()) {
|
||||
|
||||
List<String> synsetList = Arrays.asList(StreamUtils.copyToString(synsetIs, utf8)
|
||||
.split("\n")).stream().map(l -> l.trim()).collect(Collectors.toList());
|
||||
Assert.notNull(synsetList, "Failed to initialize the labels list");
|
||||
Assert.isTrue(synsetList.size() == 1000, "Labels list is expected to be of " +
|
||||
"size 1000 but was:" + synsetList.size());
|
||||
|
||||
Map<String, String> synsetToHuman = Arrays.asList(StreamUtils.copyToString(synsetToHumanIs, utf8)
|
||||
.split("\n")).stream().map(s2h -> s2h.split("\t")).collect(Collectors.toMap(s -> s[0], s -> s[1]));
|
||||
Assert.notNull(synsetToHuman, "Failed to initialize the synsetToHuman");
|
||||
Assert.isTrue(synsetToHuman.size() == 21842, "synsetToHuman is expected to be of " +
|
||||
"size 21842 but was:" + synsetToHuman.size());
|
||||
|
||||
List<String> l = synsetList.stream().map(id -> synsetToHuman.get(id)).collect(Collectors.toList());
|
||||
|
||||
List<String> ll = new ArrayList<>();
|
||||
ll.add("dummy");
|
||||
ll.addAll(l);
|
||||
System.out.println(ll.get(389));
|
||||
FileUtils.writeLines(new File("labels.txt"), ll);
|
||||
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Failed to initialize the Vocabulary", e);
|
||||
}
|
||||
}
|
||||
|
||||
public static Resource toResource(String uri) {
|
||||
return new DefaultResourceLoader().getResource(uri);
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 101 KiB |
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,103 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
import java.io.FileOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.commons.io.IOUtils;
|
||||
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class ImageRecognitionExample {
|
||||
|
||||
private ImageRecognitionExample() {
|
||||
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws IOException {
|
||||
|
||||
// You can use file:, http: or classpath: to provide the path to the input image.
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg");
|
||||
|
||||
// MmobileNetV2 models
|
||||
// https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models
|
||||
String mobilenet_v2_modelUri = "https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb";
|
||||
//String mobilenet_v2_modelUri = "https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_96.tgz#mobilenet_v2_0.35_96_frozen.pb";
|
||||
try (ImageRecognition imageRecognition = ImageRecognition.mobileNetV2(
|
||||
mobilenet_v2_modelUri,
|
||||
224,
|
||||
5,
|
||||
true)) {
|
||||
|
||||
List<RecognitionResponse> recognizedObjects =
|
||||
ImageRecognition.toRecognitionResponse(imageRecognition.recognizeTopK(inputImage));
|
||||
|
||||
// Draw the predicted labels on top of the input image.
|
||||
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
|
||||
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-mobilnetV2.jpg"));
|
||||
|
||||
|
||||
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
|
||||
System.out.println("mobilnetV2 result:" + jsonRecognizedObjects);
|
||||
}
|
||||
|
||||
|
||||
String mobilenet_v1_modelUri = "https://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz#mobilenet_v1_1.0_224_frozen.pb";
|
||||
try (ImageRecognition recognitionService = ImageRecognition.mobileNetV1(
|
||||
mobilenet_v1_modelUri,
|
||||
224,
|
||||
5,
|
||||
true)) {
|
||||
|
||||
List<RecognitionResponse> recognizedObjects =
|
||||
ImageRecognition.toRecognitionResponse(recognitionService.recognizeTopK(inputImage));
|
||||
|
||||
// Draw the predicted labels on top of the input image.
|
||||
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
|
||||
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-mobilnetV1.jpg"));
|
||||
|
||||
|
||||
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
|
||||
System.out.println("mobilnetV1 result:" + jsonRecognizedObjects);
|
||||
}
|
||||
|
||||
String inception_modelUri = "https://storage.googleapis.com/scdf-tensorflow-models/image-recognition/tensorflow_inception_graph.pb";
|
||||
try (ImageRecognition recognitionService = ImageRecognition.inception(
|
||||
inception_modelUri,
|
||||
224,
|
||||
5,
|
||||
true)) {
|
||||
|
||||
List<RecognitionResponse> recognizedObjects =
|
||||
ImageRecognition.toRecognitionResponse(recognitionService.recognizeTopK(inputImage));
|
||||
|
||||
// Draw the predicted labels on top of the input image.
|
||||
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects);
|
||||
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented-inception.jpg"));
|
||||
|
||||
|
||||
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
|
||||
System.out.println("inception result:" + jsonRecognizedObjects);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
import java.io.FileOutputStream;
|
||||
import java.io.IOException;
|
||||
|
||||
import org.apache.commons.io.IOUtils;
|
||||
|
||||
import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class ImageRecognitionExample2 {
|
||||
|
||||
private ImageRecognitionExample2() {
|
||||
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws IOException {
|
||||
|
||||
ImageRecognitionAugmenter augmenter = new ImageRecognitionAugmenter();
|
||||
|
||||
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg");
|
||||
|
||||
ImageRecognition inceptions = ImageRecognition.inception(
|
||||
"https://storage.googleapis.com/scdf-tensorflow-models/image-recognition/tensorflow_inception_graph.pb",
|
||||
224, 10, true);
|
||||
System.out.println(inceptions.recognizeMax(inputImage));
|
||||
System.out.println(inceptions.recognizeTopK(inputImage));
|
||||
System.out.println(ImageRecognition.toRecognitionResponse(inceptions.recognizeTopK(inputImage)));
|
||||
|
||||
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(inceptions.recognizeTopK(inputImage))),
|
||||
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-inceptions.jpg"));
|
||||
inceptions.close();
|
||||
|
||||
ImageRecognition mobileNetV2 = ImageRecognition.mobileNetV2(
|
||||
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb",
|
||||
224, 10, true);
|
||||
System.out.println(mobileNetV2.recognizeMax(inputImage));
|
||||
System.out.println(mobileNetV2.recognizeTopK(inputImage));
|
||||
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(mobileNetV2.recognizeTopK(inputImage))),
|
||||
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-mobilnetV2.jpg"));
|
||||
mobileNetV2.close();
|
||||
|
||||
ImageRecognition mobileNetV1 = ImageRecognition.mobileNetV1(
|
||||
"https://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz#mobilenet_v1_1.0_224_frozen.pb",
|
||||
224, 10, true);
|
||||
System.out.println(mobileNetV1.recognizeMax(inputImage));
|
||||
System.out.println(mobileNetV1.recognizeTopK(inputImage));
|
||||
IOUtils.write(augmenter.apply(inputImage, ImageRecognition.toRecognitionResponse(mobileNetV1.recognizeTopK(inputImage))),
|
||||
new FileOutputStream("./functions/function/image-recognition-function/target/image-augmented-mobilnetV1.jpg"));
|
||||
mobileNetV1.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.image.recognition;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
|
||||
import com.google.protobuf.InvalidProtocolBufferException;
|
||||
import org.tensorflow.SavedModelBundle;
|
||||
import org.tensorflow.framework.MetaGraphDef;
|
||||
import org.tensorflow.framework.SignatureDef;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public final class SavedModelTest {
|
||||
|
||||
private SavedModelTest() {
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* https://medium.com/@jsflo.dev/saving-and-loading-a-tensorflow-model-using-the-savedmodel-api-17645576527
|
||||
*
|
||||
* https://www.tensorflow.org/alpha/guide/saved_model
|
||||
*
|
||||
*/
|
||||
public static void main(String[] args) throws InvalidProtocolBufferException {
|
||||
SavedModelBundle savedModelBundle =
|
||||
SavedModelBundle.load("/Users/ctzolov/Downloads/ssd_mobilenet_v1_coco_2017_11_17/saved_model", "serve");
|
||||
//SavedModelBundle.load("/Users/ctzolov/Downloads/aiy_vision_classifier_plants_V1_1/", "serve");
|
||||
//SavedModelBundle savedModelBundle =
|
||||
// SavedModelBundle.load("/Users/ctzolov/Downloads/mnasnet-a1/saved_model", "serve");
|
||||
|
||||
MetaGraphDef meta = MetaGraphDef.parseFrom(savedModelBundle.metaGraphDef());
|
||||
|
||||
Map<String, SignatureDef> signatures = meta.getSignatureDefMap();
|
||||
|
||||
System.out.println(signatures);
|
||||
|
||||
savedModelBundle.session();
|
||||
|
||||
//Iterator<Operation> itr = savedModelBundle.graph().operations();
|
||||
//
|
||||
//while (itr.hasNext()) {
|
||||
// System.out.println("Operation: " + itr.next());
|
||||
//}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Reference in New Issue
Block a user