SpringCloud/spring-functions-catalog
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Tensorflow functions and applications

Browse Source 
* initial step
 * Tensorflow models functional model redesign

      -- Based on https://tzolov.github.io/mind-model-services
      -- Resolves #5

 * Add object detection processor README
 * Add image recognition processor README
 * Initial Tensorflow commonn README
 * Initial Tensorflow commonn README
 * Tensorflow common diagram
 * Tensorflow docs code
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Add semantic segmentation function. add object detecteion function readme
 * oo images
 * Furether oo readme improvments
 * Final obj detection readme fixes
 * Add image recognition readme
 * Add image recognition readme 2
 * Semantic segmentation readme
 * Segmentation readme
 * Semantic segmentation readme 3
 * Fix image recognition and object detcion app starter dependecies

 * Add metadata for Tensorflow apps
This commit is contained in:
Christian Tzolov
2020-06-11 17:07:44 +02:00
committed by Soby Chacko
parent 3bb9e066b9
commit dffb467da4
66 changed files with 10300 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

344
common/tensorflow-common/README.adoc Normal file
View File

@@ -0,0 +1,344 @@
:images-asciidoc: https://raw.githubusercontent.com/tzolov/stream-applications/tensorflow-redesign/functions/common/tensorflow-common/src/main/resources/images/
= Programming Model for TensorFlow Inference
Programming model builds on the https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html[Java Function API], the `TF Java Ops API` and few basic data structure that together help to unify and streamline the building of TensorFlow inference pipelines.
Quick Start: just add the following dependency:
[source,XML]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>tensorflow-common</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
----
== Programming Model
Implementing a real-time TensorFlow Inference in Java, typically leverages the TF Java API for loading and scoring the pre-trained models. But the logic used to convert the upstream data into model input Tensors (e.g. pre-processor) and in turn to convert the inferred Tensors back into application data (e.g. post-processor) is commonly implemented in plain Java:
image::{images-asciidoc}/programming_model.png[TF Architecture, scaledwidth="70%"]
The Pre and Post processing steps could become very complex (check https://github.com/ildoonet/tf-pose-estimation[Pose Estimation] or https://github.com/davidsandberg/facenet[Face Recognition]) and computationally intensive. E.g. tons of math and image operations that are better fit for optimized TF utilities rather the plain Java math or AWT/2D/Canvas such.
Additionally, the unnecessary shuffling of data between the JVM and the native TF impacts the overall performance of the flow. The issue is apparent when multiple pre-trained TF models are combined (composed) or the same TF models are evaluated iteratively. In those cases the processed data is repeatedly being moved between the JVM Heap and the TF native memory.
The `Java Ops API` exposes the native https://www.tensorflow.org/versions/r1.9/api_docs/cc?hl=en[TF C++ Core API] offering comprehensive, native math, image, io and alike utilities. Later is useful for implementing the computational intensive logic by using the same TF tools and infrastructure used for running the pre-trained models.
Proposed programming model builds on the https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html[Java Function API], the Java Ops API and a basic data structure that together help to unify and streamline the building of TensorFlow inference pipelines. While focused on model-inference, this programming model is likely to be useful for building model-training pipelines as well.
The programming model leverages the following functional definition:
[source,Java]
----
Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>>
----
and the corresponding method expression
[source,Java]
----
Map<String, Tensor<?>> apply( Map<String, Tensor<?>> feeds)
----
This function receives a map of named Tensors as an input and in turn returns a map of named Tensors. Because the input and the output formats are equivalent, functions with this signature can be reused and composed into larger, complex functions.
The names used in the inputs, and the outputs maps are strings of the form `operation_name: output_index` (output_index defaults to 0). The names must match the indexed operations inside the underlying TF graph. +
The https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/Tensor[Tensor] class is a reference to the data used natively in the TF engine. The referenced data is not moved to JVM Heap unless explicitly materialized with `Tensor.copyTo()`. Exchanging Tensor references between functions prevents unnecessarily copying the data to the JVM heap. +
Proposed data structure fits well with existing https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/Session.Runner[Session.Runner API], which accepts https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/Session.Runner.html#feed(java.lang.String,%20org.tensorflow.Tensor%3C?%3E)[indexed operations] as an input feed and returns list of tensors predefined by https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/Session.Runner.html#fetch(java.lang.String)[fetch indexed operations].
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java[GraphRunner] and the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphDefinition.java[GraphDefinition] are the core abstractions used to define, load and inference TensorFlow models. https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java[GraphRunner] implements the Function (e.g. Fn<Map<S,T>, Map<S,T>>) definition and uses the TF Java API to run the underlying TF graph. The input Tensor map is fed to the Session Runner. After the graph is evaluated, a list of predefined fetch names is used to retrieve selected Tensors from the result as a named Tensor map. The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java#L45[withGraphDefintition(GraphDefinition)] method is used to define a new or to load a pre-trained TF graph, while the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java#L60[withSavedModel(path)] method helps to load a Tensorflow SavedModel. +
The GraphDefinition argument is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
Following snippets illustrates how to use the `withGraphDefinition` to define a new TF Graph that computes the `y1 = x1 * 2` expression:
[source,Java]
----
myGraph = new GraphRunner("x1", "y1")
.withGraphDefinition( tf ->
tf.withName("y1").math.mul(
tf.withName("x1").placeholder(Integer.class),
tf.constant(2)));
----
The x1 and y1 constructor arguments define the input and output Tensor names (technically indexed operation names) used to feed in and fetch out data to and from the defined model.
The GraphRunner https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/AbstractGraphRunner.java#L49[apply] method helps evaluate/inference the so defined graph:
[source,Java]
----
Map<String, Tensor<?>> input = Collections.singletonMap("x1", Tensor.create(666));
result = myGraph.apply(input);
----
Similarly we can load a frozen/pre-trained model (https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] model in this case) from an archive using the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/ProtoBufGraphDefinition.java[ProtoBufGraphDefinitions] helper class.
[source,Java]
----
mibileNetV2 = new GraphRunner("input", "MobilenetV2/Predictions/Reshape_1")
.withGraphDefinition(
new ProtoBufGraphDefinition(
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb",
cacheModel));
----
You can load archives from `http://`, `file://` or `classpath://` locations.
For loading a `SavedModel` from the local file system use the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java#L68[withSavedModel] method like this:
[source,Java]
----
ssdMibileNetV1Coco = new GraphRunner( Arrays.asList("image_tensor"),
Arrays.asList("detection_boxes", “detection_scores”, “detection_classes”))
.withSavedModel( ”./ssd_mobilenet_v1_coco_2017_11_17/saved_model”, "serve");
----
An example inference pipeline based on the proposed Functional Programming Model would look similar to this:
image::{images-asciidoc}/tf_pipeline.png[TF pipelinne, scaledwidth="70%"]
Every GraphRunner instance in the pipeline uses either an in-place defined, or a pre-trained TF graph.
In practice, it would still be required to implement some input and output adapters for the logic that cannot be (or are not feasible to be) implemented with the native Java Ops API. But you have the freedom to choose what part of the processing logic to run as natively (e.g. Java Ops API) code and what is a plain Java.
Furthermore, we are not limited to GraphRunner but any custom https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html[Function]<Map<String, Tensor>, Map<String, Tensor>> implementations can be used in the processing pipelines. In fact the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/Functions.java[Functions] utilities use this approach.
When appropriate any custom Function, https://docs.oracle.com/javase/8/docs/api/java/util/function/Supplier.html[Supplier]<Map<String, Tensor>>, https://docs.oracle.com/javase/8/docs/api/java/util/function/Consumer.html[Consumer]<Map<String, Tensor>> or the rest of the https://docs.oracle.com/javase/8/docs/api/java/util/function/package-frame.html[java.util.function] classes and interfaces can be used.
For real time examples check the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/image-recognition/src/main/java/io/mindmodel/services/image/recognition/ImageRecognition.java[image-recognition] and https://github.com/tzolov/mind-model-services/blob/ops-programming-model/semantic-segmentation/src/main/java/io/mindmodel/services/semantic/segmentation/SemanticSegmentation.java[semantic-segmentation] implementations.
== Features
Following paragraphs discusses some features and techniques useful for composing graphs, memorizing and reusing intermediate Tensor values, managing the Tensor resources and so on.
=== Input and Output Contracts
The GraphRunner constructor expects two compulsory list fields: `feedNames` - list of names that the graph accepts as input Tensors and `fetchNames` - list of (Tensor) names that the graph would return.
[source,Java]
----
public GraphRunner(List<String> feedNames, List<String> fetchedNames)
----
Together those two lists define the input and output contract of the graph runner. +
The names used in the inputs, and the outputs maps are strings of the form `operation_name : output_index` (output_index defaults to 0). The names must match the indexed operations inside the underlying TF graph.
=== Composition
Because the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java[GraphRunner] function signature uses the same type for input and output parameters, the https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html[Functional] interface allows us compose multiple graph https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java[GraphRunner] functions into a larger composite function:
[source,Java]
----
composed-graph = graph1.andThen(graph2)....andThen(graphN)
----
For example let's take two simple graphs: `G1 (y1 = x1 * 2)` and `G2 (y2 = x2 + 20)`. The composed graph `G = G1.andThen(G2)` is equivalent to `y = (x * 2 ) + 20`.
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/test/java/io/mindmodel/services/common/examples/FunctionComposition.java[FunctionComposition example] demonstrates how this works:
[source,Java]
----
try (
GraphRunner graph1 = new GraphRunner("x1", "y1")
.withGraphDefinition(tf -> tf.withName("y1").math.mul(
tf.withName("x1").placeholder(Integer.class),
tf.constant(2)));
GraphRunner graph2 = new GraphRunner("x2", "y2")
.withGraphDefinition(tf -> tf.withName("y2").math.add(
tf.withName("x2").placeholder(Integer.class),
tf.constant(20)));
Tensor x = Tensor.create(10);
) {
Map<String, Tensor<?>> result =
graph1.andThen(graph2).apply(Collections.singletonMap("x", x));
System.out.println("Result is: " + result.get("y2").intValue()); // Result is: 40
}
----
Note that the GraphRunner https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/AbstractGraphRunner.java#L65[automatically binds] the singleton outputs (e.g fetchs) with the singleton input (e.g. feeds). In the example above the GraphRunner automatically binds the `y1` tensor produced by `graph1` to the `x2` input placeholders expected by `graph2`.
==== Multiple inputs/outputs
When the composed graphs use multiple input and output parameters we need to explicitly bind the outputs from the upstream graph to the inputs of the downstream one.
For example let’s Graph1 produces two outputs (e.g. fetchNames) y11 and y12 and Graph2 expects to inputs (e.g. feedNames) x21 and x22:
|===
|Graph1:|Graph2:
| y11 = x1 * 2 | y2 = x21 + x22
| y12 = x1 * 3 |
|===
The composed graph would look like this:
[source,Java]
----
Composed = Graph1.andThen( map: y11 -> x21 and y12 -> x22).andThen(Graph2)
----
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/Functions.java#L52[Functions#rename] utility helps to define the input/output mappings as illustrated in the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/test/java/io/mindmodel/services/common/examples/FunctionCompositionMultipleInputsOutputs.java[FunctionCompositionMultipleInputsOutputs] example:
[source,Java]
----
try (
GraphRunner graph1 = new GraphRunner(Arrays.asList("x1"), Arrays.asList("y11", "y12"))
.withGraphDefinition(tf -> {
Placeholder<Integer> x1 = tf.withName("x1").placeholder(Integer.class);
tf.withName("y11").math.mul(x1, tf.constant(2));
tf.withName("y12").math.mul(x1, tf.constant(3));
});
GraphRunner graph2 = new GraphRunner(Arrays.asList("x21", "x22"), Arrays.asList("y2"))
.withGraphDefinition(tf -> tf.withName("y2").math.add(
tf.withName("x21").placeholder(Integer.class),
tf.withName("x22").placeholder(Integer.class)));
Tensor x = Tensor.create(10);
) {
Map<String, Tensor<?>> result =
graph1
.andThen(
Functions.rename(
"y11", "x21",
"y12", "x22"
))
.andThen(graph2)
.apply(Collections.singletonMap("x", x));
System.out.println("Result is: " + result.get("y2").intValue()); // Result is: 50
}
----
The Functions#rename(String...mappings) takes an even number of string pairs, where every even parameter represents the from and to name to map. Eg. The y11 above is mapped into x21 and y12 is mapped into x22. +
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java#L120[GraphRunner#enableAutoBinding()] and https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunner.java#L115[GraphRunner#disableAutoBinding()] allow altering the autobinding behavior enforcing mapping even of singleton input/output graphs.
=== Save and Close Obsolete Tensors
The Tensors used as inputs (feeds) and outputs (fetches) by the GraphRunners have to be released (e.g. closed) when not used anymore.
Because every sub-graph in a composite pipeline produces one or more <String, Tensor> pairs we need to track those references and close them.
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunnerMemory.java[GraphRunnerMemory] is a handy utility Function implementation that keeps track of all input Tensor parameters passed through. It is https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html[AutoClosable] and will release all tracked Tensors when closed.
The GraphRunnerMemory implements the same function signatures as the GraphRunner (e.g. Fun<Map<S,T>, Map<S,T>>) and therefore can participate in composite graph definitions:
[source,Java]
----
try ( memory = new GraphRunnerMemory() ) {
composed-graph =
Graph1..andThen(memory)
.andThen(Graph2).andThen(memory)
…
.andThen(GraphN).andThen(memory)
….
} // releases all Tensors returned by the GraphRunners
----
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/test/java/io/mindmodel/services/common/examples/ReleaseTensorParameters.java[ReleaseTensorParameters] example illustrates how to use the GraphRunnerMemory:
[source,Java]
----
try (
Tensor x = Tensor.create(input);
GraphRunnerMemory memory = new GraphRunnerMemory();
) {
Map<String, Tensor<?>> result =
this.graph1.andThen(memory)
.andThen(this.graph2).andThen(memory)
.apply(Collections.singletonMap("x", x));
return result.get("y2").intValue();
}
// At that point all intermediate Tensors used by the GraphRunners are closed.
----
Note: the GraphRunnerMemory has some other very useful applications that we will highlight in the next paragraph.
=== Enrich Graph Inputs
For particular graphs in the composite pipeline, we can add an additional input parameters that were not produced by the upstream graph.
WIth the help fo the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/Functions.java#L22[Functions#enrichWith(name, Tensor)] utility function we can inject the additional parameters in the graph composition.
In the following snippet we enrich the graph2’s input with an additional parameter (newParam):
[source,Java]
----
try (
Tensor x = Tensor.create(input);
Tensor additionalTensor = Tensor.create(colorMap);
) {
Map<String, Tensor<?>> result =
graph1
.andThen(Functions.enrichWIth("newParam", additionalTensor)
.andThen(graph2)
.apply(Collections.singletonMap("x", x));
return result.get("y2").intValue();
}
----
The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/semantic-segmentation/src/main/java/io/mindmodel/services/semantic/segmentation/SemanticSegmentation.java#L141[SemanticSegmentation] implementation provides a real example how to enrich with parameters.
=== Enrich Inputs from Saved Tensors
We can combine the enricher approach with the https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/GraphRunnerMemory.java[GraphRunnerMemory]. This allows us to enrich some downstream Graphs with tensor parameters computed in some of the upstream Graphs. The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/common/src/main/java/io/mindmodel/services/common/Functions.java#L34[Functions#enrichFromMemory(memory, tensorName)] utility function can enrich a graph input parameter by extracting one stored in the memory.
For example let’s construct the following graph compositions:
----
graph1: y1 = x1 * 10 +
graph2: y2 = y1 * 200 +
graph3: y3 = y2 + y1
----
[source,Java]
----
try (
Tensor x = Tensor.create(input);
GraphRunnerMemory memory = new GraphRunnerMemory();
) {
Map<String, Tensor<?>> result =
this.graph1.andThen(memory) // memorizes y1
.andThen(graph2).andThen(memory) // memorizes y2
.andThen(Functions.enrichFromMemory(memory, "y1")) // retrieve graph1’s output y1 and adds it as an input for the next function.
.andThen(Functions.rename(
"y1", "x31", // renames the input y1 into x31
"y2", "x32" // renames the input y2 into x32
))
.andThen(graph3).andThen(memory)
.apply(Collections.singletonMap("x", x));
return result.get("y3").intValue();
}
----
=== Load Frozen Models from Remote Archives
The ProtoBufGraphDefinition extracts a pre-trained (frozen) Tensorflow model form a URI archive into byte array. It supports the `http(s)://`, `file://` and `classpath://` URI schemas. For this it uses the `ModelExtractor` and `CachedModelExtractor` utilities.
Models can be extracted either from raw files or form compressed archives. When extracted from an archive the model file name can optionally be provided as a URI fragment. For example for resource: `http://myarchive.tar.gz#model.pb`
the `myarchive.tar.gz` is traversed to uncompress and extract the model.pb file as a byte array. If the file name is not provided as URI fragment then the first file in the archive with extension .pb is extracted.
In addition, the CachedModelExtractor allows keeping a local copy (cache) of the model (protobuf) files extracted from the URI archive.
|===
|The https://github.com/tzolov/mind-model-services/blob/ops-programming-model/image-recognition/src/main/java/io/mindmodel/services/image/recognition/ImageRecognition.java[image-recognition] and https://github.com/tzolov/mind-model-services/blob/ops-programming-model/semantic-segmentation/src/main/java/io/mindmodel/services/semantic/segmentation/SemanticSegmentation.java[semantic-segmentation] inference models implementations demonstrate the suggested programming model.
|===

130
common/tensorflow-common/pom.xml Normal file
View File

@@ -0,0 +1,130 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<artifactId>tensorflow-common</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>tensorflow-common</name>
<description>tensorflow common</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<properties>
<checkstyle.skip>true</checkstyle.skip>
<spring-framework.version>5.1.6.RELEASE</spring-framework.version>
<jackson.version>2.11.0</jackson.version>
<apache.commons.compress>1.20</apache.commons.compress>
<commons-io.version>2.7</commons-io.version>
<tensorflow.version>1.15.0</tensorflow.version>
</properties>
<dependencies>
<!-- <dependency>-->
<!-- <groupId>org.tensorflow</groupId>-->
<!-- <artifactId>tensorflow-core-platform</artifactId>-->
<!-- <version>0.1.0-SNAPSHOT</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>tensorflow</artifactId>
<version>${tensorflow.version}</version>
</dependency>
<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>proto</artifactId>
<version>${tensorflow.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>${apache.commons.compress}</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.26</version>
</dependency>
<dependency>
<groupId>org.pcollections</groupId>
<artifactId>pcollections</artifactId>
<version>3.0.3</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-messaging</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<repositories>
<repository>
<id>tensorflow-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>

143
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/AbstractGraphRunner.java Normal file
View File

@@ -0,0 +1,143 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import org.tensorflow.Session;
import org.tensorflow.Tensor;
import org.springframework.util.Assert;
/**
* @author Christian Tzolov
*/
public abstract class AbstractGraphRunner implements Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>> {
public abstract Session doGetSession();
/**
* Names expected in the named Tensor inside the input {@link AbstractGraphRunner#apply(Map)}.
* If the apply method will fail if the input map is missing some of the feedNames.
*/
private final List<String> feedNames;
/**
* Names expected {@link AbstractGraphRunner#apply(Map)} result map.
*/
private final List<String> fetchNames;
/**
* When set and the input takes a single feed, then the name of the input tensor is automatically mapped
* to the expected input name. E.g. no need to rename the input names explicitly.
*/
private boolean autoBinding;
public AbstractGraphRunner(String feedName, String fetchedName) {
this(Arrays.asList(feedName), Arrays.asList(fetchedName));
}
public AbstractGraphRunner(List<String> feedNames, List<String> fetchedNames) {
this.feedNames = feedNames;
this.fetchNames = fetchedNames;
this.autoBinding = feedNames.size() == 1;
}
@Override
public Map<String, Tensor<?>> apply(Map<String, Tensor<?>> feeds) {
if (!this.isAutoBinding() && !feeds.keySet().containsAll(this.feedNames)) {
throw new IllegalArgumentException("Applied feeds:" + feeds.keySet()
+ "\n, don't match the expected feeds contract:" + this.feedNames);
}
if (this.isAutoBinding() && (feeds.size() != 1)) {
throw new IllegalArgumentException("Feed auto-binding expects a " +
"single feed tensors but found: " + feeds);
}
Session.Runner runner = this.doGetSession().runner();
// Feed in the input named tensors
for (Map.Entry<String, Tensor<?>> feedEntry : feeds.entrySet()) {
String feedName = (this.isAutoBinding()) ? this.feedNames.get(0) : feedEntry.getKey();
runner = runner.feed(feedName, feedEntry.getValue());
}
// Set the tensor name to be fetched after the evaluation
for (String fetchName : this.fetchNames) {
runner.fetch(fetchName);
}
// Evaluate the input
List<Tensor<?>> outputTensors = runner.run();
// Extract the output tensors
Map<String, Tensor<?>> outTensorMap = new HashMap<>();
for (int outputIndex = 0; outputIndex < this.fetchNames.size(); outputIndex++) {
outTensorMap.put(this.fetchNames.get(outputIndex), outputTensors.get(outputIndex));
}
return outTensorMap;
}
public List<String> getFeedNames() {
return this.feedNames;
}
public String getSingleøøƶFeedName() {
Assert.isTrue(feedNames.size() == 1, "Assumes a single feed input");
return this.feedNames.get(0);
}
public List<String> getFetchNames() {
return this.fetchNames;
}
public String getSingleFetchName() {
Assert.isTrue(this.fetchNames.size() == 1, "Assumes a single fetch output");
return this.fetchNames.get(0);
}
public boolean isAutoBinding() {
return this.autoBinding;
}
public AbstractGraphRunner disableAutoBinding() {
this.autoBinding = false;
return this;
}
public AbstractGraphRunner enableAutoBinding() {
if (this.getFeedNames().size() != 1) {
throw new IllegalArgumentException("Auto-binding is permitted for Graphs with single input feed, but " +
" found: " + this.getFeedNames());
}
this.autoBinding = true;
return this;
}
@Override
public String toString() {
return String.format("(%s) -> (%s)", String.join(",", this.feedNames),
String.join(",", this.fetchNames));
}
}

63
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/AutoCloseableSession.java Normal file
View File

@@ -0,0 +1,63 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import org.tensorflow.Graph;
import org.tensorflow.Session;
import org.tensorflow.op.Ops;
/**
* @author Christian Tzolov
*/
class AutoCloseableSession implements AutoCloseable {
private Session session;
/**
* Note: don't call this method inside the constructor.
*/
protected void init() {
Graph graph = this.doCreateGraph();
this.doGraphDefinition(Ops.create(graph));
this.session = new Session(graph);
}
protected Graph doCreateGraph() {
return new Graph();
}
protected void doGraphDefinition(Ops tf) {
}
protected Session getSession() {
if (this.session == null) {
init();
}
return this.session;
}
@Override
public void close() {
this.doClose();
if (this.session != null) {
this.session.close();
}
}
protected void doClose() {
}
}

86
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/Functions.java Normal file
View File

@@ -0,0 +1,86 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.HashMap;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.tensorflow.Tensor;
/**
* @author Christian Tzolov
*/
public final class Functions {
private Functions() {
}
/**
* On every function call enrich the input tensorMap with an addition (tensorName, tensor) pair.
*
* @param tensorName tensor key to use in the map
* @param tensor new Tensor to add to the map
* @return Returns a copy of the input tensorMap enriched with the provided (tensorName, tensor).
*/
public static Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>> enrichWith(
String tensorName, Tensor<?> tensor) {
return tensorMap -> enrich(tensorMap, tensorName, tensor);
}
/**
* On function call retrieves a named tensor from the provided {@link GraphRunnerMemory} and uses it to enrich
* the input tensorMap.
* @param memory GraphRunnerMemory to retrieve the tensor from
* @param tensorName name of the tensor in GraphRunnerMemory to retrieve.
* @return Returns copy of the input tensorMap enriched with the tensor from the memory.
*/
public static Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>> enrichFromMemory(
GraphRunnerMemory memory, String tensorName) {
return tensorMap -> enrich(tensorMap, tensorName, memory.getTensorMap().get(tensorName));
}
private static Map<String, Tensor<?>> enrich(Map<String, Tensor<?>> inputTensorMap, String key, Tensor<?> value) {
Map<String, Tensor<?>> newMap = new HashMap<>(inputTensorMap);
newMap.put(key, value);
return newMap;
}
/**
* Renames the tensor names in the incoming tensorMap with the providing mappings.
*
* @param mapping Pairs of From and To names. E.g. fromName1, toName1, fromName2, toName2, ... fromNameN, toNameN
* Must be an even number.
* @return Map that renames the input tensorMap entries according to the mapping provided
*/
public static Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>> rename(String... mapping) {
Map<String, String> mappingMap = new HashMap<>();
for (int i = 0; i < mapping.length; i = i + 2) {
mappingMap.put(mapping[i], mapping[i + 1]);
}
return tensorMap -> tensorMap.entrySet().stream()
.filter(e -> mappingMap.containsKey(e.getKey()))
.collect(Collectors.toMap(
kv -> mappingMap.get(kv.getKey()),
kv -> kv.getValue()
));
}
}

27
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/GraphDefinition.java Normal file
View File

@@ -0,0 +1,27 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import org.tensorflow.op.Ops;
/**
* @author Christian Tzolov
*/
@FunctionalInterface
public interface GraphDefinition {
void defineGraph(Ops tf);
}

106
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/GraphRunner.java Normal file
View File

@@ -0,0 +1,106 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Arrays;
import java.util.List;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.Session;
import org.tensorflow.op.Ops;
import org.springframework.util.Assert;
/**
* @author Christian Tzolov
*/
public class GraphRunner extends AbstractGraphRunner implements AutoCloseable {
private SavedModelBundle savedModelBundle;
private AutoCloseableSession autoCloseableSession;
public GraphRunner(List<String> feedNames, String fetchedName) {
super(feedNames, Arrays.asList(fetchedName));
}
public GraphRunner(String feedName, List<String> fetchedNames) {
super(Arrays.asList(feedName), fetchedNames);
}
public GraphRunner(String feedName, String fetchedName) {
super(feedName, fetchedName);
}
public GraphRunner(List<String> feedNames, List<String> fetchedNames) {
super(feedNames, fetchedNames);
}
@Override
public Session doGetSession() {
if (this.autoCloseableSession != null && this.savedModelBundle != null) {
throw new IllegalStateException("Either SavedModel or GraphDefinition can be set! But both are set!");
}
if (this.autoCloseableSession != null) {
return this.autoCloseableSession.getSession();
}
if (this.savedModelBundle != null) {
return this.savedModelBundle.session();
}
throw new IllegalStateException("Either SavedModel or GraphDefinition can be set! None found");
}
public GraphRunner withGraphDefinition(GraphDefinition graphDefinition) {
Assert.isNull(this.savedModelBundle, "Either SavedModel or GraphDefinition can be set! " +
"SavedModelBundle is found: " + this.savedModelBundle);
this.autoCloseableSession = new AutoCloseableSession() {
@Override
protected void doGraphDefinition(Ops tf) {
graphDefinition.defineGraph(tf);
}
};
return this;
}
public GraphRunner withSavedModel(String savedModelDir, String... tags) {
Assert.isNull(this.autoCloseableSession, "Either SavedModel or GraphDefinition can be set! " +
"AutoCloseableSession is found: " + this.autoCloseableSession);
this.savedModelBundle = SavedModelBundle.load(savedModelDir, tags);
return this;
}
@Override
public String toString() {
return String.format("(%s) -> (%s)", String.join(",", this.getFeedNames()),
String.join(",", this.getFetchNames()));
}
@Override
public void close() {
if (this.savedModelBundle != null) {
this.savedModelBundle.close();
}
if (this.autoCloseableSession != null) {
this.autoCloseableSession.close();
}
}
}

53
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/GraphRunnerMemory.java Normal file
View File

@@ -0,0 +1,53 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
import org.pcollections.HashTreePMap;
import org.pcollections.PMap;
import org.tensorflow.Tensor;
import org.springframework.cloud.fn.common.tensorflow.util.AutoCloseables;
/**
* Keeps all tensorMap input parameters.
*/
public class GraphRunnerMemory implements Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>>, AutoCloseable {
private AtomicReference<PMap<String, Tensor<?>>> tensorMap = new AtomicReference<>(HashTreePMap.empty());
public Map<String, Tensor<?>> getTensorMap() {
return tensorMap.get();
}
@Override
public Map<String, Tensor<?>> apply(Map<String, Tensor<?>> tensorMap) {
this.tensorMap.getAndUpdate(pmap -> pmap.plusAll(tensorMap));
return tensorMap;
}
@Override
public void close() {
AutoCloseables.all(this.tensorMap.get());
//this.tensorMap.get().clear();
}
}

75
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/ProtoBufGraphDefinition.java Normal file
View File

@@ -0,0 +1,75 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Iterator;
import org.tensorflow.Graph;
import org.tensorflow.Operation;
import org.tensorflow.op.Ops;
import org.springframework.cloud.fn.common.tensorflow.util.CachedModelExtractor;
import org.springframework.cloud.fn.common.tensorflow.util.ModelExtractor;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
/**
* @author Christian Tzolov
*/
public class ProtoBufGraphDefinition implements GraphDefinition {
/**
* Location of the pre-trained model archive.
*/
private final Resource modelLocation;
/**
* If set true the pre-trained model is cached on the local file system.
*/
private final boolean cacheModel;
public ProtoBufGraphDefinition(String modelUri, boolean cacheModel) {
this(new DefaultResourceLoader().getResource(modelUri), cacheModel);
}
public ProtoBufGraphDefinition(Resource modelLocation, boolean cacheModel) {
this.modelLocation = modelLocation;
this.cacheModel = cacheModel;
}
@Override
public void defineGraph(Ops tf) {
// Extract the pre-trained model as byte array.
byte[] model = this.cacheModel ? new CachedModelExtractor().getModel(this.modelLocation)
: new ModelExtractor().getModel(this.modelLocation);
// Import the pre-trained model
((Graph) tf.scope().env()).importGraphDef(model);
//try {
// ((Graph) tf.scope().env()).importGraphDef(GraphDef.parseFrom(model));
//}
//catch (InvalidProtocolBufferException e) {
// throw new RuntimeException(e);
//}
Graph graph = ((Graph) tf.scope().env());
Iterator<Operation> ops = graph.operations();
while (ops.hasNext()) {
System.out.println(ops.next().name());
}
System.out.println("Boza");
}
}

637
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/GraphicsUtils.java Normal file
View File

@@ -0,0 +1,637 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.deprecated;
import java.awt.BasicStroke;
import java.awt.Color;
import java.awt.Font;
import java.awt.FontMetrics;
import java.awt.Graphics2D;
import java.awt.Image;
import java.awt.RenderingHints;
import java.awt.Stroke;
import java.awt.geom.Rectangle2D;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import javax.imageio.ImageIO;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.util.StreamUtils;
/**
* Utility class used to provide some handy image manipulation functions. Among others it can provide contrast colors
* for image annotation labels and bounding boxes as well as functionality to draw later.
*
* @author Christian Tzolov
*/
public final class GraphicsUtils {
private GraphicsUtils() {
}
/**
* Default DEFAULT_FONT used in image label annotation.
*/
private static final Font DEFAULT_FONT = new Font("arial", Font.PLAIN, 12);
/**
* Bounding box default line thickness.
*/
private static final float LINE_THICKNESS = 2;
/**
* Color used when no multi-color is used.
*/
private static final Color AGNOSTIC_COLOR = new Color(167, 252, 0);
/**
* in labels text offset.
*/
public static final int TITLE_OFFSET = 3;
/**
* Predefined contrasting colors used when drawing multiple objects in the same image.
*/
public static final Color aliceblue = new Color(240, 248, 255); /* color */
/** Color. **/
public static final Color antiquewhite = new Color(250, 235, 215);
/** Color. **/
public static final Color aqua = new Color(0, 255, 255); // color
/** Color. **/
public static final Color aquamarine = new Color(127, 255, 212); // color
/** Color. **/
public static final Color azure = new Color(240, 255, 255); // color
/** Color. **/
public static final Color beige = new Color(245, 245, 220); // color
/** Color. **/
public static final Color bisque = new Color(255, 228, 196);
/** Color. **/
public static final Color black = new Color(0, 0, 0);
/** Color. **/
public static final Color blanchedalmond = new Color(255, 255, 205);
/** Color. **/
public static final Color blue = new Color(0, 0, 255);
/** Color. **/
public static final Color blueviolet = new Color(138, 43, 226);
/** Color. **/
public static final Color brown = new Color(165, 42, 42);
/** Color. **/
public static final Color burlywood = new Color(222, 184, 135);
/** Color. **/
public static final Color cadetblue = new Color(95, 158, 160);
/** Color. **/
public static final Color chartreuse = new Color(127, 255, 0);
/** Color. **/
public static final Color chocolate = new Color(210, 105, 30);
/** Color. **/
public static final Color coral = new Color(255, 127, 80);
/** Color. **/
public static final Color cornflowerblue = new Color(100, 149, 237);
/** Color. **/
public static final Color cornsilk = new Color(255, 248, 220);
/** Color. **/
public static final Color crimson = new Color(220, 20, 60);
/** Color. **/
public static final Color cyan = new Color(0, 255, 255);
/** Color. **/
public static final Color darkblue = new Color(0, 0, 139);
/** Color. **/
public static final Color darkcyan = new Color(0, 139, 139);
/** Color. **/
public static final Color darkgoldenrod = new Color(184, 134, 11);
/** Color. **/
public static final Color darkgray = new Color(169, 169, 169);
/** Color. **/
public static final Color darkgreen = new Color(0, 100, 0);
/** Color. **/
public static final Color darkkhaki = new Color(189, 183, 107);
/** Color. **/
public static final Color darkmagenta = new Color(139, 0, 139);
/** Color. **/
public static final Color darkolivegreen = new Color(85, 107, 47);
/** Color. **/
public static final Color darkorange = new Color(255, 140, 0);
/** Color. **/
public static final Color darkorchid = new Color(153, 50, 204);
/** Color. **/
public static final Color darkred = new Color(139, 0, 0);
/** Color. **/
public static final Color darksalmon = new Color(233, 150, 122);
/** Color. **/
public static final Color darkseagreen = new Color(143, 188, 143);
/** Color. **/
public static final Color darkslateblue = new Color(72, 61, 139);
/** Color. **/
public static final Color darkslategray = new Color(47, 79, 79);
/** Color. **/
public static final Color darkturquoise = new Color(0, 206, 209);
/** Color. **/
public static final Color darkviolet = new Color(148, 0, 211);
/** Color. **/
public static final Color deeppink = new Color(255, 20, 147);
/** Color. **/
public static final Color deepskyblue = new Color(0, 191, 255);
/** Color. **/
public static final Color dimgray = new Color(105, 105, 105);
/** Color. **/
public static final Color dodgerblue = new Color(30, 144, 255);
/** Color. **/
public static final Color firebrick = new Color(178, 34, 34);
/** Color. **/
public static final Color floralwhite = new Color(255, 250, 240);
/** Color. **/
public static final Color forestgreen = new Color(34, 139, 34);
/** Color. **/
public static final Color fuchsia = new Color(255, 0, 255);
/** Color. **/
public static final Color gainsboro = new Color(220, 220, 220);
/** Color. **/
public static final Color ghostwhite = new Color(248, 248, 255);
/** Color. **/
public static final Color gold = new Color(255, 215, 0);
/** Color. **/
public static final Color goldenrod = new Color(218, 165, 32);
/** Color. **/
public static final Color gray = new Color(128, 128, 128);
/** Color. **/
public static final Color green = new Color(0, 128, 0);
/** Color. **/
public static final Color greenyellow = new Color(173, 255, 47);
/** Color. **/
public static final Color honeydew = new Color(240, 255, 240);
/** Color. **/
public static final Color hotpink = new Color(255, 105, 180);
/** Color. **/
public static final Color indianred = new Color(205, 92, 92);
/** Color. **/
public static final Color indigo = new Color(75, 0, 130);
/** Color. **/
public static final Color ivory = new Color(255, 240, 240);
/** Color. **/
public static final Color khaki = new Color(240, 230, 140);
/** Color. **/
public static final Color lavender = new Color(230, 230, 250);
/** Color. **/
public static final Color lavenderblush = new Color(255, 240, 245);
/** Color. **/
public static final Color lawngreen = new Color(124, 252, 0);
/** Color. **/
public static final Color lemonchiffon = new Color(255, 250, 205);
/** Color. **/
public static final Color lightblue = new Color(173, 216, 230);
/** Color. **/
public static final Color lightcoral = new Color(240, 128, 128);
/** Color. **/
public static final Color lightcyan = new Color(224, 255, 255);
/** Color. **/
public static final Color lightgoldenrodyellow = new Color(250, 250, 210);
/** Color. **/
public static final Color lightgreen = new Color(144, 238, 144);
/** Color. **/
public static final Color lightgrey = new Color(211, 211, 211);
/** Color. **/
public static final Color lightpink = new Color(255, 182, 193);
/** Color. **/
public static final Color lightsalmon = new Color(255, 160, 122);
/** Color. **/
public static final Color lightseagreen = new Color(32, 178, 170);
/** Color. **/
public static final Color lightskyblue = new Color(135, 206, 250);
/** Color. **/
public static final Color lightslategray = new Color(119, 136, 153);
/** Color. **/
public static final Color lightsteelblue = new Color(176, 196, 222);
/** Color. **/
public static final Color lightyellow = new Color(255, 255, 224);
/** Color. **/
public static final Color lime = new Color(0, 255, 0);
/** Color. **/
public static final Color limegreen = new Color(50, 205, 50);
/** Color. **/
public static final Color linen = new Color(250, 240, 230);
/** Color. **/
public static final Color magenta = new Color(255, 0, 255);
/** Color. **/
public static final Color maroon = new Color(128, 0, 0);
/** Color. **/
public static final Color mediumaquamarine = new Color(102, 205, 170);
/** Color. **/
public static final Color mediumblue = new Color(0, 0, 205);
/** Color. **/
public static final Color mediumorchid = new Color(186, 85, 211);
/** Color. **/
public static final Color mediumpurple = new Color(147, 112, 219);
/** Color. **/
public static final Color mediumseagreen = new Color(60, 179, 113);
/** Color. **/
public static final Color mediumslateblue = new Color(123, 104, 238);
/** Color. **/
public static final Color mediumspringgreen = new Color(0, 250, 154);
/** Color. **/
public static final Color mediumturquoise = new Color(72, 209, 204);
/** Color. **/
public static final Color mediumvioletred = new Color(199, 21, 133);
/** Color. **/
public static final Color midnightblue = new Color(25, 25, 112);
/** Color. **/
public static final Color mintcream = new Color(245, 255, 250);
/** Color. **/
public static final Color mistyrose = new Color(255, 228, 225);
/** Color. **/
public static final Color mocassin = new Color(255, 228, 181);
/** Color. **/
public static final Color navajowhite = new Color(255, 222, 173);
/** Color. **/
public static final Color navy = new Color(0, 0, 128);
/** Color. **/
public static final Color oldlace = new Color(253, 245, 230);
/** Color. **/
public static final Color olive = new Color(128, 128, 0);
/** Color. **/
public static final Color olivedrab = new Color(107, 142, 35);
/** Color. **/
public static final Color orange = new Color(255, 165, 0);
/** Color. **/
public static final Color orangered = new Color(255, 69, 0);
/** Color. **/
public static final Color orchid = new Color(218, 112, 214);
/** Color. **/
public static final Color palegoldenrod = new Color(238, 232, 170);
/** Color. **/
public static final Color palegreen = new Color(152, 251, 152);
/** Color. **/
public static final Color paleturquoise = new Color(175, 238, 238);
/** Color. **/
public static final Color palevioletred = new Color(219, 112, 147);
/** Color. **/
public static final Color papayawhip = new Color(255, 239, 213);
/** Color. **/
public static final Color peachpuff = new Color(255, 218, 185);
/** Color. **/
public static final Color peru = new Color(205, 133, 63);
/** Color. **/
public static final Color pink = new Color(255, 192, 203);
/** Color. **/
public static final Color plum = new Color(221, 160, 221);
/** Color. **/
public static final Color powderblue = new Color(176, 224, 230);
/** Color. **/
public static final Color purple = new Color(128, 0, 128);
/** Color. **/
public static final Color red = new Color(255, 0, 0);
/** Color. **/
public static final Color rosybrown = new Color(188, 143, 143);
/** Color. **/
public static final Color royalblue = new Color(65, 105, 225);
/** Color. **/
public static final Color saddlebrown = new Color(139, 69, 19);
/** Color. **/
public static final Color salmon = new Color(250, 128, 114);
/** Color. **/
public static final Color sandybrown = new Color(244, 164, 96);
/** Color. **/
public static final Color seagreen = new Color(46, 139, 87);
/** Color. **/
public static final Color seashell = new Color(255, 245, 238);
/** Color. **/
public static final Color sienna = new Color(160, 82, 45);
/** Color. **/
public static final Color silver = new Color(192, 192, 192);
/** Color. **/
public static final Color skyblue = new Color(135, 206, 235);
/** Color. **/
public static final Color slateblue = new Color(106, 90, 205);
/** Color. **/
public static final Color slategray = new Color(112, 128, 144);
/** Color. **/
public static final Color snow = new Color(255, 250, 250);
/** Color. **/
public static final Color springgreen = new Color(0, 255, 127);
/** Color. **/
public static final Color steelblue = new Color(70, 138, 180);
/** Color. **/
public static final Color tan = new Color(210, 180, 140);
/** Color. **/
public static final Color teal = new Color(0, 128, 128);
/** Color. **/
public static final Color thistle = new Color(216, 191, 216);
/** Color. **/
public static final Color tomato = new Color(253, 99, 71);
/** Color. **/
public static final Color turquoise = new Color(64, 224, 208);
/** Color. **/
public static final Color violet = new Color(238, 130, 238);
/** Color. **/
public static final Color wheat = new Color(245, 222, 179);
/** Color. **/
public static final Color white = new Color(255, 255, 255);
/** Color. **/
public static final Color whitesmoke = new Color(245, 245, 245);
/** Color. **/
public static final Color yellow = new Color(255, 255, 0);
/** Color. **/
public static final Color yellowgreen = new Color(154, 205, 50);
/**
* Limbs color list.
*/
public static final Color[] LIMBS_COLORS = new Color[] {
new Color(153, 0, 0), // 0 (1 -> 2)
new Color(153, 51, 0), // 1 (1 -> 5)
new Color(153, 102, 0), // 2 (2 -> 3)
new Color(153, 153, 0), // 3 (3 -> 4)
new Color(102, 153, 0), // 4 (5 -> 6)
new Color(51, 153, 0), // 5 (6 -> 7)
new Color(0, 153, 0), // 6 (1 -> 8)
new Color(0, 153, 51), // 7 (8 -> 9)
new Color(0, 153, 102), // 8 (9 -> 10)
new Color(0, 153, 153), // 9 (1 -> 11)
new Color(0, 102, 153), // 10 (11 -> 12)
new Color(0, 51, 153), // 11 (12 -> 13)
new Color(0, 0, 153), // 12 (1 -> 0)
new Color(51, 0, 153), // 13 (0 -> 14)
new Color(102, 0, 153), // 14 (14 -> 16)
new Color(153, 0, 153), // 15 (0 -> 15)
new Color(153, 0, 102), // 16 (15 -> 17)
new Color(153, 0, 51), // 17 (2 -> 16)
new Color(153, 153, 153), // 18 (5 -> 17)
};
/**
* Constants lists.
*/
private static final Color[] CLASS_COLOR = new Color[] {
aliceblue, chartreuse, aqua, aquamarine, azure, beige, bisque,
blanchedalmond, blueviolet, burlywood, cadetblue, antiquewhite,
chocolate, coral, cornflowerblue, cornsilk, crimson, cyan,
darkcyan, darkgoldenrod, darkgray, darkkhaki, darkorange,
darkorchid, darksalmon, darkseagreen, darkturquoise, darkviolet,
deeppink, deepskyblue, dodgerblue, firebrick, floralwhite,
forestgreen, fuchsia, gainsboro, ghostwhite, gold, goldenrod,
salmon, tan, honeydew, hotpink, indianred, ivory, khaki,
lavender, lavenderblush, lawngreen, lemonchiffon, lightblue,
lightcoral, lightcyan, lightgoldenrodyellow, lightgreen, lightgrey,
lightgreen, lightpink, lightsalmon, lightseagreen, lightskyblue,
lightslategray, lightslategray, lightsteelblue, lightyellow, lime,
limegreen, linen, magenta, mediumaquamarine, mediumorchid,
mediumpurple, mediumseagreen, mediumslateblue, mediumspringgreen,
mediumturquoise, mediumvioletred, mintcream, mistyrose, mocassin,
navajowhite, oldlace, olive, olivedrab, orange, orangered,
orchid, palegoldenrod, palegreen, paleturquoise, palevioletred,
papayawhip, peachpuff, peru, pink, plum, powderblue, purple,
red, rosybrown, royalblue, saddlebrown, green, sandybrown,
seagreen, seashell, sienna, silver, skyblue, slateblue,
slategray, slategray, snow, springgreen, steelblue, greenyellow,
teal, thistle, tomato, turquoise, violet, wheat, white,
whitesmoke, yellow, yellowgreen
};
/**
* List of constants.
*/
public static final Color[] CLASS_COLOR2 = new Color[] {
yellow, yellowgreen, turquoise, springgreen, skyblue, slateblue, red, violet, olivedrab, royalblue,
darkorange, mediumblue, deeppink, chartreuse, orchid, palegreen, aqua, orange, navy
};
/**
* Return different color for each Id. It rotates when the ID exceeds the number of predefined colors.
* @param id the unique id to pick color for.
* @return a distinct color computed from the input #id
*/
public static Color getClassColor(int id) {
return CLASS_COLOR[id % CLASS_COLOR.length];
}
/**
* Augments the input image fromMemory a labeled rectangle (e.g. bounding box) fromMemory coordinates: (x1, y1, x2, y2).
*
* @param image Input image to be augmented fromMemory labeled rectangle.
* @param cid Unique id used to select the color of the rectangle. Used only if the colorAgnostic is set to false.
* @param title rectangle title
* @param x1 top left corner for the bounding box
* @param y1 top left corner for the bounding box
* @param x2 bottom right corner for the bounding box
* @param y2 bottom right corner for the bounding box
* @param colorAgnostic If set to false the cid is used to select the bounding box color. Uses the
* AGNOSTIC_COLOR otherwise.
*/
public static void drawBoundingBox(BufferedImage image, int cid, String title, int x1, int y1, int x2, int y2,
boolean colorAgnostic) {
Graphics2D g = image.createGraphics();
g.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
Color labelColor = colorAgnostic ? AGNOSTIC_COLOR : GraphicsUtils.getClassColor(cid);
g.setColor(labelColor);
g.setFont(DEFAULT_FONT);
FontMetrics fontMetrics = g.getFontMetrics();
Stroke oldStroke = g.getStroke();
g.setStroke(new BasicStroke(LINE_THICKNESS));
g.drawRect(x1, y1, (x2 - x1), (y2 - y1));
g.setStroke(oldStroke);
Rectangle2D rect = fontMetrics.getStringBounds(title, g);
g.setColor(labelColor);
g.fillRect(x1, y1 - fontMetrics.getAscent(),
(int) rect.getWidth() + 2 * TITLE_OFFSET, (int) rect.getHeight());
g.setColor(getTextColor(labelColor));
g.drawString(title, x1 + TITLE_OFFSET, y1);
}
/**
* Depends on the darkness of the background, pick a dark or light DEFAULT_FONT color.
* @param backGroundColor background color within which the text is drawn
* @return a text color, that contrast to the given background color.
*/
private static Color getTextColor(Color backGroundColor) {
double y = (299 * backGroundColor.getRed() + 587 * backGroundColor.getGreen() +
114 * backGroundColor.getBlue()) / 1000;
return y >= 128 ? Color.black : Color.white;
}
public static BufferedImage createMaskImage(float[][] maskPixels,
int scaledWidth, int scaledHeight, Color maskColor) {
int maskWidth = maskPixels.length;
int maskHeight = maskPixels[0].length;
int[] maskArray = new int[maskWidth * maskHeight];
int k = 0;
for (int i = 0; i < maskHeight; i++) {
for (int j = 0; j < maskWidth; j++) {
maskArray[k++] = grayScaleToARGB(maskPixels[i][j], maskColor);
}
}
// Turn the pixel array into image;
BufferedImage maskImage = new BufferedImage(maskWidth, maskHeight, BufferedImage.TYPE_INT_ARGB);
maskImage.setRGB(0, 0, maskWidth, maskHeight, maskArray, 0, maskWidth);
// Stretch the image to fit the target box width and height!
return toBufferedImage(maskImage.getScaledInstance(scaledWidth, scaledHeight, Image.SCALE_DEFAULT));
}
/**
* Converts an gray scale (e.g. value between 0 to 1) into ARGB.
*
* @param grayScale - value between 0 and 1
* @param maskColor - desired mask color
* @return Returns a ARGB color based on the grayscale and the mask colors
*/
private static int grayScaleToARGB(float grayScale, Color maskColor) {
if (maskColor != null) {
float r = col(maskColor.getRed(), grayScale);
float g = col(maskColor.getGreen(), grayScale);
float b = col(maskColor.getBlue(), grayScale);
float t = grayScale * 0.7f;
return new Color(r, g, b, t).getRGB();
}
return new Color(grayScale, grayScale, grayScale, grayScale).getRGB();
}
private static float col(int channelColor, float grayScale) {
//return ((float) channelColor / 255) * grayScale;
return ((float) channelColor / 255);
}
public static BufferedImage toBufferedImage(Image img) {
//if (img instanceof BufferedImage) {
// return (BufferedImage) img;
//}
// Create a buffered image fromMemory transparency
BufferedImage bimage = new BufferedImage(img.getWidth(null), img.getHeight(null), BufferedImage.TYPE_INT_ARGB);
// Draw the image on to the buffered image
Graphics2D bGr = bimage.createGraphics();
bGr.drawImage(img, 0, 0, null);
bGr.dispose();
// Return the buffered image
return bimage;
}
public static BufferedImage overlayImages(BufferedImage bgImage, BufferedImage fgImage, int fgX, int fgY) {
// Foreground image width and height cannot be greater than background image width and height.
if (fgImage.getHeight() > bgImage.getHeight()
|| fgImage.getWidth() > fgImage.getWidth()) {
throw new IllegalArgumentException(
"Foreground Image Is Bigger In One or Both Dimensions"
+ "nCannot proceed fromMemory overlay."
+ "nn Please use smaller Image for foreground");
}
// Create a Graphics from the background image
Graphics2D g = bgImage.createGraphics();
//Set Antialias Rendering
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
//Draw background image at location (0,0)
g.drawImage(bgImage, 0, 0, null);
// Draw foreground image at location (fgX,fgy)
g.drawImage(fgImage, fgX, fgY, null);
g.dispose();
return bgImage;
}
/**
* Convert {@link BufferedImage} to byte array.
*
* @param image the image to be converted
* @param format the output image format
* @return New array of bytes
*/
public static byte[] toImageByteArray(BufferedImage image, String format) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
ImageIO.write(image, format, baos);
byte[] bytes = baos.toByteArray();
return bytes;
}
catch (IOException e) {
throw new IllegalStateException(e);
}
finally {
try {
baos.close();
}
catch (IOException e) {
throw new IllegalStateException(e);
}
}
}
/**
*
* @param bufferedImage buffer to be converted in to raw array
* @return flat byte array representing the buffered image
*/
public static byte[] toRawByteArray(BufferedImage bufferedImage) {
return ((DataBufferByte) bufferedImage.getRaster().getDataBuffer()).getData();
}
/**
* BufferedImage.TYPE_3BYTE_BGR, BufferedImage.TYPE_3BYTE_BGR.
* @param image image to be converted into buffer
* @param imageType desired type
* @return
*/
public static BufferedImage toBufferedImageType(BufferedImage image, int imageType) {
if (image.getType() == imageType) {
return image;
}
BufferedImage outputImage = new BufferedImage(image.getWidth(), image.getHeight(), imageType);
outputImage.getGraphics().drawImage(image, 0, 0, null);
return outputImage;
}
public static byte[] toImageToBytes(String imageUri) throws IOException {
try (InputStream is = new DefaultResourceLoader().getResource(imageUri).getInputStream()) {
return StreamUtils.copyToByteArray(is);
}
}
/**
* Loads a resource as byte array. Supports http:, file: and classpath: URI schemas
* @param resourceUri resource URI
* @return Returns resources referred by the resourceUri as a byte array
* @throws IOException failure due to missing resource or invalid URI
*/
public static byte[] loadAsByteArray(String resourceUri) throws IOException {
Resource expectedPoseResponse = new DefaultResourceLoader().getResource(resourceUri);
try (InputStream is = expectedPoseResponse.getInputStream()) {
return StreamUtils.copyToByteArray(is);
}
}
}

46
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java Normal file
View File

@@ -0,0 +1,46 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.deprecated;
import java.util.function.Function;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* Maps domain objects into JSON strings.
*
* @author Christian Tzolov
*/
public class JsonMapperFunction implements Function<Object, String> {
private static final Log logger = LogFactory.getLog(JsonMapperFunction.class);
@Override
public String apply(Object o) {
try {
return new ObjectMapper().writeValueAsString(o);
}
catch (JsonProcessingException e) {
logger.error("Failed to encode the object detections into JSON message", e);
}
return "ERROR";
}
}

131
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/TensorFlowService.java Normal file
View File

@@ -0,0 +1,131 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.deprecated;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.function.Function;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.tensorflow.Graph;
import org.tensorflow.Session;
import org.tensorflow.Session.Runner;
import org.tensorflow.Tensor;
import org.springframework.cloud.fn.common.tensorflow.util.CachedModelExtractor;
import org.springframework.cloud.fn.common.tensorflow.util.ModelExtractor;
import org.springframework.core.io.Resource;
/**
* @author Christian Tzolov
*/
public class TensorFlowService implements Function<Map<String, Tensor<?>>, Map<String, Tensor<?>>>, AutoCloseable {
private static final Log logger = LogFactory.getLog(TensorFlowService.class);
private final Session session;
private final List<String> fetchedNames;
private final boolean autoCloseFeedTensors;
public TensorFlowService(Resource modelLocation, List<String> fetchedNames) {
this(modelLocation, fetchedNames, false);
}
public TensorFlowService(Resource modelLocation, List<String> fetchedNames, boolean cacheModel) {
this(modelLocation, fetchedNames, cacheModel, false);
}
public TensorFlowService(Resource modelLocation, List<String> fetchedNames, boolean cacheModel,
boolean autoCloseFeedTensors) {
if (logger.isInfoEnabled()) {
logger.info("Loading TensorFlow graph model: " + modelLocation);
}
this.autoCloseFeedTensors = autoCloseFeedTensors;
this.fetchedNames = fetchedNames;
Graph graph = new Graph();
byte[] model = cacheModel ? new CachedModelExtractor().getModel(modelLocation) : new ModelExtractor().getModel(modelLocation);
graph.importGraphDef(model);
this.session = new Session(graph);
}
/**
* Evaluates a pre-trained tensorflow model (encoded as {@link Graph}). Use the feeds parameter to feed in the
* model input data and fetch-names to specify the output tensors.
*
* @param feeds Named map of input tensors.
* @return Returns the computed output tensors. The names of the output tensors is defined by the fetchedNames
* argument
*/
@Override
public Map<String, Tensor<?>> apply(Map<String, Tensor<?>> feeds) {
Runner runner = this.session.runner();
// Keep tensor references to release them in the finally block
Tensor[] feedTensors = new Tensor[feeds.size()];
try {
// Feed in the input named tensors
int inputIndex = 0;
for (Entry<String, Tensor<?>> e : feeds.entrySet()) {
String feedName = e.getKey();
feedTensors[inputIndex] = e.getValue();
runner = runner.feed(feedName, feedTensors[inputIndex]);
inputIndex++;
}
// Set the tensor name to be fetched after the evaluation
for (String fetchName : this.fetchedNames) {
runner.fetch(fetchName);
}
// Evaluate the input
List<Tensor<?>> outputTensors = runner.run();
// Extract the output tensors
Map<String, Tensor<?>> outTensorMap = new HashMap<>();
for (int outputIndex = 0; outputIndex < this.fetchedNames.size(); outputIndex++) {
outTensorMap.put(this.fetchedNames.get(outputIndex), outputTensors.get(outputIndex));
}
return outTensorMap;
}
finally {
if (this.autoCloseFeedTensors) {
// Release all feed tensors
for (Tensor tensor : feedTensors) {
if (tensor != null) {
tensor.close();
}
}
}
}
}
@Override
public void close() {
logger.info("Close TensorFlow Session!");
if (this.session != null) {
this.session.close();
}
}
}

153
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/util/AutoCloseables.java Normal file
View File

@@ -0,0 +1,153 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.util;
import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
import java.util.Objects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Utilities for AutoCloseable classes.
* Based on the Apache Drill AutoCloseables implementation.
*/
public final class AutoCloseables {
private AutoCloseables() {
}
private static final Logger LOGGER = LoggerFactory.getLogger(AutoCloseables.class);
public static AutoCloseable all(final Collection<? extends AutoCloseable> autoCloseables) {
return () -> close(autoCloseables);
}
public static AutoCloseable all(final Map<?, ? extends AutoCloseable>... autoCloseables) {
return () -> close(autoCloseables);
}
/**
* Closes all autoCloseables if not null and suppresses exceptions by adding them to t.
* @param t the throwable to add suppressed exception to
* @param autoCloseables the closeables to close
*/
public static void close(Throwable t, AutoCloseable... autoCloseables) {
close(t, Arrays.asList(autoCloseables));
}
/**
* Closes all autoCloseables if not null and suppresses exceptions by adding them to t.
* @param t the throwable to add suppressed exception to
* @param autoCloseables the closeables to close
*/
public static void close(Throwable t, Collection<? extends AutoCloseable> autoCloseables) {
try {
close(autoCloseables);
}
catch (Exception e) {
t.addSuppressed(e);
}
}
/**
* Closes all autoCloseables if not null and suppresses subsequent exceptions if more than one.
* @param autoCloseables the closeables to close
*/
public static void close(AutoCloseable... autoCloseables) throws Exception {
close(Arrays.asList(autoCloseables));
}
/**
* Closes all autoCloseables if not null and suppresses subsequent exceptions if more than one.
* @param autoCloseables the closeables to close
*/
public static void close(Iterable<? extends AutoCloseable> autoCloseables) throws Exception {
Exception topLevelException = null;
for (AutoCloseable closeable : autoCloseables) {
try {
if (closeable != null) {
closeable.close();
}
}
catch (Exception e) {
if (topLevelException == null) {
topLevelException = e;
}
else {
topLevelException.addSuppressed(e);
}
}
}
if (topLevelException != null) {
throw topLevelException;
}
}
/**
* Closes all autoCloseables entry values if not null and suppresses subsequent exceptions if more than one.
* @param closableMaps the closeables to close
*/
public static void close(Map<?, ? extends AutoCloseable>... closableMaps) throws Exception {
Exception topLevelException = null;
for (Map<?, ? extends AutoCloseable> closableMap : closableMaps) {
for (Object key : closableMap.keySet()) {
AutoCloseable closeable = closableMap.get(key);
try {
if (closeable != null) {
closeable.close();
}
}
catch (Exception e) {
if (topLevelException == null) {
topLevelException = e;
}
else {
topLevelException.addSuppressed(e);
}
}
}
closableMap.keySet();
}
if (topLevelException != null) {
throw topLevelException;
}
}
/**
* Close all without caring about thrown exceptions.
* @param closeables - array containing auto closeables
*/
public static void closeSilently(AutoCloseable... closeables) {
Arrays.stream(closeables).filter(Objects::nonNull)
.forEach(target -> {
try {
target.close();
}
catch (Exception e) {
LOGGER.warn(String.format("Exception was thrown while closing auto closeable: %s", target), e);
}
});
}
}

107
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/util/CachedModelExtractor.java Normal file
View File

@@ -0,0 +1,107 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.util;
import java.io.File;
import java.io.FileInputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.util.Assert;
import org.springframework.util.FileCopyUtils;
import org.springframework.util.StreamUtils;
import org.springframework.util.StringUtils;
/**
* Extends the {@link ModelExtractor} to allow keeping a local copy (cache) of the loaded model (protobuf) files.
* @author Christian Tzolov
*/
public class CachedModelExtractor extends ModelExtractor {
private static final Log logger = LogFactory.getLog(CachedModelExtractor.class);
/**
* Parent folder under which the model files are cached.
*/
public String cacheRootDirectory = new File(System.getProperty("java.io.tmpdir"), "mind-model").getAbsolutePath();
public String getCacheRootDirectory() {
return cacheRootDirectory;
}
public void setCacheRootDirectory(String cacheRootDirectory) {
this.cacheRootDirectory = cacheRootDirectory;
}
public CachedModelExtractor() {
super();
}
public CachedModelExtractor(String frozenGraphFileExtension) {
super(frozenGraphFileExtension);
}
@Override
public byte[] getModel(String modelUri) {
return this.getModel(new DefaultResourceLoader().getResource(modelUri));
}
@Override
public byte[] getModel(Resource modelResource) {
try {
File rootFolder = new File(this.cacheRootDirectory);
if (!rootFolder.exists()) {
logger.info("Create Model Cache root folder: " + rootFolder.getAbsolutePath());
rootFolder.mkdirs();
}
Assert.isTrue(rootFolder.isDirectory(), "The cache root folder must be a Directory");
String fileName = modelResource.getFilename();
String fragment = modelResource.getURI().getFragment();
File cachedFile = StringUtils.isEmpty(fragment) ? new File(rootFolder, fileName) :
new File(rootFolder, fileName + "_" + fragment);
if (cachedFile.exists()) {
logger.info("Load model " + modelResource.toString() + " from cache: " + cacheRootDirectory);
return StreamUtils.copyToByteArray(new FileInputStream(cachedFile));
}
byte[] model = super.getModel(modelResource);
// cache the file
FileCopyUtils.copy(model, cachedFile);
logger.info("Caching the " + modelResource.toString() + " model at: " + cachedFile);
return model;
}
catch (Exception e) {
throw new IllegalStateException("Failed to extract a model from: " + modelResource.getDescription(), e);
}
}
public void emptyModelCache() {
File rootFolder = new File(this.cacheRootDirectory);
if (rootFolder.exists()) {
logger.info("Empty Model Cache at:" + rootFolder.getAbsolutePath());
rootFolder.delete();
rootFolder.mkdirs();
}
}
}

246
common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/util/ModelExtractor.java Normal file
View File

@@ -0,0 +1,246 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow.util;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.X509Certificate;
import java.util.Optional;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.commons.compress.archivers.ArchiveEntry;
import org.apache.commons.compress.archivers.ArchiveInputStream;
import org.apache.commons.compress.archivers.ArchiveStreamFactory;
import org.apache.commons.compress.compressors.CompressorInputStream;
import org.apache.commons.compress.compressors.CompressorStreamFactory;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.util.Assert;
import org.springframework.util.StreamUtils;
import org.springframework.util.StringUtils;
/**
* Extracts a pre-trained (frozen) Tensorflow model URI into byte array. The 'http://', 'file://' and 'classpath://'
* URI schemas are supported.
*
* Models can be extract either from raw files or form compressed archives. When extracted from an archive the model
* file name can optionally be provided as an URI fragment. For example for resource: http://myarchive.tar.gz#model.pb
* the myarchive.tar.gz is traversed to uncompress and extract the model.pb file as byte array.
* If the file name is not provided as URI fragment then the first file in the archive with extension .pb is extracted.
*
* @author Christian Tzolov
*/
public class ModelExtractor {
private static final String DEFAULT_FROZEN_GRAPH_FILE_EXTENSION = ".pb";
/**
* When an archive resource if referred, but no fragment URI is provided (to specify the target file name in
* the archive) then the extractor selects the first file in the archive with the extension that match
* the frozenGraphFileExtension (defaults to .pb).
*/
public final String frozenGraphFileExtension;
public ModelExtractor() {
this(DEFAULT_FROZEN_GRAPH_FILE_EXTENSION);
}
public ModelExtractor(String frozenGraphFileExtension) {
this.frozenGraphFileExtension = frozenGraphFileExtension;
}
public byte[] getModel(String modelUri) {
return getModel(new DefaultResourceLoader().getResource(modelUri));
}
public byte[] getModel(Resource modelResource) {
Assert.notNull(modelResource, "Not null model resource is required!");
try (InputStream is = modelResource.getInputStream(); InputStream bi = new BufferedInputStream(is)) {
String[] archiveCompressor = detectArchiveAndCompressor(modelResource.getFilename());
String archive = archiveCompressor[0];
String compressor = archiveCompressor[1];
String fragment = modelResource.getURI().getFragment();
if (StringUtils.hasText(compressor)) {
try (CompressorInputStream cis = new CompressorStreamFactory().createCompressorInputStream(compressor, bi)) {
if (StringUtils.hasText(archive)) {
try (ArchiveInputStream ais = new ArchiveStreamFactory().createArchiveInputStream(archive, cis)) {
// Compressor fromMemory Archive
return findInArchiveStream(fragment, ais);
}
}
else { // Compressor only
return StreamUtils.copyToByteArray(cis);
}
}
}
else if (StringUtils.hasText(archive)) { // Archive only
try (ArchiveInputStream ais = new ArchiveStreamFactory().createArchiveInputStream(archive, bi)) {
return findInArchiveStream(fragment, ais);
}
}
else {
// No compressor nor Archive
return StreamUtils.copyToByteArray(bi);
}
}
catch (Exception e) {
throw new IllegalStateException("Failed to extract a model from: " + modelResource.getDescription(), e);
}
}
/**
* Traverses the Archive to find either an entry that matches the modelFileNameInArchive name (if not empty) or
* and entry that ends in .pb if the modelFileNameInArchive is empty.
*
* @param modelFileNameInArchive Optional name of the archive entry that represents the frozen model file. If empty
* the archive will be searched for the first entry that ends in .pb
* @param archive Archive stream to be traversed
*
*/
private byte[] findInArchiveStream(String modelFileNameInArchive, ArchiveInputStream archive) throws IOException {
ArchiveEntry entry;
while ((entry = archive.getNextEntry()) != null) {
//System.out.println(entry.getName() + " : " + entry.isDirectory());
if (archive.canReadEntryData(entry) && !entry.isDirectory()) {
if ((StringUtils.hasText(modelFileNameInArchive) && entry.getName().endsWith(modelFileNameInArchive)) ||
(!StringUtils.hasText(modelFileNameInArchive) && entry.getName().endsWith(this.frozenGraphFileExtension))) {
return StreamUtils.copyToByteArray(archive);
}
}
}
throw new IllegalArgumentException("No model is found in the archive");
}
/**
* Detect the Archive and the Compressor from the file extension.
*
* @param fileName File name with extension.
* @return Returns a tuple of the detected (Archive, Compressor). Null stands for not available
* archive or detector. The (null, null) response stands for no Archive or Compressor discovered.
*/
private String[] detectArchiveAndCompressor(String fileName) {
String normalizedFileName = fileName.trim().toLowerCase();
if (normalizedFileName.endsWith(".tar.gz")
|| normalizedFileName.endsWith(".tgz")
|| normalizedFileName.endsWith(".taz")) {
return new String[] { ArchiveStreamFactory.TAR, CompressorStreamFactory.GZIP };
}
else if (normalizedFileName.endsWith(".tar.bz2")
|| normalizedFileName.endsWith(".tbz2")
|| normalizedFileName.endsWith(".tbz")) {
return new String[] { ArchiveStreamFactory.TAR, CompressorStreamFactory.BZIP2 };
}
else if (normalizedFileName.endsWith(".cpgz")) {
return new String[] { ArchiveStreamFactory.CPIO, CompressorStreamFactory.GZIP };
}
else if (hasArchive(normalizedFileName)) {
return new String[] { findArchive(normalizedFileName).get(), null };
}
else if (hasCompressor(normalizedFileName)) {
return new String[] { null, findCompressor(normalizedFileName).get() };
}
else if (normalizedFileName.endsWith(".gzip")) {
return new String[] { null, CompressorStreamFactory.GZIP };
}
else if (normalizedFileName.endsWith(".bz2")
|| normalizedFileName.endsWith(".bz")) {
return new String[] { null, CompressorStreamFactory.BZIP2 };
}
// No archived/compressed
return new String[] { null, null };
}
private boolean hasArchive(String normalizedFileName) {
return findArchive(normalizedFileName).isPresent();
}
private Optional<String> findArchive(String normalizedFileName) {
return new ArchiveStreamFactory().getInputStreamArchiveNames()
.stream().filter(arch -> normalizedFileName.endsWith("." + arch)).findFirst();
}
private boolean hasCompressor(String normalizedFileName) {
return findCompressor(normalizedFileName).isPresent();
}
private Optional<String> findCompressor(String normalizedFileName) {
return new CompressorStreamFactory().getInputStreamCompressorNames()
.stream().filter(compressor -> normalizedFileName.endsWith("." + compressor)).findFirst();
}
static {
disableSslVerification();
}
private static void disableSslVerification() {
try {
// Create a trust manager that does not validate certificate chains
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(X509Certificate[] certs, String authType) {
}
}
};
// Install the all-trusting trust manager
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
// Create all-trusting host name verifier
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
// Install the all-trusting host verifier
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
}
catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
catch (KeyManagementException e) {
e.printStackTrace();
}
}
}

BIN
common/tensorflow-common/src/main/resources/images/programming_model.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

BIN
common/tensorflow-common/src/main/resources/images/tf_pipeline.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 33 KiB

89
common/tensorflow-common/src/test/java/org/springframework/cloud/fn/common/tensorflow/EnrichFromMemory.java Normal file
View File

@@ -0,0 +1,89 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import org.tensorflow.Tensor;
/**
* @author Christian Tzolov
*/
public class EnrichFromMemory implements AutoCloseable {
private final GraphRunner graph1;
private final GraphRunner graph2;
private final GraphRunner graph3;
public EnrichFromMemory() {
this.graph1 = new GraphRunner("x1", "y1")
.withGraphDefinition(tf -> tf.withName("y1").math.mul(
tf.withName("x1").placeholder(Integer.class),
tf.constant(10)));
this.graph2 = new GraphRunner("x2", "y2")
.withGraphDefinition(tf -> tf.withName("y2").math.mul(
tf.withName("x2").placeholder(Integer.class),
tf.constant(20)));
this.graph3 = new GraphRunner(Arrays.asList("x31", "x32"), Arrays.asList("y3"))
.withGraphDefinition(tf -> tf.withName("y3").math.add(
tf.withName("x31").placeholder(Integer.class),
tf.withName("x32").placeholder(Integer.class)));
}
public int compute(Integer input) {
try (
Tensor x = Tensor.create(input);
GraphRunnerMemory memory = new GraphRunnerMemory();
) {
Map<String, Tensor<?>> result =
this.graph1.andThen(memory)
.andThen(graph2).andThen(memory)
.andThen(Functions.enrichFromMemory(memory, "y1")) // retrieves the graph1's y1 output and adds it as a parameter with the same name
.andThen(Functions.rename(
"y1", "x31", // renames the input y1 into x31
"y2", "x32" // renames the input y2 into x32
))
.andThen(graph3).andThen(memory)
.apply(Collections.singletonMap("x", x));
memory.getTensorMap().entrySet().forEach(e -> System.out.println(" " + e));
return result.get("y3").intValue();
}
}
@Override
public void close() {
this.graph1.close();
this.graph2.close();
}
public static void main(String[] args) {
try (EnrichFromMemory example = new EnrichFromMemory()) {
for (int x = 0; x < 5; x++) {
System.out.println("For x = " + x + ", y = " + example.compute(x));
}
}
}
}

58
common/tensorflow-common/src/test/java/org/springframework/cloud/fn/common/tensorflow/FunctionComposition.java Normal file
View File

@@ -0,0 +1,58 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Collections;
import java.util.Map;
import org.tensorflow.Tensor;
/**
* @author Christian Tzolov
*/
public final class FunctionComposition {
private FunctionComposition() {
}
// y = (x * 2) + 20
//
// y1 = x1 * 2 , where x1 == x
// y2 = x2 + 20 , where x2 == y1 and y = y2
public static void main(String[] args) {
try (
GraphRunner graph1 = new GraphRunner("x1", "y1")
.withGraphDefinition(tf -> tf.withName("y1").math.mul(
tf.withName("x1").placeholder(Integer.class),
tf.constant(2)));
GraphRunner graph2 = new GraphRunner("x2", "y2")
.withGraphDefinition(tf -> tf.withName("y2").math.add(
tf.withName("x2").placeholder(Integer.class),
tf.constant(20)));
Tensor x = Tensor.create(10);
) {
Map<String, Tensor<?>> result = graph1.andThen(graph2).apply(Collections.singletonMap("x", x));
System.out.println("Result is: " + result.get("y2").intValue());
// Result is: 40
}
}
}

67
common/tensorflow-common/src/test/java/org/springframework/cloud/fn/common/tensorflow/FunctionCompositionMultipleInputsOutputs.java Normal file
View File

@@ -0,0 +1,67 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import org.tensorflow.Tensor;
import org.tensorflow.op.core.Placeholder;
/**
* @author Christian Tzolov
*/
public final class FunctionCompositionMultipleInputsOutputs {
private FunctionCompositionMultipleInputsOutputs() {
}
// y = (x * 2) + (x * 3)
//
// y11 = x1 * 2 , where x1 == x
// y12 = x1 * 3 , where x1 == x
// y2 = x21 + x22 , where x21 == y11, x22 == y12 and y == y2
public static void main(String[] args) {
try (
GraphRunner graph1 = new GraphRunner(Arrays.asList("x1"), Arrays.asList("y11", "y12"))
.withGraphDefinition(tf -> {
Placeholder<Integer> x1 = tf.withName("x1").placeholder(Integer.class);
tf.withName("y11").math.mul(x1, tf.constant(2));
tf.withName("y12").math.mul(x1, tf.constant(3));
});
GraphRunner graph2 = new GraphRunner(Arrays.asList("x21", "x22"), Arrays.asList("y2"))
.withGraphDefinition(tf -> tf.withName("y2").math.add(
tf.withName("x21").placeholder(Integer.class),
tf.withName("x22").placeholder(Integer.class)));
Tensor x = Tensor.create(10);
) {
Map<String, Tensor<?>> result = graph1
.andThen(Functions.rename(
"y11", "x21",
"y12", "x22"
))
.andThen(graph2)
.apply(Collections.singletonMap("x", x));
System.out.println("Result is: " + result.get("y2").intValue()); // Result is: 50
}
}
}

77
common/tensorflow-common/src/test/java/org/springframework/cloud/fn/common/tensorflow/ReleaseTensorParameters.java Normal file
View File

@@ -0,0 +1,77 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.tensorflow;
import java.util.Collections;
import java.util.Map;
import org.tensorflow.Tensor;
/**
* @author Christian Tzolov
*/
public class ReleaseTensorParameters implements AutoCloseable {
private final GraphRunner graph1;
private final GraphRunner graph2;
public ReleaseTensorParameters() {
this.graph1 = new GraphRunner("x1", "y1")
.withGraphDefinition(tf -> tf.withName("y1").math.mul(
tf.withName("x1").placeholder(Integer.class),
tf.constant(2)));
this.graph2 = new GraphRunner("x2", "y2")
.withGraphDefinition(tf -> tf.withName("y2").math.add(
tf.withName("x2").placeholder(Integer.class),
tf.constant(20)));
}
// y = (x * 2) + 20
public int compute(Integer input) {
try (
Tensor x = Tensor.create(input);
GraphRunnerMemory memory = new GraphRunnerMemory();
) {
Map<String, Tensor<?>> result =
this.graph1.andThen(memory)
.andThen(graph2).andThen(memory)
.apply(Collections.singletonMap("x", x));
memory.getTensorMap().entrySet().forEach(e -> System.out.println(" " + e));
return result.get("y2").intValue();
}
}
@Override
public void close() {
this.graph1.close();
this.graph2.close();
}
public static void main(String[] args) {
try (ReleaseTensorParameters example = new ReleaseTensorParameters()) {
for (int x = 0; x < 5; x++) {
System.out.println("For x = " + x + ", y = " + example.compute(x));
}
}
}
}