Tensorflow functions and applications

* initial step
 * Tensorflow models functional model redesign

      -- Based on https://tzolov.github.io/mind-model-services
      -- Resolves #5

 * Add object detection processor README
 * Add image recognition processor README
 * Initial Tensorflow commonn README
 * Initial Tensorflow commonn README
 * Tensorflow common diagram
 * Tensorflow docs code
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Tensorflow docs code snippets improve
 * Add semantic segmentation function. add object detecteion function readme
 * oo images
 * Furether oo readme improvments
 * Final obj detection readme fixes
 * Add image recognition readme
 * Add image recognition readme 2
 * Semantic segmentation readme
 * Segmentation readme
 * Semantic segmentation readme 3
 * Fix image recognition and object detcion app starter dependecies

 * Add metadata for Tensorflow apps

applications/processor/object-detection-processor/README.adoc

@@ -0,0 +1,54 @@
+//tag::ref-doc[]
+:image-root: https://raw.githubusercontent.com/spring-cloud-stream-app-starters/tensorflow/master/images
+
+= Object Detection Processor
+
+The https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-object-detection[Object Detection] processor provides out-of-the-box support for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. It allows for real-time localization and identification of multiple objects in a single image or image stream. The Object Detection processor uses one of the pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[object detection] models and corresponding https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels].
+
+If the pre-trained model is not set explicitly set then following defaults are used:
+
+* `tensorflow.modelFetch` : `detection_scores,detection_classes,detection_boxes,num_detections`
+* `tensorflow.model` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/faster_rcnn_resnet101_coco_2018_01_28_frozen_inference_graph.pb`
+* `tensorflow.object.detection.labels` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt`
+
+The following diagram illustrates a Spring Cloud Data Flow streaming pipeline that predicts object types from the images in real-time.
+
+image::{image-root}/scdf-tensorflow-object-detection-arch.png[]
+
+Processor's input is an image byte array and the output is a JSON message in this format:
+
+```json
+{
+  "labels" : [
+     {"name":"person", "confidence":0.9996774,"x1":0.0,"y1":0.3940161,"x2":0.9465165,"y2":0.5592592,"cid":1},
+     {"name":"person", "confidence":0.9996604,"x1":0.047891676,"y1":0.03169123,"x2":0.941098,"y2":0.2085562,"cid":1},
+     {"name":"backpack", "confidence":0.96534747,"x1":0.15588468,"y1":0.85957795,"x2":0.5091308,"y2":0.9908878,"cid":23},
+     {"name":"backpack", "confidence":0.963343,"x1":0.1273736,"y1":0.57658505,"x2":0.47765,"y2":0.6986431,"cid":23}
+  ]
+}
+```
+
+The output format is:
+
+* *object-name*:**confidence** - human readable name of the detected object (e.g. label) with its confidence as a float between [0-1]
+* *x1*, *y1*, *x2*, *y2* - Response also provides the bounding box of the detected objects represented as (x1, y1, x2, y2). The coordinates are relative to the size of the image size.
+* *cid*  - Classification identifier as defined in the provided https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[labels] configuration file.
+
+=== Payload
+
+If the incoming type is `byte[]` and the content type is set to `application/octet-stream` , then the application process the input `byte[]` image into and outputs augmented `byte[]` image payload and json header.
+
+== Options
+
+//tag::configuration-properties[]
+$$object.detection.cache-model$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$true$$`)*
+$$object.detection.confidence$$:: $$<documentation missing>$$ *($$Float$$, default: `$$0.4$$`)*
+$$object.detection.debug-output$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
+$$object.detection.debug-output-path$$:: $$<documentation missing>$$ *($$String$$, default: `$$object-detection-result.png$$`)*
+$$object.detection.labels$$:: $$Labels URI.$$ *($$String$$, default: `$$https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt$$`)*
+$$object.detection.model$$:: $$pre-trained tensorflow object detection model.$$ *($$String$$, default: `$$https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb$$`)*
+$$object.detection.response-size$$:: $$<documentation missing>$$ *($$Integer$$, default: `$$<none>$$`)*
+$$object.detection.with-masks$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
+//end::configuration-properties[]
+
+//end::ref-doc[]

applications/processor/object-detection-processor/pom.xml

@@ -0,0 +1,97 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>object-detection-processor</artifactId>
+    <version>3.0.0-SNAPSHOT</version>
+    <name>object-detection-processor</name>
+    <description>Object Detection (tensorflow) processor apps</description>
+    <packaging>jar</packaging>
+
+    <parent>
+        <groupId>org.springframework.cloud.stream.app</groupId>
+        <artifactId>stream-applications-core</artifactId>
+        <version>3.0.0-SNAPSHOT</version>
+        <relativePath/>
+    </parent>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-configuration-processor</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-test</artifactId>
+            <scope>test</scope>
+            <exclusions>
+                <exclusion>
+                    <groupId>org.junit.vintage</groupId>
+                    <artifactId>junit-vintage-engine</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework.cloud.fn</groupId>
+            <artifactId>object-detection-function</artifactId>
+            <version>${java-functions.version}</version>
+        </dependency>
+
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.springframework.cloud</groupId>
+                <artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
+            </plugin>
+            <plugin>
+                <groupId>org.springframework.cloud.stream.app.plugin</groupId>
+                <artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
+                <configuration>
+                    <generatedApp>
+                        <name>object-detection</name>
+                        <type>processor</type>
+                        <version>${project.version}</version>
+                        <configClass>org.springframework.cloud.stream.app.processor.object.detection.ObjectDetectionProcessorConfiguration.class</configClass>
+                        <functionDefinition>objectDetection</functionDefinition>
+                    </generatedApp>
+
+                    <dependencies>
+                        <dependency>
+                            <groupId>org.springframework.cloud.stream.app</groupId>
+                            <artifactId>object-detection-processor</artifactId>
+                            <version>${project.version}</version>
+                        </dependency>
+                    </dependencies>
+
+                    <containerImage>
+                        <enableMetadata>true</enableMetadata>
+                    </containerImage>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+    <repositories>
+        <repository>
+            <snapshots>
+                <enabled>true</enabled>
+            </snapshots>
+            <id>spring-snapshots</id>
+            <name>Spring Snapshots</name>
+            <url>https://repo.spring.io/libs-snapshot-local</url>
+        </repository>
+        <repository>
+            <snapshots>
+                <enabled>false</enabled>
+            </snapshots>
+            <id>spring-milestones</id>
+            <name>Spring Milestones</name>
+            <url>https://repo.spring.io/libs-milestone-local</url>
+        </repository>
+    </repositories>
+
+</project>

applications/processor/object-detection-processor/src/main/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorConfiguration.java

@@ -0,0 +1,100 @@
+/*
+ * Copyright 2020-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.stream.app.processor.object.detection;
+
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction;
+import org.springframework.cloud.fn.object.detection.ObjectDetectionImageAugmenter;
+import org.springframework.cloud.fn.object.detection.ObjectDetectionService;
+import org.springframework.cloud.fn.object.detection.domain.ObjectDetection;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.integration.support.MessageBuilder;
+import org.springframework.messaging.Message;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * @author Christian Tzolov
+ */
+@Configuration
+@EnableConfigurationProperties(ObjectDetectionProcessorProperties.class)
+public class ObjectDetectionProcessorConfiguration {
+
+	private static final Log logger = LogFactory.getLog(ObjectDetectionProcessorConfiguration.class);
+
+	/**
+	 * Name of the Message header containing the JSON encoded detected objects.
+	 */
+	public static final String DETECTED_OBJECTS_HEADER = "detected_objects";
+
+	@Bean
+	public ObjectDetectionService objectDetectionService(ObjectDetectionProcessorProperties properties) {
+		return new ObjectDetectionService(properties.getModel(),
+				properties.getLabels(), properties.getConfidence(), properties.isWithMasks(),
+				properties.isCacheModel());
+	}
+
+	@Bean
+	public Function<Message<byte[]>, Message<byte[]>> objectDetection(
+			ObjectDetectionService objectDetectionService,
+			ObjectDetectionProcessorProperties properties) {
+
+		return input -> {
+			// You can use file:, http: or classpath: to provide the path to the input image.
+			byte[] inputImage = input.getPayload();
+
+			List<ObjectDetection> detectedObjects = objectDetectionService.detect(inputImage);
+
+			if (!CollectionUtils.isEmpty(detectedObjects) && properties.getResponseSize() < detectedObjects.size()) {
+				detectedObjects = detectedObjects.subList(0, properties.getResponseSize());
+			}
+
+			// Draw the predicted labels on top of the input image.
+			byte[] augmentedImage = new ObjectDetectionImageAugmenter().apply(inputImage, detectedObjects);
+
+			String jsonDetectedObjects = new JsonMapperFunction().apply(detectedObjects);
+
+			Message<byte[]> outMessage = MessageBuilder
+					.withPayload(augmentedImage)
+					.setHeader(DETECTED_OBJECTS_HEADER, jsonDetectedObjects)
+					.build();
+
+			if (properties.isDebugOutput()) {
+				try {
+					logger.info("detected objects = " + jsonDetectedObjects);
+					IOUtils.write(augmentedImage, new FileOutputStream(properties.getDebugOutputPath()));
+				}
+				catch (IOException e) {
+					logger.warn("Cloud not produce debug output", e);
+				}
+			}
+
+			return outMessage;
+
+		};
+	}
+
+}

applications/processor/object-detection-processor/src/main/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorProperties.java

@@ -0,0 +1,114 @@
+/*
+ * Copyright 2020-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.stream.app.processor.object.detection;
+
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.validation.annotation.Validated;
+
+/**
+ * @author Christian Tzolov
+ */
+@ConfigurationProperties("object.detection")
+@Validated
+public class ObjectDetectionProcessorProperties {
+
+	/**
+	 * pre-trained tensorflow object detection model.
+	 */
+	private String model = "https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb";
+
+	/**
+	 * Labels URI.
+	 */
+	private String labels = "https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt";
+
+	private float confidence = 0.4f;
+
+	private boolean withMasks;
+
+	private boolean cacheModel = true;
+
+	private boolean debugOutput = false;
+
+	private String debugOutputPath = "object-detection-result.png";
+
+	private int responseSize = Integer.MAX_VALUE;
+
+	public boolean isDebugOutput() {
+		return debugOutput;
+	}
+
+	public void setDebugOutput(boolean debugOutput) {
+		this.debugOutput = debugOutput;
+	}
+
+	public String getDebugOutputPath() {
+		return debugOutputPath;
+	}
+
+	public void setDebugOutputPath(String debugOutputPath) {
+		this.debugOutputPath = debugOutputPath;
+	}
+
+	public String getModel() {
+		return model;
+	}
+
+	public void setModel(String model) {
+		this.model = model;
+	}
+
+	public String getLabels() {
+		return labels;
+	}
+
+	public void setLabels(String labels) {
+		this.labels = labels;
+	}
+
+	public float getConfidence() {
+		return confidence;
+	}
+
+	public void setConfidence(float confidence) {
+		this.confidence = confidence;
+	}
+
+	public boolean isWithMasks() {
+		return withMasks;
+	}
+
+	public void setWithMasks(boolean withMasks) {
+		this.withMasks = withMasks;
+	}
+
+	public boolean isCacheModel() {
+		return cacheModel;
+	}
+
+	public void setCacheModel(boolean cacheModel) {
+		this.cacheModel = cacheModel;
+	}
+
+	public int getResponseSize() {
+		return responseSize;
+	}
+
+	public void setResponseSize(int responseSize) {
+		this.responseSize = responseSize;
+	}
+}

applications/processor/object-detection-processor/src/main/resources/META-INF/dataflow-configuration-metadata-whitelist.properties

@@ -0,0 +1,2 @@
+configuration-properties.classes=\
+  org.springframework.cloud.stream.app.processor.object.detection.ObjectDetectionProcessorProperties

applications/processor/object-detection-processor/src/test/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorTests.java

@@ -0,0 +1,100 @@
+/*
+ * Copyright 2020-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.stream.app.processor.object.detection;
+
+import java.io.IOException;
+
+import org.junit.jupiter.api.Test;
+
+import org.springframework.boot.WebApplicationType;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.boot.builder.SpringApplicationBuilder;
+import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils;
+import org.springframework.cloud.stream.binder.test.InputDestination;
+import org.springframework.cloud.stream.binder.test.OutputDestination;
+import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
+import org.springframework.context.ConfigurableApplicationContext;
+import org.springframework.context.annotation.Import;
+import org.springframework.messaging.Message;
+import org.springframework.messaging.support.GenericMessage;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * @author Christian Tzolov
+ */
+public class ObjectDetectionProcessorTests {
+
+	@Test
+	public void testObjectDetectionProcessor() throws IOException {
+		try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
+				TestChannelBinderConfiguration.getCompleteConfiguration(ObjectDetectionProcessorTestApplication.class))
+				.web(WebApplicationType.NONE)
+				.run("--spring.cloud.function.definition=objectDetection",
+						"--object.detection.model=https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb",
+						"--object.detection.labels=https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt",
+						"--object.detection.responseSize=10",
+						"--object.detection..debugOutput=true",
+						"--object.detection..debugOutputPath=./target/object-detection-1.png")) {
+
+			InputDestination processorInput = context.getBean(InputDestination.class);
+			OutputDestination processorOutput = context.getBean(OutputDestination.class);
+
+			byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/pivotal.jpeg");
+			processorInput.send(new GenericMessage<>(inputImage));
+			Message<byte[]> sourceMessage = processorOutput.receive(10000);
+			String jsonRecognizedObjects = (String) sourceMessage.getHeaders().get(ObjectDetectionProcessorConfiguration.DETECTED_OBJECTS_HEADER);
+			assertThat(jsonRecognizedObjects).isNotEmpty();
+			//assertThat(jsonRecognizedObjects)
+			//		.isEqualTo("[{\"label\":\"giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca\",\"probability\":0.962329626083374}," +
+			//				"{\"label\":\"badger\",\"probability\":0.006058811210095882}," +
+			//				"{\"label\":\"ram, tup\",\"probability\":0.0010668420000001788}]");
+		}
+	}
+
+	//@Test
+	public void testObjectDetectionProcessoriNaturalistSpecies() throws IOException {
+		try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
+				TestChannelBinderConfiguration.getCompleteConfiguration(ObjectDetectionProcessorTestApplication.class))
+				.web(WebApplicationType.NONE)
+				.run(
+						//"--spring.cloud.function.definition=objectDetection",
+						"--object.detection.model=http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_fgvc_2018_07_19.tar.gz#frozen_inference_graph.pb",
+						"--object.detection.labels=https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/fgvc_2854_classes_label_map.pbtxt",
+						"--object.detection..debugOutput=true",
+						"--object.detection..debugOutputPath=./target/object-detection-2.png")) {
+
+			InputDestination processorInput = context.getBean(InputDestination.class);
+			OutputDestination processorOutput = context.getBean(OutputDestination.class);
+
+			byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/animals2.jpg");
+			processorInput.send(new GenericMessage<>(inputImage));
+			Message<byte[]> sourceMessage = processorOutput.receive(10000);
+			String jsonRecognizedObjects = (String) sourceMessage.getHeaders().get(ObjectDetectionProcessorConfiguration.DETECTED_OBJECTS_HEADER);
+			//assertThat(jsonRecognizedObjects)
+			//		.isEqualTo("[{\"label\":\"giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca\",\"probability\":0.962329626083374}," +
+			//				"{\"label\":\"badger\",\"probability\":0.006058811210095882}," +
+			//				"{\"label\":\"ram, tup\",\"probability\":0.0010668420000001788}]");
+		}
+	}
+
+	@SpringBootApplication
+	@Import({ ObjectDetectionProcessorConfiguration.class })
+	public static class ObjectDetectionProcessorTestApplication {
+	}
+
+}