diff --git a/README.adoc b/README.adoc index 2f98a244..314899a4 100644 --- a/README.adoc +++ b/README.adoc @@ -32,25 +32,25 @@ The following are the four major components of this repository. |link:functions/function/http-request-function/README.adoc[HTTP Request] |link:functions/supplier/http-supplier/README.adoc[HTTP] |link:functions/consumer/ftp-consumer/README.adoc[FTP] -|link:functions/function/image-recognition-function/README.adoc[Image Recognition(Tensorflow)] +| |link:functions/supplier/jdbc-supplier/README.adoc[JDBC] |link:functions/consumer/geode-consumer/README.adoc[Geode] -|link:functions/function/object-detection-function/README.adoc[Object Detection(Tensorflow)] +|link:functions/function/spel-function/README.adoc[SpEL] |link:functions/supplier/jms-supplier/README.adoc[JMS] |link:functions/consumer/jdbc-consumer/README.adoc[JDBC] -|link:functions/function/spel-function/README.adoc[SpEL] +|link:functions/function/splitter-function/README.adoc[Splitter] |link:functions/supplier/mongodb-supplier/README.adoc[MongoDB] |link:functions/consumer/log-consumer/README.adoc[Log] -|link:functions/function/splitter-function/README.adoc[Splitter] +|link:functions/function/task-launch-request-function/README.adoc[Task Launch Request] |link:functions/supplier/mqtt-supplier/README.adoc[MQTT] |link:functions/consumer/mongodb-consumer/README.adoc[MongoDB] -|link:functions/function/task-launch-request-function/README.adoc[Task Launch Request] +|link:functions/function/tasklauncher-function/README.adoc[Task Launcher] |link:functions/supplier/rabbit-supplier/README.adoc[RabbitMQ] |link:functions/consumer/mqtt-consumer/README.adoc[MQTT] -|link:functions/function/tasklauncher-function/README.adoc[Task Launcher] +|link:functions/function/twitter-function/README.adoc[Twitter] |link:functions/supplier/s3-supplier/README.adoc[AWS S3] |link:functions/consumer/rabbit-consumer/README.adoc[RabbitMQ] -|link:functions/function/twitter-function/README.adoc[Twitter] +| |link:functions/supplier/tcp-supplier/README.adoc[TCP] |link:functions/consumer/redis-consumer/README.adoc[Redis] | @@ -93,22 +93,22 @@ The following are the four major components of this repository. |link:applications/processor/http-request-processor/README.adoc[HTTP Request] |link:applications/source/jms-source/README.adoc[JMS] |link:applications/sink/jdbc-sink/README.adoc[JDBC] -|link:applications/processor/image-recognition-processor/README.adoc[Image Recognition(Tensorflow)] +|link:applications/processor/script-processor/README.adoc[Script] |link:applications/source/load-generator-source/README.adoc[Load-Generator] |link:applications/sink/log-sink/README.adoc[Log] -|link:applications/processor/object-detection-processor/README.adoc[Object Detection(Tensorflow)] +|link:applications/processor/splitter-processor/README.adoc[Splitter] |link:applications/source/mongodb-source/README.adoc[MongoDB] |link:applications/sink/mongodb-sink/README.adoc[MongoDB] -|link:applications/processor/script-processor/README.adoc[Script] +|link:applications/processor/transform-processor/README.adoc[Transform] |link:applications/source/mqtt-source/README.adoc[MQTT] |link:applications/sink/mqtt-sink/README.adoc[MQTT] -|link:applications/processor/splitter-processor/README.adoc[Splitter] +|link:applications/processor/twitter-trend-processor/README.adoc[Twitter Trend] |link:applications/source/rabbit-source/README.adoc[RabbitMQ] |link:applications/sink/rabbit-sink/README.adoc[RabbitMQ] -|link:applications/processor/transform-processor/README.adoc[Transform] +| |link:applications/source/s3-source/README.adoc[AWS S3] |link:applications/sink/redis-sink/README.adoc[Redis] -|link:applications/processor/twitter-trend-processor/README.adoc[Twitter Trend] +| |link:applications/source/tcp-source/README.adoc[TCP] |link:applications/sink/router-sink/README.adoc[Router] | diff --git a/applications/processor/object-detection-processor/README.adoc b/applications/processor/object-detection-processor/README.adoc index 06937cb2..51614323 100644 --- a/applications/processor/object-detection-processor/README.adoc +++ b/applications/processor/object-detection-processor/README.adoc @@ -5,17 +5,17 @@ The https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-object-detection[Object Detection] processor provides out-of-the-box support for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. It allows for real-time localization and identification of multiple objects in a single image or image stream. The Object Detection processor uses one of the pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[object detection] models and corresponding https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels]. -If the pre-trained model is not set explicitly set then following defaults are used: +Sensible defaults when the pre-trained model is not configured: -* `tensorflow.modelFetch` : `detection_scores,detection_classes,detection_boxes,num_detections` -* `tensorflow.model` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/faster_rcnn_resnet101_coco_2018_01_28_frozen_inference_graph.pb` -* `tensorflow.object.detection.labels` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt` +* `object.detection.model` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/faster_rcnn_resnet101_coco_2018_01_28_frozen_inference_graph.pb` +* `object.detection.labels` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt` +* `object.detection.with-masks` : `false` -The following diagram illustrates a Spring Cloud Data Flow streaming pipeline that predicts object types from the images in real-time. +The following diagram illustrates a Spring Cloud Data Flow a streaming pipeline that predicts object types from the images in real-time. image::{image-root}/scdf-tensorflow-object-detection-arch.png[] -Processor's input is an image byte array and the output is a JSON message in this format: +Processor's input is an image byte array, and the output is an augmented image byte array and a JSON header `detected_objects` in this format: ```json { @@ -28,7 +28,7 @@ Processor's input is an image byte array and the output is a JSON message in thi } ``` -The output format is: +The output JSON format is: * *object-name*:**confidence** - human readable name of the detected object (e.g. label) with its confidence as a float between [0-1] * *x1*, *y1*, *x2*, *y2* - Response also provides the bounding box of the detected objects represented as (x1, y1, x2, y2). The coordinates are relative to the size of the image size. @@ -36,7 +36,7 @@ The output format is: === Payload -If the incoming type is `byte[]` and the content type is set to `application/octet-stream` , then the application process the input `byte[]` image into and outputs augmented `byte[]` image payload and json header. +The incoming type is `byte[]`, and the content type is `application/octet-stream`. The processor processes the input `byte[]` image and outputs augmented `byte[]` image payload and json header. == Options diff --git a/applications/processor/object-detection-processor/src/test/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorTests.java b/applications/processor/object-detection-processor/src/test/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorTests.java index cf2b423a..decb828d 100644 --- a/applications/processor/object-detection-processor/src/test/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorTests.java +++ b/applications/processor/object-detection-processor/src/test/java/org/springframework/cloud/stream/app/processor/object/detection/ObjectDetectionProcessorTests.java @@ -48,8 +48,8 @@ public class ObjectDetectionProcessorTests { "--object.detection.model=https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb", "--object.detection.labels=https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt", "--object.detection.responseSize=10", - "--object.detection..debugOutput=true", - "--object.detection..debugOutputPath=./target/object-detection-1.png")) { + "--object.detection.debugOutput=true", + "--object.detection.debugOutputPath=./target/object-detection-1.png")) { InputDestination processorInput = context.getBean(InputDestination.class); OutputDestination processorOutput = context.getBean(OutputDestination.class); diff --git a/applications/processor/pom.xml b/applications/processor/pom.xml index eb740eb1..bde1c4ab 100644 --- a/applications/processor/pom.xml +++ b/applications/processor/pom.xml @@ -21,6 +21,7 @@ twitter-trend-processor image-recognition-processor object-detection-processor + semantic-segmentation-processor diff --git a/applications/processor/semantic-segmentation-processor/README.adoc b/applications/processor/semantic-segmentation-processor/README.adoc new file mode 100644 index 00000000..811ffb58 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/README.adoc @@ -0,0 +1,41 @@ +//tag::ref-doc[] + += Semantic Segmentation Processor + +Image Semantic Segmentation based on the state-of-art https://github.com/tensorflow/models/tree/master/research/deeplab[DeepLab] Tensorflow model. + +The `Semantic Segmentation` is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car). +Unlike the `Instance Segmentation`, which produces instance-aware region masks, the `Semantic Segmentation` produces class-aware masks. +For implementing `Instance Segmentation` consult the https://github.com/spring-cloud/stream-applications/tree/master/functions/function/object-detection-function[Object Detection Service] instead. + +The `Semantic Segmentation Processor` uses the https://github.com/spring-cloud/stream-applications/tree/master/functions/function/semantic-segmentation-function[Semantic Segmentation Function] library and the https://github.com/spring-cloud/stream-applications/tree/master/functions/common/tensorflow-common[TensorFlow Service]. + +=== Payload + +The incoming type is `byte[]`, and the content type is `application/octet-stream`. The processor processes the input `byte[]` image and outputs augmented `byte[]` image payload and json header. + +Processor's input is an image byte array, and the output is an augmented image byte array, and a JSON header `semantic_segmentation` in this format: + +```json +[ + [ 0, 0, 0 ], + [ 127, 127, 127 ], + [ 255, 255, 255 ] + ... +] +``` + +The output header json format represents the color pixel map computed from the input image. + +== Options + +//tag::configuration-properties[] +$$semantic.segmentation.color-map-uri$$:: $$Every pre-trained model is based on certain object color maps. The pre-defined options are: - classpath:/colormap/citymap_colormap.json - classpath:/colormap/ade20k_colormap.json - classpath:/colormap/black_white_colormap.json - classpath:/colormap/mapillary_colormap.json$$ *($$String$$, default: `$$classpath:/colormap/citymap_colormap.json$$`)* +$$semantic.segmentation.debug-output$$:: $$save output image inn the local debugOutputPath path.$$ *($$Boolean$$, default: `$$false$$`)* +$$semantic.segmentation.debug-output-path$$:: $$$$ *($$String$$, default: `$$semantic-segmentation-result.png$$`)* +$$semantic.segmentation.mask-transparency$$:: $$The alpha color of the computed segmentation mask image.$$ *($$Float$$, default: `$$0.45$$`)* +$$semantic.segmentation.model$$:: $$pre-trained tensorflow semantic segmentation model.$$ *($$String$$, default: `$$http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb$$`)* +$$semantic.segmentation.output-type$$:: $$Specifies the output image type. You can return either the input image with the computed mask overlay, or the mask alone.$$ *($$OutputType$$, default: `$$$$`, possible values: `blended`,`mask`)* +//end::configuration-properties[] + +//end::ref-doc[] diff --git a/applications/processor/semantic-segmentation-processor/pom.xml b/applications/processor/semantic-segmentation-processor/pom.xml new file mode 100644 index 00000000..f95c51b5 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/pom.xml @@ -0,0 +1,102 @@ + + + 4.0.0 + semantic-segmentation-processor + 3.0.0-SNAPSHOT + semantic-segmentation-processor + Semantic Segmentation (tensorflow) processor apps + jar + + + org.springframework.cloud.stream.app + stream-applications-core + 3.0.0-SNAPSHOT + + + + + + org.springframework.boot + spring-boot-configuration-processor + + + + org.springframework.boot + spring-boot-starter-test + test + + + org.junit.vintage + junit-vintage-engine + + + + + + org.springframework.cloud.fn + semantic-segmentation-function + ${java-functions.version} + + + org.springframework.cloud.fn + object-detection-function + ${java-functions.version} + + + + + + + + org.springframework.cloud + spring-cloud-app-starter-doc-maven-plugin + + + org.springframework.cloud.stream.app.plugin + spring-cloud-stream-app-maven-plugin + + + semantic-segmentation + processor + ${project.version} + org.springframework.cloud.stream.app.processor.semantic.segmentation.SemanticSegmentationProcessorConfiguration.class + semanticSegmentationFunction + + + + + org.springframework.cloud.stream.app + semantic-segmentation-processor + ${project.version} + + + + + true + + + + + + + + + + true + + spring-snapshots + Spring Snapshots + https://repo.spring.io/libs-snapshot-local + + + + false + + spring-milestones + Spring Milestones + https://repo.spring.io/libs-milestone-local + + + + diff --git a/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorConfiguration.java b/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorConfiguration.java new file mode 100644 index 00000000..7a6e5554 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorConfiguration.java @@ -0,0 +1,89 @@ +/* + * Copyright 2020-2020 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.springframework.cloud.stream.app.processor.semantic.segmentation; + +import java.io.FileOutputStream; +import java.io.IOException; +import java.util.function.Function; + +import org.apache.commons.io.IOUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.cloud.fn.common.tensorflow.deprecated.JsonMapperFunction; +import org.springframework.cloud.fn.semantic.segmentation.SegmentationColorMap; +import org.springframework.cloud.fn.semantic.segmentation.SemanticSegmentation; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.integration.support.MessageBuilder; +import org.springframework.messaging.Message; + +/** + * @author Christian Tzolov + */ +@Configuration +@EnableConfigurationProperties(SemanticSegmentationProcessorProperties.class) +public class SemanticSegmentationProcessorConfiguration { + + private static final Log logger = LogFactory.getLog(SemanticSegmentationProcessorConfiguration.class); + + /** + * Output header name. + */ + public static final String SEMANTIC_SEGMENTATION_HEADER = "semantic_segmentation"; + + @Bean + public SemanticSegmentation semanticSegmentation(SemanticSegmentationProcessorProperties properties) { + return new SemanticSegmentation(properties.getModel(), + SegmentationColorMap.loadColorMap(properties.getColorMapUri()), null, + properties.getMaskTransparency()); + } + + @Bean + public Function, Message> semanticSegmentationFunction( + SemanticSegmentation semanticSegmentation, + SemanticSegmentationProcessorProperties properties) { + + return input -> { + // You can use file:, http: or classpath: to provide the path to the input image. + byte[] inputImage = input.getPayload(); + + byte[] outputImage = (properties.getOutputType() == SemanticSegmentationProcessorProperties.OutputType.blended) ? + semanticSegmentation.blendMask(inputImage) : semanticSegmentation.maskImage(inputImage); + + long[][] maskPixels = semanticSegmentation.maskPixels(inputImage); + String jsonMaskPixels = new JsonMapperFunction().apply(maskPixels); + + Message outMessage = MessageBuilder + .withPayload(outputImage) + .setHeader(SEMANTIC_SEGMENTATION_HEADER, jsonMaskPixels) + .build(); + + if (properties.isDebugOutput()) { + try { + IOUtils.write(outputImage, new FileOutputStream(properties.getDebugOutputPath())); + } + catch (IOException e) { + logger.warn("Cloud not produce debug output", e); + } + } + + return outMessage; + }; + } +} diff --git a/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorProperties.java b/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorProperties.java new file mode 100644 index 00000000..8eb58fe2 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/src/main/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorProperties.java @@ -0,0 +1,117 @@ +/* + * Copyright 2020-2020 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.springframework.cloud.stream.app.processor.semantic.segmentation; + +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.validation.annotation.Validated; + +/** + * @author Christian Tzolov + */ +@ConfigurationProperties("semantic.segmentation") +@Validated +public class SemanticSegmentationProcessorProperties { + + enum OutputType { + /** Input image augmented with the segmentation mask on top. */ + blended, + /** Image of the segmentation mask. */ + mask + } + + /** + * pre-trained tensorflow semantic segmentation model. + */ + private String model = "http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb"; + + /** + * Specifies the output image type. You can return either the input image with the computed mask overlay, or + * the mask alone. + */ + private OutputType outputType = OutputType.blended; + + /** + * Every pre-trained model is based on certain object color maps. + * The pre-defined options are: + * - classpath:/colormap/citymap_colormap.json + * - classpath:/colormap/ade20k_colormap.json + * - classpath:/colormap/black_white_colormap.json + * - classpath:/colormap/mapillary_colormap.json + */ + private String colorMapUri = "classpath:/colormap/citymap_colormap.json"; + + /** + * The alpha color of the computed segmentation mask image. + */ + private float maskTransparency = 0.45f; + + /** + * save output image inn the local debugOutputPath path. + */ + private boolean debugOutput = false; + + private String debugOutputPath = "semantic-segmentation-result.png"; + + public OutputType getOutputType() { + return outputType; + } + + public void setOutputType(OutputType outputType) { + this.outputType = outputType; + } + + public boolean isDebugOutput() { + return debugOutput; + } + + public void setDebugOutput(boolean debugOutput) { + this.debugOutput = debugOutput; + } + + public String getDebugOutputPath() { + return debugOutputPath; + } + + public void setDebugOutputPath(String debugOutputPath) { + this.debugOutputPath = debugOutputPath; + } + + public String getModel() { + return model; + } + + public void setModel(String model) { + this.model = model; + } + + public String getColorMapUri() { + return colorMapUri; + } + + public void setColorMapUri(String colorMapUri) { + this.colorMapUri = colorMapUri; + } + + public float getMaskTransparency() { + return maskTransparency; + } + + public void setMaskTransparency(float maskTransparency) { + this.maskTransparency = maskTransparency; + } + +} diff --git a/applications/processor/semantic-segmentation-processor/src/main/resources/META-INF/dataflow-configuration-metadata-whitelist.properties b/applications/processor/semantic-segmentation-processor/src/main/resources/META-INF/dataflow-configuration-metadata-whitelist.properties new file mode 100644 index 00000000..104f7436 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/src/main/resources/META-INF/dataflow-configuration-metadata-whitelist.properties @@ -0,0 +1,2 @@ +configuration-properties.classes=\ + org.springframework.cloud.stream.app.processor.semantic.segmentation.SemanticSegmentationProcessorProperties diff --git a/applications/processor/semantic-segmentation-processor/src/test/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorTests.java b/applications/processor/semantic-segmentation-processor/src/test/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorTests.java new file mode 100644 index 00000000..e85a9561 --- /dev/null +++ b/applications/processor/semantic-segmentation-processor/src/test/java/org/springframework/cloud/stream/app/processor/semantic/segmentation/SemanticSegmentationProcessorTests.java @@ -0,0 +1,76 @@ +/* + * Copyright 2020-2020 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.springframework.cloud.stream.app.processor.semantic.segmentation; + +import java.io.IOException; + +import org.junit.jupiter.api.Test; + +import org.springframework.boot.WebApplicationType; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.builder.SpringApplicationBuilder; +import org.springframework.cloud.fn.common.tensorflow.deprecated.GraphicsUtils; +import org.springframework.cloud.stream.binder.test.InputDestination; +import org.springframework.cloud.stream.binder.test.OutputDestination; +import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration; +import org.springframework.context.ConfigurableApplicationContext; +import org.springframework.context.annotation.Import; +import org.springframework.messaging.Message; +import org.springframework.messaging.support.GenericMessage; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * @author Christian Tzolov + */ +public class SemanticSegmentationProcessorTests { + + @Test + public void testSemanticSegmentationProcessor() throws IOException { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration(SemanticSegmentationTestApplication.class)) + .web(WebApplicationType.NONE) + .run("--spring.cloud.function.definition=semanticSegmentationFunction", + "--semantic.segmentation.model=http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb", + "--semantic.segmentation.colorMapUri=classpath:/colormap/citymap_colormap.json", + "--semantic.segmentation.outputType=blended", + "--semantic.segmentation.debugOutput=true", + "--semantic.segmentation.debugOutputPath=./target/semantic-segmentation-1.png")) { + + InputDestination processorInput = context.getBean(InputDestination.class); + OutputDestination processorOutput = context.getBean(OutputDestination.class); + + byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/amsterdam-cityscape1.jpg"); + processorInput.send(new GenericMessage<>(inputImage)); + Message sourceMessage = processorOutput.receive(10000); + String jsonRecognizedObjects = (String) sourceMessage.getHeaders().get( + SemanticSegmentationProcessorConfiguration.SEMANTIC_SEGMENTATION_HEADER); + assertThat(jsonRecognizedObjects).isNotEmpty(); + //assertThat(jsonRecognizedObjects) + // .isEqualTo("[{\"label\":\"giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca\",\"probability\":0.962329626083374}," + + // "{\"label\":\"badger\",\"probability\":0.006058811210095882}," + + // "{\"label\":\"ram, tup\",\"probability\":0.0010668420000001788}]"); + } + } + + + @SpringBootApplication + @Import(SemanticSegmentationProcessorConfiguration.class) + public static class SemanticSegmentationTestApplication { + } + +} diff --git a/applications/processor/semantic-segmentation-processor/src/test/resources/images/amsterdam-cityscape1.jpg b/applications/processor/semantic-segmentation-processor/src/test/resources/images/amsterdam-cityscape1.jpg new file mode 100644 index 00000000..d77cae40 Binary files /dev/null and b/applications/processor/semantic-segmentation-processor/src/test/resources/images/amsterdam-cityscape1.jpg differ diff --git a/applications/stream-applications-build/pom.xml b/applications/stream-applications-build/pom.xml index cc0d7939..880896e4 100644 --- a/applications/stream-applications-build/pom.xml +++ b/applications/stream-applications-build/pom.xml @@ -61,8 +61,6 @@ 3.0.0-SNAPSHOT 3.0.0-SNAPSHOT 3.0.0-SNAPSHOT - 3.0.0-SNAPSHOT - 3.0.0-SNAPSHOT 3.0.0-SNAPSHOT 3.0.0-SNAPSHOT 3.0.0-SNAPSHOT diff --git a/applications/stream-applications-build/stream-applications-descriptor/pom.xml b/applications/stream-applications-build/stream-applications-descriptor/pom.xml index 9088c8ed..12211d10 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/pom.xml +++ b/applications/stream-applications-build/stream-applications-descriptor/pom.xml @@ -127,10 +127,6 @@ "${header-enricher-processor.version}".contains('SNAPSHOT') ? 'latest' : "${header-enricher-processor.version}" pom.properties['http-request-processor-docker.tag']= "${http-request-processor.version}".contains('SNAPSHOT') ? 'latest' : "${http-request-processor.version}" - pom.properties['image-recognition-processor-docker.tag']= - "${image-recognition-processor.version}".contains('SNAPSHOT') ? 'latest' : "${image-recognition-processor.version}" - pom.properties['object-detection-processor-docker.tag']= - "${object-detection-processor.version}".contains('SNAPSHOT') ? 'latest' : "${object-detection-processor.version}" pom.properties['script-processor-docker.tag']= "${script-processor.version}".contains('SNAPSHOT') ? 'latest' : "${script-processor.version}" pom.properties['splitter-processor-docker.tag']= diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-docker.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-docker.properties index 77cc617a..d1f82a74 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-docker.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-docker.properties @@ -40,8 +40,6 @@ processor.filter=docker:springcloudstream/filter-processor-kafka:@filter-process processor.groovy=docker:springcloudstream/groovy-processor-kafka:@groovy-processor-docker.tag@ processor.header-enricher=docker:springcloudstream/header-enricher-processor-kafka:@header-enricher-processor-docker.tag@ processor.http-request=docker:springcloudstream/http-request-processor-kafka:@http-request-processor-docker.tag@ -processor.image-recognition=docker:springcloudstream/image-recognition-processor-kafka:@image-recognition-processor-docker.tag@ -processor.object-detection=docker:springcloudstream/object-detection-processor-kafka:@object-detection-processor-docker.tag@ processor.script=docker:springcloudstream/script-processor-kafka:@script-processor-docker.tag@ processor.splitter=docker:springcloudstream/splitter-processor-kafka:@splitter-processor-docker.tag@ processor.transform=docker:springcloudstream/transform-processor-kafka:@transform-processor-docker.tag@ diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven-repo-url.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven-repo-url.properties index 2bf3c811..d9a46ce0 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven-repo-url.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven-repo-url.properties @@ -82,10 +82,6 @@ processor.header-enricher=https://@repo-spring-io@/org/springframework/cloud/str processor.header-enricher.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/header-enricher-processor-kafka/@header-enricher-processor.version@/header-enricher-processor-kafka-@header-enricher-processor.version@-metadata.jar processor.http-request=https://@repo-spring-io@/org/springframework/cloud/stream/app/http-request-processor-kafka/@http-request-processor.version@/http-request-processor-kafka-@http-request-processor.version@.jar processor.http-request.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/http-request-processor-kafka/@http-request-processor.version@/http-request-processor-kafka-@http-request-processor.version@-metadata.jar -processor.image-recognition=https://@repo-spring-io@/org/springframework/cloud/stream/app/image-recognition-processor-kafka/@image-recognition-processor.version@/image-recognition-processor-kafka-@image-recognition-processor.version@.jar -processor.image-recognition.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/image-recognition-processor-kafka/@image-recognition-processor.version@/image-recognition-processor-kafka-@image-recognition-processor.version@-metadata.jar -processor.object-detection=https://@repo-spring-io@/org/springframework/cloud/stream/app/object-detection-processor-kafka/@object-detection-processor.version@/object-detection-processor-kafka-@object-detection-processor.version@.jar -processor.object-detection.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/object-detection-processor-kafka/@object-detection-processor.version@/object-detection-processor-kafka-@object-detection-processor.version@-metadata.jar processor.script=https://@repo-spring-io@/org/springframework/cloud/stream/app/script-processor-kafka/@script-processor.version@/script-processor-kafka-@script-processor.version@.jar processor.script.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/script-processor-kafka/@script-processor.version@/script-processor-kafka-@script-processor.version@-metadata.jar processor.splitter=https://@repo-spring-io@/org/springframework/cloud/stream/app/splitter-processor-kafka/@splitter-processor.version@/splitter-processor-kafka-@splitter-processor.version@.jar diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven.properties index 80083881..105c02ad 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/kafka-apps-maven.properties @@ -82,10 +82,6 @@ processor.header-enricher=maven://org.springframework.cloud.stream.app:header-en processor.header-enricher.metadata=maven://org.springframework.cloud.stream.app:header-enricher-processor-kafka:jar:metadata:@header-enricher-processor.version@ processor.http-request=maven://org.springframework.cloud.stream.app:http-request-processor-kafka:@http-request-processor.version@ processor.http-request.metadata=maven://org.springframework.cloud.stream.app:http-request-processor-kafka:jar:metadata:@http-request-processor.version@ -processor.image-recognition=maven://org.springframework.cloud.stream.app:image-recognition-processor-kafka:@image-recognition-processor.version@ -processor.image-recognition.metadata=maven://org.springframework.cloud.stream.app:image-recognition-processor-kafka:jar:metadata:@image-recognition-processor.version@ -processor.object-detection=maven://org.springframework.cloud.stream.app:object-detection-processor-kafka:@object-detection-processor.version@ -processor.object-detection.metadata=maven://org.springframework.cloud.stream.app:object-detection-processor-kafka:jar:metadata:@object-detection-processor.version@ processor.script=maven://org.springframework.cloud.stream.app:script-processor-kafka:@script-processor.version@ processor.script.metadata=maven://org.springframework.cloud.stream.app:script-processor-kafka:jar:metadata:@script-processor.version@ processor.splitter=maven://org.springframework.cloud.stream.app:splitter-processor-kafka:@splitter-processor.version@ diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-docker.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-docker.properties index ecb89321..118e08a3 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-docker.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-docker.properties @@ -40,8 +40,6 @@ processor.filter=docker:springcloudstream/filter-processor-rabbit:@filter-proces processor.groovy=docker:springcloudstream/groovy-processor-rabbit:@groovy-processor-docker.tag@ processor.header-enricher=docker:springcloudstream/header-enricher-processor-rabbit:@header-enricher-processor-docker.tag@ processor.http-request=docker:springcloudstream/http-request-processor-rabbit:@http-request-processor-docker.tag@ -processor.image-recognition=docker:springcloudstream/image-recognition-processor-rabbit:@image-recognition-processor-docker.tag@ -processor.object-detection=docker:springcloudstream/object-detection-processor-rabbit:@object-detection-processor-docker.tag@ processor.script=docker:springcloudstream/script-processor-rabbit:@script-processor-docker.tag@ processor.splitter=docker:springcloudstream/splitter-processor-rabbit:@splitter-processor-docker.tag@ processor.transform=docker:springcloudstream/transform-processor-rabbit:@transform-processor-docker.tag@ diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven-repo-url.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven-repo-url.properties index 3cd8e5d5..6d4b0e36 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven-repo-url.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven-repo-url.properties @@ -82,10 +82,6 @@ processor.header-enricher=https://@repo-spring-io@/org/springframework/cloud/str processor.header-enricher.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/header-enricher-processor-rabbit/@header-enricher-processor.version@/header-enricher-processor-rabbit-@header-enricher-processor.version@-metadata.jar processor.http-request=https://@repo-spring-io@/org/springframework/cloud/stream/app/http-request-processor-rabbit/@http-request-processor.version@/http-request-processor-rabbit-@http-request-processor.version@.jar processor.http-request.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/http-request-processor-rabbit/@http-request-processor.version@/http-request-processor-rabbit-@http-request-processor.version@-metadata.jar -processor.image-recognition=https://@repo-spring-io@/org/springframework/cloud/stream/app/image-recognition-processor-rabbit/@image-recognition-processor.version@/image-recognition-processor-rabbit-@image-recognition-processor.version@.jar -processor.image-recognition.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/image-recognition-processor-rabbit/@image-recognition-processor.version@/image-recognition-processor-rabbit-@image-recognition-processor.version@-metadata.jar -processor.object-detection=https://@repo-spring-io@/org/springframework/cloud/stream/app/object-detection-processor-rabbit/@object-detection-processor.version@/object-detection-processor-rabbit-@object-detection-processor.version@.jar -processor.object-detection.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/object-detection-processor-rabbit/@object-detection-processor.version@/object-detection-processor-rabbit-@object-detection-processor.version@-metadata.jar processor.script=https://@repo-spring-io@/org/springframework/cloud/stream/app/script-processor-rabbit/@script-processor.version@/script-processor-rabbit-@script-processor.version@.jar processor.script.metadata=https://@repo-spring-io@/org/springframework/cloud/stream/app/script-processor-rabbit/@script-processor.version@/script-processor-rabbit-@script-processor.version@-metadata.jar processor.splitter=https://@repo-spring-io@/org/springframework/cloud/stream/app/splitter-processor-rabbit/@splitter-processor.version@/splitter-processor-rabbit-@splitter-processor.version@.jar diff --git a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven.properties b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven.properties index 5c0a3812..d0518d22 100644 --- a/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven.properties +++ b/applications/stream-applications-build/stream-applications-descriptor/src/main/resources/META-INF/rabbit-apps-maven.properties @@ -82,8 +82,6 @@ processor.header-enricher=maven://org.springframework.cloud.stream.app:header-en processor.header-enricher.metadata=maven://org.springframework.cloud.stream.app:header-enricher-processor-rabbit:jar:metadata:@header-enricher-processor.version@ processor.http-request=maven://org.springframework.cloud.stream.app:http-request-processor-rabbit:@http-request-processor.version@ processor.http-request.metadata=maven://org.springframework.cloud.stream.app:http-request-processor-rabbit:jar:metadata:@http-request-processor.version@ -processor.object-detection=maven://org.springframework.cloud.stream.app:object-detection-processor-rabbit:@object-detection-processor.version@ -processor.object-detection.metadata=maven://org.springframework.cloud.stream.app:object-detection-processor-rabbit:jar:metadata:@object-detection-processor.version@ processor.script=maven://org.springframework.cloud.stream.app:script-processor-rabbit:@script-processor.version@ processor.script.metadata=maven://org.springframework.cloud.stream.app:script-processor-rabbit:jar:metadata:@script-processor.version@ processor.splitter=maven://org.springframework.cloud.stream.app:splitter-processor-rabbit:@splitter-processor.version@ diff --git a/applications/stream-applications-build/stream-applications-docs/pom.xml b/applications/stream-applications-build/stream-applications-docs/pom.xml index 174c04a1..d3518de8 100644 --- a/applications/stream-applications-build/stream-applications-docs/pom.xml +++ b/applications/stream-applications-build/stream-applications-docs/pom.xml @@ -224,16 +224,6 @@ http-request-function ${java-functions.version} - - org.springframework.cloud.fn - image-recognition-function - ${java-functions.version} - - - org.springframework.cloud.fn - object-detection-function - ${java-functions.version} - org.springframework.cloud.fn spel-function diff --git a/applications/stream-applications-build/stream-applications-docs/src/main/asciidoc/processors.adoc b/applications/stream-applications-build/stream-applications-docs/src/main/asciidoc/processors.adoc index cb3699da..f64e670c 100644 --- a/applications/stream-applications-build/stream-applications-docs/src/main/asciidoc/processors.adoc +++ b/applications/stream-applications-build/stream-applications-docs/src/main/asciidoc/processors.adoc @@ -18,12 +18,6 @@ include::{stream-apps-root}/{branch}/applications/processor/header-enricher-proc [[spring-cloud-stream-modules-http-request-processor]] include::{stream-apps-root}/{branch}/applications/processor/http-request-processor/README.adoc[tags=ref-doc] -[[spring-cloud-stream-modules-image-recognition-processor]] -include::{stream-apps-root}/{branch}/applications/processor/image-recognition-processor/README.adoc[tags=ref-doc] - -[[spring-cloud-stream-modules-object-detection-processor]] -include::{stream-apps-root}/{branch}/applications/processor/object-detection-processor/README.adoc[tags=ref-doc] - [[spring-cloud-stream-modules-script-processor]] include::{stream-apps-root}/{branch}/applications/processor/script-processor/README.adoc[tags=ref-doc] @@ -33,8 +27,8 @@ include::{stream-apps-root}/{branch}/applications/processor/splitter-processor/R [[spring-cloud-stream-modules-transform-processor]] include::{stream-apps-root}/{branch}/applications/processor/transform-processor/README.adoc[tags=ref-doc] -[[spring-cloud-stream-modules-twitter-trend-processor]] -include::{stream-apps-root}/{branch}/applications/processor/twitter-trend-processor/README.adoc[tags=ref-doc] +[[spring-cloud-stream-modules-transform-processor]] +include::{stream-apps-root}/{branch}/applications/processor/transform-processor/README.adoc[tags=ref-doc] :leveloffset: -2 diff --git a/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/ProtoBufGraphDefinition.java b/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/ProtoBufGraphDefinition.java index a1cb7d49..e79be4b1 100644 --- a/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/ProtoBufGraphDefinition.java +++ b/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/ProtoBufGraphDefinition.java @@ -70,6 +70,5 @@ public class ProtoBufGraphDefinition implements GraphDefinition { while (ops.hasNext()) { System.out.println(ops.next().name()); } - System.out.println("Boza"); } } diff --git a/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SegmentationColorMap.java b/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SegmentationColorMap.java index 3a958cf1..7c42e179 100644 --- a/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SegmentationColorMap.java +++ b/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SegmentationColorMap.java @@ -16,6 +16,13 @@ package org.springframework.cloud.fn.semantic.segmentation; +import java.io.InputStream; +import java.util.Arrays; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import org.springframework.core.io.DefaultResourceLoader; + /** * * Visualizes the segmentation results via specified color map. @@ -303,4 +310,54 @@ public final class SegmentationColorMap { System.arraycopy(_CITYMAP_COLORMAP[i], 0, CITYMAP_COLORMAP[i], 0, _CITYMAP_COLORMAP[i].length); } } + + public static int[][] loadColorMap(String resourceUri) { + try { + InputStream colorMapIs = new DefaultResourceLoader().getResource(resourceUri).getInputStream(); + ColorMap colorMap = new ObjectMapper().readValue(colorMapIs, ColorMap.class); + return colorMap.getColormap(); + } + catch (Exception exception) { + throw new RuntimeException(exception); + } + } + + public static class ColorMap { + private String name; + private String info; + private int[][] colormap; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getInfo() { + return info; + } + + public void setInfo(String info) { + this.info = info; + } + + public int[][] getColormap() { + return colormap; + } + + public void setColormap(int[][] colormap) { + this.colormap = colormap; + } + + @Override + public String toString() { + return "ColorMap{" + + "name='" + name + '\'' + + "info='" + info + '\'' + + ", colormap=" + Arrays.deepToString(colormap) + + '}'; + } + } } diff --git a/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SemanticSegmentation.java b/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SemanticSegmentation.java index bfe5e317..fb872ae4 100644 --- a/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SemanticSegmentation.java +++ b/functions/function/semantic-segmentation-function/src/main/java/org/springframework/cloud/fn/semantic/segmentation/SemanticSegmentation.java @@ -25,6 +25,7 @@ import java.util.Map; import javax.imageio.ImageIO; +import com.fasterxml.jackson.databind.ObjectMapper; import org.tensorflow.Operand; import org.tensorflow.Tensor; import org.tensorflow.op.Ops; @@ -219,33 +220,31 @@ public class SemanticSegmentation implements AutoCloseable { public static void main(String[] args) throws IOException { - //String inputImageUri = "file:/Users/ctzolov/Dev/projects/mindmodel/mind-model-services/semantic-segmentation/src/test/resources/images/VikiMaxiAdi.jpg"; - String outputBlendedImagePath = "./semantic-segmentation/target/blendedImage.png"; - String outputMaskImagePath = "./semantic-segmentation/target/maskImage.png"; - - try (SemanticSegmentation segmentationService = new SemanticSegmentation( "http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz#frozen_inference_graph.pb", - SegmentationColorMap.CITYMAP_COLORMAP, null, 0.45f) + SegmentationColorMap.loadColorMap("classpath:/colormap/citymap_colormap.json"), null, 0.45f) ) { byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/amsterdam-cityscape1.jpg"); // 1. Mask pixels long[][] maskPixels = segmentationService.maskPixels(inputImage); + String json = new ObjectMapper().writeValueAsString(maskPixels); + // 2. Alpha Blending byte[] blended = segmentationService.blendMask(inputImage); - ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", new File(outputBlendedImagePath)); + ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", + new File("./functions/function/semantic-segmentation-function/target/blendedImage.png")); // 3. Mask Image byte[] maskImage = segmentationService.maskImage(inputImage); - ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", new File(outputMaskImagePath)); + ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", + new File("./functions/function/semantic-segmentation-function/target/maskImage.png")); } - try (SemanticSegmentation segmentationService = new SemanticSegmentation( "http://download.tensorflow.org/models/deeplabv3_xception_ade20k_train_2018_05_29.tar.gz#frozen_inference_graph.pb", - SegmentationColorMap.ADE20K_COLORMAP, null, 0.45f) + SegmentationColorMap.loadColorMap("classpath:/colormap/ade20k_colormap.json"), null, 0.45f) ) { byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/interior.jpg"); @@ -255,17 +254,17 @@ public class SemanticSegmentation implements AutoCloseable { // 2. Alpha Blending byte[] blended = segmentationService.blendMask(inputImage); ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", - new File("./semantic-segmentation/target/inventory-blendedImage.png")); + new File("./functions/function/semantic-segmentation-function/target/inventory-blendedImage.png")); // 3. Mask Image byte[] maskImage = segmentationService.maskImage(inputImage); ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", - new File("./semantic-segmentation/target/inventory-MaskImage.png")); + new File("./functions/function/semantic-segmentation-function/target/inventory-MaskImage.png")); } try (SemanticSegmentation segmentationService = new SemanticSegmentation( "http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", - SegmentationColorMap.BLACK_WHITE_COLORMAP, null, 0.45f) + SegmentationColorMap.loadColorMap("classpath:/colormap/black_white_colormap.json"), null, 0.45f) ) { byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); @@ -275,12 +274,12 @@ public class SemanticSegmentation implements AutoCloseable { // 2. Alpha Blending byte[] blended = segmentationService.blendMask(inputImage); ImageIO.write(ImageIO.read(new ByteArrayInputStream(blended)), "png", - new File("./semantic-segmentation/target/pascal-blendedImage.png")); + new File("./functions/function/semantic-segmentation-function/target/pascal-blendedImage.png")); // 3. Mask Image byte[] maskImage = segmentationService.maskImage(inputImage); ImageIO.write(ImageIO.read(new ByteArrayInputStream(maskImage)), "png", - new File("./semantic-segmentation/target/pascal-MaskImage.png")); + new File("./functions/function/semantic-segmentation-function/target/pascal-MaskImage.png")); } } diff --git a/functions/function/semantic-segmentation-function/src/main/resources/colormap/ade20k_colormap.json b/functions/function/semantic-segmentation-function/src/main/resources/colormap/ade20k_colormap.json new file mode 100644 index 00000000..51648f4e --- /dev/null +++ b/functions/function/semantic-segmentation-function/src/main/resources/colormap/ade20k_colormap.json @@ -0,0 +1,156 @@ +{ + "name" : "ade20k", + "info" : "ADE20K (http://groups.csail.mit.edu/vision/datasets/ADE20K/)", + "colormap" :[ + [ 0, 0, 0 ], + [ 120, 120, 120 ], + [ 180, 120, 120 ], + [ 6, 230, 230 ], + [ 80, 50, 50 ], + [ 4, 200, 3 ], + [ 120, 120, 80 ], + [ 140, 140, 140 ], + [ 204, 5, 255 ], + [ 230, 230, 230 ], + [ 4, 250, 7 ], + [ 224, 5, 255 ], + [ 235, 255, 7 ], + [ 150, 5, 61 ], + [ 120, 120, 70 ], + [ 8, 255, 51 ], + [ 255, 6, 82 ], + [ 143, 255, 140 ], + [ 204, 255, 4 ], + [ 255, 51, 7 ], + [ 204, 70, 3 ], + [ 0, 102, 200 ], + [ 61, 230, 250 ], + [ 255, 6, 51 ], + [ 11, 102, 255 ], + [ 255, 7, 71 ], + [ 255, 9, 224 ], + [ 9, 7, 230 ], + [ 220, 220, 220 ], + [ 255, 9, 92 ], + [ 112, 9, 255 ], + [ 8, 255, 214 ], + [ 7, 255, 224 ], + [ 255, 184, 6 ], + [ 10, 255, 71 ], + [ 255, 41, 10 ], + [ 7, 255, 255 ], + [ 224, 255, 8 ], + [ 102, 8, 255 ], + [ 255, 61, 6 ], + [ 255, 194, 7 ], + [ 255, 122, 8 ], + [ 0, 255, 20 ], + [ 255, 8, 41 ], + [ 255, 5, 153 ], + [ 6, 51, 255 ], + [ 235, 12, 255 ], + [ 160, 150, 20 ], + [ 0, 163, 255 ], + [ 140, 140, 140 ], + [ 250, 10, 15 ], + [ 20, 255, 0 ], + [ 31, 255, 0 ], + [ 255, 31, 0 ], + [ 255, 224, 0 ], + [ 153, 255, 0 ], + [ 0, 0, 255 ], + [ 255, 71, 0 ], + [ 0, 235, 255 ], + [ 0, 173, 255 ], + [ 31, 0, 255 ], + [ 11, 200, 200 ], + [ 255, 82, 0 ], + [ 0, 255, 245 ], + [ 0, 61, 255 ], + [ 0, 255, 112 ], + [ 0, 255, 133 ], + [ 255, 0, 0 ], + [ 255, 163, 0 ], + [ 255, 102, 0 ], + [ 194, 255, 0 ], + [ 0, 143, 255 ], + [ 51, 255, 0 ], + [ 0, 82, 255 ], + [ 0, 255, 41 ], + [ 0, 255, 173 ], + [ 10, 0, 255 ], + [ 173, 255, 0 ], + [ 0, 255, 153 ], + [ 255, 92, 0 ], + [ 255, 0, 255 ], + [ 255, 0, 245 ], + [ 255, 0, 102 ], + [ 255, 173, 0 ], + [ 255, 0, 20 ], + [ 255, 184, 184 ], + [ 0, 31, 255 ], + [ 0, 255, 61 ], + [ 0, 71, 255 ], + [ 255, 0, 204 ], + [ 0, 255, 194 ], + [ 0, 255, 82 ], + [ 0, 10, 255 ], + [ 0, 112, 255 ], + [ 51, 0, 255 ], + [ 0, 194, 255 ], + [ 0, 122, 255 ], + [ 0, 255, 163 ], + [ 255, 153, 0 ], + [ 0, 255, 10 ], + [ 255, 112, 0 ], + [ 143, 255, 0 ], + [ 82, 0, 255 ], + [ 163, 255, 0 ], + [ 255, 235, 0 ], + [ 8, 184, 170 ], + [ 133, 0, 255 ], + [ 0, 255, 92 ], + [ 184, 0, 255 ], + [ 255, 0, 31 ], + [ 0, 184, 255 ], + [ 0, 214, 255 ], + [ 255, 0, 112 ], + [ 92, 255, 0 ], + [ 0, 224, 255 ], + [ 112, 224, 255 ], + [ 70, 184, 160 ], + [ 163, 0, 255 ], + [ 153, 0, 255 ], + [ 71, 255, 0 ], + [ 255, 0, 163 ], + [ 255, 204, 0 ], + [ 255, 0, 143 ], + [ 0, 255, 235 ], + [ 133, 255, 0 ], + [ 255, 0, 235 ], + [ 245, 0, 255 ], + [ 255, 0, 122 ], + [ 255, 245, 0 ], + [ 10, 190, 212 ], + [ 214, 255, 0 ], + [ 0, 204, 255 ], + [ 20, 0, 255 ], + [ 255, 255, 0 ], + [ 0, 153, 255 ], + [ 0, 41, 255 ], + [ 0, 255, 204 ], + [ 41, 0, 255 ], + [ 41, 255, 0 ], + [ 173, 0, 255 ], + [ 0, 245, 255 ], + [ 71, 0, 255 ], + [ 122, 0, 255 ], + [ 0, 255, 184 ], + [ 0, 92, 255 ], + [ 184, 255, 0 ], + [ 0, 133, 255 ], + [ 255, 214, 0 ], + [ 25, 194, 194 ], + [ 102, 255, 0 ], + [ 92, 0, 255 ]] +} diff --git a/functions/function/semantic-segmentation-function/src/main/resources/colormap/black_white_colormap.json b/functions/function/semantic-segmentation-function/src/main/resources/colormap/black_white_colormap.json new file mode 100644 index 00000000..ca1c33ea --- /dev/null +++ b/functions/function/semantic-segmentation-function/src/main/resources/colormap/black_white_colormap.json @@ -0,0 +1,8 @@ +{ + "name" : "black_white", + "info" : "Black and white color map", + "colormap" :[ + [ 0, 0, 0 ], + [ 127, 127, 127 ], + [ 255, 255, 255 ]] +} diff --git a/functions/function/semantic-segmentation-function/src/main/resources/colormap/citymap_colormap.json b/functions/function/semantic-segmentation-function/src/main/resources/colormap/citymap_colormap.json new file mode 100644 index 00000000..dbd1ef2c --- /dev/null +++ b/functions/function/semantic-segmentation-function/src/main/resources/colormap/citymap_colormap.json @@ -0,0 +1,24 @@ +{ + "name" : "citymap", + "info" : "Cityscapes dataset (https://www.cityscapes-dataset.com).", + "colormap" :[ + [ 128, 64, 128 ], + [ 244, 35, 232 ], + [ 70, 70, 70 ], + [ 102, 102, 156 ], + [ 190, 153, 153 ], + [ 153, 153, 153 ], + [ 250, 170, 30 ], + [ 220, 220, 0 ], + [ 107, 142, 35 ], + [ 152, 251, 152 ], + [ 70, 130, 180 ], + [ 220, 20, 60 ], + [ 255, 0, 0 ], + [ 0, 0, 142 ], + [ 0, 0, 70 ], + [ 0, 60, 100 ], + [ 0, 80, 100 ], + [ 0, 0, 230 ], + [ 119, 11, 32 ]] +} diff --git a/functions/function/semantic-segmentation-function/src/main/resources/colormap/mapillary_colormap.json b/functions/function/semantic-segmentation-function/src/main/resources/colormap/mapillary_colormap.json new file mode 100644 index 00000000..f089268b --- /dev/null +++ b/functions/function/semantic-segmentation-function/src/main/resources/colormap/mapillary_colormap.json @@ -0,0 +1,71 @@ +{ + "name" : "mapillary", + "info" : "Mapillary Vistas (https://research.mapillary.com).", + "colormap" :[ + [ 165, 42, 42 ], + [ 0, 192, 0 ], + [ 196, 196, 196 ], + [ 190, 153, 153 ], + [ 180, 165, 180 ], + [ 102, 102, 156 ], + [ 102, 102, 156 ], + [ 128, 64, 255 ], + [ 140, 140, 200 ], + [ 170, 170, 170 ], + [ 250, 170, 160 ], + [ 96, 96, 96 ], + [ 230, 150, 140 ], + [ 128, 64, 128 ], + [ 110, 110, 110 ], + [ 244, 35, 232 ], + [ 150, 100, 100 ], + [ 70, 70, 70 ], + [ 150, 120, 90 ], + [ 220, 20, 60 ], + [ 255, 0, 0 ], + [ 255, 0, 0 ], + [ 255, 0, 0 ], + [ 200, 128, 128 ], + [ 255, 255, 255 ], + [ 64, 170, 64 ], + [ 128, 64, 64 ], + [ 70, 130, 180 ], + [ 255, 255, 255 ], + [ 152, 251, 152 ], + [ 107, 142, 35 ], + [ 0, 170, 30 ], + [ 255, 255, 128 ], + [ 250, 0, 30 ], + [ 0, 0, 0 ], + [ 220, 220, 220 ], + [ 170, 170, 170 ], + [ 222, 40, 40 ], + [ 100, 170, 30 ], + [ 40, 40, 40 ], + [ 33, 33, 33 ], + [ 170, 170, 170 ], + [ 0, 0, 142 ], + [ 170, 170, 170 ], + [ 210, 170, 100 ], + [ 153, 153, 153 ], + [ 128, 128, 128 ], + [ 0, 0, 142 ], + [ 250, 170, 30 ], + [ 192, 192, 192 ], + [ 220, 220, 0 ], + [ 180, 165, 180 ], + [ 119, 11, 32 ], + [ 0, 0, 142 ], + [ 0, 60, 100 ], + [ 0, 0, 142 ], + [ 0, 0, 90 ], + [ 0, 0, 230 ], + [ 0, 80, 100 ], + [ 128, 64, 64 ], + [ 0, 0, 110 ], + [ 0, 0, 70 ], + [ 0, 0, 192 ], + [ 32, 32, 32 ], + [ 0, 0, 0 ], + [ 0, 0, 0 ]] +} diff --git a/functions/function/semantic-segmentation-function/src/main/resources/images/VikiMaxiAdi.jpg b/functions/function/semantic-segmentation-function/src/main/resources/images/VikiMaxiAdi.jpg new file mode 100644 index 00000000..5d901702 Binary files /dev/null and b/functions/function/semantic-segmentation-function/src/main/resources/images/VikiMaxiAdi.jpg differ diff --git a/functions/function/semantic-segmentation-function/src/main/resources/images/amsterdam-cityscape1.jpg b/functions/function/semantic-segmentation-function/src/main/resources/images/amsterdam-cityscape1.jpg new file mode 100644 index 00000000..d77cae40 Binary files /dev/null and b/functions/function/semantic-segmentation-function/src/main/resources/images/amsterdam-cityscape1.jpg differ diff --git a/functions/function/semantic-segmentation-function/src/main/resources/images/giant_panda_in_beijing_zoo_1.jpg b/functions/function/semantic-segmentation-function/src/main/resources/images/giant_panda_in_beijing_zoo_1.jpg new file mode 100644 index 00000000..e7a44cae Binary files /dev/null and b/functions/function/semantic-segmentation-function/src/main/resources/images/giant_panda_in_beijing_zoo_1.jpg differ diff --git a/functions/function/semantic-segmentation-function/src/main/resources/images/interior.jpg b/functions/function/semantic-segmentation-function/src/main/resources/images/interior.jpg new file mode 100644 index 00000000..f5516015 Binary files /dev/null and b/functions/function/semantic-segmentation-function/src/main/resources/images/interior.jpg differ