:images-asciidoc: https://raw.githubusercontent.com/spring-cloud/stream-applications/master/functions/function/object-detection-function/src/main/resources/images/ # Object Detection Function Java model inference library for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. Allows real-time localization and identification of multiple objects in a single or batch of images. Works with all https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[pre-trained zoo models] and ttps://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels]. [cols="1,2", frame=none, grid=none] |=== | image:{images-asciidoc}/object_detection_1.jpg[alt=Object Detection 1, width=100%] |The https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionService.java[ObjectDetectionService] takes an image or a batch of images and outputs a list of predicted objects bounding boxes represented by https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/domain/ObjectDetection.java[ObjectDetection]. For the models supporting https://github.com/tensorflow/models/tree/master/research/object_detection#february-9-2018[Instance Segmentation], the `ObjectDetectionService` can predict the instance segmentation `masks` in addition to object bounding boxes. The https://github.com/spring-cloud/stream-applications/blob/master/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits converting the `List` into JSON objects and the https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter] allow to augment the input image with the detected bounding boxes and segmentation masks. |=== ## Usage Add the `object-detection` dependency to the pom (use the latest version available): [source,xml] ---- org.springframework.cloud.fn object-detection-function ${revision} ---- #### Example 1: Object Detection The https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleObjectDetection.java[ExampleObjectDetection.java] sample demonstrates how to use the `ObjectDetectionService` for detecting objects in input images. It also shows how to convert the result into JSON format and augment the input image with the detected object bounding boxes. [source,java,linenums] ---- ObjectDetectionService detectionService = new ObjectDetectionService( "https://download.tensorflow.org/models/object_detection/faster_rcnn_nas_coco_2018_01_28.tar.gz#frozen_inference_graph.pb", //<1> "https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt", //<2> 0.4f, //<3> false, //<4> true); //<5> byte[] image = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg"); //<6> List detectedObjects = detectionService.detect(image); //<7> ---- <1> Downloads and loads a pre-trained `frozen_inference_graph.pb` model directly from the `faster_rcnn_nas_coco.tar.gz` archive in the Tensorflow model zoo. Mind that on first attempt it will download few hundreds of MBs. The consecutive runs will use the cached copy (5) instead. <2> Object category labels (e.g. names) for the model <3> Confidence threshold - Only object with estimate above the threshold are returned <4> Indicate that this is not a `mask` (e.g. not an instance segmentation) model type <5> Cache the model on the local file system. <6> Load the input image to evaluate <7> Detect the objects in the image and represent the result as a list of ObjectDetection instances. Next you can convert the result in JSON format. [source,java,linenums] ---- String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects); System.out.println(jsonObjectDetections); ---- .Sample Object Detection JSON representation [source,json] ---- [{"name":"person","estimate":0.998,"x1":0.160,"y1":0.774,"x2":0.201,"y2":0.946,"cid":1}, {"name":"kite","estimate":0.998,"x1":0.437,"y1":0.089,"x2":0.495,"y2":0.169,"cid":38}, {"name":"person","estimate":0.997,"x1":0.084,"y1":0.681,"x2":0.121,"y2":0.848,"cid":1}, {"name":"kite","estimate":0.988,"x1":0.206,"y1":0.263,"x2":0.225,"y2":0.314,"cid":38}]] ---- Use the https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/main/java/org/springframework/cloud/fn/object/detection/ObjectDetectionImageAugmenter.java[ObjectDetectionImageAugmenter] to draw the detected objects on top of the input image. [source,java,linenums] ---- byte[] annotatedImage = new ObjectDetectionImageAugmenter().apply(image, detectedObjects); // <1> IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-augmented.jpg")); //<2> ---- <1> Augment the image with the detected object bounding boxes (Uses Java2D internally). <2> Stores the augmented image as `object-detection-augmented.jpg` image file. .Augmented object-detection-augmented.jpg file image:{images-asciidoc}/object-detection-augmented.jpg[alt=Object Detection, width=60%] TIP: Set the `ObjectDetectionImageAugmenter#agnosticColors` property to `true` to use a monochrome color schema. #### Example 2: Instance Segmentation The https://github.com/spring-cloud/stream-applications/blob/master/functions/function/object-detection-function/src/test/java/org/springframework/cloud/fn/object/detection/examples/ExampleInstanceSegmentation.java[ExampleInstanceSegmentation.java] sample shows how to use the `ObjectDetectionService` for `Instance Segmentation`. NOTE: It requires a trained model that supports `Masks` as well as setting the instance segmentation (e.g. `useMasks`) flag to `true`. [source,java,linenums] ---- ObjectDetectionService detectionService = new ObjectDetectionService( "https://download.tensorflow.org/models/object_detection/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz#frozen_inference_graph.pb", // <1> "https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt", // <2> 0.4f, // <3> true, // <4> true); // <5> byte[] image = GraphicsUtils.loadAsByteArray("classpath:/images/object-detection.jpg"); List detectedObjects = detectionService.detect(image); // <6> String jsonObjectDetections = new JsonMapperFunction().apply(detectedObjects); // <7> System.out.println(jsonObjectDetections); byte[] annotatedImage = new ObjectDetectionImageAugmenter(true) // <8> .apply(image, detectedObjects); IOUtils.write(annotatedImage, new FileOutputStream("./object-detection-function/target/object-detection-segmentation-augmented.jpg")); ---- <1> Uses one of the 4 MASK pre-trained models <2> Object category labels (e.g. names) for the model <3> Confidence threshold - Only object with estimate above the threshold are returned. <4> Use masks output - For the pre-trained models instruct to use the extended fetch names that include instance segmentation masks as well. <5> Cache model - Create a local copy of the model to speed up consecutive runs. <6> Evaluate the model to predict the object in the input image. <7> Convert the detected object in to JSON array. NOTE: that with mask there is an additional field: `mask` <8> Draw the detected object on top of the input image. Mind the `true` constructor parameter stands for draw detected masks. If false only the bounding boxes will be shown. .Result augmented object-detection-segmentation-augmented.jpg file image:{images-asciidoc}/object-detection-segmentation-augmented.jpg[alt=Object Detection Augmented, width=60%] ## Models All pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[detection_model_zoo.md] models are supported. Following URI notation can be used to download any of the models directly from the zoo. ---- http://#frozen_inference_graph.pb ---- The `frozen_inference_graph.pb` is the frozen model file name within the archive. NOTE: For some models this name may differ. You have to download and open the archive to find the real name. TIP: To speedup the bootstrap performance you may consider extracting the `frozen_inference_graph.pb` and caching it locally. Then you can use the `file://path-to-my-local-copy` URI schema to access it. Following models can be used for `Instance Segmentation` as well: [frame=none, grid=none] |=== | https://download.tensorflow.org/models/object_detection/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz[mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz] | https://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz[mask_rcnn_inception_v2_coco_2018_01_28.tar.gz] | https://download.tensorflow.org/models/object_detection/mask_rcnn_resnet101_atrous_coco_2018_01_28.tar.gz[mask_rcnn_resnet101_atrous_coco_2018_01_28.tar.gz] | https://download.tensorflow.org/models/object_detection/mask_rcnn_resnet50_atrous_coco_2018_01_28.tar.gz[mask_rcnn_resnet50_atrous_coco_2018_01_28.tar.gz] |=== In addition to the model, the `ObjectDetectionService` requires a list of labels that correspond to the categories detectable by the selected model. All labels files are available in the https://github.com/tensorflow/models/tree/master/research/object_detection/data[object_detection/data] folder. NOTE: It is important to use the labels that correspond to the model being used! Table below highlights this mapping. .Relationsip between trained model types and category labels [%header, cols="1,2", frame=none, grid=none] |=== | Model | Labels | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#coco-trained-models[coco] | https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxt[mscoco_label_map.pbtxt] | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#kitti-trained-models[kitti] | https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/kitti_label_map.pbtxt[kitti_label_map.pbtxt] | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#open-images-trained-models[open-images] | https://github.com/tensorflow/models/blob/master/research/object_detection/data/oid_bbox_trainable_label_map.pbtxt[oid_bbox_trainable_label_map.pbtxt] | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#inaturalist-species-trained-models[inaturalist-species] | https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/fgvc_2854_classes_label_map.pbtxt[fgvc_2854_classes_label_map.pbtxt] | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#ava-v21-trained-models[ava] | https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/ava_label_map_v2.1.pbtxt[ava_label_map_v2.1.pbtxt] |=== TIP: For performance reasons you may consider downloading the required label files to the local file system.