//tag::ref-doc[]
:image-root: https://raw.githubusercontent.com/spring-cloud-stream-app-starters/tensorflow/master/images
= Object Detection Processor
The https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-object-detection[Object Detection] processor provides out-of-the-box support for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. It allows for real-time localization and identification of multiple objects in a single image or image stream. The Object Detection processor uses one of the pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[object detection] models and corresponding https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels].
Sensible defaults when the pre-trained model is not configured:
* `object.detection.model` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/faster_rcnn_resnet101_coco_2018_01_28_frozen_inference_graph.pb`
* `object.detection.labels` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt`
* `object.detection.with-masks` : `false`
The following diagram illustrates a Spring Cloud Data Flow a streaming pipeline that predicts object types from the images in real-time.
image::{image-root}/scdf-tensorflow-object-detection-arch.png[]
Processor's input is an image byte array, and the output is an augmented image byte array and a JSON header `detected_objects` in this format:
```json
{
"labels" : [
{"name":"person", "confidence":0.9996774,"x1":0.0,"y1":0.3940161,"x2":0.9465165,"y2":0.5592592,"cid":1},
{"name":"person", "confidence":0.9996604,"x1":0.047891676,"y1":0.03169123,"x2":0.941098,"y2":0.2085562,"cid":1},
{"name":"backpack", "confidence":0.96534747,"x1":0.15588468,"y1":0.85957795,"x2":0.5091308,"y2":0.9908878,"cid":23},
{"name":"backpack", "confidence":0.963343,"x1":0.1273736,"y1":0.57658505,"x2":0.47765,"y2":0.6986431,"cid":23}
]
}
```
The output JSON format is:
* *object-name*:**confidence** - human readable name of the detected object (e.g. label) with its confidence as a float between [0-1]
* *x1*, *y1*, *x2*, *y2* - Response also provides the bounding box of the detected objects represented as (x1, y1, x2, y2). The coordinates are relative to the size of the image size.
* *cid* - Classification identifier as defined in the provided https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[labels] configuration file.
=== Payload
The incoming type is `byte[]`, and the content type is `application/octet-stream`. The processor processes the input `byte[]` image and outputs augmented `byte[]` image payload and json header.
== Options
//tag::configuration-properties[]
$$object.detection.cache-model$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$true$$`)*
$$object.detection.confidence$$:: $$<documentation missing>$$ *($$Float$$, default: `$$0.4$$`)*
$$object.detection.debug-output$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
$$object.detection.debug-output-path$$:: $$<documentation missing>$$ *($$String$$, default: `$$object-detection-result.png$$`)*
$$object.detection.labels$$:: $$Labels URI.$$ *($$String$$, default: `$$https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt$$`)*
$$object.detection.model$$:: $$pre-trained tensorflow object detection model.$$ *($$String$$, default: `$$https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb$$`)*
$$object.detection.response-size$$:: $$<documentation missing>$$ *($$Integer$$, default: `$$<none>$$`)*
$$object.detection.with-masks$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
//end::configuration-properties[]
//end::ref-doc[]
