//tag::ref-doc[]
:image-root: https://github.com/spring-cloud/stream-applications/raw/master/applications/processor/object-detection-processor/src/test/resources/images

= Object Detection Processor

The Object Detection processor provides out-of-the-box support for the https://github.com/tensorflow/models/blob/master/research/object_detection/README.md[TensorFlow Object Detection API]. It allows for real-time localization and identification of multiple objects in a single image or image stream. The Object Detection processor is built on top of the https://github.com/spring-cloud/stream-applications/tree/master/functions/function/object-detection-function[Object Detection Function].

You have to provide the Processor with a pre-trained https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md[object detection model], and the corresponding https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[object labels].

Here are some sensible configuration defaults:

* `object.detection.model` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/faster_rcnn_resnet101_coco_2018_01_28_frozen_inference_graph.pb`
* `object.detection.labels` : `https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt`
* `object.detection.with-masks` : `false`

The following diagram shows a https://dataflow.spring.io/docs/concepts/streams/[Spring Cloud Data Flow], streaming pipeline, that predicts, in real-time,  the object types in input image stream.

image::{image-root}/scdf-tensorflow-object-detection-arch.png[]

Processor's input is an image byte array, and the output is an augmented image, and a header, called `detected_objects`, that provides textual description of the detected objects:

```json
{
  "labels" : [
     {"name":"person", "confidence":0.9996774,"x1":0.0,"y1":0.3940161,"x2":0.9465165,"y2":0.5592592,"cid":1},
     {"name":"person", "confidence":0.9996604,"x1":0.047891676,"y1":0.03169123,"x2":0.941098,"y2":0.2085562,"cid":1},
     {"name":"backpack", "confidence":0.96534747,"x1":0.15588468,"y1":0.85957795,"x2":0.5091308,"y2":0.9908878,"cid":23},
     {"name":"backpack", "confidence":0.963343,"x1":0.1273736,"y1":0.57658505,"x2":0.47765,"y2":0.6986431,"cid":23}
  ]
}
```

The `detected_objects` header format is:

* *object-name*:**confidence** - human readable name of the detected object (e.g. label) with its confidence as a float between [0-1]
* *x1*, *y1*, *x2*, *y2* - Response also provides the bounding box of the detected objects represented as `(x1, y1, x2, y2)`. The coordinates are relative to the size of the image size.
* *cid*  - Classification identifier as defined in the provided https://github.com/tensorflow/models/tree/865c14c/research/object_detection/data[labels] configuration file.

=== Payload

The incoming type is `byte[]`, and the content type is `application/octet-stream`. The processor processes the input `byte[]` image and outputs an augmented `byte[]` image payload and a JSON header (`detected_objects`).

== Options

//tag::configuration-properties[]
$$object.detection.cache-model$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$true$$`)*
$$object.detection.confidence$$:: $$<documentation missing>$$ *($$Float$$, default: `$$0.4$$`)*
$$object.detection.debug-output$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
$$object.detection.debug-output-path$$:: $$<documentation missing>$$ *($$String$$, default: `$$object-detection-result.png$$`)*
$$object.detection.labels$$:: $$Labels URI.$$ *($$String$$, default: `$$https://storage.googleapis.com/scdf-tensorflow-models/object-detection/mscoco_label_map.pbtxt$$`)*
$$object.detection.model$$:: $$pre-trained tensorflow object detection model.$$ *($$String$$, default: `$$https://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz#frozen_inference_graph.pb$$`)*
$$object.detection.response-size$$:: $$<documentation missing>$$ *($$Integer$$, default: `$$<none>$$`)*
$$object.detection.with-masks$$:: $$<documentation missing>$$ *($$Boolean$$, default: `$$false$$`)*
//end::configuration-properties[]

//end::ref-doc[]