:images-asciidoc: https://raw.githubusercontent.com/spring-cloud/stream-applications/master/functions/function/image-recognition-function/src/main/resources/images/
# Image Recognition
[.lead]
Java model inference library for the https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] image recognition architectures.
Provides real-time recognition of the https://dl.bintray.com/big-data/generic/imagenet_comp_graph_label_strings.txt[LSVRC-2012-CLS categories] in the input images.
[cols="1,2",frame=none,grid=none]
|===
| image:{images-asciidoc}/image-augmented.jpg[alt=Inception 1,width=100%]
|The https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognition.java[ImageRecognition] takes an image and outputs a list of probable categories the image contains. The response is represented by https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/RecognitionResponse.java[RecognitionResponse] class.
The https://github.com/spring-cloud/stream-applications/blob/master/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
converting the `RecognitionResponse` into JSON objects and the
https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionAugmenter.java[ImageRecognitionAugmenter] can augment the input image with the detected categories (as shown in pic. 1).
|===
## Usage
Add the `image-recognition` dependency to the pom (use the latest version available):
[source,xml]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>image-recognition-function</artifactId>
<version>${revision}</version>
</dependency>
----
#### Example 1: Image Recognition
The following snippet demonstrates how to use the `ImageRecognition` for detecting the categories present in an input image.
It also shows how to convert the result into JSON format and augment the input image with the detected category labels.
[source,java,linenums]
----
ImageRecognition recognitionService = ImageRecognition.mobilenetModeV2(
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb", //<1>
224, //<2>
5, //<3>
true); //<4>
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg"); //<5>
List<RecognitionResponse> recognizedObjects = recognitionService.recognize(inputImage); //<6>
----
<1> Downloads and loads a pre-trained `mobilenet_v2_1.4_224_frozen.pb` model.
Mind that on first attempt it will download few hundreds of MBs.
The consecutive runs will use the cached copy (5) instead.
The category labels for the MobileNetV2 are resolved from `src/main/resources/labels/mobilenet_labels.txt`.
<2> The wxh seize of the input normalized image.
<3> Top K result to return.
<4> Cache the model on the local file system.
<5> Load the image to recognise.
<6> Return a map of the top-k most probable category names and their probabilities.
The `ImageRecognition.mobilenetModeV1` and `ImageRecognition.inception` factory methods help to load and configure pre-trained mobilenetModeV1 and and Inception models.
Next you can convert the result in JSON format.
[source,java,linenums]
----
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
----
.Sample Image Recognition JSON representation
[source,json]
----
[{"label":"giant panda","probability":0.9946687817573547},{"label":"Arctic fox","probability":0.0036631098482757807},{"label":"ice bear","probability":3.3782739774324E-4},{"label":"American black bear","probability":2.3452856112271547E-4},{"label":"skunk","probability":1.6454080468975008E-4}]
----
Use the `ImageRecognitionAugmenter` to draw the recognise categories on top of the input image.
[source,java,linenums]
----
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects); //<1>
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented.jpg"));//<2>
----
<1> Augment the image with the recognized categories (uses Java2D internally).
<2> Stores the augmented image as `image-augmented.jpg` image file.
.Augmented image-augmented.jpg file
image:{images-asciidoc}/image-recognition-panda-augmented.jpg[alt=Augmented,width=30%]
## Models
This implementation supports all pre-trained https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] models.
Following URI notation can be used to download any of the models directly from the zoo.
----
http://<zoo model tar.gz url>#<frozen inference graph name.pb>
----
The `<frozen inference graph name.pb>` is the frozen model file name within the archive.
TIP: To speedup the bootstrap performance you may consider extracting the model and caching it locally.
Then you can use the `file://path-to-my-local-copy` URI schema to access it.
NOTE: It is important to use the labels that correspond to the model being used!
Table below highlights this mapping.
