84e732da08
* Add `spring-` prefix to function names Fixes: #9 This commit renames each sub-module in the common, consumer, function and supplier groups with a prefix of `spring-`. * Update README.adoc links to new prefixed names
104 lines
5.7 KiB
Plaintext
104 lines
5.7 KiB
Plaintext
:images-asciidoc: https://raw.githubusercontent.com/spring-cloud/stream-applications/master/functions/function/image-recognition-function/src/main/resources/images/
|
|
|
|
# Image Recognition
|
|
|
|
[.lead]
|
|
Java model inference library for the https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] image recognition architectures.
|
|
Provides real-time recognition of the https://dl.bintray.com/big-data/generic/imagenet_comp_graph_label_strings.txt[LSVRC-2012-CLS categories] in the input images.
|
|
|
|
[cols="1,2",frame=none,grid=none]
|
|
|===
|
|
| image:{images-asciidoc}/image-augmented.jpg[alt=Inception 1,width=100%]
|
|
|The https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognition.java[ImageRecognition] takes an image and outputs a list of probable categories the image contains. The response is represented by https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/RecognitionResponse.java[RecognitionResponse] class.
|
|
|
|
The https://github.com/spring-cloud/stream-applications/blob/master/functions/common/tensorflow-common/src/main/java/org/springframework/cloud/fn/common/tensorflow/deprecated/JsonMapperFunction.java[JsonMapperFunction] permits
|
|
converting the `RecognitionResponse` into JSON objects and the
|
|
https://github.com/spring-cloud/stream-applications/blob/master/functions/function/image-recognition-function/src/main/java/org/springframework/cloud/fn/image/recognition/ImageRecognitionAugmenter.java[ImageRecognitionAugmenter] can augment the input image with the detected categories (as shown in pic. 1).
|
|
|===
|
|
|
|
## Usage
|
|
|
|
Add the `image-recognition` dependency to the pom (use the latest version available):
|
|
|
|
[source,xml]
|
|
----
|
|
<dependency>
|
|
<groupId>org.springframework.cloud.fn</groupId>
|
|
<artifactId>image-recognition-function</artifactId>
|
|
<version>${revision}</version>
|
|
</dependency>
|
|
----
|
|
|
|
#### Example 1: Image Recognition
|
|
|
|
The following snippet demonstrates how to use the `ImageRecognition` for detecting the categories present in an input image.
|
|
It also shows how to convert the result into JSON format and augment the input image with the detected category labels.
|
|
|
|
[source,java,linenums]
|
|
----
|
|
ImageRecognition recognitionService = ImageRecognition.mobilenetModeV2(
|
|
"https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz#mobilenet_v2_1.4_224_frozen.pb", //<1>
|
|
224, //<2>
|
|
5, //<3>
|
|
true); //<4>
|
|
|
|
byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/giant_panda_in_beijing_zoo_1.jpg"); //<5>
|
|
|
|
List<RecognitionResponse> recognizedObjects = recognitionService.recognize(inputImage); //<6>
|
|
----
|
|
<1> Downloads and loads a pre-trained `mobilenet_v2_1.4_224_frozen.pb` model.
|
|
Mind that on first attempt it will download few hundreds of MBs.
|
|
The consecutive runs will use the cached copy (5) instead.
|
|
The category labels for the MobileNetV2 are resolved from `src/main/resources/labels/mobilenet_labels.txt`.
|
|
<2> The wxh seize of the input normalized image.
|
|
<3> Top K result to return.
|
|
<4> Cache the model on the local file system.
|
|
<5> Load the image to recognise.
|
|
<6> Return a map of the top-k most probable category names and their probabilities.
|
|
|
|
The `ImageRecognition.mobilenetModeV1` and `ImageRecognition.inception` factory methods help to load and configure pre-trained mobilenetModeV1 and and Inception models.
|
|
|
|
Next you can convert the result in JSON format.
|
|
|
|
[source,java,linenums]
|
|
----
|
|
String jsonRecognizedObjects = new JsonMapperFunction().apply(recognizedObjects);
|
|
----
|
|
|
|
.Sample Image Recognition JSON representation
|
|
[source,json]
|
|
----
|
|
[{"label":"giant panda","probability":0.9946687817573547},{"label":"Arctic fox","probability":0.0036631098482757807},{"label":"ice bear","probability":3.3782739774324E-4},{"label":"American black bear","probability":2.3452856112271547E-4},{"label":"skunk","probability":1.6454080468975008E-4}]
|
|
----
|
|
|
|
Use the `ImageRecognitionAugmenter` to draw the recognise categories on top of the input image.
|
|
|
|
[source,java,linenums]
|
|
----
|
|
byte[] augmentedImage = new ImageRecognitionAugmenter().apply(inputImage, recognizedObjects); //<1>
|
|
IOUtils.write(augmentedImage, new FileOutputStream("./image-recognition/target/image-augmented.jpg"));//<2>
|
|
----
|
|
<1> Augment the image with the recognized categories (uses Java2D internally).
|
|
<2> Stores the augmented image as `image-augmented.jpg` image file.
|
|
|
|
.Augmented image-augmented.jpg file
|
|
image:{images-asciidoc}/image-recognition-panda-augmented.jpg[alt=Augmented,width=30%]
|
|
|
|
## Models
|
|
|
|
This implementation supports all pre-trained https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models[Inception], https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md#pre-trained-models[MobileNetV1] and https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet#pretrained-models[MobileNetV2] models.
|
|
Following URI notation can be used to download any of the models directly from the zoo.
|
|
|
|
----
|
|
http://<zoo model tar.gz url>#<frozen inference graph name.pb>
|
|
----
|
|
|
|
The `<frozen inference graph name.pb>` is the frozen model file name within the archive.
|
|
|
|
|
|
TIP: To speedup the bootstrap performance you may consider extracting the model and caching it locally.
|
|
Then you can use the `file://path-to-my-local-copy` URI schema to access it.
|
|
|
|
NOTE: It is important to use the labels that correspond to the model being used!
|
|
Table below highlights this mapping.
|