Add Ollama multimodality support

- Requires LLaVa or Bakllava models.
  - Extend the Ollama chat client to support the Spring AI Medi type inputs.
  - Add OllamaChatClientMultimodalIT.java
  - Add Multimodal section in Ollama's docs.

 Resolves #421
This commit is contained in:
Christian Tzolov
2024-03-20 21:04:05 +01:00
parent dbf0f9e9e1
commit 7a55d66d77
12 changed files with 192 additions and 14 deletions

View File

@@ -144,7 +144,7 @@ Below is a simple code example extracted from https://github.com/spring-projects
----
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
var userMessage = new UserMessage("Explain what do you see o this picture?",
var userMessage = new UserMessage("Explain what do you see on this picture?",
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage)));

View File

@@ -123,6 +123,48 @@ ChatResponse response = chatClient.call(
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
== Multimodal
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.
Presently, the https://ollama.com/library/llava[LLaVa] and https://ollama.com/library/bakllava[bakllava] Ollama models offer multimodal support.
For further details, refer to the link:https://llava-vl.github.io/[LLaVA: Large Language and Vision Assistant].
The Ollama link:https://github.com/ollama/ollama/blob/main/docs/api.md#parameters-1[Message API] provides an "images" parameter to incorporate a list of base64-encoded images with the message.
Spring AIs link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Message.java[Message] interface facilitates multimodal AI models by introducing the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Media.java[Media] type.
This type encompasses data and details regarding media attachments in messages, utilizing Springs `org.springframework.util.MimeType` and a `java.lang.Object` for the raw media data.
Below is a straightforward code example excerpted from link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatClientMultimodalIT.java[OllamaChatClientMultimodalIT.java], illustrating the fusion of user text with an image.
[source,java]
----
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
var userMessage = new UserMessage("Explain what do you see on this picture?",
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
ChatResponse response = chatClient.call(
new Prompt(List.of(userMessage), OllamaOptions.create().withModel("llava")));
logger.info(response.getResult().getOutput().getContent());
----
It takes as an input the `multimodal.test.png` image:
image::multimodal.test.png[Multimodal Test Image, 200, 200, align="left"]
along with the text message "Explain what do you see on this picture?", and generates a response like this:
----
The image shows a small metal basket filled with ripe bananas and red apples. The basket is placed on a surface,
which appears to be a table or countertop, as there's a hint of what seems like a kitchen cabinet or drawer in
the background. There's also a gold-colored ring visible behind the basket, which could indicate that this
photo was taken in an area with metallic decorations or fixtures. The overall setting suggests a home environment
where fruits are being displayed, possibly for convenience or aesthetic purposes.
----
== Sample Controller
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.