Add Ollama multimodality support
- Requires LLaVa or Bakllava models. - Extend the Ollama chat client to support the Spring AI Medi type inputs. - Add OllamaChatClientMultimodalIT.java - Add Multimodal section in Ollama's docs. Resolves #421
This commit is contained in:
@@ -144,7 +144,7 @@ Below is a simple code example extracted from https://github.com/spring-projects
|
||||
----
|
||||
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
|
||||
|
||||
var userMessage = new UserMessage("Explain what do you see o this picture?",
|
||||
var userMessage = new UserMessage("Explain what do you see on this picture?",
|
||||
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
|
||||
|
||||
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage)));
|
||||
|
||||
@@ -123,6 +123,48 @@ ChatResponse response = chatClient.call(
|
||||
|
||||
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
|
||||
== Multimodal
|
||||
|
||||
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.
|
||||
|
||||
Presently, the https://ollama.com/library/llava[LLaVa] and https://ollama.com/library/bakllava[bakllava] Ollama models offer multimodal support.
|
||||
For further details, refer to the link:https://llava-vl.github.io/[LLaVA: Large Language and Vision Assistant].
|
||||
|
||||
The Ollama link:https://github.com/ollama/ollama/blob/main/docs/api.md#parameters-1[Message API] provides an "images" parameter to incorporate a list of base64-encoded images with the message.
|
||||
|
||||
Spring AI’s link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Message.java[Message] interface facilitates multimodal AI models by introducing the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Media.java[Media] type.
|
||||
This type encompasses data and details regarding media attachments in messages, utilizing Spring’s `org.springframework.util.MimeType` and a `java.lang.Object` for the raw media data.
|
||||
|
||||
Below is a straightforward code example excerpted from link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatClientMultimodalIT.java[OllamaChatClientMultimodalIT.java], illustrating the fusion of user text with an image.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
|
||||
|
||||
var userMessage = new UserMessage("Explain what do you see on this picture?",
|
||||
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(List.of(userMessage), OllamaOptions.create().withModel("llava")));
|
||||
|
||||
logger.info(response.getResult().getOutput().getContent());
|
||||
----
|
||||
|
||||
It takes as an input the `multimodal.test.png` image:
|
||||
|
||||
image::multimodal.test.png[Multimodal Test Image, 200, 200, align="left"]
|
||||
|
||||
along with the text message "Explain what do you see on this picture?", and generates a response like this:
|
||||
|
||||
----
|
||||
The image shows a small metal basket filled with ripe bananas and red apples. The basket is placed on a surface,
|
||||
which appears to be a table or countertop, as there's a hint of what seems like a kitchen cabinet or drawer in
|
||||
the background. There's also a gold-colored ring visible behind the basket, which could indicate that this
|
||||
photo was taken in an area with metallic decorations or fixtures. The overall setting suggests a home environment
|
||||
where fruits are being displayed, possibly for convenience or aesthetic purposes.
|
||||
----
|
||||
|
||||
== Sample Controller
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Reference in New Issue
Block a user