Add Multi-Modality Support for OpenAI

- Implemented a MediaContent abstraction within the OpenAiApi to handle text and image inputs.
 - Response message content remains a plain String, ensuring backward compatibility.
 - Extended the OpenAiChatClient request creation process to seamlessly map Spring AI Messages with
   Media content to the low-level OpenAiApi MediaContent types.
 - Added integration tests for embedded and URL images, covering both synchronous and streaming calls.
 - Updated the OpenAI class diagram to reflect the new media content types provided by the OpenAI API.
 - Incorporated a chapter on multi-modality within the openai-chat.adoc documentation.
 - Improve the openai multimoality doc
This commit is contained in:
Christian Tzolov
2024-03-21 11:38:31 +01:00
committed by Mark Pollack
parent b8f773cc78
commit 834d2d0487
13 changed files with 241 additions and 38 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 532 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 424 KiB

View File

@@ -141,6 +141,57 @@ You can register custom Java functions with the OpenAiChatClient and have the Op
This is a powerful technique to connect the LLM capabilities with external tools and APIs.
Read more about xref:api/chat/functions/openai-chat-functions.adoc[OpenAI Function Calling].
== Multimodal
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.
Presently, the OpenAI `gpt-4-visual-preview` model offers multimodal support. Refer to the link:https://platform.openai.com/docs/guides/vision[Vision] guide for more information.
The OpenAI link:https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages[User Message API] can incorporate a list of base64-encoded images or image urls with the message.
Spring AIs link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Message.java[Message] interface facilitates multimodal AI models by introducing the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Media.java[Media] type.
This type encompasses data and details regarding media attachments in messages, utilizing Springs `org.springframework.util.MimeType` and a `java.lang.Object` for the raw media data.
Below is a code example excerpted from link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/test/java/org/springframework/ai/openai/OpenAiChatClientIT.java[OpenAiChatClientIT.java], illustrating the fusion of user text with an image.
[source,java]
----
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
var userMessage = new UserMessage("Explain what do you see on this picture?",
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage),
OpenAiChatOptions.builder().withModel(OpenAiApi.ChatModel.GPT_4_VISION_PREVIEW.getValue()).build()));
----
or the image URL equivalent:
[source,java]
----
var userMessage = new UserMessage("Explain what do you see on this picture?",
List.of(new Media(MimeTypeUtils.IMAGE_PNG,
"https://docs.spring.io/spring-ai/reference/1.0-SNAPSHOT/_images/multimodal.test.png")));
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage),
OpenAiChatOptions.builder().withModel(OpenAiApi.ChatModel.GPT_4_VISION_PREVIEW.getValue()).build()));
----
TIP: you can pass multiple images as well.
It takes as an input the `multimodal.test.png` image:
image::multimodal.test.png[Multimodal Test Image, 200, 200, align="left"]
along with the text message "Explain what do you see on this picture?", and generates a response like this:
----
This is an image of a fruit bowl with a simple design. The bowl is made of metal with curved wire edges that
create an open structure, allowing the fruit to be visible from all angles. Inside the bowl, there are two
yellow bananas resting on top of what appears to be a red apple. The bananas are slightly overripe, as
indicated by the brown spots on their peels. The bowl has a metal ring at the top, likely to serve as a handle
for carrying. The bowl is placed on a flat surface with a neutral-colored background that provides a clear
view of the fruit inside.
----
== Sample Controller
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.
@@ -239,7 +290,7 @@ The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-open
Following class diagram illustrates the `OpenAiApi` chat interfaces and building blocks:
image::openai-chat-api.png[OpenAiApi Chat API Diagram]
image::openai-chat-api.jpg[OpenAiApi Chat API Diagram, width=1000, align="center"]
Here is a simple snippet how to use the api programmatically: