Add Multi-Modality Support for OpenAI
- Implemented a MediaContent abstraction within the OpenAiApi to handle text and image inputs. - Response message content remains a plain String, ensuring backward compatibility. - Extended the OpenAiChatClient request creation process to seamlessly map Spring AI Messages with Media content to the low-level OpenAiApi MediaContent types. - Added integration tests for embedded and URL images, covering both synchronous and streaming calls. - Updated the OpenAI class diagram to reflect the new media content types provided by the OpenAI API. - Incorporated a chapter on multi-modality within the openai-chat.adoc documentation. - Improve the openai multimoality doc
This commit is contained in:
committed by
Mark Pollack
parent
b8f773cc78
commit
834d2d0487
Binary file not shown.
|
After Width: | Height: | Size: 532 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 424 KiB |
@@ -141,6 +141,57 @@ You can register custom Java functions with the OpenAiChatClient and have the Op
|
||||
This is a powerful technique to connect the LLM capabilities with external tools and APIs.
|
||||
Read more about xref:api/chat/functions/openai-chat-functions.adoc[OpenAI Function Calling].
|
||||
|
||||
== Multimodal
|
||||
|
||||
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.
|
||||
Presently, the OpenAI `gpt-4-visual-preview` model offers multimodal support. Refer to the link:https://platform.openai.com/docs/guides/vision[Vision] guide for more information.
|
||||
|
||||
The OpenAI link:https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages[User Message API] can incorporate a list of base64-encoded images or image urls with the message.
|
||||
Spring AI’s link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Message.java[Message] interface facilitates multimodal AI models by introducing the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Media.java[Media] type.
|
||||
This type encompasses data and details regarding media attachments in messages, utilizing Spring’s `org.springframework.util.MimeType` and a `java.lang.Object` for the raw media data.
|
||||
|
||||
Below is a code example excerpted from link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/test/java/org/springframework/ai/openai/OpenAiChatClientIT.java[OpenAiChatClientIT.java], illustrating the fusion of user text with an image.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
byte[] imageData = new ClassPathResource("/multimodal.test.png").getContentAsByteArray();
|
||||
|
||||
var userMessage = new UserMessage("Explain what do you see on this picture?",
|
||||
List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageData)));
|
||||
|
||||
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage),
|
||||
OpenAiChatOptions.builder().withModel(OpenAiApi.ChatModel.GPT_4_VISION_PREVIEW.getValue()).build()));
|
||||
----
|
||||
|
||||
or the image URL equivalent:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
var userMessage = new UserMessage("Explain what do you see on this picture?",
|
||||
List.of(new Media(MimeTypeUtils.IMAGE_PNG,
|
||||
"https://docs.spring.io/spring-ai/reference/1.0-SNAPSHOT/_images/multimodal.test.png")));
|
||||
|
||||
ChatResponse response = chatClient.call(new Prompt(List.of(userMessage),
|
||||
OpenAiChatOptions.builder().withModel(OpenAiApi.ChatModel.GPT_4_VISION_PREVIEW.getValue()).build()));
|
||||
----
|
||||
|
||||
TIP: you can pass multiple images as well.
|
||||
|
||||
It takes as an input the `multimodal.test.png` image:
|
||||
|
||||
image::multimodal.test.png[Multimodal Test Image, 200, 200, align="left"]
|
||||
|
||||
along with the text message "Explain what do you see on this picture?", and generates a response like this:
|
||||
|
||||
----
|
||||
This is an image of a fruit bowl with a simple design. The bowl is made of metal with curved wire edges that
|
||||
create an open structure, allowing the fruit to be visible from all angles. Inside the bowl, there are two
|
||||
yellow bananas resting on top of what appears to be a red apple. The bananas are slightly overripe, as
|
||||
indicated by the brown spots on their peels. The bowl has a metal ring at the top, likely to serve as a handle
|
||||
for carrying. The bowl is placed on a flat surface with a neutral-colored background that provides a clear
|
||||
view of the fruit inside.
|
||||
----
|
||||
|
||||
== Sample Controller
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
@@ -239,7 +290,7 @@ The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-open
|
||||
|
||||
Following class diagram illustrates the `OpenAiApi` chat interfaces and building blocks:
|
||||
|
||||
image::openai-chat-api.png[OpenAiApi Chat API Diagram]
|
||||
image::openai-chat-api.jpg[OpenAiApi Chat API Diagram, width=1000, align="center"]
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user