Spring/spring-ai
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update AnthropicChatModel and Spring AI Documentation for Multimodal Support

Browse Source 
- Enhance AnthropicChatModel to support PDF and document content types
- Introduce getContentBlockTypeByMedia method for flexible media type handling
- Update ContentBlock handling to dynamically determine content type for media
- Add multimodal PDF support test case for Claude 3.5 Sonnet
- Update documentation to reflect PDF and multimodal capabilities
- Modify comparison chart to show PDF support for Anthropic Claude

Fixese #1819

review
This commit is contained in:
Christian Tzolov
2024-11-25 16:13:56 +01:00
committed by Mark Pollack
parent 9d207e37be
commit e543cd082d
8 changed files with 357 additions and 276 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chat/anthropic-chat.adoc
View File

@@ -146,10 +146,12 @@ Read more about xref:api/chat/functions/anthropic-chat-functions.adoc[Anthropic
== Multimodal
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats. This paradigm represents a significant advancement in AI models.
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, pdf, images, data formats.
=== Images
Currently, Anthropic Claude 3 supports the `base64` source type for `images`, and the `image/jpeg`, `image/png`, `image/gif`, and `image/webp` media types.
Check the link:https://docs.anthropic.com/claude/docs/vision[Vision guide] for more information.
Anthropic Claude 3.5 Sonnet also supports the `pdf` source type for `application/pdf` files.
Spring AI's `Message` interface supports multimodal AI models by introducing the Media type.
This type contains data and information about media attachments in messages, using Spring's `org.springframework.util.MimeType` and a `java.lang.Object` for the raw media data.
@@ -176,14 +178,23 @@ along with the text message "Explain what do you see on this picture?", and gene
----
The image shows a close-up view of a wire fruit basket containing several pieces of fruit.
The basket appears to be made of thin metal wires formed into a round shape with an elevated handle.
...
----
Inside the basket, there are a few yellow bananas and a couple of red apples or possibly tomatoes.
The vibrant colors of the fruit contrast nicely against the metallic tones of the wire basket.
=== PDF
The shallow depth of field in the photograph puts the focus squarely on the fruit in the foreground, while the basket handle extending upwards is slightly blurred, creating a pleasing bokeh effect in the background.
Starting with Sonnet 3.5 https://docs.anthropic.com/en/docs/build-with-claude/pdf-support[PDF support (beta)] is provided.
Use the `application/pdf` media type to attach a PDF file to the message:
The composition and lighting give the image a clean, minimalist aesthetic that highlights the natural beauty and freshness of the fruit displayed in this elegant wire basket.
[source,java]
----
var pdfData = new ClassPathResource("/spring-ai-reference-overview.pdf");
var userMessage = new UserMessage(
"You are a very professional document summarization specialist. Please summarize the given document.",
List.of(new Media(new MimeType("application", "pdf"), pdfData)));
var response = this.chatModel.call(new Prompt(List.of(userMessage)));
----
== Sample Controller
@@ -195,7 +206,7 @@ Add a `application.properties` file, under the `src/main/resources` directory, t
[source,application.properties]
----
spring.ai.anthropic.api-key=YOUR_API_KEY
spring.ai.anthropic.chat.options.model=claude-3-5-sonnet-20241022
spring.ai.anthropic.chat.options.model=claude-3-5-sonnet-latest
spring.ai.anthropic.chat.options.temperature=0.7
spring.ai.anthropic.chat.options.max-tokens=450
----

2
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chat/comparison.adoc
View File

@@ -19,7 +19,7 @@ This table compares various Chat Models supported by Spring AI, detailing their
|====
| Provider | Multimodality ^| Tools/Functions ^| Streaming ^| Retry ^| Observability ^| Built-in JSON ^| Local ^| OpenAI API Compatible
| xref::api/chat/anthropic-chat.adoc[Anthropic Claude] | text, image ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::no.svg[width=12] ^a| image::no.svg[width=12] ^a| image::no.svg[width=12]
| xref::api/chat/anthropic-chat.adoc[Anthropic Claude] | text, pdf, image ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::no.svg[width=12] ^a| image::no.svg[width=12] ^a| image::no.svg[width=12]
| xref::api/chat/azure-openai-chat.adoc[Azure OpenAI] | text, image ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::no.svg[width=12] ^a| image::yes.svg[width=16]
| xref::api/chat/vertexai-gemini-chat.adoc[Google VertexAI Gemini] | text, pdf, image, audio, video ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::no.svg[width=12] ^a| image::yes.svg[width=16]
| xref::api/chat/groq-chat.adoc[Groq (OpenAI-proxy)] | text, image ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::yes.svg[width=16] ^a| image::no.svg[width=12] ^a| image::no.svg[width=12] ^a| image::yes.svg[width=16]

19
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chat/vertexai-gemini-chat.adoc
View File

@@ -125,8 +125,8 @@ Read more about xref:api/chat/functions/vertexai-gemini-chat-functions.adoc[Vert
== Multimodal
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including `text`, `pdf`, `images`, `audio`, and other data formats.
This paradigm represents a significant advancement in AI models.
=== Image, Audio, Video
Google's Gemini AI models support this capability by comprehending and integrating text, code, audio, images, and video.
For more details, refer to the blog post https://blog.google/technology/ai/google-gemini-ai/#introducing-gemini[Introducing Gemini].
@@ -146,6 +146,23 @@ var userMessage = new UserMessage("Explain what do you see on this picture?",
ChatResponse response = chatModel.call(new Prompt(List.of(this.userMessage)));
----
=== PDF
Latest Vertex Gemini provides support for PDF input types..
Use the `application/pdf` media type to attach a PDF file to the message:
[source,java]
----
var pdfData = new ClassPathResource("/spring-ai-reference-overview.pdf");
var userMessage = new UserMessage(
"You are a very professional document summarization specialist. Please summarize the given document.",
List.of(new Media(new MimeType("application", "pdf"), pdfData)));
var response = this.chatModel.call(new Prompt(List.of(userMessage)));
----
== Sample Controller
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-vertex-ai-gemini-spring-boot-starter` to your pom (or gradle) dependencies.