Fix Azure chat client image handling for byte arrays

Resolve issue with AzureOpenAiChatModel processing non-public images as
byte arrays. Implement handling for both URL strings and byte arrays,
converting latter to base64 encoded data URLs. Add test case for
resource-based media data. Update documentation with sample for
classpath resources.
This commit is contained in:
Szymon Ochnio
2024-08-01 23:52:05 +02:00
committed by Mark Pollack
parent 8469d7dc27
commit 197fe8105c
4 changed files with 50 additions and 2 deletions

View File

@@ -215,6 +215,21 @@ for carrying. The bowl is placed on a flat surface with a neutral-colored backgr
view of the fruit inside.
----
You can also pass in a classpath resource instead of a URL as shown in the example below
[source,java]
----
Resource resource = new ClassPathResource("multimodality/multimodal.test.png");
String response = ChatClient.create(chatModel).prompt()
.options(AzureOpenAiChatOptions.builder()
.withDeploymentName("gpt-4o").build())
.user(u -> u.text("Explain what do you see on this picture?")
.media(MimeTypeUtils.IMAGE_PNG, resource))
.call()
.content();
----
== Sample Controller
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-azure-openai-spring-boot-starter` to your pom (or gradle) dependencies.