diff --git a/models/spring-ai-bedrock/README_ANTHROPIC_CHAT.md b/models/spring-ai-bedrock/README_ANTHROPIC_CHAT.md index 6a63678cb..4c55598e2 100644 --- a/models/spring-ai-bedrock/README_ANTHROPIC_CHAT.md +++ b/models/spring-ai-bedrock/README_ANTHROPIC_CHAT.md @@ -2,122 +2,4 @@ Provides Bedrock Anthropic Chat API and Spring-AI chat clients. -## BedrockAnthropicChatClient - -The [BedrockAnthropicChatClient](./src/main/java/org/springframework/ai/bedrock/anthropic/BedrockAnthropicChatClient.java) implements the `ChatClient` and `StreamingChatClient` and uses the `AnthropicChatBedrockApi` library to connect to the Bedrock Anthropic service. - -Add the `spring-ai-` dependency to your project's Maven `pom.xml` file: - -```xml - - org.springframework.ai - spring-ai-bedrock - 0.8.0-SNAPSHOT - -``` - -or to your Gradle `build.gradle` build file. - -```gradle -dependencies { - implementation 'org.springframework.ai:spring-ai-bedrock:0.8.0-SNAPSHOT' -} -``` - -Next, create an `BedrockAnthropicChatClient` instance and use it to text generations requests: - -```java - AnthropicChatBedrockApi anthropicApi = new AnthropicChatBedrockApi( - AnthropicChatBedrockApi.AnthropicModel.CLAUDE_V2.id(), - EnvironmentVariableCredentialsProvider.create(), - Region.EU_CENTRAL_1.id(), - new ObjectMapper()); - - BedrockAnthropicChatClient chatClient = new BedrockAnthropicChatClient(anthropicApi, - AnthropicChatOptions.builder() - .withTemperature(0.6f) - .withTopK(10) - .withTopP(0.8f) - .withMaxTokensToSample(100) - .withAnthropicVersion(AnthropicChatBedrockApi.DEFAULT_ANTHROPIC_VERSION) - .build()); - -ChatResponse response = chatClient.call( - new Prompt("Generate the names of 5 famous pirates.")); - -// Or with streaming responses -Flux response = chatClient.stream( - new Prompt("Generate the names of 5 famous pirates.")); -``` - -or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Spring Boot starter: - -```xml - - org.springframework.ai - spring-ai-bedrock-ai-spring-boot-starter - 0.8.0-SNAPSHOT - -``` - -And set `spring.ai.bedrock.anthropic.chat.enabled=true`. -By default the client is disabled. - -Use the `BedrockAnthropicChatProperties` to configure the Bedrock Anthropic Chat client: - -| Property | Description | Default | -| ------------- | ------------- | ------------- | -| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 | -| spring.ai.bedrock.aws.accessKey | AWS credentials access key. | | -| spring.ai.bedrock.aws.secretKey | AWS credentials secret key. | | -| spring.ai.bedrock.anthropic.chat.enable | Enable Bedrock Anthropic chat client. Disabled by default | false | -| spring.ai.bedrock.anthropic.chat.model | The model id to use. See the `AnthropicChatModel` for the supported models. | anthropic.claude-v2 | -| spring.ai.bedrock.anthropic.chat.options.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.8 | -| spring.ai.bedrock.anthropic.chat.options.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default | -| spring.ai.bedrock.anthropic.chat.options.topK | Specify the number of token choices the generative uses to generate the next token. | AWS Bedrock default | -| spring.ai.bedrock.anthropic.chat.options.stopSequences | Configure up to four sequences that the generative recognizes. After a stop sequence, the generative stops generating further tokens. The returned text doesn't contain the stop sequence. | 10 | -| spring.ai.bedrock.anthropic.chat.options.anthropicVersion | The version of the generative to use. | bedrock-2023-05-31 | -| spring.ai.bedrock.anthropic.chat.options.maxTokensToSample | Specify the maximum number of tokens to use in the generated response. Note that the models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate. We recommend a limit of 4,000 tokens for optimal performance. | 500 | - -## Appendices - -## Using low-level AnthropicChatBedrockApi Library - -[AnthropicChatBedrockApi](./src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java) provides is lightweight Java client on top of AWS Bedrock [Anthropic Claude models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html). - -Following class diagram illustrates the AnthropicChatBedrockApi interface and building blocks: - -![AnthropicChatBedrockApi Class Diagram](./src/test/resources/doc/Bedrock-Anthropic-Chat-API.jpg) - -The AnthropicChatBedrockApi supports the `anthropic.claude-instant-v1` and `anthropic.claude-v2` models. - -Also the AnthropicChatBedrockApi supports both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses. - -Here is a simple snippet how to use the api programmatically: - -```java -AnthropicChatBedrockApi anthropicChatApi = new AnthropicChatBedrockApi( - AnthropicModel.CLAUDE_V2.id(), - Region.EU_CENTRAL_1.id()); - -AnthropicChatRequest request = AnthropicChatRequest - .builder(String.format(AnthropicChatBedrockApi.PROMPT_TEMPLATE, "Name 3 famous pirates")) - .withTemperature(0.8f) - .withMaxTokensToSample(300) - .withTopK(10) - // .withStopSequences(List.of("\n\nHuman:")) - .build(); - -AnthropicChatResponse response = anthropicChatApi.chatCompletion(request); - -System.out.println(response.completion()); - -// Streaming response -Flux responseStream = anthropicChatApi.chatCompletionStream(request); - -List responses = responseStream.collectList().block(); - -System.out.println(responses); -``` - -Follow the [AnthropicChatBedrockApi.java](./src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java)'s JavaDoc for further information. +Visit the Spring AI [Bedrock Anthropic Chat Documentation](https://docs.spring.io/spring-ai/reference/api/clients/bedrock/bedrock-anthropic.html). \ No newline at end of file diff --git a/models/spring-ai-bedrock/README_COHERE_CHAT.md b/models/spring-ai-bedrock/README_COHERE_CHAT.md index 9b438c862..e539a0a58 100644 --- a/models/spring-ai-bedrock/README_COHERE_CHAT.md +++ b/models/spring-ai-bedrock/README_COHERE_CHAT.md @@ -1,110 +1,5 @@ -# 1. Bedrock Cohere Chat +# Bedrock Cohere Chat Provides Bedrock Cohere Chat clients. -## 1.1 CohereChatBedrockApi - -[CohereChatBedrockApi](./src/main/java/org/springframework/ai/bedrock/cohere/api/CohereChatBedrockApi.java) provides is lightweight Java client on top of AWS Bedrock [Cohere Command models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-cohere-command.html). - -Following class diagram illustrates the Llama2ChatBedrockApi interface and building blocks: - -![CohereChatBedrockApi Class Diagram](./src/test/resources/doc/Bedrock%20Cohere%20Chat%20API.jpg) - -The CohereChatBedrockApi supports the `cohere.command-light-text-v14` and `cohere.command-text-v14` models for bot synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses. - -Here is a simple snippet how to use the api programmatically: - -```java -CohereChatBedrockApi cohereChatApi = new CohereChatBedrockApi( - CohereChatModel.COHERE_COMMAND_V14.id(), - Region.US_EAST_1.id()); - -var request = CohereChatRequest - .builder("What is the capital of Bulgaria and what is the size? What it the national anthem?") - .withStream(false) - .withTemperature(0.5f) - .withTopP(0.8f) - .withTopK(15) - .withMaxTokens(100) - .withStopSequences(List.of("END")) - .withReturnLikelihoods(CohereChatRequest.ReturnLikelihoods.ALL) - .withNumGenerations(3) - .withLogitBias(null) - .withTruncate(Truncate.NONE) - .build(); - -CohereChatResponse response = cohereChatApi.chatCompletion(request); - -var request = CohereChatRequest - .builder("What is the capital of Bulgaria and what is the size? What it the national anthem?") - .withStream(true) - .withTemperature(0.5f) - .withTopP(0.8f) - .withTopK(15) - .withMaxTokens(100) - .withStopSequences(List.of("END")) - .withReturnLikelihoods(CohereChatRequest.ReturnLikelihoods.ALL) - .withNumGenerations(3) - .withLogitBias(null) - .withTruncate(Truncate.NONE) - .build(); - -Flux responseStream = cohereChatApi.chatCompletionStream(request); -List responses = responseStream.collectList().block(); -``` - -## 1.2 BedrockCohereChatClient - -[BedrockCohereChatClient](./src/main/java/org/springframework/ai/bedrock/cohere/BedrockCohereChatClient.java) implements the Spring-Ai `ChatClient` and `StreamingChatClient` on top of the `CohereChatBedrockApi`. - -You can use like this: - -```java -@Bean -public CohereChatBedrockApi cohereApi() { - return new CohereChatBedrockApi( - CohereChatModel.COHERE_COMMAND_V14.id(), - EnvironmentVariableCredentialsProvider.create(), - Region.US_EAST_1.id(), - new ObjectMapper()); -} - -@Bean -public BedrockCohereChatClient cohereChatClient(CohereChatBedrockApi cohereApi) { - return new BedrockCohereChatClient(cohereApi); -} -``` - -or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Boot starter. For this add the following dependency: - -```xml - - spring-ai-bedrock-ai-spring-boot-starter - org.springframework.ai - 0.8.0-SNAPSHOT - -``` - -**NOTE:** You have to enable the Bedrock Cohere chat client with `spring.ai.bedrock.cohere.chat.enabled=true`. -By default the client is disabled. - -Use the `BedrockCohereChatProperties` to configure the Bedrock Cohere Chat client: - -| Property | Description | Default | -| ------------- | ------------- | ------------- | -| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 | -| spring.ai.bedrock.aws.accessKey | AWS credentials access key. | | -| spring.ai.bedrock.aws.secretKey | AWS credentials secret key. | | -| spring.ai.bedrock.cohere.chat.enable | Enable Bedrock Cohere chat client. Disabled by default | false | -| spring.ai.bedrock.cohere.chat.model | The model id to use. See the `CohereChatModel` for the supported models. | cohere.command-text-v14 | -| spring.ai.bedrock.cohere.chat.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7 | -| spring.ai.bedrock.cohere.chat.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.topK | Specify the number of token choices the model uses to generate the next token | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.maxTokens | Specify the maximum number of tokens to use in the generated response. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.stopSequences | Configure up to four sequences that the model recognizes. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.returnLikelihoods | The token likelihoods are returned with the response. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.numGenerations | The maximum number of generations that the model should return. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.logitBiasToken | Prevents the model from generating unwanted tokens or incentivize the model to include desired tokens. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.logitBiasBias | Prevents the model from generating unwanted tokens or incentivize the model to include desired tokens. | AWS Bedrock default | -| spring.ai.bedrock.cohere.chat.truncate | Specifies how the API handles inputs longer than the maximum token length | AWS Bedrock default | - +Visit the Spring AI [Bedrock Cohere Chat Documentation](https://docs.spring.io/spring-ai/reference/api/clients/bedrock/bedrock-cohere.html). \ No newline at end of file diff --git a/models/spring-ai-bedrock/README_LLAMA2_CHAT.md b/models/spring-ai-bedrock/README_LLAMA2_CHAT.md index d3d3a83ea..fa7a39a82 100644 --- a/models/spring-ai-bedrock/README_LLAMA2_CHAT.md +++ b/models/spring-ai-bedrock/README_LLAMA2_CHAT.md @@ -1,91 +1,5 @@ -# 1. Bedrock Llama2 +# Bedrock Llama2 Provides Bedrock Llama2 Chat API and Spring-AI chat clients. -## 1.1 Llama2ChatBedrockApi - -[Llama2ChatBedrockApi](./src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java) provides is lightweight Java client on top of AWS Bedrock [Meta Llama 2 and Llama 2 Chat models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-meta.html). - -Following class diagram illustrates the Llama2ChatBedrockApi interface and building blocks: - -![Llama2ChatBedrockApi Class Diagram](./src/test/resources/doc/Bedrock-Llama2-Chat-API.jpg) - -The Llama2ChatBedrockApi supports the `meta.llama2-13b-chat-v1` and `meta.llama2-70b-chat-v1` models. - -Also the Llama2ChatBedrockApi supports both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses. - -Here is a simple snippet how to use the api programmatically: - -```java -Llama2ChatBedrockApi llama2ChatApi = new Llama2ChatBedrockApi( - Llama2ChatModel.LLAMA2_70B_CHAT_V1.id(), - Region.US_EAST_1.id()); - -Llama2ChatRequest request = Llama2ChatRequest.builder("Hello, my name is") - .withTemperature(0.9f) - .withTopP(0.9f) - .withMaxGenLen(20) - .build(); - -Llama2ChatResponse response = llama2ChatApi.chatCompletion(request); - -System.out.println(response.generation()); - -// Streaming response -Flux responseStream = llama2ChatApi.chatCompletionStream(request); - -List responses = responseStream.collectList().block(); - -System.out.println(responses); -``` - -Follow the [Llama2ChatBedrockApi.java](./src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java)'s JavaDoc for further information. - -## 1.2 BedrockLlama2ChatClient - -[BedrockLlama2ChatClient](./src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java) implements the Spring-Ai `ChatClient` and `StreamingChatClient` on top of the `Llama2ChatBedrockApi`. - -You can use like this: - -```java -@Bean -public Llama2ChatBedrockApi llama2Api() { - return new Llama2ChatBedrockApi( - Llama2ChatModel.LLAMA2_70B_CHAT_V1.id(), - EnvironmentVariableCredentialsProvider.create(), - Region.US_EAST_1.id(), - new ObjectMapper()); -} - -@Bean -public BedrockLlama2ChatClient llama2ChatClient(Llama2ChatBedrockApi llama2Api) { - return new BedrockLlama2ChatClient(llama2Api); -} -``` - -or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Spring Boot starter: - -```xml - - spring-ai-bedrock-ai-spring-boot-starter - org.springframework.ai - 0.8.0-SNAPSHOT - -``` - -And set `spring.ai.bedrock.llama2.chat.enabled=true`. -By default the client is disabled. - -Use the `BedrockLlama2ChatProperties` to configure the Bedrock Llama2 Chat client: - -| Property | Description | Default | -| ------------- | ------------- | ------------- | -| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 | -| spring.ai.bedrock.aws.accessKey | AWS credentials access key. | | -| spring.ai.bedrock.aws.secretKey | AWS credentials secret key. | | -| spring.ai.bedrock.llama2.chat.enable | Enable Bedrock Llama2 chat client. Disabled by default | false | -| spring.ai.bedrock.llama2.chat.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7 | -| spring.ai.bedrock.llama2.chat.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default | -| spring.ai.bedrock.llama2.chat.maxGenLen | Specify the maximum number of tokens to use in the generated response. | 300 | -| spring.ai.bedrock.llama2.chat.model | The model id to use. See the `Llama2ChatModel` for the supported models. | meta.llama2-70b-chat-v1 | - +Visit the Spring AI [Bedrock Llama2 Chat Documentation](https://docs.spring.io/spring-ai/reference/api/clients/bedrock/bedrock-llama2.html). \ No newline at end of file diff --git a/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java b/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java index 20c81e536..e427459b6 100644 --- a/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java +++ b/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java @@ -18,19 +18,17 @@ package org.springframework.ai.bedrock.llama2; import java.util.List; -import org.springframework.ai.chat.ChatResponse; import reactor.core.publisher.Flux; import org.springframework.ai.bedrock.MessageToPromptConverter; -import org.springframework.ai.bedrock.anthropic.AnthropicChatOptions; -import org.springframework.ai.bedrock.anthropic.api.AnthropicChatBedrockApi.AnthropicChatRequest; import org.springframework.ai.bedrock.llama2.api.Llama2ChatBedrockApi; import org.springframework.ai.bedrock.llama2.api.Llama2ChatBedrockApi.Llama2ChatRequest; import org.springframework.ai.bedrock.llama2.api.Llama2ChatBedrockApi.Llama2ChatResponse; import org.springframework.ai.chat.ChatClient; import org.springframework.ai.chat.ChatOptions; -import org.springframework.ai.chat.StreamingChatClient; +import org.springframework.ai.chat.ChatResponse; import org.springframework.ai.chat.Generation; +import org.springframework.ai.chat.StreamingChatClient; import org.springframework.ai.chat.metadata.ChatGenerationMetadata; import org.springframework.ai.chat.metadata.Usage; import org.springframework.ai.chat.prompt.Prompt; diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatClient.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatClient.java index 89162ab2d..ca60a65b7 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatClient.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatClient.java @@ -28,6 +28,7 @@ public interface ChatClient extends ModelClient { return call(prompt).getResult().getOutput().getContent(); } + @Override ChatResponse call(Prompt prompt); } diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatResponse.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatResponse.java index 6ad2dab3e..c28630999 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatResponse.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatResponse.java @@ -77,6 +77,7 @@ public class ChatResponse implements ModelResponse { * @return Returns {@link ChatResponseMetadata} containing information about the use * of the AI provider's API. */ + @Override public ChatResponseMetadata getMetadata() { return this.chatResponseMetadata; } diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/Generation.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/Generation.java index 6234a0d56..cc2863c3f 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/Generation.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/Generation.java @@ -46,6 +46,7 @@ public class Generation implements ModelResult { return this.assistantMessage; } + @Override public ChatGenerationMetadata getMetadata() { ChatGenerationMetadata chatGenerationMetadata = this.chatGenerationMetadata; return chatGenerationMetadata != null ? chatGenerationMetadata : ChatGenerationMetadata.NULL; diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java index a650c9d68..018c6d379 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java @@ -16,10 +16,15 @@ package org.springframework.ai.chat; +import reactor.core.publisher.Flux; + import org.springframework.ai.chat.prompt.Prompt; import org.springframework.ai.model.StreamingModelClient; @FunctionalInterface public interface StreamingChatClient extends StreamingModelClient { + @Override + Flux stream(Prompt prompt); + } diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/Prompt.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/Prompt.java index 99c362fb5..76c749140 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/Prompt.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/Prompt.java @@ -18,6 +18,7 @@ package org.springframework.ai.chat.prompt; import org.springframework.ai.model.ModelOptions; import org.springframework.ai.model.ModelRequest; +import org.springframework.ai.chat.ChatOptions; import org.springframework.ai.chat.messages.Message; import org.springframework.ai.chat.messages.UserMessage; @@ -29,7 +30,7 @@ public class Prompt implements ModelRequest> { private final List messages; - private ModelOptions modelOptions; + private ChatOptions modelOptions; public Prompt(String contents) { this(new UserMessage(contents)); @@ -43,15 +44,15 @@ public class Prompt implements ModelRequest> { this.messages = messages; } - public Prompt(String contents, ModelOptions modelOptions) { + public Prompt(String contents, ChatOptions modelOptions) { this(new UserMessage(contents), modelOptions); } - public Prompt(Message message, ModelOptions modelOptions) { + public Prompt(Message message, ChatOptions modelOptions) { this(Collections.singletonList(message), modelOptions); } - public Prompt(List messages, ModelOptions modelOptions) { + public Prompt(List messages, ChatOptions modelOptions) { this.messages = messages; this.modelOptions = modelOptions; } diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/images/bedrock/bedrock-llama2-chat-api.jpg b/spring-ai-docs/src/main/antora/modules/ROOT/images/bedrock/bedrock-llama2-chat-api.jpg index 2c0d3c4f6..b836546eb 100644 Binary files a/spring-ai-docs/src/main/antora/modules/ROOT/images/bedrock/bedrock-llama2-chat-api.jpg and b/spring-ai-docs/src/main/antora/modules/ROOT/images/bedrock/bedrock-llama2-chat-api.jpg differ diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/images/chat-options-flow.jpg b/spring-ai-docs/src/main/antora/modules/ROOT/images/chat-options-flow.jpg new file mode 100644 index 000000000..4d259d79a Binary files /dev/null and b/spring-ai-docs/src/main/antora/modules/ROOT/images/chat-options-flow.jpg differ diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-api.jpg b/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-api.jpg new file mode 100644 index 000000000..371b3f4e9 Binary files /dev/null and b/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-api.jpg differ diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-completions-clients.jpg b/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-completions-clients.jpg new file mode 100644 index 000000000..bf529080b Binary files /dev/null and b/spring-ai-docs/src/main/antora/modules/ROOT/images/spring-ai-chat-completions-clients.jpg differ diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc index a416978e4..24814e73d 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc @@ -2,6 +2,12 @@ * xref:concepts.adoc[AI Concepts] * xref:getting-started.adoc[Getting Started] * xref:api/index.adoc[] +** xref:api/embeddings.adoc[] +*** xref:api/embeddings/onnx.adoc[] +*** xref:api/embeddings/openai-embeddings.adoc[] +*** xref:api/embeddings/ollama-embeddings.adoc[] +*** xref:api/embeddings/azure-openai-embeddings.adoc[] +*** xref:api/embeddings/postgresml-embeddings.adoc[] ** xref:api/chatclient.adoc[] *** xref:api/clients/openai-chat.adoc[] *** xref:api/clients/azure-openai-chat.adoc[] @@ -14,12 +20,6 @@ ** xref:api/prompt.adoc[] ** xref:api/output-parser.adoc[] ** xref:api/etl-pipeline.adoc[] -** xref:api/embeddings.adoc[] -*** xref:api/embeddings/onnx.adoc[] -*** xref:api/embeddings/openai-embeddings.adoc[] -*** xref:api/embeddings/ollama-embeddings.adoc[] -*** xref:api/embeddings/azure-openai-embeddings.adoc[] -*** xref:api/embeddings/postgresml-embeddings.adoc[] ** xref:api/vectordbs.adoc[] *** xref:api/vectordbs/azure.adoc[] *** xref:api/vectordbs/chroma.adoc[] diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc index a2d01e56c..3a5442b93 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc @@ -1,60 +1,87 @@ [[ChatClient]] -= ChatClient += Chat Completion API -The `ChatClient` interface streamlines interactions with xref:concepts.adoc#_models[AI Models]. -It simplifies connecting to various AI Models -— each with potentially unique APIs -— by offering a uniform interface for interaction. +The Chat Completion API offers developers the ability to integrate AI-powered chat completion capabilities into their applications. It leverages pre-trained language models, such as GPT (Generative Pre-trained Transformer), to generate human-like responses to user inputs in natural language. -Currently, the interface supports only text-based input and output. -You should expect some of the classes and interfaces to change as we add other input and output types. +The API typically works by sending a prompt or partial conversation to the AI model, which then generates a completion or continuation of the conversation based on its training data and understanding of natural language patterns. The completed response is then returned to the application, which can present it to the user or use it for further processing. -The design of the `ChatClient` interface centers around two primary goals: - -* *Portability*: It allows easy integration with different AI Models, letting developers switch between differing AI models with minimal code changes. +The `Spring AI Chat Completion API` is designed to be a simple and portable interface for interacting with various xref:concepts.adoc#_models[AI Models], allowing developers to switch between different models with minimal code changes. This design aligns with Spring's philosophy of modularity and interchangeability. -* *Simplicity*: By using companion classes like `Prompt` for input encapsulation and `ChatResponse` for output handling, the `ChatClient` interface simplifies communication with AI Models. It manages the complexity of request preparation and response parsing, offering a direct and simplified API interaction. +Also with the help of companion classes like `Prompt` for input encapsulation and `ChatResponse` for output handling, the Chat Completion API unifies the communication with AI Models. +It manages the complexity of request preparation and response parsing, offering a direct and simplified API interaction. + +The Spring AI Chat Completion API is build on top of the Spring AI `Generic Model API` providing Chat specific abstractions and implementations. Following class diagram illustrates the main classes and interfaces of the Spring AI Chat Completion API. + +image::spring-ai-chat-api.jpg[align="center", width="900px"] == API Overview -This section provides a guide to the `ChatClient` interface and associated classes. +This section provides a guide to the Spring AI Chat Completion API interface and associated classes. === ChatClient -Here is the `ChatClient` interface definition: -```java -public interface ChatClient { +Here is the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatClient.java[ChatClient] interface definition: - default String generate(String message) { // implementation omitted - } +[source,java] +---- +public interface ChatClient extends ModelClient { - ChatResponse generate(Prompt prompt); + default String call(String message) {// implementation omitted + } + @Override + ChatResponse call(Prompt prompt); } -``` -The `generate` method with a `String` parameter simplifies initial use, avoiding the complexities of the more sophisticated `Prompt` and `ChatResponse` classes. +---- -In real-world applications, it is more common to use the `generate` method that takes a `Prompt` instance and returns an `ChatResponse`. +The `call` method with a `String` parameter simplifies initial use, avoiding the complexities of the more sophisticated `Prompt` and `ChatResponse` classes. +In real-world applications, it is more common to use the `call` method that takes a `Prompt` instance and returns an `ChatResponse`. + +=== StreamingChatClient + +Here is the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java[StreamingChatClient] interface definition: + +[source,java] +---- +public interface StreamingChatClient extends StreamingModelClient { + @Override + Flux stream(Prompt prompt); +} +---- + +The `stream` method takes a `Prompt` request similar to `ChatClient` but it streams the responses using the reactive Flux API. === Prompt -The `Prompt` class encapsulates a list of `Message` objects. +The https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/Prompt.java[Prompt] is a `ModelRequest` that encapsulates a list of https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/messages/Message.java[Message] objects and optional model request options. The following listing shows a truncated version of the Prompt class, excluding constructors and other utility methods: -```java -public class Prompt { +[source,java] +---- +public class Prompt implements ModelRequest> { private final List messages; + private ChatOptions modelOptions; + + @Override + public ChatOptions getOptions() {..} + + @Override + public List getInstructions() {...} + // constructors and utility methods omitted } -``` +---- -=== Message +==== Message The `Message` interface encapsulates a textual message, a collection of attributes as a `Map`, and a categorization known as `MessageType`. The interface is defined as follows: -```java +[source,java] +---- public interface Message { String getContent(); @@ -62,181 +89,106 @@ public interface Message { Map getProperties(); MessageType getMessageType(); - } -``` +---- The `Message` interface has various implementations that correspond to the categories of messages that an AI model can process. Some models, like OpenAI's chat completion endpoint, distinguish between message categories based on conversational roles, effectively mapped by the `MessageType`. -For instance, OpenAI recognizes message categories for distinct conversational roles such as "`system,`" "`user,`" or "`assistant.`" +For instance, OpenAI recognizes message categories for distinct conversational roles such as `system`,`user`, `function` or `assistant`. While the term `MessageType` might imply a specific message format, in this context it effectively designates the role a message plays in the dialogue. For AI models that do not use specific roles, the `UserMessage` implementation acts as a standard category, typically representing user-generated inquiries or instructions. To understand the practical application and the relationship between `Prompt` and `Message`, especially in the context of these roles or message categories, see the detailed explanations in the xref:api/prompt.adoc[Prompts] section. +==== Chat Options + +Represents the options that can be passed to the AI model. The `ChatOptions` class is a subclass of `ModelOptions` and is used to define few portable options that can be passed to the AI model. +The `ChatOptions` class is defined as follows: + +[source,java] +---- +public interface ChatOptions extends ModelOptions { + + Float getTemperature(); + void setTemperature(Float temperature); + Float getTopP(); + void setTopP(Float topP); + Integer getTopK(); + void setTopK(Integer topK); +} +---- + +Additionally, every model specific ChatClient/StreamingChatClient implementation can have its own options that can be passed to the AI model. For example, the OpenAI Chat Completion model has its own options like `presencePenalty`, `frequencyPenalty`, `bestOf` etc. + +This is a powerful feature that allows developers to use model specific options when starting the application and then override them with at runtime using the Prompt request: + +image::chat-options-flow.jpg[align="center", width="800px"] + === ChatResponse The structure of the `ChatResponse` class is as follows: -```java -public class ChatResponse { +[source,java] +---- +public class ChatResponse implements ModelResponse { + private final ChatResponseMetadata chatResponseMetadata; private final List generations; + @Override + public ChatResponseMetadata getMetadata() {...} + + @Override + public List getResults() {...} + // other methods omitted } -``` +---- -The `ChatResponse` class holds the AI Model's output, with each `Generation` instance containing one of potentially multiple outputs resulting from a single prompt. +The https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatResponse.java[ChatResponse] class holds the AI Model's output, with each `Generation` instance containing one of potentially multiple outputs resulting from a single prompt. -The `ChatResponse` class also carries a map of key-value pairs providing metadata about the AI Model's response. This feature is still in progress and is not elaborated on in this document. +The `ChatResponse` class also carries a `ChatResponseMetadata` metadata about the AI Model's response. === Generation -Finally, the `Generation` class contains a `String` that represents the output text and a map that provides metadata about this response: - - -```java -public class Generation { - - private final String text; - - private Map info; - -} -``` - -== Available Implementations - -There is a `ChatClient` implementation for the following Model providers: - -* OpenAI -* Azure OpenAI: Using https://learn.microsoft.com/en-us/java/api/overview/azure/ai-openai-readme?view=azure-java-preview[Microsoft's OpenAI client library]. -* Hugging Face: Using the https://huggingface.co/inference-endpoints[Hugging Face Hosted Inference Service]. This gives you access to hundreds of models. -* https://ollama.ai/[Ollama]: Run large language models locally. - -Planned implementations - -* Amazon Bedrock: This can provide access to many AI models. -* Google Vertex: Providing access to 'Bard' (AKA Palm2). - -Others are welcome. The list is not at all closed. - -== OpenAI-Compatible Models - -A variety of models compatible with the OpenAI API are available, including those that can be operated locally, such as https://github.com/mudler/LocalAI[LocalAI]. The standard configuration for connecting to the OpenAI API is through the `spring.ai.openai.baseUrl` property, which defaults to `https://api.openai.com`. - -To link the OpenAI client to a compatible model that uses the OpenAI API, you should adjust the `spring.ai.openai.baseUrl` property to the corresponding URL of the model you wish to connect to. - -== Configuration - -This section describes how to configure models, including: - -* <> -* <> -* <> -* <> - -[[openan-api]] -=== OpenAI - -Add the Spring Boot starter to you project's dependencies: - -[source, xml] ----- - - org.springframework.ai - spring-ai-azure-openai-spring-boot-starter - 0.8.0-SNAPSHOT - ----- - -This makes an instance of the `ChatClient` that is backed by the https://github.com/TheoKanning/openai-java[Theo Kanning client library] available for injection in your application classes. - -The Spring AI project defines a configuration property named `spring.ai.openai.api-key` that you should set to the value of the `API Key` obtained from `openai.com`. - -Exporting an environment variable is one way to set that configuration property. - -[source,shell] ----- -export SPRING_AI_OPENAI_API_KEY= ----- - -[[azure-openai-api]] -=== Azure OpenAI - -This makes an instance of the `ChatClient` that is backed by https://learn.microsoft.com/en-us/java/api/overview/azure/ai-openai-readme?view=azure-java-preview[Microsoft's OpenAI client library] available for injection in your application classes. - -The Spring AI project defines a configuration property named `spring.ai.azure.openai.api-key` that you should set to the value of the `API Key` obtained from Azure. -There is also a configuration property named `spring.ai.azure.openai.endpoint` that you should set to the endpoint URL obtained when provisioning your model in Azure. - -Exporting environment variables is one way to set these configuration properties. - -[source,shell] ----- -export SPRING_AI_AZURE_OPENAI_API_KEY= -export SPRING_AI_AZURE_OPENAI_ENDPOINT= ----- - -[[hugging-face-api]] -=== Hugging Face - -There is not yet a Spring Boot Starter for this client implementation, so you should add the dependency to the HuggingFace client implementation to your project's dependencies and export an environment variable: - -[source, xml] ----- - - org.springframework.ai - spring-ai-huggingface - 0.8.0-SNAPSHOT - ----- - -[source,shell] ----- -export HUGGINGFACE_API_KEY=your_api_key_here ----- - -Obtain the endpoint URL of the inference endpoint. You can find this on the Inference Endpoint's UI https://ui.endpoints.huggingface.co/[here]. - -[[ollama-api]] -=== Ollama - -There is not yet a Spring Boot Starter for this client implementation, so you should add the dependency to the Ollama client implementation to your project's dependencies: - -[source, xml] ----- - - org.springframework.ai - spring-ai-ollama - 0.8.0-SNAPSHOT - ----- - -== Example Usage - -The following listing shows a simple "Hello, world" example. It uses the `ChatClient.generate` method that takes a `String` as input and returns a `String` as output: +Finally, the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/Generation.java[Generation] class extends from the `ModelResult` to represent the output assistant message response and related metadata about this result: [source,java] ---- -@RestController -public class SimpleAiController { +public class Generation implements ModelResult { - private final ChatClient chatClient; + private AssistantMessage assistantMessage; + private ChatGenerationMetadata chatGenerationMetadata; - @Autowired - public SimpleAiController(ChatClient chatClient) { - this.chatClient = chatClient; - } + @Override + public AssistantMessage getOutput() {...} - @GetMapping("/ai/generate") - public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { - return Map.of("generation", chatClient.generate(message)); - } + @Override + public ChatGenerationMetadata getMetadata() {...} + + // other methods omitted } ---- +== Available Implementations + +The `ChatClient` and `StreamingChatClient` implementations are provided for the following Model providers: + +image::spring-ai-chat-completions-clients.jpg[align="center", width="800px"] + +* link:clients/openai-chat.html[OpenAI Chat Completion] +* link:clients/azure-openai-chat.html[Microsoft Azure Open AI Chat Completion] +* link:clients/ollama-chat.html[Ollama Chat Completion] +* link:clients/huggingface.html[HuggingFace Chat Completion] (no streaming support) +* link:clients/vertex-chat.html[Google Vertex Chat Completion] (no streaming support) +* link:clients/bedrock.html[Amazon Bedrock] +** link:clients/bedrock/bedrock-cohere.html[Cohere Chat Completion] +** link:clients/bedrock/bedrock-llama2.html[Llama2 Chat Completion] +** link:clients/bedrock/bedrock-titan.html[Titan Chat Completion] +** link:clients/bedrock/bedrock-anthropic.html[Anthropic Chat Completion] +** link:clients/bedrock/bedrock-jurassic.html[Jurassic2 Chat Completion] (WIP, no streaming support) + // == Best Practices // // TBD @@ -245,7 +197,7 @@ public class SimpleAiController { // // TBD -== API Docs +== API Docs (OUTDATED!!!) You can find the Javadoc https://docs.spring.io/spring-ai/docs/current-SNAPSHOT/[here]. diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/embeddings.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/embeddings.adoc index b59567307..f1c53aa08 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/embeddings.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/embeddings.adoc @@ -1,5 +1,5 @@ [[EmbeddingClient]] -= EmbeddingClient += Embedding API The `EmbeddingClient` interface is designed for straightforward integration with embedding models in AI and machine learning. Its primary function is to convert text into numerical vectors, commonly referred to as embeddings.