Add Bedrock Anthropic Chat Options support
- Improve Anthropic tests - Add anthrpic docs - Restructure the docs for Azure OpenAI, OpenAI, Ollama, Bedrock Cohere and Bedrock Lllam2
This commit is contained in:
Christian Tzolov
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 194 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 424 KiB |
@@ -8,6 +8,7 @@
|
||||
*** xref:api/clients/bedrock.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-anthropic.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-llama2.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-cohere.adoc[]
|
||||
*** xref:api/clients/huggingface.adoc[]
|
||||
*** xref:api/clients/ollama-chat.adoc[]
|
||||
** xref:api/prompt.adoc[]
|
||||
|
||||
@@ -7,10 +7,8 @@ Azure offers Java developers the opportunity to leverage AI's full potential by
|
||||
== Prerequisites
|
||||
|
||||
Obtain your Azure OpenAI `endpoint` and `api-key` from the Azure OpenAI Service section on the link:https://portal.azure.com[Azure Portal].
|
||||
|
||||
Spring AI defines a configuration property named `spring.ai.azure.openai.api-key` that you should set to the value of the `API Key` obtained from Azure.
|
||||
There is also a configuration property named `spring.ai.azure.openai.endpoint` that you should set to the endpoint URL obtained when provisioning your model in Azure.
|
||||
|
||||
Exporting environment variables is one way to set these configuration properties:
|
||||
|
||||
[source,shell]
|
||||
@@ -42,6 +40,8 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
The prefix `spring.ai.azure.openai` is the property prefix to configure the connection to Azure OpenAI.
|
||||
@@ -73,10 +73,36 @@ The prefix `spring.ai.azure.openai.chat` is the property prefix that configures
|
||||
| spring.ai.azure.openai.chat.options.frequencyPenalty | A value that influences the probability of generated tokens appearing based on their cumulative frequency in generated text. Positive values will make tokens less likely to appear as their frequency increases and decrease the likelihood of the model repeating the same statements verbatim. | -
|
||||
|====
|
||||
|
||||
=== Sample Code
|
||||
TIP: All properties prefixed with `spring.ai.azure.openai.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
The link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-azure-openai/src/main/java/org/springframework/ai/azure/openai/AzureOpenAiChatOptions.java[AzureOpenAiChatOptions.java] provides model configurations, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `AzureOpenAiChatClient(api, options)` constructor or the `spring.ai.azure.openai.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default model and temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
AzureOpenAiChatOptions.builder()
|
||||
.withModel("gpt-4-32k")
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-azure-openai/src/main/java/org/springframework/ai/azure/openai/AzureOpenAiChatOptions.java[AzureOpenAiChatOptions.java] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-azure-openai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the OpenAi Chat client:
|
||||
|
||||
[source,application.properties]
|
||||
----
|
||||
@@ -86,15 +112,21 @@ spring.ai.azure.openai.chat.options.model=gpt-35-turbo
|
||||
spring.ai.azure.openai.chat.options.temperature=0.7
|
||||
----
|
||||
|
||||
TIP: replace the `api-key` and `endpoint` with your Azure OpenAI credentials.
|
||||
|
||||
This will create a `AzureOpenAiChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final AzureOpenAiChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
public ChatController(AzureOpenAiChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@@ -102,12 +134,20 @@ public class ChatController {
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
Add the `spring-ai-azure-openai` dependency to your project's Maven `pom.xml` file:
|
||||
The link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-azure-openai/src/main/java/org/springframework/ai/azure/openai/AzureOpenAiChatClient.java[AzureOpenAiChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the link:https://learn.microsoft.com/en-us/java/api/overview/azure/ai-openai-readme?view=azure-java-preview[Azure OpenAI Java Client].
|
||||
|
||||
To enable it, add the `spring-ai-azure-openai` dependency to your project's Maven `pom.xml` file:
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
@@ -126,7 +166,7 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-azure-openai` dependency also provide the access to the `AzureOpenAiChatClient`. For more information about the `AzureOpenAiChatClient` refer to the link:../clients/azure-openai-chat.html[Azure OpenAI Chat] section.
|
||||
TIP: The `spring-ai-azure-openai` dependency also provide the access to the `AzureOpenAiChatClient`. For more information about the `AzureOpenAiChatClient` refer to the link:../clients/azure-openai-chat.html[Azure OpenAI Chat] section.
|
||||
|
||||
Next, create an `AzureOpenAiChatClient` instance and use it to generate text responses:
|
||||
|
||||
@@ -155,21 +195,3 @@ Flux<ChatResponse> response = chatClient.stream(
|
||||
|
||||
NOTE: the `gpt-35-turbo` is actually the `Deployment Name` as presented in the Azure AI Portal.
|
||||
|
||||
=== Chat Options
|
||||
|
||||
The `AzureOpenAiChatOptions` provides the configuration information for the chat requests.
|
||||
The `AzureOpenAiChatOptions` offers a builder to create the options.
|
||||
|
||||
At start time use the `AzureOpenAiChatClient` constructor to set the default options used for all char requests.
|
||||
At runtime, you can override the default options by passing a `AzureOpenAiChatOptions` instance with your to the `Prompt` request.
|
||||
|
||||
For example to override the default model name for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
AzureOpenAiChatOptions.builder().withModel("gpt-4-32k").build()
|
||||
));
|
||||
----
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
= Amazon Bedrock
|
||||
= Amazon Bedrock Chat
|
||||
|
||||
link:https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock] is a managed service that provides foundation models from various AI providers, available through a unified API.
|
||||
|
||||
@@ -86,6 +86,7 @@ For more information, refer to the documentation below for each supported model.
|
||||
|
||||
* xref:api/clients/bedrock/bedrock-anthropic.adoc[Spring AI Bedrock Anthropic Chat]: `spring.ai.bedrock.anthropic.chat.enabled=true`
|
||||
* xref:api/clients/bedrock/bedrock-llama2.adoc[Spring AI Bedrock Llama2 Chat]: `spring.ai.bedrock.llama2.chat.enabled=true`
|
||||
* xref:api/clients/bedrock/bedrock-cohere.adoc[Spring AI Bedrock Cohere Chat]: `spring.ai.bedrock.cohere.chat.enabled=true`
|
||||
|
||||
|
||||
// * [Spring AI Bedrock Cohere Chat](./README_COHERE_CHAT.md) - `spring.ai.bedrock.cohere.chat.enabled=true`
|
||||
|
||||
@@ -1,22 +1,21 @@
|
||||
= Anthropic Chat
|
||||
|
||||
https://www.anthropic.com/product[Anthropic's Claude] is an AI assistant based on Anthropic’s research into training helpful, honest, and harmless AI systems.
|
||||
|
||||
The Claude model has the following high level features
|
||||
|
||||
* 200k Token Context Window: Claude boasts a generous token capacity of 200,000, making it ideal for handling extensive information in applications like technical documentation, codebases, and literary works.
|
||||
* 200k Token Context Window: Claude boasts a generous token capacity of 200,000, making it ideal for handling extensive information in applications like technical documentation, codebase, and literary works.
|
||||
* Supported Tasks: Claude's versatility spans tasks such as summarization, Q&A, trend forecasting, and document comparisons, enabling a wide range of applications from dialogues to content generation.
|
||||
* AI Safety Features: Built on Anthropic's safety research, Claude prioritizes helpfulness, honesty, and harmlessness in its interactions, reducing brand risk and ensuring responsible AI behavior.
|
||||
|
||||
The https://aws.amazon.com/bedrock/claude[AWS Bedrock Anthropic Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model.
|
||||
|
||||
== Pre-requisites
|
||||
== Prerequisites
|
||||
|
||||
Refer to the xref:api/clients/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access.
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Spring Boot starter:
|
||||
Add the `spring-ai-bedrock-ai-spring-boot-starter` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
@@ -27,10 +26,20 @@ or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Spring Boot s
|
||||
</dependency>
|
||||
----
|
||||
|
||||
=== Enable Anthropic Support
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
Spring AI defines a configuration property named `spring.ai.bedrock.anthropic.chat.enabled` that you should set to `true` to enable support for Anthropic.
|
||||
Exporting environment variables in one way to set this configuration property.
|
||||
[source,gradle]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
=== Enable Anthropic Chat
|
||||
|
||||
By default the Anthropic model is disabled.
|
||||
To enable it set the `spring.ai.bedrock.anthropic.chat.enabled` property to `true`.
|
||||
Exporting environment variable is one way to set this configuration property:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
@@ -45,19 +54,19 @@ The prefix `spring.ai.bedrock.aws` is the property prefix to configure the conne
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1
|
||||
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1
|
||||
| spring.ai.bedrock.aws.access-key | AWS access key. | -
|
||||
| spring.ai.bedrock.aws.secret-key | AWS secret key. | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.bedrock.anthropic.chat` is the property prefix that configures the `ChatClient` implementation for Claude.
|
||||
The prefix `spring.ai.bedrock.anthropic.chat` is the property prefix that configures the chat client implementation for Claude.
|
||||
|
||||
[cols="2,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.anthropic.chat.enable | Enable Bedrock Anthropic chat client. Disabled by default | false
|
||||
| spring.ai.bedrock.anthropic.chat.model | The model id to use. See the `AnthropicChatModel` for the supported models. | anthropic.claude-v2
|
||||
| spring.ai.bedrock.anthropic.chat.model | The model id to use. See the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatModel] for the supported models. | anthropic.claude-v2
|
||||
| spring.ai.bedrock.anthropic.chat.options.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.8
|
||||
| spring.ai.bedrock.anthropic.chat.options.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default
|
||||
| spring.ai.bedrock.anthropic.chat.options.topK | Specify the number of token choices the generative uses to generate the next token. | AWS Bedrock default
|
||||
@@ -66,46 +75,84 @@ The prefix `spring.ai.bedrock.anthropic.chat` is the property prefix that config
|
||||
| spring.ai.bedrock.anthropic.chat.options.maxTokensToSample | Specify the maximum number of tokens to use in the generated response. Note that the models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate. We recommend a limit of 4,000 tokens for optimal performance. | 500
|
||||
|====
|
||||
|
||||
Look at the Spring AI enumeration `AnthropicChatModel` for other model IDs. The other value supported is `anthropic.claude-instant-v1`.
|
||||
|
||||
Look at the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatModel] for other model IDs.
|
||||
Supported values are: `anthropic.claude-instant-v1`, `anthropic.claude-v2` and `anthropic.claude-v2:1`.
|
||||
Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs].
|
||||
|
||||
=== Sample Code
|
||||
TIP: All properties prefixed with `spring.ai.bedrock.anthropic.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
Create an `application.properties` file in the `src/main/resources` directory and add the following properties to configure the Anthropic Chat client.
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/AnthropicChatOptions.java[AnthropicChatOptions.java] provides model configurations, such as temperature, topK, topP, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `BedrockAnthropicChatClient(api, options)` constructor or the `spring.ai.bedrock.anthropic.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
AnthropicChatOptions.builder()
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/AnthropicChatOptions.java[AnthropicChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-bedrock-ai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the Anthropic Chat client:
|
||||
|
||||
[source]
|
||||
----
|
||||
spring.ai.bedrock.aws.region=eu-central-1
|
||||
spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID}
|
||||
spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
|
||||
|
||||
spring.ai.bedrock.anthropic.chat.enabled=true
|
||||
spring.ai.bedrock.anthropic.chat.options.temperature=0.8
|
||||
spring.ai.bedrock.anthropic.chat.options.top-k=15
|
||||
----
|
||||
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
TIP: replace the `regions`, `access-key` and `secret-key` with your AWS credentials.
|
||||
|
||||
This will create a `BedrockAnthropicChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final BedrockAnthropicChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
public ChatController(BedrockAnthropicChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
return Map.of("generation", chatClient.call(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
The link:./src/main/java/org/springframework/ai/bedrock/anthropic/BedrockAnthropicChatClient.java[BedrockAnthropicChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the `AnthropicChatBedrockApi` library to connect to the Bedrock Anthropic service.
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/BedrockAnthropicChatClient.java[BedrockAnthropicChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Bedrock Anthropic service.
|
||||
|
||||
Add the `spring-ai-bedrock` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
@@ -127,9 +174,9 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
Next, create an `BedrockAnthropicChatClient` instance and use it to text generations requests:
|
||||
Next, create an https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/BedrockAnthropicChatClient.java[BedrockAnthropicChatClient] and use it for text generations:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@@ -156,47 +203,36 @@ Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
=== Low-level AnthropicChatBedrockApi Client [[low-level-api]]
|
||||
|
||||
== Appendices
|
||||
|
||||
=== Using low-level AnthropicChatBedrockApi Library
|
||||
|
||||
The link:./src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock link:https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html[Anthropic Claude models].
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock link:https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html[Anthropic Claude models].
|
||||
|
||||
Following class diagram illustrates the AnthropicChatBedrockApi interface and building blocks:
|
||||
|
||||
image::bedrock/bedrock-anthropic-chat-api.png[AnthropicChatBedrockApi Class Diagram]
|
||||
|
||||
The AnthropicChatBedrockApi supports the `anthropic.claude-instant-v1` and `anthropic.claude-v2` models.
|
||||
|
||||
Also the AnthropicChatBedrockApi supports both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
|
||||
Client supports the `anthropic.claude-instant-v1`, `anthropic.claude-v2` and `anthropic.claude-v2:1` models for both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
AnthropicChatBedrockApi anthropicChatApi = new AnthropicChatBedrockApi(
|
||||
AnthropicModel.CLAUDE_V2.id(),
|
||||
Region.EU_CENTRAL_1.id());
|
||||
AnthropicModel.CLAUDE_V2.id(), Region.EU_CENTRAL_1.id());
|
||||
|
||||
AnthropicChatRequest request = AnthropicChatRequest
|
||||
.builder(String.format(AnthropicChatBedrockApi.PROMPT_TEMPLATE, "Name 3 famous pirates"))
|
||||
.withTemperature(0.8f)
|
||||
.withMaxTokensToSample(300)
|
||||
.withTopK(10)
|
||||
// .withStopSequences(List.of("\n\nHuman:"))
|
||||
.build();
|
||||
|
||||
// Sync request
|
||||
AnthropicChatResponse response = anthropicChatApi.chatCompletion(request);
|
||||
|
||||
System.out.println(response.completion());
|
||||
|
||||
// Streaming response
|
||||
// Streaming request
|
||||
Flux<AnthropicChatResponse> responseStream = anthropicChatApi.chatCompletionStream(request);
|
||||
|
||||
List<AnthropicChatResponse> responses = responseStream.collectList().block();
|
||||
|
||||
System.out.println(responses);
|
||||
----
|
||||
|
||||
Follow the link:./src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatBedrockApi.java]'s JavaDoc for further information.
|
||||
Follow the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApi.java[AnthropicChatBedrockApi.java]'s JavaDoc for further information.
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
= Cohere Chat
|
||||
|
||||
Provides Bedrock Cohere Chat client.
|
||||
Integrate generative AI capabilities into essential apps and workflows that improve business outcomes.
|
||||
|
||||
The https://aws.amazon.com/bedrock/cohere-command-embed/[AWS Bedrock Cohere Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
Refer to the xref:api/clients/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access.
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
Add the `spring-ai-bedrock-ai-spring-boot-starter` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bedrock-ai-spring-boot-starter</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,gradle]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
=== Enable Cohere Chat Support
|
||||
|
||||
By default the Cohere model is disabled.
|
||||
To enable it set the `spring.ai.bedrock.cohere.chat.enabled` property to `true`.
|
||||
Exporting environment variable is one way to set this configuration property:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
export SPRING_AI_BEDROCK_COHERE_CHAT_ENABLED=true
|
||||
----
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
The prefix `spring.ai.bedrock.aws` is the property prefix to configure the connection to AWS Bedrock.
|
||||
|
||||
[cols="3,3,3"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1
|
||||
| spring.ai.bedrock.aws.access-key | AWS access key. | -
|
||||
| spring.ai.bedrock.aws.secret-key | AWS secret key. | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.bedrock.cohere.chat` is the property prefix that configures the chat client implementation for Cohere.
|
||||
|
||||
[cols="2,5,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.cohere.chat.enabled | Enable or disable support for Cohere | false
|
||||
| spring.ai.bedrock.cohere.chat.model | The model id to use. See the https://github.com/spring-projects/spring-ai/blob/4ba9a3cd689b9fd3a3805f540debe398a079c6ef/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/api/CohereChatBedrockApi.java#L326C14-L326C29[CohereChatModel] for the supported models. | cohere.command-text-v14
|
||||
| spring.ai.bedrock.cohere.chat.options.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7
|
||||
| spring.ai.bedrock.cohere.chat.options.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.topK | Specify the number of token choices the model uses to generate the next token | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.maxTokens | Specify the maximum number of tokens to use in the generated response. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.stopSequences | Configure up to four sequences that the model recognizes. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.returnLikelihoods | The token likelihoods are returned with the response. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.numGenerations | The maximum number of generations that the model should return. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.logitBias | Prevents the model from generating unwanted tokens or incentivize the model to include desired tokens. | AWS Bedrock default
|
||||
| spring.ai.bedrock.cohere.chat.options.truncate | Specifies how the API handles inputs longer than the maximum token length | AWS Bedrock default
|
||||
|====
|
||||
|
||||
Look at the https://github.com/spring-projects/spring-ai/blob/4ba9a3cd689b9fd3a3805f540debe398a079c6ef/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/api/CohereChatBedrockApi.java#L326C14-L326C29[CohereChatModel] for other model IDs.
|
||||
Supported values are: `cohere.command-light-text-v14` and `cohere.command-text-v14`.
|
||||
Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs].
|
||||
|
||||
TIP: All properties prefixed with `spring.ai.bedrock.cohere.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/BedrockCohereChatOptions.java[BedrockCohereChatOptions.java] provides model configurations, such as temperature, topK, topP, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `BedrockCohereChatClient(api, options)` constructor or the `spring.ai.bedrock.cohere.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
BedrockCohereChatOptions.builder()
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/BedrockCohereChatOptions.java[BedrockCohereChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-bedrock-ai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the Anthropic Chat client:
|
||||
|
||||
[source]
|
||||
----
|
||||
spring.ai.bedrock.aws.region=eu-central-1
|
||||
spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID}
|
||||
spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
|
||||
|
||||
spring.ai.bedrock.cohere.chat.enabled=true
|
||||
spring.ai.bedrock.cohere.chat.options.temperature=0.8
|
||||
----
|
||||
|
||||
TIP: replace the `regions`, `access-key` and `secret-key` with your AWS credentials.
|
||||
|
||||
This will create a `BedrockCohereChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final BedrockCohereChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(BedrockCohereChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.call(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/BedrockCohereChatClient.java[BedrockCohereChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Bedrock Anthropic service.
|
||||
|
||||
Add the `spring-ai-bedrock` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bedrock</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,gradle]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-bedrock:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
Next, create an https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/BedrockCohereChatClient.java[BedrockCohereChatClient] and use it for text generations:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
CohereChatBedrockApi api = new CohereChatBedrockApi(CohereChatModel.COHERE_COMMAND_V14.id(),
|
||||
EnvironmentVariableCredentialsProvider.create(), Region.US_EAST_1.id(), new ObjectMapper());
|
||||
|
||||
BedrockCohereChatClient chatClient = new BedrockCohereChatClient(api,
|
||||
BedrockCohereChatOptions.builder()
|
||||
.withTemperature(0.6f)
|
||||
.withTopK(10)
|
||||
.withTopP(0.5f)
|
||||
.withMaxTokens(678)
|
||||
.build()
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
|
||||
// Or with streaming responses
|
||||
Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
== Low-level CohereChatBedrockApi Client [[low-level-api]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/cohere/api/CohereChatBedrockApi.java[CohereChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-cohere-command.html[Cohere Command models].
|
||||
|
||||
Following class diagram illustrates the CohereChatBedrockApi interface and building blocks:
|
||||
|
||||
image::bedrock/bedrock-cohere-chat-api.jpg[CohereChatBedrockApi Class Diagram]
|
||||
|
||||
The CohereChatBedrockApi supports the `cohere.command-light-text-v14` and `cohere.command-text-v14` models for both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) requests.
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
CohereChatBedrockApi cohereChatApi = new CohereChatBedrockApi(
|
||||
CohereChatModel.COHERE_COMMAND_V14.id(),
|
||||
Region.US_EAST_1.id());
|
||||
|
||||
var request = CohereChatRequest
|
||||
.builder("What is the capital of Bulgaria and what is the size? What it the national anthem?")
|
||||
.withStream(false)
|
||||
.withTemperature(0.5f)
|
||||
.withTopP(0.8f)
|
||||
.withTopK(15)
|
||||
.withMaxTokens(100)
|
||||
.withStopSequences(List.of("END"))
|
||||
.withReturnLikelihoods(CohereChatRequest.ReturnLikelihoods.ALL)
|
||||
.withNumGenerations(3)
|
||||
.withLogitBias(null)
|
||||
.withTruncate(Truncate.NONE)
|
||||
.build();
|
||||
|
||||
CohereChatResponse response = cohereChatApi.chatCompletion(request);
|
||||
|
||||
var request = CohereChatRequest
|
||||
.builder("What is the capital of Bulgaria and what is the size? What it the national anthem?")
|
||||
.withStream(true)
|
||||
.withTemperature(0.5f)
|
||||
.withTopP(0.8f)
|
||||
.withTopK(15)
|
||||
.withMaxTokens(100)
|
||||
.withStopSequences(List.of("END"))
|
||||
.withReturnLikelihoods(CohereChatRequest.ReturnLikelihoods.ALL)
|
||||
.withNumGenerations(3)
|
||||
.withLogitBias(null)
|
||||
.withTruncate(Truncate.NONE)
|
||||
.build();
|
||||
|
||||
Flux<CohereChatResponse.Generation> responseStream = cohereChatApi.chatCompletionStream(request);
|
||||
List<CohereChatResponse.Generation> responses = responseStream.collectList().block();
|
||||
----
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ Refer to the xref:api/clients/bedrock.adoc[Spring AI documentation on Amazon Bed
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Spring Boot starter:
|
||||
Add the `spring-ai-bedrock-ai-spring-boot-starter` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
@@ -37,8 +37,9 @@ dependencies {
|
||||
|
||||
=== Enable Llama2 Chat Support
|
||||
|
||||
Spring AI defines a configuration property named `spring.ai.bedrock.llama2.chat.enabled` that you should set to `true` to enable support for Llama2.
|
||||
Exporting environment variables in one way to set this configuration property.
|
||||
By default the Bedrock Llama2 model is disabled.
|
||||
To enable it set the `spring.ai.bedrock.llama2.chat.enabled` property to `true`.
|
||||
Exporting environment variable is one way to set this configuration property:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
@@ -59,7 +60,7 @@ The prefix `spring.ai.bedrock.aws` is the property prefix to configure the conne
|
||||
|====
|
||||
|
||||
|
||||
The prefix `spring.ai.bedrock.llama2.chat` is the property prefix that configures the `ChatClient` implementation for Llama2.
|
||||
The prefix `spring.ai.bedrock.llama2.chat` is the property prefix that configures the chat client implementation for Llama2.
|
||||
|
||||
[cols="2,5,1"]
|
||||
|====
|
||||
@@ -72,45 +73,83 @@ The prefix `spring.ai.bedrock.llama2.chat` is the property prefix that configure
|
||||
| spring.ai.bedrock.llama2.chat.options.max-gen-len | Specify the maximum number of tokens to use in the generated response. The model truncates the response once the generated text exceeds maxGenLen. | 300
|
||||
|====
|
||||
|
||||
Look at the Spring AI enumeration, `Llama2ChatModel` for other model IDs. The other value supported is `meta.llama2-13b-chat-v1`.
|
||||
|
||||
Look at https://github.com/spring-projects/spring-ai/blob/4ba9a3cd689b9fd3a3805f540debe398a079c6ef/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java#L164[Llama2ChatBedrockApi#Llama2ChatModel] for other model IDs. The other value supported is `meta.llama2-13b-chat-v1`.
|
||||
Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs].
|
||||
|
||||
=== Sample Code
|
||||
TIP: All properties prefixed with `spring.ai.bedrock.llama2.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
Create an `application.properties` file in the `src/main/resources` directory and add the following properties to configure the Llama2 Chat client.
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatOptions.java[BedrockLlama2ChatOptions.java] provides model configurations, such as temperature, topK, topP, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `BedrockLlama2ChatClient(api, options)` constructor or the `spring.ai.bedrock.llama2.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
BedrockLlama2ChatOptions.builder()
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatOptions.java[BedrockLlama2ChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-bedrock-ai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the Anthropic Chat client:
|
||||
|
||||
[source]
|
||||
----
|
||||
spring.ai.bedrock.aws.region=eu-central-1
|
||||
spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID}
|
||||
spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
|
||||
|
||||
spring.ai.bedrock.llama2.chat.enabled=true
|
||||
spring.ai.bedrock.llama2.chat.options.temperature=0.8
|
||||
----
|
||||
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
TIP: replace the `regions`, `access-key` and `secret-key` with your AWS credentials.
|
||||
|
||||
This will create a `BedrockLlama2ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final BedrockLlama2ChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
public ChatController(BedrockLlama2ChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
return Map.of("generation", chatClient.call(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java[BedrockLlama2ChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Bedrock Anthropic service.
|
||||
|
||||
Add the `spring-ai-bedrock` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
@@ -131,11 +170,9 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
The link:./src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java[BedrockLlama2ChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the `Llama2ChatBedrockApi` library to connect to the Bedrock Llama2 service.
|
||||
|
||||
Here is how to create and use a `BedrockLlama2ChatClient`:
|
||||
Next, create an https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/BedrockLlama2ChatClient.java[BedrockLlama2ChatClient] and use it for text generations:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@@ -143,10 +180,10 @@ Llama2ChatBedrockApi api = new Llama2ChatBedrockApi(Llama2ChatModel.LLAMA2_70B_C
|
||||
EnvironmentVariableCredentialsProvider.create(), Region.US_EAST_1.id(), new ObjectMapper());
|
||||
|
||||
BedrockLlama2ChatClient chatClient = new BedrockLlama2ChatClient(api,
|
||||
BedrockLlama2ChatOptions.builder()
|
||||
.withTemperature(0.5f)
|
||||
.withMaxGenLen(100)
|
||||
.withTopP(0.9f).build());
|
||||
BedrockLlama2ChatOptions.builder()
|
||||
.withTemperature(0.5f)
|
||||
.withMaxGenLen(100)
|
||||
.withTopP(0.9f).build());
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
@@ -156,28 +193,23 @@ Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
== Low-level Llama2ChatBedrockApi Client [[low-level-api]]
|
||||
|
||||
== Appendices
|
||||
|
||||
=== Using low-level Llama2ChatBedrockApi Library
|
||||
|
||||
link:./src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java[Llama2ChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-meta.html[Meta Llama 2 and Llama 2 Chat models].
|
||||
https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java[Llama2ChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-meta.html[Meta Llama 2 and Llama 2 Chat models].
|
||||
|
||||
Following class diagram illustrates the Llama2ChatBedrockApi interface and building blocks:
|
||||
|
||||
image::bedrock/bedrock-llama2-chat-api.jpg[Llama2ChatBedrockApi Class Diagram]
|
||||
|
||||
The Llama2ChatBedrockApi supports the `meta.llama2-13b-chat-v1` and `meta.llama2-70b-chat-v1` models.
|
||||
|
||||
Also the Llama2ChatBedrockApi supports both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
|
||||
The Llama2ChatBedrockApi supports the `meta.llama2-13b-chat-v1` and `meta.llama2-70b-chat-v1` models for both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
Llama2ChatBedrockApi llama2ChatApi = new Llama2ChatBedrockApi(
|
||||
Llama2ChatModel.LLAMA2_70B_CHAT_V1.id(),
|
||||
Region.US_EAST_1.id());
|
||||
Llama2ChatModel.LLAMA2_70B_CHAT_V1.id(),
|
||||
Region.US_EAST_1.id());
|
||||
|
||||
Llama2ChatRequest request = Llama2ChatRequest.builder("Hello, my name is")
|
||||
.withTemperature(0.9f)
|
||||
@@ -187,16 +219,11 @@ Llama2ChatRequest request = Llama2ChatRequest.builder("Hello, my name is")
|
||||
|
||||
Llama2ChatResponse response = llama2ChatApi.chatCompletion(request);
|
||||
|
||||
System.out.println(response.generation());
|
||||
|
||||
// Streaming response
|
||||
Flux<Llama2ChatResponse> responseStream = llama2ChatApi.chatCompletionStream(request);
|
||||
|
||||
List<Llama2ChatResponse> responses = responseStream.collectList().block();
|
||||
|
||||
System.out.println(responses);
|
||||
----
|
||||
|
||||
Follow the link:./src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java[Llama2ChatBedrockApi.java]'s JavaDoc for further information.
|
||||
Follow the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java[Llama2ChatBedrockApi.java]'s JavaDoc for further information.
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
= HuggingFace
|
||||
= HuggingFace Chat
|
||||
|
||||
HuggingFace Inference Endpoints allow you to deploy and serve machine learning models in the cloud, making them accessible via an API.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Spring AI supports the Ollama text generation with `OllamaChatClient`.
|
||||
You first need to run Ollama on your local machine.
|
||||
Refer to the official Ollama project link:https://github.com/jmorganca/ollama[README] to get started running models on your local machine.
|
||||
|
||||
Note, installing `ollama run llama2` will download a 4GB docker image.
|
||||
NOTE: installing `ollama run llama2` will download a 4GB docker image.
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
@@ -33,7 +33,7 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
@@ -46,9 +46,9 @@ The prefix `spring.ai.ollama` is the property prefix to configure the connection
|
||||
| spring.ai.ollama.base-url | Base URL where Ollama API server is running. | `http://localhost:11434`
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.ollama.chat.options` is the property prefix that configures the `ChatClient` implementation for Ollama.
|
||||
The prefix `spring.ai.ollama.chat.options` is the property prefix that configures the chat client implementation for Ollama.
|
||||
|
||||
NOTE: The listed properties are based on the https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values[Ollama Valid Parameters and Values] and https://github.com/jmorganca/ollama/blob/main/api/types.go[Ollama Types]. And the default values are based on: https://github.com/ollama/ollama/blob/b538dc3858014f94b099730a592751a5454cab0a/api/types.go#L364[Ollama type defaults].
|
||||
NOTE: The `options` properties are based on the link:https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values[Ollama Valid Parameters and Values] and link:https://github.com/jmorganca/ollama/blob/main/api/types.go[Ollama Types]. The default values are based on: link:https://github.com/ollama/ollama/blob/b538dc3858014f94b099730a592751a5454cab0a/api/types.go#L364[Ollama type defaults].
|
||||
|
||||
[cols="3,6,1"]
|
||||
|====
|
||||
@@ -94,34 +94,80 @@ NOTE: The listed properties are based on the https://github.com/jmorganca/ollama
|
||||
|
||||
NOTE: The list of options for chat is to be reviewed. This https://github.com/spring-projects/spring-ai/issues/230[issue] will track progress.
|
||||
|
||||
=== Sample Code
|
||||
TIP: All properties prefixed with `spring.ai.ollama.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides model configurations, such as the model to use, the temperature, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `OllamaChatClient(api, options)` constructor or the `spring.ai.ollama.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default model and temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OllamaOptions.create()
|
||||
.withModel("llama2")
|
||||
.withTemperature(0.4)
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the OpenAi Chat client:
|
||||
|
||||
[source,application.properties]
|
||||
----
|
||||
spring.ai.ollama.base-url=http://localhost:11434
|
||||
spring.ai.ollama.chat.model=mistral
|
||||
spring.ai.ollama.chat.options.temperature=0.7
|
||||
----
|
||||
|
||||
TIP: replace the `base-url` with your Ollama server URL.
|
||||
|
||||
This will create a `OllamaChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final OllamaChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
public ChatController(OllamaChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
return Map.of("generation", chatClient.call(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
If you don't want to use the Spring Boot auto-configuration, you can manually configure the `OllamaChatClient` in your application.
|
||||
For this add the spring-ai-ollama dependency to your project’s Maven pom.xml file:
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/OllamaChatClient.java[OllamaChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Ollama service.
|
||||
|
||||
To use it add the `spring-ai-ollama` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
@@ -141,7 +187,7 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-ollama` dependency provides access also to the `OllamaEmbeddingClient`.
|
||||
TIP: The `spring-ai-ollama` dependency provides access also to the `OllamaEmbeddingClient`.
|
||||
For more information about the `OllamaEmbeddingClient` refer to the link:../embeddings/ollama-embeddings.html[Ollama Embedding Client] section.
|
||||
|
||||
Next, create an `OllamaChatClient` instance and use it to text generations requests:
|
||||
@@ -165,27 +211,45 @@ Flux<ChatResponse> response = chatClient.stream(
|
||||
|
||||
The `OllamaOptions` provides the configuration information for all chat requests.
|
||||
|
||||
=== Chat Options
|
||||
=== Low-level OpenAiApi Client [[low-level-api]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides provides configuration information for the chat requests, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
The link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaApi.java[OllamaApi] provides is lightweight Java client for Ollama Chat API link:https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion[Ollama Chat Completion API].
|
||||
|
||||
The default options can be configured using the `spring.ai.ollama.chat.options` properties as well.
|
||||
Following class diagram illustrates the `OllamaApi` chat interfaces and building blocks:
|
||||
|
||||
On start-time use the `OllamaChatClient#withDefaultOptions()` to set the default options applicable for all chat completion requests.
|
||||
At run-time you can override the default options with `OllamaOptions` instance in the request `Prompt`.
|
||||
image::ollama-chat-completion-api.png[OllamaApi Chat Completion API Diagram]
|
||||
|
||||
For example to override the default model name and temperature for a specific request:
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OllamaOptions.create()
|
||||
.withModel("llama2")
|
||||
.withTemperature(0.4)
|
||||
));
|
||||
----
|
||||
OllamaApi ollamaApi =
|
||||
new OllamaApi("YOUR_HOST:YOUR_PORT");
|
||||
|
||||
You can use as prompt options any instance that implements the portable `ChatOptions` interface.
|
||||
For example you can use the `ChatOptionsBuilder` to create a portable prompt options.
|
||||
// Sync request
|
||||
var request = ChatRequest.builder("orca-mini")
|
||||
.withStream(false) // not streaming
|
||||
.withMessages(List.of(
|
||||
Message.builder(Role.SYSTEM)
|
||||
.withContent("You are geography teacher. You are talking to a student.")
|
||||
.build(),
|
||||
Message.builder(Role.USER)
|
||||
.withContent("What is the capital of Bulgaria and what is the size? "
|
||||
+ "What it the national anthem?")
|
||||
.build()))
|
||||
.withOptions(OllamaOptions.create().withTemperature(0.9f))
|
||||
.build();
|
||||
|
||||
ChatResponse response = ollamaApi.chat(request);
|
||||
|
||||
// Streaming request
|
||||
var request2 = ChatRequest.builder("orca-mini")
|
||||
.withStream(true) // streaming
|
||||
.withMessages(List.of(Message.builder(Role.USER)
|
||||
.withContent("What is the capital of Bulgaria and what is the size? " + "What it the national anthem?")
|
||||
.build()))
|
||||
.withOptions(OllamaOptions.create().withTemperature(0.9f).toMap())
|
||||
.build();
|
||||
|
||||
Flux<ChatResponse> streamingResponse = ollamaApi.streamingChat(request2);
|
||||
----
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Spring AI supports ChatGPT, the AI language model by OpenAI. ChatGPT has been instrumental in sparking interest in AI-driven text generation, thanks to its creation of industry-leading text generation models and embeddings.
|
||||
|
||||
== Pre-requisites
|
||||
== Prerequisites
|
||||
|
||||
You will need to create an API with OpenAI to access ChatGPT models.
|
||||
Create an account at https://platform.openai.com/signup[OpenAI signup page] and generate the token on the https://platform.openai.com/account/api-keys[API Keys page].
|
||||
@@ -37,7 +37,7 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
@@ -51,7 +51,7 @@ The prefix `spring.ai.openai` is used as the property prefix that lets you conne
|
||||
| spring.ai.openai.api-key | The API Key | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.openai.chat` is the property prefix that lets you configure the `ChatClient` implementation for OpenAI.
|
||||
The prefix `spring.ai.openai.chat` is the property prefix that lets you configure the chat client implementation for OpenAI.
|
||||
|
||||
[cols="3,5,1"]
|
||||
|====
|
||||
@@ -59,7 +59,7 @@ The prefix `spring.ai.openai.chat` is the property prefix that lets you configur
|
||||
|
||||
| spring.ai.openai.chat.base-url | Optional overrides the spring.ai.openai.base-url to provide chat specific url | -
|
||||
| spring.ai.openai.chat.api-key | Optional overrides the spring.ai.openai.api-key to provide chat specific api-key | -
|
||||
| spring.ai.openai.chat.options.model | This is the OpenAI Chat model to use | `gpt-35-turbo` (the `gpt-3.5-turbo`, `gpt-4`, and `gpt-4-32k` point to the latest model versions)
|
||||
| spring.ai.openai.chat.options.model | This is the OpenAI Chat model to use | `gpt-3.5-turbo` (the `gpt-3.5-turbo`, `gpt-4`, and `gpt-4-32k` point to the latest model versions)
|
||||
| spring.ai.openai.chat.options.temperature | The sampling temperature to use that controls the apparent creativity of generated completions. Higher values will make output more random while lower values will make results more focused and deterministic. It is not recommended to modify temperature and top_p for the same completions request as the interaction of these two settings is difficult to predict. | 0.8
|
||||
| spring.ai.openai.chat.options.frequencyPenalty | Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim. | 0.0f
|
||||
| spring.ai.openai.chat.options.logitBias | Modify the likelihood of specified tokens appearing in the completion. | -
|
||||
@@ -79,30 +79,59 @@ NOTE: You can override the common `spring.ai.openai.base-url` and `spring.ai.ope
|
||||
The `spring.ai.openai.chat.base-url` and `spring.ai.openai.chat.api-key` properties if set take precedence over the common properties.
|
||||
This is useful if you want to use different OpenAI accounts for different models and different model endpoints.
|
||||
|
||||
=== Sample Code
|
||||
TIP: All properties prefixed with `spring.ai.openai.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiChatOptions.java[OpenAiChatOptions.java] provides model configurations, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `OpenAiChatClient(api, options)` constructor or the `spring.ai.openai.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default model and temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OpenAiChatOptions.builder()
|
||||
.withModel("gpt-4-32k")
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiChatOptions.java[OpenAiChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-openai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the OpenAi Chat client:
|
||||
|
||||
[source,application.properties]
|
||||
----
|
||||
spring.ai.openai.api-key=YOUR_API_KEY
|
||||
spring.ai.openai.chat.options.model=gpt-35-turbo
|
||||
spring.ai.openai.chat.options.model=gpt-3.5-turbo
|
||||
spring.ai.openai.chat.options.temperature=0.7
|
||||
----
|
||||
|
||||
TIP: replace the `api-key` with your OpenAI credentials.
|
||||
|
||||
This will create a `OpenAiChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
private final StreamingChatClient streamingChatClient;
|
||||
private final OpenAiChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient, StreamingChatClient streamingChatClient) {
|
||||
public ChatController(OpenAiChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
this.streamingChatClient = streamingChatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generate")
|
||||
@@ -113,15 +142,17 @@ public class ChatController {
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return streamingChatClient.stream(prompt);
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
If you are not using Spring Boot, you can manually configure the `OpenAiChatClient` by creating the beans in your configuration class.
|
||||
For this add the `spring-ai-openai` dependency to your project's Maven `pom.xml` file:
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiChatClient.java[OpenAiChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the OpenAI service.
|
||||
|
||||
Add the `spring-ai-openai` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
@@ -140,9 +171,9 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-openai` dependency provides access also to the `OpenAiEmbeddingClient`. For more information about the `OpenAiEmbeddingClient` refer to the link:../embeddings/openai-embeddings.html[OpenAI Embeddings Client] section.
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
Next, create an `OpenAiChatClient` instance and use it to text generations requests:
|
||||
Next, create a `OpenAiChatClient` and use it for text generations:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@@ -166,28 +197,35 @@ Flux<ChatResponse> response = chatClient.stream(
|
||||
The `OpenAiChatOptions` provides the configuration information for the chat requests.
|
||||
The `OpenAiChatOptions.Builder` is fluent options builder.
|
||||
|
||||
=== Chat Options
|
||||
=== Low-level OpenAiApi Client [[low-level-api]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiChatOptions.java[OpenAiChatOptions.java] provides provides the configuration information for the chat requests, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/api/OpenAiApi.java[OpenAiApi] provides is lightweight Java client for OpenAI Chat API link:https://platform.openai.com/docs/api-reference/chat[OpenAI Chat API].
|
||||
|
||||
The default options can be configured using the `spring.ai.openai.chat.options` properties as well.
|
||||
On start-time use the `OpenAiChatClient#withDefaultOptions()` to set the default options applicable for all chat completion requests.
|
||||
At run-time you can override the default options with `OpenAiChatOptions` instance in the request `Prompt`.
|
||||
Following class diagram illustrates the `OpenAiApi` chat interfaces and building blocks:
|
||||
|
||||
For example to override the default model name and temperature for a specific request:
|
||||
image::openai-chat-api.png[OpenAiApi Chat API Diagram]
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
AzureOpenAiChatOptions.builder()
|
||||
.withModel("gpt-4-32k")
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
OpenAiApi openAiApi =
|
||||
new OpenAiApi(System.getenv("OPENAI_API_KEY"));
|
||||
|
||||
ChatCompletionMessage chatCompletionMessage =
|
||||
new ChatCompletionMessage("Hello world", Role.USER);
|
||||
|
||||
// Sync request
|
||||
ResponseEntity<ChatCompletion> response = openAiApi.chatCompletionEntity(
|
||||
new ChatCompletionRequest(List.of(chatCompletionMessage), "gpt-3.5-turbo", 0.8f, false));
|
||||
|
||||
// Streaming request
|
||||
Flux<ChatCompletionChunk> streamResponse = openAiApi.chatCompletionStream(
|
||||
new ChatCompletionRequest(List.of(chatCompletionMessage), "gpt-3.5-turbo", 0.8f, true));
|
||||
----
|
||||
|
||||
You can use as prompt options any instance that implements the portable `ChatOptions` interface.
|
||||
For example you can use the `ChatOptionsBuilder` to create a portable prompt options.
|
||||
Check the link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/test/java/org/springframework/ai/openai/chat/api/OpenAiApiIT.java[OpenAiApiIT.java] integration test for more examples.
|
||||
|
||||
Follow the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-openai/src/main/java/org/springframework/ai/openai/api/OpenAiApi.java[OpenAiApi.java]'s JavaDoc for further information.
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user