Spring/spring-ai
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add Bedrock Titan Chat Options + Docs

Browse Source
This commit is contained in:
Christian Tzolov
2024-02-11 15:32:33 +01:00
parent 4839a6175c
commit ed52a3e0a1
12 changed files with 522 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -6,6 +6,8 @@ The Spring AI project provides a Spring-friendly API and abstractions for develo
Let's make your `@Beans` intelligent!
For further information go to our [Spring AI documentation](https://docs.spring.io/spring-ai/reference/).
## Project Update
:partying_face: The Spring AI project has graduated out of the repository!

84
models/spring-ai-bedrock/README_TITAN_CHAT.md
View File

@@ -1,83 +1,3 @@
# 1. Bedrock Titan Chat
# Bedrock Titan Chat
## 1.1 TitanChatBedrockApi
[TitanChatBedrockApi](./src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java) provides is lightweight Java client on top of AWS Bedrock [Titan text models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-text.html).
Following class diagram illustrates the Llama2ChatBedrockApi interface and building blocks:
![TitanChatBedrockApi Class Diagram](./src/test/resources/doc/Bedrock%20Titan%20Chat%20API.jpg)
The TitanChatBedrockApi supports the `amazon.titan-text-lite-v1` and `amazon.titan-text-express-v1` models for bot synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
Here is a simple snippet how to use the api programmatically:
```java
TitanChatBedrockApi titanBedrockApi = new TitanChatBedrockApi(TitanChatCompletionModel.TITAN_TEXT_EXPRESS_V1.id(),
Region.EU_CENTRAL_1.id());
TitanChatRequest titanChatRequest = TitanChatRequest.builder("Give me the names of 3 famous pirates?")
.withTemperature(0.5f)
.withTopP(0.9f)
.withMaxTokenCount(100)
.withStopSequences(List.of("|"))
.build();
TitanChatResponse response = titanBedrockApi.chatCompletion(titanChatRequest);
assertThat(response.results()).hasSize(1);
assertThat(response.results().get(0).outputText()).contains("Blackbeard");
Flux<TitanChatResponseChunk> response = titanBedrockApi.chatCompletionStream(titanChatRequest);
List<TitanChatResponseChunk> results = response.collectList().block();
assertThat(results.stream().map(TitanChatResponseChunk::outputText).collect(Collectors.joining("\n")))
.contains("Blackbeard");
```
## 1.2 BedrockTitanChatClient
[BedrockTitanChatClient](./src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java) implements the Spring-Ai `ChatClient` and `StreamingChatClient` on top of the `TitanChatBedrockApi`.
You can use like this:
```java
@Bean
public TitanChatBedrockApi titanApi() {
return new TitanChatBedrockApi(TitanChatModel.TITAN_TEXT_EXPRESS_V1.id(),
EnvironmentVariableCredentialsProvider.create(), Region.US_EAST_1.id(), new ObjectMapper());
}
@Bean
public BedrockTitanChatClient titanChatClient(TitanChatBedrockApi titanApi) {
return new BedrockTitanChatClient(titanApi);
}
```
or you can leverage the `spring-ai-bedrock-ai-spring-boot-starter` Boot starter. For this add the following dependency:
```xml
<dependency>
<artifactId>spring-ai-bedrock-ai-spring-boot-starter</artifactId>
<groupId>org.springframework.ai</groupId>
<version>0.8.0-SNAPSHOT</version>
</dependency>
```
**NOTE:** You have to enable the Bedrock Titan chat client with `spring.ai.bedrock.titan.chat.enabled=true`.
By default the client is disabled.
Use the `BedrockTitanChatProperties` to configure the Bedrock Titan Chat client:
| Property | Description | Default |
| ------------- | ------------- | ------------- |
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 |
| spring.ai.bedrock.aws.accessKey | AWS credentials access key. | |
| spring.ai.bedrock.aws.secretKey | AWS credentials secret key. | |
| spring.ai.bedrock.titan.chat.enable | Enable Bedrock Titan chat client. Disabled by default | false |
| spring.ai.bedrock.titan.chat.model | The model id to use. See the `TitanChatModel` for the supported models. | amazon.titan-text-express-v1 |
| spring.ai.bedrock.titan.chat.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7 |
| spring.ai.bedrock.titan.chat.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default |
| spring.ai.bedrock.titan.chat.maxTokenCount | Specify the maximum number of tokens to use in the generated response. | AWS Bedrock default |
| spring.ai.bedrock.titan.chat.stopSequences | Configure up to four sequences that the model recognizes. | AWS Bedrock default |
Visit the Spring AI [Bedrock Titan Chat Documentation](https://docs.spring.io/spring-ai/reference/api/clients/bedrock/bedrock-titan.html).

97
models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2023-2023 the original author or authors.
* Copyright 2023-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -18,8 +18,6 @@ package org.springframework.ai.bedrock.titan;
import java.util.List;
import org.springframework.ai.chat.ChatClient;
import org.springframework.ai.chat.metadata.ChatGenerationMetadata;
import reactor.core.publisher.Flux;
import org.springframework.ai.bedrock.MessageToPromptConverter;
@@ -27,11 +25,16 @@ import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi.TitanChatRequest;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi.TitanChatResponse;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi.TitanChatResponseChunk;
import org.springframework.ai.chat.ChatClient;
import org.springframework.ai.chat.ChatOptions;
import org.springframework.ai.chat.ChatResponse;
import org.springframework.ai.chat.StreamingChatClient;
import org.springframework.ai.chat.Generation;
import org.springframework.ai.chat.StreamingChatClient;
import org.springframework.ai.chat.metadata.ChatGenerationMetadata;
import org.springframework.ai.chat.metadata.Usage;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.model.ModelOptionsUtils;
import org.springframework.util.Assert;
/**
* @author Christian Tzolov
@@ -41,41 +44,22 @@ public class BedrockTitanChatClient implements ChatClient, StreamingChatClient {
private final TitanChatBedrockApi chatApi;
private Float temperature;
private Float topP;
private Integer maxTokenCount;
private List<String> stopSequences;
private final BedrockTitanChatOptions defaultOptions;
public BedrockTitanChatClient(TitanChatBedrockApi chatApi) {
this(chatApi, BedrockTitanChatOptions.builder().withTemperature(0.8f).build());
}
public BedrockTitanChatClient(TitanChatBedrockApi chatApi, BedrockTitanChatOptions defaultOptions) {
Assert.notNull(chatApi, "ChatApi must not be null");
Assert.notNull(defaultOptions, "DefaultOptions must not be null");
this.chatApi = chatApi;
}
public BedrockTitanChatClient withTemperature(Float temperature) {
this.temperature = temperature;
return this;
}
public BedrockTitanChatClient withTopP(Float topP) {
this.topP = topP;
return this;
}
public BedrockTitanChatClient withMaxTokenCount(Integer maxTokens) {
this.maxTokenCount = maxTokens;
return this;
}
public BedrockTitanChatClient withStopSequences(List<String> stopSequences) {
this.stopSequences = stopSequences;
return this;
this.defaultOptions = defaultOptions;
}
@Override
public ChatResponse call(Prompt prompt) {
TitanChatResponse response = this.chatApi.chatCompletion(this.createRequest(prompt, false));
TitanChatResponse response = this.chatApi.chatCompletion(this.createRequest(prompt));
List<Generation> generations = response.results().stream().map(result -> {
return new Generation(result.outputText());
}).toList();
@@ -85,7 +69,7 @@ public class BedrockTitanChatClient implements ChatClient, StreamingChatClient {
@Override
public Flux<ChatResponse> stream(Prompt prompt) {
return this.chatApi.chatCompletionStream(this.createRequest(prompt, true)).map(chunk -> {
return this.chatApi.chatCompletionStream(this.createRequest(prompt)).map(chunk -> {
Generation generation = new Generation(chunk.outputText());
@@ -104,15 +88,48 @@ public class BedrockTitanChatClient implements ChatClient, StreamingChatClient {
});
}
private TitanChatRequest createRequest(Prompt prompt, boolean stream) {
/**
* Test access.
*/
TitanChatRequest createRequest(Prompt prompt) {
final String promptValue = MessageToPromptConverter.create().toPrompt(prompt.getInstructions());
return TitanChatRequest.builder(promptValue)
.withTemperature(this.temperature)
.withTopP(this.topP)
.withMaxTokenCount(this.maxTokenCount)
.withStopSequences(this.stopSequences)
.build();
var requestBuilder = TitanChatRequest.builder(promptValue);
if (this.defaultOptions != null) {
requestBuilder = update(requestBuilder, this.defaultOptions);
}
if (prompt.getOptions() != null) {
if (prompt.getOptions() instanceof ChatOptions runtimeOptions) {
BedrockTitanChatOptions updatedRuntimeOptions = ModelOptionsUtils.copyToTarget(runtimeOptions,
ChatOptions.class, BedrockTitanChatOptions.class);
requestBuilder = update(requestBuilder, updatedRuntimeOptions);
}
else {
throw new IllegalArgumentException("Prompt options are not of type ChatOptions: "
+ prompt.getOptions().getClass().getSimpleName());
}
}
return requestBuilder.build();
}
private TitanChatRequest.Builder update(TitanChatRequest.Builder builder, BedrockTitanChatOptions options) {
if (options.getTemperature() != null) {
builder.withTemperature(options.getTemperature());
}
if (options.getTopP() != null) {
builder.withTopP(options.getTopP());
}
if (options.getMaxTokenCount() != null) {
builder.withMaxTokenCount(options.getMaxTokenCount());
}
if (options.getStopSequences() != null) {
builder.withStopSequences(options.getStopSequences());
}
return builder;
}
private Usage extractUsage(TitanChatResponseChunk response) {

133
models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatOptions.java Normal file
View File

@@ -0,0 +1,133 @@
/*
* Copyright 2024-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.ai.bedrock.titan;
import java.util.List;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonInclude.Include;
import org.springframework.ai.chat.ChatOptions;
import com.fasterxml.jackson.annotation.JsonProperty;
/**
* @author Christian Tzolov
* @since 0.8.0
*/
@JsonInclude(Include.NON_NULL)
public class BedrockTitanChatOptions implements ChatOptions {
// @formatter:off
/**
* The temperature value controls the randomness of the generated text.
*/
private @JsonProperty("temperature") Float temperature;
/**
* The topP value controls the diversity of the generated text. Use a lower value to ignore less probable options.
*/
private @JsonProperty("topP") Float topP;
/**
* Maximum number of tokens to generate.
*/
private @JsonProperty("maxTokenCount") Integer maxTokenCount;
/**
* A list of tokens that the model should stop generating after.
*/
private @JsonProperty("stopSequences") List<String> stopSequences;
// @formatter:on
public static Builder builder() {
return new Builder();
}
public static class Builder {
private BedrockTitanChatOptions options = new BedrockTitanChatOptions();
public Builder withTemperature(Float temperature) {
this.options.temperature = temperature;
return this;
}
public Builder withTopP(Float topP) {
this.options.topP = topP;
return this;
}
public Builder withMaxTokenCount(Integer maxTokenCount) {
this.options.maxTokenCount = maxTokenCount;
return this;
}
public Builder withStopSequences(List<String> stopSequences) {
this.options.stopSequences = stopSequences;
return this;
}
public BedrockTitanChatOptions build() {
return this.options;
}
}
public Float getTemperature() {
return temperature;
}
public void setTemperature(Float temperature) {
this.temperature = temperature;
}
public Float getTopP() {
return topP;
}
public void setTopP(Float topP) {
this.topP = topP;
}
public Integer getMaxTokenCount() {
return maxTokenCount;
}
public void setMaxTokenCount(Integer maxTokenCount) {
this.maxTokenCount = maxTokenCount;
}
public List<String> getStopSequences() {
return stopSequences;
}
public void setStopSequences(List<String> stopSequences) {
this.stopSequences = stopSequences;
}
@Override
public Integer getTopK() {
throw new UnsupportedOperationException("Bedrock Titian Chat does not support the 'TopK' option.");
}
@Override
public void setTopK(Integer topK) {
throw new UnsupportedOperationException("Bedrock Titian Chat does not support the 'TopK' option.'");
}
}

76
models/spring-ai-bedrock/src/test/java/org/springframework/ai/bedrock/titan/BedrockTitanChatCreateRequestTests.java Normal file
View File

@@ -0,0 +1,76 @@
/*
* Copyright 2024-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.ai.bedrock.titan;
import java.util.List;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi.TitanChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import static org.assertj.core.api.Assertions.assertThat;
/**
* @author Christian Tzolov
*/
public class BedrockTitanChatCreateRequestTests {
private TitanChatBedrockApi api = new TitanChatBedrockApi(TitanChatModel.TITAN_TEXT_EXPRESS_V1.id(),
EnvironmentVariableCredentialsProvider.create(), Region.US_EAST_1.id(), new ObjectMapper());
@Test
public void createRequestWithChatOptions() {
var client = new BedrockTitanChatClient(api,
BedrockTitanChatOptions.builder()
.withTemperature(66.6f)
.withTopP(0.66f)
.withMaxTokenCount(666)
.withStopSequences(List.of("stop1", "stop2"))
.build());
var request = client.createRequest(new Prompt("Test message content"));
assertThat(request.inputText()).isNotEmpty();
assertThat(request.textGenerationConfig().temperature()).isEqualTo(66.6f);
assertThat(request.textGenerationConfig().topP()).isEqualTo(0.66f);
assertThat(request.textGenerationConfig().maxTokenCount()).isEqualTo(666);
assertThat(request.textGenerationConfig().stopSequences()).containsExactly("stop1", "stop2");
request = client.createRequest(new Prompt("Test message content",
BedrockTitanChatOptions.builder()
.withTemperature(99.9f)
.withTopP(0.99f)
.withMaxTokenCount(999)
.withStopSequences(List.of("stop3", "stop4"))
.build()
));
assertThat(request.inputText()).isNotEmpty();
assertThat(request.textGenerationConfig().temperature()).isEqualTo(99.9f);
assertThat(request.textGenerationConfig().topP()).isEqualTo(0.99f);
assertThat(request.textGenerationConfig().maxTokenCount()).isEqualTo(999);
assertThat(request.textGenerationConfig().stopSequences()).containsExactly("stop3", "stop4");
}
}

BIN
spring-ai-docs/src/main/antora/modules/ROOT/images/bedrock/bedrock-titan-chat-low-level-api.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 237 KiB

1
spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc
View File

@@ -17,6 +17,7 @@
**** xref:api/clients/bedrock/bedrock-anthropic.adoc[]
**** xref:api/clients/bedrock/bedrock-llama2.adoc[]
**** xref:api/clients/bedrock/bedrock-cohere.adoc[]
**** xref:api/clients/bedrock/bedrock-titan.adoc[]
*** xref:api/clients/huggingface.adoc[]
*** xref:api/clients/ollama-chat.adoc[]
** xref:api/prompt.adoc[]

2
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/bedrock.adoc
View File

@@ -88,4 +88,6 @@ For more information, refer to the documentation below for each supported model.
* xref:api/clients/bedrock/bedrock-llama2.adoc[Spring AI Bedrock Llama2 Chat]: `spring.ai.bedrock.llama2.chat.enabled=true`
* xref:api/clients/bedrock/bedrock-cohere.adoc[Spring AI Bedrock Cohere Chat]: `spring.ai.bedrock.cohere.chat.enabled=true`
* xref:api/embeddings/bedrock-cohere-embedding.adoc[Spring AI Bedrock Cohere Embeddings]: `spring.ai.bedrock.cohere.embedding.enabled=true`
* xref:api/clients/bedrock/bedrock-titan.adoc[Spring AI Bedrock Titan Chat]: `spring.ai.bedrock.titan.chat.enabled=true`
// * xref:api/clients/bedrock/bedrock-jurassic2-chat.adoc[(WIP)Spring AI Bedrock Jurassic Chat]: `spring.ai.bedrock.jurassic2.chat.enabled=true`

228
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock/bedrock-titan.adoc Normal file
View File

@@ -0,0 +1,228 @@
= Titan Chat
link:https://aws.amazon.com/bedrock/titan/[Amazon Titan] foundation models (FMs) provide customers with a breadth of high-performing image, multimodal, and text model choices, via a fully managed API.
Amazon Titan models are created by AWS and pretrained on large datasets, making them powerful, general-purpose models built to support a variety of use cases, while also supporting the responsible use of AI.
Use them as is or privately customize them with your own data.
The https://aws.amazon.com/bedrock/titan/[AWS Bedrock Titan Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model.
== Prerequisites
Refer to the xref:api/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access.
== Auto-configuration
Add the `spring-ai-bedrock-ai-spring-boot-starter` dependency to your project's Maven `pom.xml` file:
[source,xml]
----
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bedrock-ai-spring-boot-starter</artifactId>
<version>0.8.0-SNAPSHOT</version>
</dependency>
----
or to your Gradle `build.gradle` build file.
[source,gradle]
----
dependencies {
implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT'
}
----
=== Enable Titan Chat
By default the Titan model is disabled.
To enable it set the `spring.ai.bedrock.titan.chat.enabled` property to `true`.
Exporting environment variable is one way to set this configuration property:
[source,shell]
----
export SPRING_AI_BEDROCK_TITAN_CHAT_ENABLED=true
----
=== Chat Properties
The prefix `spring.ai.bedrock.aws` is the property prefix to configure the connection to AWS Bedrock.
[cols="3,4,1"]
|====
| Property | Description | Default
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1
| spring.ai.bedrock.aws.access-key | AWS access key. | -
| spring.ai.bedrock.aws.secret-key | AWS secret key. | -
|====
The prefix `spring.ai.bedrock.titan.chat` is the property prefix that configures the chat client implementation for Titan.
[cols="3,4,1"]
|====
| Property | Description | Default
| spring.ai.bedrock.titan.chat.enable | Enable Bedrock Titan chat client. Disabled by default | false
| spring.ai.bedrock.titan.chat.model | The model id to use. See the link:https://github.com/spring-projects/spring-ai/blob/4839a6175cd1ec89498b97d3efb6647022c3c7cb/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java#L220[TitanChatBedrockApi#TitanChatModel] for the supported models. | amazon.titan-text-lite-v1
| spring.ai.bedrock.titan.chat.options.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7
| spring.ai.bedrock.titan.chat.options.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default
| spring.ai.bedrock.titan.chat.options.stopSequences | Configure up to four sequences that the generative recognizes. After a stop sequence, the generative stops generating further tokens. The returned text doesn't contain the stop sequence. | AWS Bedrock default
| spring.ai.bedrock.titan.chat.options.maxTokenCount | Specify the maximum number of tokens to use in the generated response. Note that the models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate. We recommend a limit of 4,000 tokens for optimal performance. | AWS Bedrock default
|====
Look at the https://github.com/spring-projects/spring-ai/blob/4839a6175cd1ec89498b97d3efb6647022c3c7cb/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java#L220[TitanChatBedrockApi#TitanChatModel] for other model IDs.
Supported values are: `amazon.titan-text-lite-v1` and `amazon.titan-text-express-v1`.
Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs].
TIP: All properties prefixed with `spring.ai.bedrock.titan.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
=== Chat Options [[chat-options]]
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatOptions.java[BedrockTitanChatOptions.java] provides model configurations, such as temperature, topP, etc.
On start-up, the default options can be configured with the `BedrockTitanChatClient(api, options)` constructor or the `spring.ai.bedrock.titan.chat.options.*` properties.
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
For example to override the default temperature for a specific request:
[source,java]
----
ChatResponse response = chatClient.call(
new Prompt(
"Generate the names of 5 famous pirates.",
BedrockTitanChatOptions.builder()
.withTemperature(0.4)
.build()
));
----
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatOptions.java[BedrockTitanChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
=== Sample Controller (Auto-configuration)
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-bedrock-ai-spring-boot-starter` to your pom (or gradle) dependencies.
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the Titan Chat client:
[source]
----
spring.ai.bedrock.aws.region=eu-central-1
spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID}
spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
spring.ai.bedrock.titan.chat.enabled=true
spring.ai.bedrock.titan.chat.options.temperature=0.8
----
TIP: replace the `regions`, `access-key` and `secret-key` with your AWS credentials.
This will create a `BedrockTitanChatClient` implementation that you can inject into your class.
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
[source,java]
----
@RestController
public class ChatController {
private final BedrockTitanChatClient chatClient;
@Autowired
public ChatController(BedrockTitanChatClient chatClient) {
this.chatClient = chatClient;
}
@GetMapping("/ai/generate")
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
return Map.of("generation", chatClient.call(message));
}
@GetMapping("/open-ai/generateStream")
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
Prompt prompt = new Prompt(new UserMessage(message));
return chatClient.stream(prompt);
}
}
----
== Manual Configuration
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java[BedrockTitanChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Bedrock Titanic service.
Add the `spring-ai-bedrock` dependency to your project's Maven `pom.xml` file:
[source,xml]
----
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bedrock</artifactId>
<version>0.8.0-SNAPSHOT</version>
</dependency>
----
or to your Gradle `build.gradle` build file.
[source,gradle]
----
dependencies {
implementation 'org.springframework.ai:spring-ai-bedrock:0.8.0-SNAPSHOT'
}
----
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
Next, create an https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java[BedrockTitanChatClient] and use it for text generations:
[source,java]
----
TitanChatBedrockApi titanApi = new TitanChatBedrockApi(
TitanChatModel.TITAN_TEXT_EXPRESS_V1.id(),
EnvironmentVariableCredentialsProvider.create(),
Region.US_EAST_1.id(), new ObjectMapper());
BedrockTitanChatClient chatClient = new BedrockTitanChatClient(titanApi,
BedrockTitanChatOptions.builder()
.withTemperature(0.6f)
.withTopP(0.8f)
.withMaxTokenCount(100)
.build());
ChatResponse response = chatClient.call(
new Prompt("Generate the names of 5 famous pirates."));
// Or with streaming responses
Flux<ChatResponse> response = chatClient.stream(
new Prompt("Generate the names of 5 famous pirates."));
----
=== Low-level TitanChatBedrockApi Client [[low-level-api]]
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java[TitanChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock link:https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-text.html[Bedrock Titan models].
Following class diagram illustrates the TitanChatBedrockApi interface and building blocks:
image::bedrock/bedrock-titan-chat-low-level-api.jpg[width=800,align="center"]
Client supports the `amazon.titan-text-lite-v1` and `amazon.titan-text-express-v1` models for both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
Here is a simple snippet how to use the api programmatically:
[source,java]
----
TitanChatBedrockApi titanBedrockApi = new TitanChatBedrockApi(TitanChatCompletionModel.TITAN_TEXT_EXPRESS_V1.id(),
Region.EU_CENTRAL_1.id());
TitanChatRequest titanChatRequest = TitanChatRequest.builder("Give me the names of 3 famous pirates?")
.withTemperature(0.5f)
.withTopP(0.9f)
.withMaxTokenCount(100)
.withStopSequences(List.of("|"))
.build();
TitanChatResponse response = titanBedrockApi.chatCompletion(titanChatRequest);
Flux<TitanChatResponseChunk> response = titanBedrockApi.chatCompletionStream(titanChatRequest);
List<TitanChatResponseChunk> results = response.collectList().block();
----
Follow the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java[TitanChatBedrockApi]'s JavaDoc for further information.

7
spring-ai-spring-boot-autoconfigure/src/main/java/org/springframework/ai/autoconfigure/bedrock/titan/BedrockTitanChatAutoConfiguration.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2023 the original author or authors.
* Copyright 2023-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -59,10 +59,7 @@ public class BedrockTitanChatAutoConfiguration {
public BedrockTitanChatClient titanChatClient(TitanChatBedrockApi titanChatApi,
BedrockTitanChatProperties properties) {
return new BedrockTitanChatClient(titanChatApi).withTemperature(properties.getTemperature())
.withTopP(properties.getTopP())
.withMaxTokenCount(properties.getMaxTokenCount())
.withStopSequences(properties.getStopSequences());
return new BedrockTitanChatClient(titanChatApi, properties.getOptions());
}
}

64
spring-ai-spring-boot-autoconfigure/src/main/java/org/springframework/ai/autoconfigure/bedrock/titan/BedrockTitanChatProperties.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2023 the original author or authors.
* Copyright 2023-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,10 +16,10 @@
package org.springframework.ai.autoconfigure.bedrock.titan;
import java.util.List;
import org.springframework.ai.bedrock.titan.BedrockTitanChatOptions;
import org.springframework.ai.bedrock.titan.api.TitanChatBedrockApi.TitanChatModel;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
/**
* Bedrock Titan Chat autoconfiguration properties.
@@ -42,30 +42,8 @@ public class BedrockTitanChatProperties {
*/
private String model = TitanChatModel.TITAN_TEXT_EXPRESS_V1.id();
/**
* (optional) Use a lower value to decrease randomness in the response. Defaults to
* 0.7.
*/
private Float temperature = 0.7f;
/**
* (optional) The maximum cumulative probability of tokens to consider when sampling.
* The generative uses combined Top-k and nucleus sampling. Nucleus sampling considers
* the smallest set of tokens whose probability sum is at least topP.
*/
private Float topP;
/**
* (optional) Specify the maximum number of tokens to use in the generated response.
*/
private Integer maxTokenCount;
/**
* (optional) Configure up to four sequences that the generative recognizes. After a
* stop sequence, the generative stops generating further tokens. The returned text
* doesn't contain the stop sequence.
*/
private List<String> stopSequences;
@NestedConfigurationProperty
private BedrockTitanChatOptions options = BedrockTitanChatOptions.builder().withTemperature(0.7f).build();
public boolean isEnabled() {
return enabled;
@@ -83,36 +61,12 @@ public class BedrockTitanChatProperties {
this.model = model;
}
public Float getTemperature() {
return temperature;
public BedrockTitanChatOptions getOptions() {
return options;
}
public void setTemperature(Float temperature) {
this.temperature = temperature;
}
public Float getTopP() {
return topP;
}
public void setTopP(Float topP) {
this.topP = topP;
}
public Integer getMaxTokenCount() {
return maxTokenCount;
}
public void setMaxTokenCount(Integer maxTokens) {
this.maxTokenCount = maxTokens;
}
public List<String> getStopSequences() {
return stopSequences;
}
public void setStopSequences(List<String> stopSequences) {
this.stopSequences = stopSequences;
public void setOptions(BedrockTitanChatOptions options) {
this.options = options;
}
}

18
spring-ai-spring-boot-autoconfigure/src/test/java/org/springframework/ai/autoconfigure/bedrock/titan/BedrockTitanChatAutoConfigurationIT.java
View File

@@ -54,7 +54,8 @@ public class BedrockTitanChatAutoConfigurationIT {
"spring.ai.bedrock.aws.secret-key=" + System.getenv("AWS_SECRET_ACCESS_KEY"),
"spring.ai.bedrock.aws.region=" + Region.US_EAST_1.id(),
"spring.ai.bedrock.titan.chat.model=" + TitanChatModel.TITAN_TEXT_EXPRESS_V1.id(),
"spring.ai.bedrock.titan.chat.temperature=0.5", "spring.ai.bedrock.titan.chat.maxTokens=500")
"spring.ai.bedrock.titan.chat.options.temperature=0.5",
"spring.ai.bedrock.titan.chat.options.maxTokenCount=500")
.withConfiguration(AutoConfigurations.of(BedrockTitanChatAutoConfiguration.class));
private final Message systemMessage = new SystemPromptTemplate("""
@@ -106,9 +107,10 @@ public class BedrockTitanChatAutoConfigurationIT {
"spring.ai.bedrock.aws.access-key=ACCESS_KEY", "spring.ai.bedrock.aws.secret-key=SECRET_KEY",
"spring.ai.bedrock.titan.chat.model=MODEL_XYZ",
"spring.ai.bedrock.aws.region=" + Region.EU_CENTRAL_1.id(),
"spring.ai.bedrock.titan.chat.temperature=0.55", "spring.ai.bedrock.titan.chat.topP=0.55",
"spring.ai.bedrock.titan.chat.stopSequences=END1,END2",
"spring.ai.bedrock.titan.chat.maxTokenCount=123")
"spring.ai.bedrock.titan.chat.options.temperature=0.55",
"spring.ai.bedrock.titan.chat.options.topP=0.55",
"spring.ai.bedrock.titan.chat.options.stopSequences=END1,END2",
"spring.ai.bedrock.titan.chat.options.maxTokenCount=123")
.withConfiguration(AutoConfigurations.of(BedrockTitanChatAutoConfiguration.class))
.run(context -> {
var chatProperties = context.getBean(BedrockTitanChatProperties.class);
@@ -118,11 +120,11 @@ public class BedrockTitanChatAutoConfigurationIT {
assertThat(aswProperties.getRegion()).isEqualTo(Region.EU_CENTRAL_1.id());
assertThat(chatProperties.getModel()).isEqualTo("MODEL_XYZ");
assertThat(chatProperties.getTemperature()).isEqualTo(0.55f);
assertThat(chatProperties.getTopP()).isEqualTo(0.55f);
assertThat(chatProperties.getOptions().getTemperature()).isEqualTo(0.55f);
assertThat(chatProperties.getOptions().getTopP()).isEqualTo(0.55f);
assertThat(chatProperties.getStopSequences()).isEqualTo(List.of("END1", "END2"));
assertThat(chatProperties.getMaxTokenCount()).isEqualTo(123);
assertThat(chatProperties.getOptions().getStopSequences()).isEqualTo(List.of("END1", "END2"));
assertThat(chatProperties.getOptions().getMaxTokenCount()).isEqualTo(123);
assertThat(aswProperties.getAccessKey()).isEqualTo("ACCESS_KEY");
assertThat(aswProperties.getSecretKey()).isEqualTo("SECRET_KEY");