Add Bedrock Titan Chat Options + Docs
This commit is contained in:
Christian Tzolov
Binary file not shown.
|
After Width: | Height: | Size: 237 KiB |
@@ -17,6 +17,7 @@
|
||||
**** xref:api/clients/bedrock/bedrock-anthropic.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-llama2.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-cohere.adoc[]
|
||||
**** xref:api/clients/bedrock/bedrock-titan.adoc[]
|
||||
*** xref:api/clients/huggingface.adoc[]
|
||||
*** xref:api/clients/ollama-chat.adoc[]
|
||||
** xref:api/prompt.adoc[]
|
||||
|
||||
@@ -88,4 +88,6 @@ For more information, refer to the documentation below for each supported model.
|
||||
* xref:api/clients/bedrock/bedrock-llama2.adoc[Spring AI Bedrock Llama2 Chat]: `spring.ai.bedrock.llama2.chat.enabled=true`
|
||||
* xref:api/clients/bedrock/bedrock-cohere.adoc[Spring AI Bedrock Cohere Chat]: `spring.ai.bedrock.cohere.chat.enabled=true`
|
||||
* xref:api/embeddings/bedrock-cohere-embedding.adoc[Spring AI Bedrock Cohere Embeddings]: `spring.ai.bedrock.cohere.embedding.enabled=true`
|
||||
* xref:api/clients/bedrock/bedrock-titan.adoc[Spring AI Bedrock Titan Chat]: `spring.ai.bedrock.titan.chat.enabled=true`
|
||||
|
||||
// * xref:api/clients/bedrock/bedrock-jurassic2-chat.adoc[(WIP)Spring AI Bedrock Jurassic Chat]: `spring.ai.bedrock.jurassic2.chat.enabled=true`
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
= Titan Chat
|
||||
|
||||
link:https://aws.amazon.com/bedrock/titan/[Amazon Titan] foundation models (FMs) provide customers with a breadth of high-performing image, multimodal, and text model choices, via a fully managed API.
|
||||
Amazon Titan models are created by AWS and pretrained on large datasets, making them powerful, general-purpose models built to support a variety of use cases, while also supporting the responsible use of AI.
|
||||
Use them as is or privately customize them with your own data.
|
||||
|
||||
The https://aws.amazon.com/bedrock/titan/[AWS Bedrock Titan Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
Refer to the xref:api/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access.
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
Add the `spring-ai-bedrock-ai-spring-boot-starter` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bedrock-ai-spring-boot-starter</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,gradle]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
=== Enable Titan Chat
|
||||
|
||||
By default the Titan model is disabled.
|
||||
To enable it set the `spring.ai.bedrock.titan.chat.enabled` property to `true`.
|
||||
Exporting environment variable is one way to set this configuration property:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
export SPRING_AI_BEDROCK_TITAN_CHAT_ENABLED=true
|
||||
----
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
The prefix `spring.ai.bedrock.aws` is the property prefix to configure the connection to AWS Bedrock.
|
||||
|
||||
[cols="3,4,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1
|
||||
| spring.ai.bedrock.aws.access-key | AWS access key. | -
|
||||
| spring.ai.bedrock.aws.secret-key | AWS secret key. | -
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.bedrock.titan.chat` is the property prefix that configures the chat client implementation for Titan.
|
||||
|
||||
[cols="3,4,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.bedrock.titan.chat.enable | Enable Bedrock Titan chat client. Disabled by default | false
|
||||
| spring.ai.bedrock.titan.chat.model | The model id to use. See the link:https://github.com/spring-projects/spring-ai/blob/4839a6175cd1ec89498b97d3efb6647022c3c7cb/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java#L220[TitanChatBedrockApi#TitanChatModel] for the supported models. | amazon.titan-text-lite-v1
|
||||
| spring.ai.bedrock.titan.chat.options.temperature | Controls the randomness of the output. Values can range over [0.0,1.0] | 0.7
|
||||
| spring.ai.bedrock.titan.chat.options.topP | The maximum cumulative probability of tokens to consider when sampling. | AWS Bedrock default
|
||||
| spring.ai.bedrock.titan.chat.options.stopSequences | Configure up to four sequences that the generative recognizes. After a stop sequence, the generative stops generating further tokens. The returned text doesn't contain the stop sequence. | AWS Bedrock default
|
||||
| spring.ai.bedrock.titan.chat.options.maxTokenCount | Specify the maximum number of tokens to use in the generated response. Note that the models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate. We recommend a limit of 4,000 tokens for optimal performance. | AWS Bedrock default
|
||||
|====
|
||||
|
||||
Look at the https://github.com/spring-projects/spring-ai/blob/4839a6175cd1ec89498b97d3efb6647022c3c7cb/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java#L220[TitanChatBedrockApi#TitanChatModel] for other model IDs.
|
||||
Supported values are: `amazon.titan-text-lite-v1` and `amazon.titan-text-express-v1`.
|
||||
Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs].
|
||||
|
||||
TIP: All properties prefixed with `spring.ai.bedrock.titan.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
|
||||
|
||||
=== Chat Options [[chat-options]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatOptions.java[BedrockTitanChatOptions.java] provides model configurations, such as temperature, topP, etc.
|
||||
|
||||
On start-up, the default options can be configured with the `BedrockTitanChatClient(api, options)` constructor or the `spring.ai.bedrock.titan.chat.options.*` properties.
|
||||
|
||||
At run-time you can override the default options by adding new, request specific, options to the `Prompt` call.
|
||||
For example to override the default temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
BedrockTitanChatOptions.builder()
|
||||
.withTemperature(0.4)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: In addition to the model specific https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatOptions.java[BedrockTitanChatOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
=== Sample Controller (Auto-configuration)
|
||||
|
||||
https://start.spring.io/[Create] a new Spring Boot project and add the `spring-ai-bedrock-ai-spring-boot-starter` to your pom (or gradle) dependencies.
|
||||
|
||||
Add a `application.properties` file, under the `src/main/resources` directory, to enable and configure the Titan Chat client:
|
||||
|
||||
[source]
|
||||
----
|
||||
spring.ai.bedrock.aws.region=eu-central-1
|
||||
spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID}
|
||||
spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
|
||||
|
||||
spring.ai.bedrock.titan.chat.enabled=true
|
||||
spring.ai.bedrock.titan.chat.options.temperature=0.8
|
||||
----
|
||||
|
||||
TIP: replace the `regions`, `access-key` and `secret-key` with your AWS credentials.
|
||||
|
||||
This will create a `BedrockTitanChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the chat client for text generations.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final BedrockTitanChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(BedrockTitanChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.call(message));
|
||||
}
|
||||
|
||||
@GetMapping("/open-ai/generateStream")
|
||||
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
Prompt prompt = new Prompt(new UserMessage(message));
|
||||
return chatClient.stream(prompt);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java[BedrockTitanChatClient] implements the `ChatClient` and `StreamingChatClient` and uses the <<low-level-api>> to connect to the Bedrock Titanic service.
|
||||
|
||||
Add the `spring-ai-bedrock` dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bedrock</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,gradle]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-bedrock:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
Next, create an https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/BedrockTitanChatClient.java[BedrockTitanChatClient] and use it for text generations:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
TitanChatBedrockApi titanApi = new TitanChatBedrockApi(
|
||||
TitanChatModel.TITAN_TEXT_EXPRESS_V1.id(),
|
||||
EnvironmentVariableCredentialsProvider.create(),
|
||||
Region.US_EAST_1.id(), new ObjectMapper());
|
||||
|
||||
BedrockTitanChatClient chatClient = new BedrockTitanChatClient(titanApi,
|
||||
BedrockTitanChatOptions.builder()
|
||||
.withTemperature(0.6f)
|
||||
.withTopP(0.8f)
|
||||
.withMaxTokenCount(100)
|
||||
.build());
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
|
||||
// Or with streaming responses
|
||||
Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
=== Low-level TitanChatBedrockApi Client [[low-level-api]]
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java[TitanChatBedrockApi] provides is lightweight Java client on top of AWS Bedrock link:https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-text.html[Bedrock Titan models].
|
||||
|
||||
Following class diagram illustrates the TitanChatBedrockApi interface and building blocks:
|
||||
|
||||
image::bedrock/bedrock-titan-chat-low-level-api.jpg[width=800,align="center"]
|
||||
|
||||
Client supports the `amazon.titan-text-lite-v1` and `amazon.titan-text-express-v1` models for both synchronous (e.g. `chatCompletion()`) and streaming (e.g. `chatCompletionStream()`) responses.
|
||||
|
||||
Here is a simple snippet how to use the api programmatically:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
TitanChatBedrockApi titanBedrockApi = new TitanChatBedrockApi(TitanChatCompletionModel.TITAN_TEXT_EXPRESS_V1.id(),
|
||||
Region.EU_CENTRAL_1.id());
|
||||
|
||||
TitanChatRequest titanChatRequest = TitanChatRequest.builder("Give me the names of 3 famous pirates?")
|
||||
.withTemperature(0.5f)
|
||||
.withTopP(0.9f)
|
||||
.withMaxTokenCount(100)
|
||||
.withStopSequences(List.of("|"))
|
||||
.build();
|
||||
|
||||
TitanChatResponse response = titanBedrockApi.chatCompletion(titanChatRequest);
|
||||
|
||||
Flux<TitanChatResponseChunk> response = titanBedrockApi.chatCompletionStream(titanChatRequest);
|
||||
|
||||
List<TitanChatResponseChunk> results = response.collectList().block();
|
||||
----
|
||||
|
||||
Follow the https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanChatBedrockApi.java[TitanChatBedrockApi]'s JavaDoc for further information.
|
||||
Reference in New Issue
Block a user