diff --git a/README.md b/README.md index 268e7bbbd..6bfca6b37 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ Please use the 0.7.1-SNAPSHOT if you don't want to be on the bleeding edge. * Azure OpenAI ```xml - org.springframework.ai + org.springframework.experimental.ai spring-ai-azure-openai-spring-boot-starter 0.7.1-SNAPSHOT @@ -337,7 +337,7 @@ Examples of Vector Databases include Chroma, Postgres, Pinecone, Weaviate, Mongo ## Building -To build with only unit tests +To build with running unit tests ```shell ./mvnw clean package diff --git a/models/spring-ai-bedrock/README.md b/models/spring-ai-bedrock/README.md index e1e4b0788..1b2b38072 100644 --- a/models/spring-ai-bedrock/README.md +++ b/models/spring-ai-bedrock/README.md @@ -2,6 +2,7 @@ [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a managed service that provides foundation models from various AI providers, available through a unified API. + Spring AI implements `API` clients for the [Bedrock models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html) along with implementations for the `ChatClient`, `StreamingChatClient` and the `EmbeddingClient`. The API clients provide structured, type-safe implementation for the Bedrock models, while the `ChatClient`, `StreamingChatClient` and the `EmbeddingClient` implementations provide Chat and Embedding clients compliant with the Spring-AI API. Later can be used interchangeably with the other (e.g. OpenAI, Azure OpenAI, diff --git a/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java b/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java index b441ade99..d69763615 100644 --- a/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java +++ b/spring-ai-core/src/main/java/org/springframework/ai/chat/StreamingChatClient.java @@ -23,6 +23,6 @@ import org.springframework.ai.prompt.Prompt; @FunctionalInterface public interface StreamingChatClient { - public Flux generateStream(Prompt prompt); + Flux generateStream(Prompt prompt); } diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc index 8a41bf606..78c9e7400 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc @@ -2,8 +2,9 @@ * xref:concepts.adoc[AI Concepts] * xref:getting-started.adoc[Getting Started] * xref:api/index.adoc[] -** xref:api/aiclient.adoc[] +** xref:api/chatclient.adoc[] *** xref:api/clients/huggingface.adoc[] +*** xref:api/clients/ollama.adoc[] ** xref:api/prompt.adoc[] ** xref:api/output-parser.adoc[] ** xref:api/etl-pipeline.adoc[] diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/aiclient.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc similarity index 97% rename from spring-ai-docs/src/main/antora/modules/ROOT/pages/api/aiclient.adoc rename to spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc index f5871552d..747026606 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/aiclient.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc @@ -108,9 +108,9 @@ public class Generation { == Available Implementations -The `ChatClient` interface has the following available implementations: +There is a `ChatClient` implementation for the following Model providers: -* OpenAI: Using the https://github.com/TheoKanning/openai-java[Theo Kanning client library]. +* OpenAI * Azure OpenAI: Using https://learn.microsoft.com/en-us/java/api/overview/azure/ai-openai-readme?view=azure-java-preview[Microsoft's OpenAI client library]. * Hugging Face: Using the https://huggingface.co/inference-endpoints[Hugging Face Hosted Inference Service]. This gives you access to hundreds of models. * https://ollama.ai/[Ollama]: Run large language models locally. @@ -236,13 +236,13 @@ public class SimpleAiController { } ---- -== Best Practices - -TBD - -== Troubleshooting - -TBD +// == Best Practices +// +// TBD +// +// == Troubleshooting +// +// TBD == API Docs @@ -252,6 +252,6 @@ You can find the Javadoc https://docs.spring.io/spring-ai/docs/current-SNAPSHOT/ The project's https://github.com/spring-projects/spring-ai/discussions[GitHub discussions] is a great place to send feedback. -== Related Resources - -TBD +// == Related Resources +// +// TBD diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/azure-openai.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/azure-openai.adoc new file mode 100644 index 000000000..f9b99fd9e --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/azure-openai.adoc @@ -0,0 +1,103 @@ += Azure OpenAI + +Azure's OpenAI offering, powered by ChatGPT, extends beyond traditional OpenAI capabilities, delivering AI-driven text generation with enhanced functionality. Azure offers additional AI safety and responsible AI features, as highlighted in their recent update https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-new-ai-safety-amp-responsible-ai-features-in-azure/ba-p/3983686[here] + +Azure offers Java developers the opportunity to leverage AI's full potential by integrating it with an array of Azure services, which includes AI-related resources such as Vector Stores on Azure. + + +== Gettting Started + +Obtain your Azure OpenAI `endpoint` and `api-key` from the Azure OpenAI Service section on the link:https://portal.azure.com[Azure Portal]. + +Spring AI defines a configuration property named `spring.ai.azure.openai.api-key` that you should set to the value of the `API Key` obtained from Azure. +There is also a configuration property named `spring.ai.azure.openai.endpoint` that you should set to the endpoint URL obtained when provisioning your model in Azure. + +Exporting environment variables is one way to set these configuration properties: + +[source,shell] +---- +export SPRING_AI_AZURE_OPENAI_API_KEY= +export SPRING_AI_AZURE_OPENAI_ENDPOINT= +---- + +== Project Dependencies + +Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file. + +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: + +[source, xml] +---- + + org.springframework.ai + spring-ai-azure-openai-spring-boot-starter + 0.8.0-SNAPSHOT + +---- + +or to your Gradle `build.gradle` build file. + +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-azure-openai-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + +== Sample Code + +This will create a `ChatClient` implementation that you can inject into your class. +Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation. + +[source,java] +---- +@RestController +public class ChatController { + + private final ChatClient chatClient; + + @Autowired + public ChatController(ChatClient chatClient) { + this.chatClient = chatClient; + } + + @GetMapping("/ai/generate") + public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { + return Map.of("generation", chatClient.generate(message)); + } +} +---- + + +== Azure OpenAI Properties + +The prefix `spring.ai.azure.openai` is the property prefix to configure the connection to Azure OpenAI. + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.azure.openai.api-key | The Key from Azure AI OpenAI `Keys and Endpoint` section under `Resource Management` | - +| spring.ai.azure.openai.endpoint | The endpoint from the Azure AI OpenAI `Keys and Endpoint` section under `Resource Management` | - +|==== + +The prefix `spring.ai.azure.openai.chat` is the property prefix that configures the `ChatClient` implementation for Azure OpenAI. + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.azure.openai.chat.model | This is the value of the 'Deployment Name' as presented in the Azure AI Portal | gpt-35-turbo +| spring.ai.azure.openai.chat.temperature | The sampling temperature to use that controls the apparent creativity of generated completions. Higher values will make output more random while lower values will make results more focused and deterministic. It is not recommended to modify temperature and top_p for the same completions request as the interaction of these two settings is difficult to predict. | 0.7 +| spring.ai.azure.openai.chat.top-p | An alternative to sampling with temperature called nucleus sampling. This value causes the model to consider the results of tokens with the provided probability mass. As an example, a value of 0.15 will cause only the tokens comprising the top 15% of probability mass to be considered. It is not recommended to modify temperature and top_p for the same completions request as the interaction of these two settings is difficult to predict. | - +| spring.ai.azure.openai.chat.max-tokens | The maximum number of tokens to generate | - +|==== + +The prefix `spring.ai.azure.openai.embeddings` is the property prefix that configures the `EmbeddingClient` implementation for Azure OoenAI + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.azure.openai.embedding.model | This is the value of the 'Deployment Name' as presented in the Azure AI Portal | text-embedding-ada-002 +|==== \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-anthropic.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-anthropic.adoc new file mode 100644 index 000000000..fd37c29f1 --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-anthropic.adoc @@ -0,0 +1,109 @@ += Amazon Bedrock Anthropic + +https://www.anthropic.com/product[Anthropic's Claude] is an AI assistant based on Anthropic’s research into training helpful, honest, and harmless AI systems. + +The Claude model has the following high level features + +* 200k Token Context Window: Claude boasts a generous token capacity of 200,000, making it ideal for handling extensive information in applications like technical documentation, codebases, and literary works. +* Supported Tasks: Claude's versatility spans tasks such as summarization, Q&A, trend forecasting, and document comparisons, enabling a wide range of applications from dialogues to content generation. +* AI Safety Features: Built on Anthropic's safety research, Claude prioritizes helpfulness, honesty, and harmlessness in its interactions, reducing brand risk and ensuring responsible AI behavior. + +The https://aws.amazon.com/bedrock/claude[AWS Bedrock Anthropic Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model. + +== Getting Started + +Refer to the xref:api/clients/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access. + +== Project Dependencies + +Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file. + +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: + +[source, xml] +---- + + org.springframework.ai + spring-ai-bedrock-ai-spring-boot-starter + 0.8.0-SNAPSHOT + +---- + +or to your Gradle `build.gradle` build file. + +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + +== Enable Anthropic Support + +Spring AI defines a configuration property named `spring.ai.bedrock.llama2.chat.enabled` that you should set to `true` to enable support for Llama2. + +Exporting environment variables in one way to set this configuration property. + +[source,shell] +---- +export SPRING_AI_BEDROCK_LLAMA2_CHAT_ENABLED=true +---- + +== Sample Code + +This will create a `ChatClient` implementation that you can inject into your class. +Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation. + +[source,java] +---- +@RestController +public class ChatController { + + private final ChatClient chatClient; + + @Autowired + public ChatController(ChatClient chatClient) { + this.chatClient = chatClient; + } + + @GetMapping("/ai/generate") + public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { + return Map.of("generation", chatClient.generate(message)); + } +} +---- + +== Bedrock Properties + +The prefix `spring.ai.bedrock.aws` is the property prefix to configure the connection to AWS Bedrock. + + +[cols="3,3,3"] +|==== +| Property | Description | Default + +| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 +| spring.ai.bedrock.aws.access-key | AWS access key. | - +| spring.ai.bedrock.aws.secret-key | AWS secret key. | - +|==== + + +The prefix `spring.ai.bedrock.anthropic.chat` is the property prefix that configures the `ChatClient` implementation for Claude. + +[cols="8,4,3"] +|==== +| Property | Description | Default + +| spring.ai.bedrock.anthropic.chat.enabled | Enable or disable support for Anthropic | false +| spring.ai.bedrock.anthropic.chat.model | The model id to use (See Below) | anthropic.claude-v2 +| spring.ai.bedrock.anthropic.chat.anthropic-version | The version of the model to use | bedrock-2023-05-31 +| spring.ai.bedrock.anthropic.chat.temperature | Controls the randomness of the output. Values can range over [0.0,1.0], inclusive. A value closer to 1.0 will produce responses that are more varied, while a value closer to 0.0 will typically result in less surprising responses from the model. This value specifies default to be used by the backend while making the call to the model.| 0.7 +| spring.ai.bedrock.anthropic.chat.top-p | The maximum cumulative probability of tokens to consider when sampling. The model uses combined Top-k and nucleus sampling. Nucleus sampling considers the smallest set of tokens whose probability sum is at least topP.| AWS Bedrock default +| spring.ai.bedrock.anthropic.chat.max-tokens-to-sample | Specify the maximum number of tokens to use in the generated response. Note that the models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate. We recommend a limit of 4,000 tokens for optimal performance. | 300 +| spring.ai.bedrock.anthropic.chat.top-k | Specify the number of token choices the model uses to generate the next token. | 10 +| spring.ai.bedrock.anthropic.chat.stop-sequences | Configure up to four sequences that the model recognizes. After a stop sequence, the model stops generating further tokens. The returned text doesn't contain the stop sequence. | "\n\Human:" +|==== + +Look at the Spring AI enumeration `AnthropicChatModel` for other model IDs. The other value supported is `anthropic.claude-instant-v1`. + +Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs]. \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-llama2.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-llama2.adoc new file mode 100644 index 000000000..c955f6b56 --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock-llama2.adoc @@ -0,0 +1,106 @@ += Amazon Bedrock Anthropic + +https://ai.meta.com/llama/[Meta's Llama 2 Chat] is part of the Llama 2 collection of large language models. +It excels in dialogue-based applications with a parameter scale ranging from 7 billion to 70 billion. +Leveraging public datasets and over 1 million human annotations, Llama Chat offers context-aware dialogues. + +Trained on 2 trillion tokens from public data sources, Llama-2-Chat provides extensive knowledge for insightful conversations. +Rigorous testing, including over 1,000 hours of red-teaming and annotation, ensures both performance and safety, making it a reliable choice for AI-driven dialogues. + +The https://aws.amazon.com/bedrock/llama-2/[AWS Llama 2 Model Page] and https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock User Guide] contains detailed information on how to use the AWS hosted model. + + +== Getting Started + +Refer to the xref:api/clients/bedrock.adoc[Spring AI documentation on Amazon Bedrock] for setting up API access. + +== Project Dependencies + +Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file. + +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: + +[source, xml] +---- + + org.springframework.ai + spring-ai-bedrock-ai-spring-boot-starter + 0.8.0-SNAPSHOT + +---- + +or to your Gradle `build.gradle` build file. + +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + +== Enable Anthropic Support + +Spring AI defines a configuration property named `spring.ai.bedrock.anthropic.chat.enabled` that you should set to `true` to enable support for Anthropic. + +Exporting environment variables in one way to set this configuration property. + +[source,shell] +---- +export SPRING_AI_BEDROCK_ANTHROPIC_CHAT_ENABLED=true +---- + +== Sample Code + +This will create a `ChatClient` implementation that you can inject into your class. +Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation. + +[source,java] +---- +@RestController +public class ChatController { + + private final ChatClient chatClient; + + @Autowired + public ChatController(ChatClient chatClient) { + this.chatClient = chatClient; + } + + @GetMapping("/ai/generate") + public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { + return Map.of("generation", chatClient.generate(message)); + } +} +---- + +== Bedrock Properties + +The prefix `spring.ai.bedrock.aws` is the property prefix to configure the connection to AWS Bedrock. + + +[cols="3,3,3"] +|==== +| Property | Description | Default + +| spring.ai.bedrock.aws.region | AWS region to use. | us-east-1 +| spring.ai.bedrock.aws.access-key | AWS access key. | - +| spring.ai.bedrock.aws.secret-key | AWS secret key. | - +|==== + + +The prefix `spring.ai.bedrock.llama2.chat` is the property prefix that configures the `ChatClient` implementation for Llama2. + +[cols="8,4,3"] +|==== +| Property | Description | Default + +| spring.ai.bedrock.llama2.chat.enabled | Enable or disable support for Llama2 | false +| spring.ai.bedrock.llama2.chat.model | The model id to use (See Below) | meta.llama2-70b-chat-v1 +| spring.ai.bedrock.llama2.chat.temperature | Controls the randomness of the output. Values can range over [0.0,1.0], inclusive. A value closer to 1.0 will produce responses that are more varied, while a value closer to 0.0 will typically result in less surprising responses from the model. This value specifies default to be used by the backend while making the call to the model. | 0.7 +| spring.ai.bedrock.llama2.chat.top-p | The maximum cumulative probability of tokens to consider when sampling. The model uses combined Top-k and nucleus sampling. Nucleus sampling considers the smallest set of tokens whose probability sum is at least topP. | AWS Bedrock default +| spring.ai.bedrock.llama2.chat.max-gen-len | Specify the maximum number of tokens to use in the generated response. The model truncates the response once the generated text exceeds maxGenLen. | 300 +|==== + +Look at the Spring AI enumeration, `Llama2ChatCompletionModel` for other model IDs. The other value supported is `meta.llama2-13b-chat-v1`. + +Model ID values can also be found in the https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[AWS Bedrock documentation for base model IDs]. \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock.adoc new file mode 100644 index 000000000..51d3ff8d3 --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/bedrock.adoc @@ -0,0 +1,95 @@ += Amazon Bedrock + +link:https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html[Amazon Bedrock] is a managed service that provides foundation models from various AI providers, available through a unified API. + +Spring AI supports https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html[all the Chat and Embedding AI models] available through Amazon Bedrock by implementing the Spring interfaces `ChatClient`, `StreamingChatClient`, and `EmbeddingClient`. + +Additionally, Spring AI provides Spring Auto-Configurations and Boot Starters for all clients, making it easy to bootstrap and configure for the Bedrock models. + +== Getting Started + +There are a few steps to get started + +* *Add Boot Starter*: Add the Spring Boot starter for Bedrock to your project. +* *Obtain AWS credentials*: If you don't have an AWS account and AWS CLI configured yet, this video guide can help you configure it: link:https://youtu.be/gswVHTrRX8I?si=buaY7aeI0l3-bBVb[AWS CLI & SDK Setup in Less Than 4 Minutes!]. You should be able to obtain your access and security keys. +* *Enable Bedrock models to use*: Go to link:https://us-east-1.console.aws.amazon.com/bedrock/home[Amazon Bedrock] and from the link:https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/modelaccess[Model Access] menu on the left, configure access to the models you are going to use. + +=== Project Dependencies + +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: + +[source,xml] +---- + + spring-ai-bedrock-ai-spring-boot-starter + org.springframework.ai + 0.8.0-SNAPSHOT + +---- + +or to your Gradle `build.gradle` build file. + +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-bedrock-ai-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + + +=== Connect to AWS Bedrock + +Use the `BedrockAwsConnectionProperties` to configure AWS credentials and region: + +[source,shell] +---- +spring.ai.bedrock.aws.region=us-east-1 + +spring.ai.bedrock.aws.access-key=YOUR_ACCESS_KEY +spring.ai.bedrock.aws.secret-key=YOUR_SECRET_KEY +---- + +The `region` property is compulsory. + +AWS credentials are resolved in the following order: + +1. Spring-AI Bedrock `spring.ai.bedrock.aws.access-key` and `spring.ai.bedrock.aws.secret-key` properties. +2. Java System Properties - `aws.accessKeyId` and `aws.secretAccessKey`. +3. Environment Variables - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. +4. Web Identity Token credentials from system properties or environment variables. +5. Credential profiles file at the default location (`~/.aws/credentials`) shared by all AWS SDKs and the AWS CLI. +6. Credentials delivered through the Amazon EC2 container service if the `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` environment variable is set and the security manager has permission to access the variable. +7. Instance profile credentials delivered through the Amazon EC2 metadata service or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. + +=== Enable selected Bedrock model + +NOTE: By default, all models are disabled. You have to enable the chosen Bedrock models explicitly using the `spring.ai.bedrock...enabled=true` property. + +Here are the supported `` and `` combinations: + +[cols="|,|,|,|"] +|==== +| Model | Chat | Chat Streaming | Embedding + +| llama2 | Yes | Yes | No +| cohere | Yes | Yes | Yes +| anthropic | Yes | Yes | No +| jurassic2 | Yes | No | No +| titan | Yes | Yes | Yes (however, no batch support) +|==== + +For example, to enable the Bedrock Llama2 Chat client, you need to set `spring.ai.bedrock.llama2.chat.enabled=true`. + +Next, you can use the `spring.ai.bedrock...*` properties to configure each model as provided. + +For more information, refer to the documentation below for each supported model. + +* xref:api/clients/bedrock-anthropic.adoc[Spring AI Bedrock Anthropic Chat]: `spring.ai.bedrock.anthropic.chat.enabled=true` +* xref:api/clients/bedrock-llama2.adoc[Spring AI Bedrock Llama2 Chat]: `spring.ai.bedrock.llama2.chat.enabled=true` + + +// * [Spring AI Bedrock Cohere Chat](./README_COHERE_CHAT.md) - `spring.ai.bedrock.cohere.chat.enabled=true` +// * [Spring AI Bedrock Cohere Embedding](./README_COHERE_EMBEDDING.md) - `spring.ai.bedrock.cohere.embedding.enabled=true` +// * [Spring AI Bedrock Titan Chat](./README_TITAN_CHAT.md) - `spring.ai.bedrock.titan.chat.enabled=true` +// * [Spring AI Bedrock Titan Embedding](./README_TITAN_EMBEDDING.md) - `spring.ai.bedrock.titan.embedding.enabled=true` +// * (WIP) [Spring AI Bedrock Ai21 Jurassic2 Chat](./README_JURASSIC2_CHAT.md) - `spring.ai.bedrock.jurassic2.chat.enabled=true` \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/huggingface.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/huggingface.adoc index 20268fbcd..629b10f57 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/huggingface.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/huggingface.adoc @@ -1,6 +1,10 @@ = HuggingFace -HuggingFace Inference Endpoints allow you to deploy and serve machine learning models in the cloud, making them accessible via an API. Further details on HuggingFace Inference Endpoints can be found link:https://huggingface.co/docs/inference-endpoints/index[here]. +HuggingFace Inference Endpoints allow you to deploy and serve machine learning models in the cloud, making them accessible via an API. + +== Getting Started + +Further details on HuggingFace Inference Endpoints can be found link:https://huggingface.co/docs/inference-endpoints/index[here]. == Prerequisites diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/ollama.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/ollama.adoc index a474bcfc6..3f546bd36 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/ollama.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/ollama.adoc @@ -2,43 +2,19 @@ Ollama lets you get up and running with large language models locally. -Refer to the official link:https://github.com/jmorganca/ollama[README] to get started. +== Getting Started + +You first need to run Ollama on your local machine. + +Refer to the official Ollama project link:https://github.com/jmorganca/ollama[README] to get started running models on your local machine. Note, installing `ollama run llama2` will download a 4GB docker image. -You can run the disabled test in `OllamaClientTests.java` to kick the tires. +== Project Dependencies -== How to use +Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file. -Add the `spring-ai-ollama` dependency to your project's pom: - -[source,xml] ----- - - org.springframework.ai - spring-ai-ollama - 0.8.0-SNAPSHOT - ----- - -then create a client and use generate response: - -[source,java] ----- -var ollamaClient = new OllamaClient("http://127.0.0.1:11434", "llama2", - ollamaResult -> { - if (ollamaResult.getDone()) { - .... - } - }); - -AiResponse aiResponse = ollamaClient.generate(new Prompt("Hello")); ----- - -=== Spring Boot Starter - -For convenience, you can opt for the Ollama Boot starter. -For this add the following dependency: +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: [source,xml] ---- @@ -49,13 +25,140 @@ For this add the following dependency: ---- -and use the `spring.ai.ollama.*` properties to configure it if you want to use something other than the default values. +or to your Gradle `build.gradle` build file. -The complete list of supported properties are: +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-ollama-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + +== Sample Code + +This will create a `ChatClient` implementation that you can inject into your class. +Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation. + +[source,java] +---- +@RestController +public class ChatController { + + private final ChatClient chatClient; + + @Autowired + public ChatController(ChatClient chatClient) { + this.chatClient = chatClient; + } + + @GetMapping("/ai/generate") + public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { + return Map.of("generation", chatClient.generate(message)); + } +} +---- + +=== Ollama Properties + +The prefix `spring.ai.ollama` is the property prefix to configure the connection to Ollama [cols="3,5,3"] |==== | Property | Description | Default + | spring.ai.ollama.base-url | Base URL where Ollama API server is running. | `http://localhost:11434` | spring.ai.ollama.model | Language model to use. | `llama2` |==== + +[NOTE] +==== +The list of options for chat and embedding is to be reviewd. This https://github.com/spring-projects/spring-ai/issues/230[issue] will track progress. +==== + +The prefix `spring.ai.ollama.chat` is the property prefix that configures the `ChatClient` implementation for Ollama. + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.ollama.chat.enabled | Enable the Ollama Chat Client | true +| spring.ai.ollama.chat.model | The name of the model to use | llama2 +| spring.ai.ollama.chat.options.numa | Whether to use NUMA. | false +| spring.ai.ollama.chat.options.num-ctx | Sets the size of the context window used to generate the next token. | 2048 +| spring.ai.ollama.chat.options.num-batch | ??? | - +| spring.ai.ollama.chat.options.num-gqa | The number of GQA groups in the transformer layer. Required for some models, for example, it is 8 for llama2:70b. | - +| spring.ai.ollama.chat.options.num-gpu | The number of layers to send to the GPU(s). On macOS it defaults to 1 to enable metal support, 0 to disable. | - +| spring.ai.ollama.chat.options.main-gpu | ??? | - +| spring.ai.ollama.chat.options.low-vram | ??? | - +| spring.ai.ollama.chat.options.f16-kv | ??? | - +| spring.ai.ollama.chat.options.logits-all | ??? | - +| spring.ai.ollama.chat.options.vocab-only | ??? | - +| spring.ai.ollama.chat.options.use-mmap | ??? | - +| spring.ai.ollama.chat.options.use-mlock | ??? | - +| spring.ai.ollama.chat.options.embedding-only | ??? | - +| spring.ai.ollama.chat.options.rope-frequency-base | ??? | - +| spring.ai.ollama.chat.options.rope-frequency-scale | ??? | - +| spring.ai.ollama.chat.options.num-thread | Sets the number of threads to use during computation. By default, Ollama will detect this for optimal performance. It is recommended to set this value to the number of physical CPU cores your system has (as opposed to the logical number of cores). | - +| spring.ai.ollama.chat.options.num-keep | ??? | - +| spring.ai.ollama.chat.options.seed | Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. | 0 +| spring.ai.ollama.chat.options.num-predict | Maximum number of tokens to predict when generating text. (Default: 128, -1 = infinite generation, -2 = fill context) | 128 +| spring.ai.ollama.chat.options.top-k | Reduces the probability of generating nonsense. A higher value (e.g., 100) will give more diverse answers, while a lower value (e.g., 10) will be more conservative. | 40 +| spring.ai.ollama.chat.options.top-p | Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. | 0.9 +| spring.ai.ollama.chat.options.tfs-z | Tail-free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting. | 1 +| spring.ai.ollama.chat.options.typical-p | ??? | - +| spring.ai.ollama.chat.options.repeat-last-n | Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx) | 64 +| spring.ai.ollama.chat.options.temperature | The temperature of the model. Increasing the temperature will make the model answer more creatively. | 0.8 +| spring.ai.ollama.chat.options.repeat-penalty | Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. | 1.1 +| spring.ai.ollama.chat.options.presence-penalty | ??? | - +| spring.ai.ollama.chat.options.frequency-penalty | ??? | - +| spring.ai.ollama.chat.options.mirostat | Enable Mirostat sampling for controlling perplexity. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) | 0 +| spring.ai.ollama.chat.options.mirostat-tau | Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. | 0.1 +| spring.ai.ollama.chat.options.mirostat-eta | Controls the balance between coherence and diversity of the output. A lower value will result in more focused and coherent text. | 5.0 +| spring.ai.ollama.chat.options.penalize-newline | ??? | - +| spring.ai.ollama.chat.options.stop | Sets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile. | - +|==== + + +The prefix `spring.ai.ollama.embedding` is the property prefix that configures the `EmbeddingClient` implementation for Ollama. + + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.ollama.embedding.enabled | Enable the Ollama Chat Client | true +| spring.ai.ollama.embedding.model | The name of the model to use | llama2 +| spring.ai.ollama.embedding.options.numa | Whether to use NUMA. | false +| spring.ai.ollama.embedding.options.num-ctx | Sets the size of the context window used to generate the next token. | 2048 +| spring.ai.ollama.embedding.options.num-batch | ??? | - +| spring.ai.ollama.embedding.options.num-gqa | The number of GQA groups in the transformer layer. Required for some models, for example, it is 8 for llama2:70b. | - +| spring.ai.ollama.embedding.options.num-gpu | The number of layers to send to the GPU(s). On macOS it defaults to 1 to enable metal support, 0 to disable. | - +| spring.ai.ollama.embedding.options.main-gpu | ??? | - +| spring.ai.ollama.embedding.options.low-vram | ??? | - +| spring.ai.ollama.embedding.options.f16-kv | ??? | - +| spring.ai.ollama.embedding.options.logits-all | ??? | - +| spring.ai.ollama.embedding.options.vocab-only | ??? | - +| spring.ai.ollama.embedding.options.use-mmap | ??? | - +| spring.ai.ollama.embedding.options.use-mlock | ??? | - +| spring.ai.ollama.embedding.options.embedding-only | ??? | - +| spring.ai.ollama.embedding.options.rope-frequency-base | ??? | - +| spring.ai.ollama.embedding.options.rope-frequency-scale | ??? | - +| spring.ai.ollama.chat.options.num-thread | Sets the number of threads to use during computation. By default, Ollama will detect this for optimal performance. It is recommended to set this value to the number of physical CPU cores your system has (as opposed to the logical number of cores). | - +| spring.ai.ollama.embedding.options.num-keep | ??? | - +| spring.ai.ollama.embedding.options.seed | Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. | 0 +| spring.ai.ollama.embedding.options.num-predict | Maximum number of tokens to predict when generating text. (Default: 128, -1 = infinite generation, -2 = fill context) | 128 +| spring.ai.ollama.embedding.options.top-k | Reduces the probability of generating nonsense. A higher value (e.g., 100) will give more diverse answers, while a lower value (e.g., 10) will be more conservative. | 40 +| spring.ai.ollama.embedding.options.top-p | Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. | 0.9 +| spring.ai.ollama.embedding.options.tfs-z | Tail-free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting. | 1 +| spring.ai.ollama.embedding.options.typical-p | ??? | - +| spring.ai.ollama.embedding.options.repeat-last-n | Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx) | 64 +| spring.ai.ollama.embedding.options.temperature | The temperature of the model. Increasing the temperature will make the model answer more creatively. | 0.8 +| spring.ai.ollama.embedding.options.repeat-penalty | Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. | 1.1 +| spring.ai.ollama.embedding.options.presence-penalty | ??? | - +| spring.ai.ollama.embedding.options.frequency-penalty | ??? | - +| spring.ai.ollama.embedding.options.mirostat | Enable Mirostat sampling for controlling perplexity. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) | 0 +| spring.ai.ollama.embedding.options.mirostat-tau | Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. | 0.1 +| spring.ai.ollama.embedding.options.mirostat-eta | Controls the balance between coherence and diversity of the output. A lower value will result in more focused and coherent text. | 5.0 +| spring.ai.ollama.embedding.options.penalize-newline | ??? | - +| spring.ai.ollama.embedding.options.stop | Sets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile. | - +|==== \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/openai.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/openai.adoc new file mode 100644 index 000000000..3c665a85c --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/clients/openai.adoc @@ -0,0 +1,103 @@ += OpenAI + +Spring AI supports ChatGPT, the AI language model by OpenAI. ChatGPT has been instrumental in sparking interest in AI-driven text generation, thanks to its creation of industry-leading text generation models and embeddings. + +== Getting Started + +You will need to create an API with OpenAI to access ChatGPT models. + +Create an account at https://platform.openai.com/signup[OpenAI signup page] and generate the token on the https://platform.openai.com/account/api-keys[API Keys page]. + +The Spring AI project defines a configuration property named `spring.ai.openai.api-key` that you should set to the value of the `API Key` obtained from openai.com. + +Exporting an environment variable is one way to set that configuration property: + +[source,shell] +---- +export SPRING_AI_OPENAI_API_KEY= +---- + +== Project Dependencies + +Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file. + +Then add the Spring Boot Starter dependency to your project's Maven `pom.xml` build file: + +[source, xml] +---- + + org.springframework.ai + spring-ai-openai-spring-boot-starter + 0.8.0-SNAPSHOT + +---- + +or to your Gradle `build.gradle` build file. + +[source,groovy] +---- +dependencies { + implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter:0.8.0-SNAPSHOT' +} +---- + + +== Sample Code + +This will create a `ChatClient` implementation that you can inject into your class. +Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation. + +[source,java] +---- +@RestController +public class ChatController { + + private final ChatClient chatClient; + + @Autowired + public ChatController(ChatClient chatClient) { + this.chatClient = chatClient; + } + + @GetMapping("/ai/generate") + public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) { + return Map.of("generation", chatClient.generate(message)); + } +} +---- + +== OpenAI Properties + +The prefix `spring.ai.openai` is used as the property prefix that lets you connect to OpenAI. + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.openai.base-url | The URL to connect to | https://api.openai.com" +| spring.ai.openai.model | The model to use | gpt-3.5-turbo +| spring.ai.openai.api-key | The API Key | - +|==== + +The prefix `spring.ai.openai.chat` is the property prefix that lets you configure the `ChatClient` implementation for OpenAI. + +[cols="3,5,3"] +|==== +| Property | Description | Default + +| spring.ai.azure.openai.chat.model | This is the value of the 'Deployment Name' as presented in the Azure AI Portal | gpt-35-turbo +| spring.ai.azure.openai.chat.temperature | The sampling temperature to use that controls the apparent creativity of generated completions. Higher values will make output more random while lower values will make results more focused and deterministic. It is not recommended to modify temperature and top_p for the same completions request as the interaction of these two settings is difficult to predict. | 0.7 +| spring.ai.azure.openai.chat.top-p | An alternative to sampling with temperature called nucleus sampling. This value causes the model to consider the results of tokens with the provided probability mass. As an example, a value of 0.15 will cause only the tokens comprising the top 15% of probability mass to be considered. It is not recommended to modify temperature and top_p for the same completions request as the interaction of these two settings is difficult to predict. | - +|==== + + +The prefix `spring.ai.openai.embedding` is property prefix that configures the `EmbeddingClient` implementation for OpenAI. + + +[cols="3,5,3"] +|==== +| Property | Description | Default +| spring.ai.openai.embedding.base-url | The URL to connect to | https://api.openai.com" +| spring.ai.openai.embedding.model | The model to use | text-embedding-ada-002 +| spring.ai.openai.embedding.api-key | The API Key | - +|==== \ No newline at end of file diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/getting-started.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/getting-started.adoc index 145f78964..fc3f956a6 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/getting-started.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/getting-started.adoc @@ -1,48 +1,27 @@ [[getting-started]] = Getting Started -This section offers quick guidance on how to get started with using Spring AI. +This section offers jumping off points for how to get started using Spring AI. -== API Keys +== Dependency Management -To use OpenAI or Azure OpenAI, you need to generate an API key. +The Spring AI project provides artifacts in the Spring Milestone and Snapshot repositories. -=== OpenAI +You need to add configuration to add a reference to the Spring Milestone or Snapshot repository in your build file. -Create an account at https://platform.openai.com/signup[OpenAI signup page] and generate the token on the https://platform.openai.com/account/api-keys[API Keys page]. - -The Spring AI project defines a configuration property named `spring.ai.openai.api-key` that you should set to the value of the `API Key` obtained from openai.com. - -Exporting an environment variable is one way to set that configuration property: - -[source,shell] ----- -export SPRING_AI_OPENAI_API_KEY= ----- - -=== Azure OpenAI -Obtain your Azure OpenAI `endpoint` and `api-key` from the Azure OpenAI Service section on the link:https://portal.azure.com[Azure Portal]. - -The Spring AI project defines a configuration property named `spring.ai.azure.openai.api-key` that you should set to the value of the `API Key` obtained from Azure. -There is also a configuration property named `spring.ai.azure.openai.endpoint` that you should set to the endpoint URL obtained when provisioning your model in Azure. - -Exporting environment variables is one way to set these configuration properties: - -[source,shell] ----- -export SPRING_AI_AZURE_OPENAI_API_KEY= -export SPRING_AI_AZURE_OPENAI_ENDPOINT= ----- - -== Dependencies - -The Spring AI project provides artifacts in the Spring Milestone Repository. -You need to add configuration to add a reference to the Spring Milestone repository in your build file. -For example, in Maven, add the following repository definition: +For Maven, add the following repository definitions as needed: [source,xml] ---- + + spring-milestones + Spring Milestones + https://repo.spring.io/milestone + + false + + spring-snapshots Spring Snapshots @@ -54,27 +33,31 @@ For example, in Maven, add the following repository definition: ---- -Add the Spring Boot Starter, depending on whether you use Azure Open AI or Open AI: +For Gradle, add the following repository definitions as needed: -* Azure OpenAI -[source, xml] +[source,groovy] ---- - - org.springframework.ai - spring-ai-azure-openai-spring-boot-starter - 0.8.0-SNAPSHOT - +repositories { + mavenCentral() + maven { url 'https://repo.spring.io/milestone' } + maven { url 'https://repo.spring.io/snapshot' } +} ---- -* OpenAI -[source, xml] ----- - - org.springframework.ai - spring-ai-openai-spring-boot-starter - 0.8.0-SNAPSHOT - ----- + + +== Chat Models + +* xref:api/clients/openai.adoc#_getting_started[OpenAI] +* xref:api/clients/azure-openai.adoc#_gettting_started[Azure OpenAI] +* xref:api/clients/huggingface.adoc#_getting_started[HuggingFace] +* xref:api/clients/bedrock.adoc +* xref:api/clients/ollama.adoc#_getting_started[Ollama] +* + +== Embedding Models + + == Spring CLI