Update ollama chat/embedding documentations
This commit is contained in:
Christian Tzolov
@@ -3,86 +3,14 @@
|
||||
With https://ollama.ai/[Ollama] you can run various Large Language Models (LLMs) locally and generate text from them.
|
||||
Spring AI supports the Ollama text generation with `OllamaChatClient`.
|
||||
|
||||
== Getting Started
|
||||
== Prerequisites
|
||||
|
||||
You first need to run Ollama on your local machine.
|
||||
|
||||
Refer to the official Ollama project link:https://github.com/jmorganca/ollama[README] to get started running models on your local machine.
|
||||
|
||||
Note, installing `ollama run llama2` will download a 4GB docker image.
|
||||
|
||||
=== Configure the Ollama Chat Client Manually
|
||||
|
||||
Add the spring-ai-ollama dependency to your project’s Maven pom.xml file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-ollama</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-ollama:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-ollama` dependency provides access also to the `OllamaEmbeddingClient`.
|
||||
For more information about the `OllamaEmbeddingClient` refer to the link:../embeddings/ollama-embeddings.html[Ollama Embedding Client] section.
|
||||
|
||||
Next, create an `OllamaChatClient` instance and use it to text generations requests:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
var ollamaApi = new OllamaApi();
|
||||
|
||||
var chatClient = new OllamaChatClient(ollamaApi).withModel(MODEL)
|
||||
.withDefaultOptions(OllamaOptions.create()
|
||||
.withModel(OllamaOptions.DEFAULT_MODEL)
|
||||
.withTemperature(0.9f));
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
|
||||
// Or with streaming responses
|
||||
Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
The `OllamaOptions` provides the configuration information for all chat requests.
|
||||
|
||||
==== ChatOptions and OllamaOptions
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides provides configuration information for the chat requests, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
|
||||
The default options can be configured using the `spring.ai.ollama.chat.options` properties as well.
|
||||
|
||||
On start-time use the `OllamaChatClient#withDefaultOptions()` to set the default options applicable for all chat completion requests.
|
||||
At run-time you can override the default options with `OllamaOptions` instance in the request `Prompt`.
|
||||
|
||||
For example to override the default model name and temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OllamaOptions.create()
|
||||
.withModel("llama2")
|
||||
.withTemperature(0.4)
|
||||
));
|
||||
----
|
||||
|
||||
You can use as prompt options any instance that implements the portable `ChatOptions` interface.
|
||||
For example you can use the `ChatOptionsBuilder` to create a portable prompt options.
|
||||
|
||||
=== OllamaChatClient Auto-configuration
|
||||
== Auto-configuration
|
||||
|
||||
Spring AI provides Spring Boot auto-configuration for the Ollama Chat Client.
|
||||
To enable it add the following dependency to your project's Maven `pom.xml` file:
|
||||
@@ -107,32 +35,7 @@ dependencies {
|
||||
|
||||
NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency Management] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
|
||||
==== Sample Code
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
=== Ollama Properties
|
||||
=== Chat Properties
|
||||
|
||||
The prefix `spring.ai.ollama` is the property prefix to configure the connection to Ollama
|
||||
|
||||
@@ -191,3 +94,98 @@ NOTE: The listed properties are based on the https://github.com/jmorganca/ollama
|
||||
|
||||
NOTE: The list of options for chat is to be reviewed. This https://github.com/spring-projects/spring-ai/issues/230[issue] will track progress.
|
||||
|
||||
=== Sample Code
|
||||
|
||||
This will create a `ChatClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `ChatClient` implementation.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class ChatController {
|
||||
|
||||
private final ChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
public ChatController(ChatClient chatClient) {
|
||||
this.chatClient = chatClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/generate")
|
||||
public Map generate(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
return Map.of("generation", chatClient.generate(message));
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
If you don't want to use the Spring Boot auto-configuration, you can manually configure the `OllamaChatClient` in your application.
|
||||
For this add the spring-ai-ollama dependency to your project’s Maven pom.xml file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-ollama</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-ollama:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-ollama` dependency provides access also to the `OllamaEmbeddingClient`.
|
||||
For more information about the `OllamaEmbeddingClient` refer to the link:../embeddings/ollama-embeddings.html[Ollama Embedding Client] section.
|
||||
|
||||
Next, create an `OllamaChatClient` instance and use it to text generations requests:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
var ollamaApi = new OllamaApi();
|
||||
|
||||
var chatClient = new OllamaChatClient(ollamaApi).withModel(MODEL)
|
||||
.withDefaultOptions(OllamaOptions.create()
|
||||
.withModel(OllamaOptions.DEFAULT_MODEL)
|
||||
.withTemperature(0.9f));
|
||||
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
|
||||
// Or with streaming responses
|
||||
Flux<ChatResponse> response = chatClient.stream(
|
||||
new Prompt("Generate the names of 5 famous pirates."));
|
||||
----
|
||||
|
||||
The `OllamaOptions` provides the configuration information for all chat requests.
|
||||
|
||||
=== Chat Options
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides provides configuration information for the chat requests, such as the model to use, the temperature, the frequency penalty, etc.
|
||||
|
||||
The default options can be configured using the `spring.ai.ollama.chat.options` properties as well.
|
||||
|
||||
On start-time use the `OllamaChatClient#withDefaultOptions()` to set the default options applicable for all chat completion requests.
|
||||
At run-time you can override the default options with `OllamaOptions` instance in the request `Prompt`.
|
||||
|
||||
For example to override the default model name and temperature for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
ChatResponse response = chatClient.call(
|
||||
new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OllamaOptions.create()
|
||||
.withModel("llama2")
|
||||
.withTemperature(0.4)
|
||||
));
|
||||
----
|
||||
|
||||
You can use as prompt options any instance that implements the portable `ChatOptions` interface.
|
||||
For example you can use the `ChatOptionsBuilder` to create a portable prompt options.
|
||||
|
||||
@@ -7,7 +7,7 @@ An embedding is a vector (list) of floating point numbers.
|
||||
The distance between two vectors measures their relatedness.
|
||||
Small distances suggest high relatedness and large distances suggest low relatedness.
|
||||
|
||||
== Getting Started
|
||||
== Prerequisites
|
||||
|
||||
You first need to run Ollama on your local machine.
|
||||
|
||||
@@ -15,66 +15,6 @@ Refer to the official Ollama project link:https://github.com/jmorganca/ollama[RE
|
||||
|
||||
Note, installing `ollama run llama2` will download a 4GB docker image.
|
||||
|
||||
=== Configure the Ollama Embedding Client Manually
|
||||
|
||||
Add the spring-ai-ollama dependency to your project’s Maven pom.xml file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-ollama</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-ollama:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-ollama` dependency provides access also to the `OllamaChatClient`.
|
||||
For more information about the `OllamaChatClient` refer to the link:../clients/ollama-chat.html[Ollama Chat Client] section.
|
||||
|
||||
Next, create an `OllamaEmbeddingClient` instance and use it to compute the similarity between two input texts:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
var ollamaApi = new OllamaApi();
|
||||
|
||||
var embeddingClient = new OllamaEmbeddingClient(ollamaApi)
|
||||
.withDefaultOptions(OllamaOptions.create()
|
||||
.withModel(OllamaOptions.DEFAULT_MODEL)
|
||||
.toMap());
|
||||
|
||||
EmbeddingResponse embeddingResponse = embeddingClient
|
||||
.embedForResponse(List.of("Hello World", "World is big and salvation is near"));
|
||||
----
|
||||
|
||||
The `OllamaOptions` provides the configuration information for all embedding requests.
|
||||
|
||||
==== OllamaOptions
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides the Ollama configurations, such as the model to use, the low level GPU and CPU tunning, etc.
|
||||
|
||||
The default options can be configured using the `spring.ai.ollama.embedding.options` properties as well.
|
||||
|
||||
At start-time use the `OllamaEmbeddingClient#withDefaultOptions()` to configure the default options used for all embedding requests.
|
||||
At run-time you can override the default options, using a `OllamaOptions` instance as part of your `EmbeddingRequest`.
|
||||
|
||||
For example to override the default model name for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
EmbeddingResponse embeddingResponse = embeddingClient.call(
|
||||
new EmbeddingRequest(List.of("Hello World", "World is big and salvation is near"),
|
||||
OllamaOptions.create()
|
||||
.withModel("Different-Embedding-Model-Deployment-Name"));
|
||||
----
|
||||
|
||||
=== OllamaEmbeddingClient Auto-configuration
|
||||
|
||||
@@ -104,33 +44,7 @@ NOTE: Refer to the xref:getting-started.adoc#_dependency_management[Dependency M
|
||||
The `spring.ai.ollama.embedding.options.*` properties are used to configure the default options used for all embedding requests.
|
||||
(It is used as `OllamaEmbeddingClient#withDefaultOptions()` instance).
|
||||
|
||||
|
||||
==== Sample Embedding Controller
|
||||
|
||||
This will create a `EmbeddingClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `EmbeddingClient` implementation.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class EmbeddingController {
|
||||
|
||||
private final EmbeddingClient embeddingClient;
|
||||
|
||||
@Autowired
|
||||
public EmbeddingController(EmbeddingClient embeddingClient) {
|
||||
this.embeddingClient = embeddingClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/embedding")
|
||||
public Map embed(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
EmbeddingResponse embeddingResponse = this.embeddingClient.embedForResponse(List.of(message));
|
||||
return Map.of("embedding", embeddingResponse);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Ollama Embedding Properties
|
||||
== Embedding Properties
|
||||
|
||||
The prefix `spring.ai.ollama` is the property prefix to configure the connection to Ollama
|
||||
|
||||
@@ -185,3 +99,90 @@ The prefix `spring.ai.ollama.embedding.options` is the property prefix that conf
|
||||
|====
|
||||
|
||||
NOTE: The `spring.ai.ollama.embedding.options.*` properties are based on the https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values[Ollama Valid Parameters and Values] and https://github.com/jmorganca/ollama/blob/main/api/types.go[Ollama Types]
|
||||
|
||||
=== Sample Controller
|
||||
|
||||
This will create a `EmbeddingClient` implementation that you can inject into your class.
|
||||
Here is an example of a simple `@Controller` class that uses the `EmbeddingClient` implementation.
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@RestController
|
||||
public class EmbeddingController {
|
||||
|
||||
private final EmbeddingClient embeddingClient;
|
||||
|
||||
@Autowired
|
||||
public EmbeddingController(EmbeddingClient embeddingClient) {
|
||||
this.embeddingClient = embeddingClient;
|
||||
}
|
||||
|
||||
@GetMapping("/ai/embedding")
|
||||
public Map embed(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
|
||||
EmbeddingResponse embeddingResponse = this.embeddingClient.embedForResponse(List.of(message));
|
||||
return Map.of("embedding", embeddingResponse);
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
If you are not using Spring Boot, you can manually configure the `OllamaEmbeddingClient`.
|
||||
For this add the spring-ai-ollama dependency to your project’s Maven pom.xml file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-ollama</artifactId>
|
||||
<version>0.8.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-ollama:0.8.0-SNAPSHOT'
|
||||
}
|
||||
----
|
||||
|
||||
NOTE: The `spring-ai-ollama` dependency provides access also to the `OllamaChatClient`.
|
||||
For more information about the `OllamaChatClient` refer to the link:../clients/ollama-chat.html[Ollama Chat Client] section.
|
||||
|
||||
Next, create an `OllamaEmbeddingClient` instance and use it to compute the similarity between two input texts:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
var ollamaApi = new OllamaApi();
|
||||
|
||||
var embeddingClient = new OllamaEmbeddingClient(ollamaApi)
|
||||
.withDefaultOptions(OllamaOptions.create()
|
||||
.withModel(OllamaOptions.DEFAULT_MODEL)
|
||||
.toMap());
|
||||
|
||||
EmbeddingResponse embeddingResponse = embeddingClient
|
||||
.embedForResponse(List.of("Hello World", "World is big and salvation is near"));
|
||||
----
|
||||
|
||||
The `OllamaOptions` provides the configuration information for all embedding requests.
|
||||
|
||||
=== Chat Options
|
||||
|
||||
The https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions.java] provides the Ollama configurations, such as the model to use, the low level GPU and CPU tunning, etc.
|
||||
|
||||
The default options can be configured using the `spring.ai.ollama.embedding.options` properties as well.
|
||||
|
||||
At start-time use the `OllamaEmbeddingClient#withDefaultOptions()` to configure the default options used for all embedding requests.
|
||||
At run-time you can override the default options, using a `OllamaOptions` instance as part of your `EmbeddingRequest`.
|
||||
|
||||
For example to override the default model name for a specific request:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
EmbeddingResponse embeddingResponse = embeddingClient.call(
|
||||
new EmbeddingRequest(List.of("Hello World", "World is big and salvation is near"),
|
||||
OllamaOptions.create()
|
||||
.withModel("Different-Embedding-Model-Deployment-Name"));
|
||||
----
|
||||
|
||||
Reference in New Issue
Block a user