Ollama: Pull models automatically at startup
* Introduce support for Ollama model auto-pull at startup time * Enhance support for Ollama model auto-pull at run time * Update documentation about integrating with Ollama and managing models * Adopt Builder pattern in Ollama Model classes for better code readability * Unify Ollama model auto-pull functionality in production and test code * Improve integration tests for Ollama with Testcontainers
This commit is contained in:
Thomas Vitale
committed by
Christian Tzolov
Christian Tzolov
parent
5dfdd8ddcd
commit
8eef6e6da5
@@ -8,12 +8,27 @@ The xref:_openai_api_compatibility[OpenAI API compatibility] section explains ho
|
||||
|
||||
== Prerequisites
|
||||
|
||||
You first need to xref:https://ollama.com/download[Download and install Ollama] on your local machine.
|
||||
You first need access to an Ollama instance. There are a few options, including the following:
|
||||
|
||||
Also you can pull the models you want to use from the xref:https://ollama.com/library[Ollama model repository]: `ollama pull <desired model name>`.
|
||||
Alternatively, you can enable the `pullMissingModel` option to automatically download missing models: xref:auto-pulling-models[Auto-pulling Models].
|
||||
* xref:https://ollama.com/download[Download and install Ollama] on your local machine.
|
||||
* Configure and xref:api/testcontainers.adoc[run Ollama via Testcontainers].
|
||||
* Bind to an Ollama instance via xref:api/cloud-bindings.adoc[Kubernetes Service Bindings].
|
||||
|
||||
TIP: you can also pull, by name, any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF HuggingFace Models]
|
||||
You can pull the models you want to use in your application from the xref:https://ollama.com/library[Ollama model library]:
|
||||
|
||||
[source,shellscript]
|
||||
----
|
||||
ollama pull <model-name>
|
||||
----
|
||||
|
||||
You can also pull any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF Hugging Face Models]:
|
||||
|
||||
[source,shellscript]
|
||||
----
|
||||
ollama pull hf.co/<username>/<model-repository>
|
||||
----
|
||||
|
||||
Alternatively, you can enable the option to download automatically any needed model: xref:auto-pulling-models[Auto-pulling Models].
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
@@ -39,18 +54,29 @@ dependencies {
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
|
||||
=== Chat Properties
|
||||
=== Base Properties
|
||||
|
||||
The prefix `spring.ai.ollama` is the property prefix to configure the connection to Ollama.
|
||||
|
||||
[cols="3,6,1", stripes=even]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.ollama.base-url | Base URL where Ollama API server is running. | `http://localhost:11434`
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.ollama.chat.options` is the property prefix that configures the Ollama chat model .
|
||||
Here are the properties for initializing the Ollama integration and xref:auto-pulling-models[auto-pulling models].
|
||||
|
||||
[cols="3,6,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
| spring.ai.ollama.init.pull-model-strategy | Whether to pull models at startup-time and how. | `never`
|
||||
| spring.ai.ollama.init.timeout | How long to wait for a model to be pulled. | `5m`
|
||||
| spring.ai.ollama.init.max-retries | Maximum number of retries for the model pull operation. | `0`
|
||||
|====
|
||||
|
||||
=== Chat Properties
|
||||
|
||||
The prefix `spring.ai.ollama.chat.options` is the property prefix that configures the Ollama chat model.
|
||||
It includes the Ollama request (advanced) parameters such as the `model`, `keep-alive`, and `format` as well as the Ollama model `options` properties.
|
||||
|
||||
Here are the advanced request parameter for the Ollama chat model:
|
||||
@@ -58,12 +84,11 @@ Here are the advanced request parameter for the Ollama chat model:
|
||||
[cols="3,6,1", stripes=even]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.ollama.chat.enabled | Enable Ollama chat model. | true
|
||||
| spring.ai.ollama.chat.options.model | The name of the https://github.com/ollama/ollama?tab=readme-ov-file#model-library[supported model] to use. | mistral
|
||||
| spring.ai.ollama.chat.options.pull-missing-model | Automatically pull missing models from Ollama repository | false
|
||||
| spring.ai.ollama.chat.options.format | The format to return a response in. Currently, the only accepted value is `json` | -
|
||||
| spring.ai.ollama.chat.options.keep_alive | Controls how long the model will stay loaded into memory following the request | 5m
|
||||
| spring.ai.ollama.chat.options.pull-model-strategy | Strategy for pulling models at run-time. | `never`
|
||||
|====
|
||||
|
||||
The remaining `options` properties are based on the link:https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values[Ollama Valid Parameters and Values] and link:https://github.com/ollama/ollama/blob/main/api/types.go[Ollama Types]. The default values are based on the link:https://github.com/ollama/ollama/blob/b538dc3858014f94b099730a592751a5454cab0a/api/types.go#L364[Ollama Types Defaults].
|
||||
@@ -101,7 +126,7 @@ The remaining `options` properties are based on the link:https://github.com/olla
|
||||
| spring.ai.ollama.chat.options.penalize-newline | - | true
|
||||
| spring.ai.ollama.chat.options.stop | Sets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile. | -
|
||||
| spring.ai.ollama.chat.options.functions | List of functions, identified by their names, to enable for function calling in a single prompt requests. Functions with those names must exist in the functionCallbacks registry. | -
|
||||
| spring.ai.ollama.chat.options.proxy-tool-calls | If true, the Spring AI will not handle the function calls internally, but will proxy them to the client. Then is the client's responsibility to handle the function calls, dispatch them to the appropriate function, and return the results. If false (the default), the Spring AI will handle the function calls internally. Applicable only for chat models with function calling support | false
|
||||
| spring.ai.ollama.chat.options.proxy-tool-calls | If true, the Spring AI will not handle the function calls internally, but will proxy them to the client. Then is the client's responsibility to handle the function calls, dispatch them to the appropriate function, and return the results. If false (the default), the Spring AI will handle the function calls internally. Applicable only for chat models with function calling support | false
|
||||
|====
|
||||
|
||||
TIP: All properties prefixed with `spring.ai.ollama.chat.options` can be overridden at runtime by adding request-specific <<chat-options>> to the `Prompt` call.
|
||||
@@ -130,28 +155,57 @@ ChatResponse response = chatModel.call(
|
||||
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/ChatOptions.java[ChatOptions] instance, created with https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
|
||||
|
||||
[[auto-pulling-models]]
|
||||
=== Auto-pulling Models
|
||||
== Auto-pulling Models
|
||||
|
||||
The `pullMissingModel` option allows you to automatically download and use models that are not currently available on your local Ollama instance.
|
||||
Spring AI Ollama can automatically pull models when not available in your Ollama instance.
|
||||
This feature is particularly useful when working with different models or when deploying your application to new environments.
|
||||
|
||||
To enable auto-pulling of missing models, you can set the `pullMissingModel` option to `true` in your `OllamaOptions`:
|
||||
TIP: you can also pull, by name, any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF Hugging Face Models].
|
||||
|
||||
There are three strategies for pulling models:
|
||||
|
||||
* `always` (defined in `PullModelStrategy.ALWAYS`). Always pull the model, even if it's already available. Useful to ensure you're using the latest version of that model.
|
||||
* `when_missing` (defined in `PullModelStrategy.WHEN_MISSING`). Only pull the model if it's not already available. It might be an older version of the model.
|
||||
* `never` (defined in `PullModelStrategy.NEVER`). Never pull the model.
|
||||
|
||||
CAUTION: Due to the unexpected delays while downloading models, this feature is not recommended for production environments. Instead, consider to assess and pre-download the necessary models in advance.
|
||||
|
||||
=== Pulling models at startup time
|
||||
|
||||
All models defined via configuration properties and default options can be automatically pulled at startup time.
|
||||
You can configure strategy, timeout, and max number of retries via configuration properties.
|
||||
|
||||
[source,yaml]
|
||||
----
|
||||
spring:
|
||||
ai:
|
||||
ollama:
|
||||
init:
|
||||
pull-model-strategy: always
|
||||
timeout: 60s
|
||||
max-retries: 1
|
||||
----
|
||||
|
||||
CAUTION: The application will not complete its initialization until all the models become available in Ollama. Depending on the model size and the speed of the Internet connection, your application might be slow at starting up.
|
||||
|
||||
=== Pulling models at runtime
|
||||
|
||||
To enable auto-pulling of models at runtime, you can configure the `pullModelStrategy` option in your `OllamaOptions`:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
OllamaOptions options = OllamaOptions.builder()
|
||||
.withModel("all-minilm:latest")
|
||||
.withPullMissingModel(true)
|
||||
.build();
|
||||
ChatResponse response = chatModel.call(new Prompt(
|
||||
"Generate the names of 5 famous pirates.",
|
||||
OllamaOptions.builder()
|
||||
.withModel("llama3.2")
|
||||
.withPullModelStrategy(PullModelStrategy.ALWAYS)
|
||||
.build()
|
||||
));
|
||||
----
|
||||
|
||||
TIP: you can also pull, by name, any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF HuggingFace Models]
|
||||
You can also configure this option using the following property: `spring.ai.ollama.chat.options.pull-model-strategy=always`.
|
||||
|
||||
You can also configure this option using the following property: `spring.ai.ollama.chat.options.pull-missing-model=true`
|
||||
|
||||
When `pullMissingModel` is set to `true`, the system will attempt to download the specified model if it's not already available locally. This process may take some time depending on the size of the model and your internet connection speed.
|
||||
|
||||
CAUTION: Be aware that enabling this option may lead to unexpected delays in your application if it needs to download large model files. It's recommended to pre-download commonly used models in production environments.
|
||||
CAUTION: The time to process an incoming request might incur unexpected delays, waiting for the needed model to become available in Ollama. Depending on the model size and the speed of the Internet connection, your application might be slow at processing requests.
|
||||
|
||||
== Function Calling
|
||||
|
||||
|
||||
@@ -11,10 +11,27 @@ TIP: you can also pull, by name, any of the thousands, free, xref:https://huggin
|
||||
|
||||
== Prerequisites
|
||||
|
||||
You first need to xref:https://ollama.com/download[Download and install Ollama] on your local machine.
|
||||
You first need access to an Ollama instance. There are a few options, including the following:
|
||||
|
||||
Also you can pull the models you want to use from the https://ollama.com/search?c=embedding[Ollama Embedding Models]: `ollama pull <selected embedding model name>`.
|
||||
Alternatively, you can enable the `pullMissingModel` option to automatically download missing models: xref:auto-pulling-models[Auto-pulling Models].
|
||||
* xref:https://ollama.com/download[Download and install Ollama] on your local machine.
|
||||
* Configure and xref:api/testcontainers.adoc[run Ollama via Testcontainers].
|
||||
* Bind to an Ollama instance via xref:api/cloud-bindings.adoc[Kubernetes Service Bindings].
|
||||
|
||||
You can pull the models you want to use in your application from the https://ollama.com/search?c=embedding[Ollama model library]:
|
||||
|
||||
[source,shellscript]
|
||||
----
|
||||
ollama pull <model-name>
|
||||
----
|
||||
|
||||
You can also pull any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF Hugging Face Models]:
|
||||
|
||||
[source,shellscript]
|
||||
----
|
||||
ollama pull hf.co/<username>/<model-repository>
|
||||
----
|
||||
|
||||
Alternatively, you can enable the option to download automatically any needed model: xref:auto-pulling-models[Auto-pulling Models].
|
||||
|
||||
== Auto-configuration
|
||||
|
||||
@@ -41,18 +58,29 @@ dependencies {
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
Spring AI artifacts are published in Spring Milestone and Snapshot repositories. Refer to the Repositories section to add these repositories to your build system.
|
||||
|
||||
=== Embedding Properties
|
||||
=== Base Properties
|
||||
|
||||
The prefix `spring.ai.ollama` is the property prefix to configure the connection to Ollama
|
||||
|
||||
[cols="3,6,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
|
||||
| spring.ai.ollama.base-url | Base URL where Ollama API server is running. | `http://localhost:11434`
|
||||
|====
|
||||
|
||||
The prefix `spring.ai.ollama.embedding.options` is the property prefix that configures the Ollama embedding model .
|
||||
Here are the properties for initializing the Ollama integration and xref:auto-pulling-models[auto-pulling models].
|
||||
|
||||
[cols="3,6,1"]
|
||||
|====
|
||||
| Property | Description | Default
|
||||
| spring.ai.ollama.init.pull-model-strategy | Whether to pull models at startup-time and how. | `never`
|
||||
| spring.ai.ollama.init.timeout | How long to wait for a model to be pulled. | `5m`
|
||||
| spring.ai.ollama.init.max-retries | Maximum number of retries for the model pull operation. | `0`
|
||||
|====
|
||||
|
||||
=== Embedding Properties
|
||||
|
||||
The prefix `spring.ai.ollama.embedding.options` is the property prefix that configures the Ollama embedding model.
|
||||
It includes the Ollama request (advanced) parameters such as the `model`, `keep-alive`, and `truncate` as well as the Ollama model `options` properties.
|
||||
|
||||
Here are the advanced request parameter for the Ollama embedding model:
|
||||
@@ -63,9 +91,9 @@ Here are the advanced request parameter for the Ollama embedding model:
|
||||
| spring.ai.ollama.embedding.enabled | Enables the Ollama embedding model auto-configuration. | true
|
||||
| spring.ai.ollama.embedding.options.model | The name of the https://github.com/ollama/ollama?tab=readme-ov-file#model-library[supported model] to use.
|
||||
You can use dedicated https://ollama.com/search?c=embedding[Embedding Model] types | mistral
|
||||
| spring.ai.ollama.embedding.options.pull-missing-model | Automatically pull missing models from Ollama repository | false
|
||||
| spring.ai.ollama.embedding.options.keep_alive | Controls how long the model will stay loaded into memory following the request | 5m
|
||||
| spring.ai.ollama.embedding.options.truncate | Truncates the end of each input to fit within context length. Returns error if false and context length is exceeded. | true
|
||||
| spring.ai.ollama.embedding.options.pull-model-strategy | Strategy for pulling models at run-time. | `never`
|
||||
|====
|
||||
|
||||
The remaining `options` properties are based on the link:https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values[Ollama Valid Parameters and Values] and link:https://github.com/ollama/ollama/blob/main/api/types.go[Ollama Types]. The default values are based on: link:https://github.com/ollama/ollama/blob/b538dc3858014f94b099730a592751a5454cab0a/api/types.go#L364[Ollama type defaults].
|
||||
@@ -129,31 +157,56 @@ EmbeddingResponse embeddingResponse = embeddingModel.call(
|
||||
----
|
||||
|
||||
[[auto-pulling-models]]
|
||||
=== Auto-pulling Models
|
||||
== Auto-pulling Models
|
||||
|
||||
The `pullMissingModel` option allows you to automatically download and use models that are not currently available on your local Ollama instance.
|
||||
Spring AI Ollama can automatically pull models when not available in your Ollama instance.
|
||||
This feature is particularly useful when working with different models or when deploying your application to new environments.
|
||||
|
||||
To enable auto-pulling of missing models, you can set the `pullMissingModel` option to `true` in your `OllamaOptions`:
|
||||
TIP: you can also pull, by name, any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF Hugging Face Models].
|
||||
|
||||
There are three strategies for pulling models:
|
||||
|
||||
* `always` (defined in `PullModelStrategy.ALWAYS`). Always pull the model, even if it's already available. Useful to ensure you're using the latest version of that model.
|
||||
* `when_missing` (defined in `PullModelStrategy.WHEN_MISSING`). Only pull the model if it's not already available. It might be an older version of the model.
|
||||
* `never` (defined in `PullModelStrategy.NEVER`). Never pull the model.
|
||||
|
||||
CAUTION: Due to the unexpected delays while downloading models, this feature is not recommended for production environments. Instead, consider to assess and pre-download the necessary models in advance.
|
||||
|
||||
=== Pulling models at startup time
|
||||
|
||||
All models defined via configuration properties and default options can be automatically pulled at startup time.
|
||||
You can configure strategy, timeout, and max number of retries via configuration properties.
|
||||
|
||||
[source,yaml]
|
||||
----
|
||||
spring:
|
||||
ai:
|
||||
ollama:
|
||||
init:
|
||||
pull-model-strategy: always
|
||||
timeout: 60s
|
||||
max-retries: 1
|
||||
----
|
||||
|
||||
CAUTION: The application will not complete its initialization until all the models become available in Ollama. Depending on the model size and the speed of the Internet connection, your application might be slow at starting up.
|
||||
|
||||
=== Pulling models at runtime
|
||||
|
||||
To enable auto-pulling of models at runtime, you can configure the `pullModelStrategy` option in your `OllamaOptions`:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
EmbeddingResponse embeddingResponse = embeddingModel
|
||||
.call(new EmbeddingRequest(List.of("Hello World", "Something else"),
|
||||
OllamaOptions.builder()
|
||||
.withModel("all-minilm:latest")
|
||||
.withPullMissingModel(true)
|
||||
.withTruncate(false)
|
||||
.withModel("all-minilm")
|
||||
.withPullModelStrategy(PullModelStrategy.ALWAYS)
|
||||
.build()));
|
||||
----
|
||||
|
||||
TIP: you can also pull, by name, any of the thousands, free, xref:https://huggingface.co/models?library=gguf&sort=trending[GGUF HuggingFace Models]
|
||||
You can also configure this option using the following property: `spring.ai.ollama.embedding.options.pull-model-strategy=always`.
|
||||
|
||||
You can also configure this option using the following property: `spring.ai.ollama.embedding.options.pull-missing-model=true`
|
||||
|
||||
When `pullMissingModel` is set to `true`, the system will attempt to download the specified model if it's not already available locally. This process may take some time depending on the size of the model and your internet connection speed.
|
||||
|
||||
CAUTION: Be aware that enabling this option may lead to unexpected delays in your application if it needs to download large model files. It's recommended to pre-download commonly used models in production environments.
|
||||
CAUTION: The time to process an incoming request might incur unexpected delays, waiting for the needed model to become available in Ollama. Depending on the model size and the speed of the Internet connection, your application might be slow at processing requests.
|
||||
|
||||
== Sample Controller
|
||||
|
||||
|
||||
Reference in New Issue
Block a user