Add Ollama Function Call support

- extend the OllamaApi with Tool, ToolCalls and Function and add Tool Role.
- make OllamaChatModel extend the AbstractToolCallSupport.
- extend the OllamaChatModel to convert the Spring AI messages to OllamaApi messages, including Tools.
- OllamaOptions implements FunctionCallingOptions.
- add OllamaApiToolFunctionCallIT for testing function calling.
- patch the AbstractToolCallSupport#isToolCall to take set of stop resons.
- add FunctionCallbackInPromptIT and FunctionCallbackWrapperIT function calling auto-configuration tests.
- extend OllamAutoConfiguration to support function calling registration.
- add function call tests to OllamChatAutoConfigurationIT.
- add OllamaWithOpenAiChatModelIT to OpenAi that uses the OpenAI API to call Ollama.
- move buildToolCallConversation and handleToolCalls to parent AbstractToolCallSupport.
- update Docs. add Ollama function call docs.
- update Ollama diagrams.

Resolves #720
This commit is contained in:
Christian Tzolov
2024-07-24 20:25:14 +02:00
committed by Mark Pollack
parent ee696f33dd
commit d53876a624
24 changed files with 1873 additions and 135 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 379 KiB

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 777 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

View File

@@ -28,6 +28,7 @@
*** xref:api/chat/moonshot-chat.adoc[Moonshot AI]
//// **** xref:api/chat/functions/moonshot-chat-functions.adoc[Function Calling]
*** xref:api/chat/ollama-chat.adoc[Ollama]
**** xref:api/chat/functions/ollama-chat-functions.adoc[Function Calling]
*** xref:api/chat/openai-chat.adoc[OpenAI]
**** xref:api/chat/functions/openai-chat-functions.adoc[Function Calling]
*** xref:api/chat/qianfan-chat.adoc[QianFan]

View File

@@ -0,0 +1,222 @@
= Function Calling
TIP: You need Ollama 0.2.8 or newer.
NOTE: Currently, the Ollama API (0.2.8) does not support function calling in streaming mode.
You can register custom Java functions with the `OllamaChatModel` and have the Ollama deployed model intelligently choose to output a JSON object containing arguments to call one or many of the registered functions.
This allows you to connect the LLM capabilities with external tools and APIs.
The Ollama models tagged with the `Tools` label are trained to detect when a function should be called and to respond with JSON that adheres to the function signature.
The Ollama API does not call the function directly; instead, the model generates JSON that you can use to call the function in your code and return the result back to the model to complete the conversation.
Spring AI provides flexible and user-friendly ways to register and call custom functions.
In general, the custom functions need to provide a function `name`, `description`, and the function call `signature` (as JSON schema) to let the model know what arguments the function expects.
The `description` helps the model to understand when to call the function.
As a developer, you need to implement a function that takes the function call arguments sent from the AI model, and responds with the result back to the model.
Your function can in turn invoke other 3rd party services to provide the results.
Spring AI makes this as easy as defining a `@Bean` definition that returns a `java.util.Function` and supplying the bean name as an option when invoking the `ChatModel`.
Under the hood, Spring wraps your POJO (the function) with the appropriate adapter code that enables interaction with the AI Model, saving you from writing tedious boilerplate code.
The basis of the underlying infrastructure is the link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/model/function/FunctionCallback.java[FunctionCallback.java] interface and the companion link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/model/function/FunctionCallbackWrapper.java[FunctionCallbackWrapper.java] utility class to simplify the implementation and registration of Java callback functions.
== How it works
Suppose we want the AI model to respond with information that it does not have, for example the current temperature at a given location.
We can provide the AI model with metadata about our own functions that it can use to retrieve that information as it processes your prompt.
For example, if during the processing of a prompt, the AI Model determines that it needs additional information about the temperature in a given location, it will start a server side generated request/response interaction.
The AI Model invokes a client side function.
The AI Model provides method invocation details as JSON and it is the responsibility of the client to execute that function and return the response.
The model-client interaction is illustrated in the <<spring-ai-function-calling-flow>> diagram.
Spring AI greatly simplifies the code you need to write to support function invocation.
It brokers the function invocation conversation for you.
You can simply provide your function definition as a `@Bean` and then provide the bean name of the function in your prompt options.
You can also reference multiple function bean names in your prompt.
== Quick Start
Let's create a chatbot that answer questions by calling our own function.
To support the response of the chatbot, we will register our own function that takes a location and returns the current weather in that location.
When the response to the prompt to the model needs to answer a question such as `"Whats the weather like in Boston?"` the AI model will invoke the client providing the location value as an argument to be passed to the function.
This RPC-like data is passed as JSON.
Our function calls some SaaS based weather service API and returns the weather response back to the model to complete the conversation.
In this example we will use a simple implementation named `MockWeatherService` that hard codes the temperature for various locations.
The following `MockWeatherService.java` represents the weather service API:
[source,java]
----
public class MockWeatherService implements Function<Request, Response> {
public enum Unit { C, F }
public record Request(String location, Unit unit) {}
public record Response(double temp, Unit unit) {}
public Response apply(Request request) {
return new Response(30.0, Unit.C);
}
}
----
=== Registering Functions as Beans
With the link:../ollama-chat.html#_auto_configuration[OllamaChatModel Auto-Configuration] you have multiple ways to register custom functions as beans in the Spring context.
We start with describing the most POJO friendly options.
==== Plain Java Functions
In this approach you define `@Beans` in your application context as you would any other Spring managed object.
Internally, Spring AI `ChatModel` will create an instance of a `FunctionCallbackWrapper` wrapper that adds the logic for it being invoked via the AI model.
The name of the `@Bean` is passed as a `ChatOption`.
[source,java]
----
@Configuration
static class Config {
@Bean
@Description("Get the weather in location") // function description
public Function<MockWeatherService.Request, MockWeatherService.Response> weatherFunction1() {
return new MockWeatherService();
}
...
}
----
The `@Description` annotation is optional and provides a function description (2) that helps the model understand when to call the function. It is an important property to set to help the AI model determine what client side function to invoke.
Another option to provide the description of the function is to use the `@JsonClassDescription` annotation on the `MockWeatherService.Request` to provide the function description:
[source,java]
----
@Configuration
static class Config {
@Bean
public Function<Request, Response> currentWeather3() { // (1) bean name as function name.
return new MockWeatherService();
}
...
}
@JsonClassDescription("Get the weather in location") // (2) function description
public record Request(String location, Unit unit) {}
----
It is a best practice to annotate the request object with information such that the generated JSON schema of that function is as descriptive as possible to help the AI model pick the correct function to invoke.
The link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-spring-boot-autoconfigure/src/test/java/org/springframework/ai/autoconfigure/openai/tool/FunctionCallbackWithPlainFunctionBeanIT.java[FunctionCallbackWithPlainFunctionBeanIT.java] demonstrates this approach.
==== FunctionCallback Wrapper
Another way to register a function is to create a `FunctionCallbackWrapper` wrapper like this:
[source,java]
----
@Configuration
static class Config {
@Bean
public FunctionCallback weatherFunctionInfo() {
return FunctionCallbackWrapper.builder(new MockWeatherService())
.withName("CurrentWeather") // (1) function name
.withDescription("Get the weather in location") // (2) function description
.build();
}
...
}
----
It wraps the 3rd party `MockWeatherService` function and registers it as a `CurrentWeather` function with the `OllamaChatModel`.
It also provides a description (2) and an optional response converter (3) to convert the response into a text as expected by the model.
NOTE: By default, the response converter does a JSON serialization of the Response object.
NOTE: The `FunctionCallbackWrapper` internally resolves the function call signature based on the `MockWeatherService.Request` class.
=== Specifying functions in Chat Options
To let the model know and call your `CurrentWeather` function you need to enable it in your prompt requests:
[source,java]
----
OllamaChatModel chatModel = ...
UserMessage userMessage = new UserMessage("What's the weather like in San Francisco, Tokyo, and Paris?");
ChatResponse response = chatModel.call(new Prompt(List.of(userMessage),
OllamaOptions.builder().withFunction("CurrentWeather").build())); // (1) Enable the function
logger.info("Response: {}", response);
----
// NOTE: You can can have multiple functions registered in your `ChatModel` but only those enabled in the prompt request will be considered for the function calling.
Above user question will trigger 3 calls to `CurrentWeather` function (one for each city) and the final response will be something like this:
----
Here is the current weather for the requested cities:
- San Francisco, CA: 30.0°C
- Tokyo, Japan: 10.0°C
- Paris, France: 15.0°C
----
The link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-spring-boot-autoconfigure/src/test/java/org/springframework/ai/autoconfigure/openai/tool/FunctionCallbackWrapperIT.java[FunctionCallbackWrapperIT.java] test demo this approach.
=== Register/Call Functions with Prompt Options
In addition to the auto-configuration you can register callback functions, dynamically, with your Prompt requests:
[source,java]
----
OllamaChatModel chatModel = ...
UserMessage userMessage = new UserMessage("What's the weather like in San Francisco, Tokyo, and Paris?");
var promptOptions = OllamaOptions.builder()
.withFunctionCallbacks(List.of(new FunctionCallbackWrapper<>(
"CurrentWeather", // name
"Get the weather in location", // function description
new MockWeatherService()))) // function code
.build();
ChatResponse response = chatModel.call(new Prompt(List.of(userMessage), promptOptions));
----
NOTE: The in-prompt registered functions are enabled by default for the duration of this request.
This approach allows to dynamically chose different functions to be called based on the user input.
The https://github.com/spring-projects/spring-ai/blob/main/spring-ai-spring-boot-autoconfigure/src/test/java/org/springframework/ai/autoconfigure/ollama/tool/FunctionCallbackInPromptIT.java[FunctionCallbackInPromptIT.java] integration test provides a complete example of how to register a function with the `OllamaChatModel` and use it in a prompt request.
== Appendices:
=== Spring AI Function Calling Flow [[spring-ai-function-calling-flow]]
The following diagram illustrates the flow of the OllamaChatModel Function Calling:
image:ollama-chatmodel-function-call.jpg[width=800, title="OllamaChatModel Function Calling Flow"]
=== OllamaAPI Function Calling Flow
The following diagram illustrates the flow of the Ollama API:
image:ollama-function-calling-flow.jpg[title="Ollama API Function Calling Flow", width=800]
The link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/test/java/org/springframework/ai/ollama/chat/api/tool/OpenAiApiToolFunctionCallIT.java[OllamaApiToolFunctionCallIT.java] provides a complete example on how to use the Ollama API function calling.

View File

@@ -102,6 +102,7 @@ The remaining `options` properties are based on the link:https://github.com/olla
| spring.ai.ollama.chat.options.mirostat-eta | Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. | 0.1
| spring.ai.ollama.chat.options.penalize-newline | ??? | true
| spring.ai.ollama.chat.options.stop | Sets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile. | -
| spring.ai.ollama.chat.options.functions | List of functions, identified by their names, to enable for function calling in a single prompt requests. Functions with those names must exist in the functionCallbacks registry. | -
|====
TIP: All properties prefixed with `spring.ai.ollama.chat.options` can be overridden at runtime by adding a request specific <<chat-options>> to the `Prompt` call.
@@ -129,6 +130,16 @@ ChatResponse response = chatModel.call(
TIP: In addition to the model specific link:https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaOptions.java[OllamaOptions] you can use a portable https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/ChatOptions.java[ChatOptions] instance, created with the https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/chat/prompt/ChatOptionsBuilder.java[ChatOptionsBuilder#builder()].
== Function Calling
You can register custom Java functions with the OllamaChatModel and have the Ollama model intelligently choose to output a JSON object containing arguments to call one or many of the registered functions.
This is a powerful technique to connect the LLM capabilities with external tools and APIs.
Read more about xref:api/chat/functions/ollama-chat-functions.adoc[Ollama Function Calling].
TIP: You need Ollama 0.2.8 or newer.
NOTE: Currently, the Ollama API (0.2.8) does not support function calling in streaming mode.
== Multimodal
Multimodality refers to a model's ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.