* Created spring-ai-commons module that contains the document classes and Media
* Created spring-ai-document-ingestion containing ETL related classes
* Created spring-ai-chat-document-transformer module containing transformers that utilitze the ChatModel
Started to refactor package names for document ETL classes to have 'document' in their package name.
Major Changes:
- Created new module spring-ai-vector-store from spring-ai-core functionality
- Split advisor functionality into three new modules:
* advisor-memory: Memory-based chat advisors
* advisor-rag: Retrieval Augmentation Generation advisors
* advisor-vector-store: Vector store based advisors
- Refactor builder methods in `ChatResponseMetadata` to remove `with` prefixes and introduce new methods with updated naming conventions.
- Deprecate existing `with*` methods while maintaining backward compatibility.
- Ensure dependent files and tests are compatible with the updated `ChatResponseMetadata` class.
- Deprecate existing ChatOptionsBuilder and its inner class DefaultChatOptions
- Create a new builder interface ChatOptions.Builder for building the Chat options
- Create an explicit DefaultChatOptions
- Create DefaultChatOptionBuilder which can create DefaultChatOptions
- Add javadoc for the deprecated Builder
Resolves#1875
- Fix OpenAI ChatModel's call() operation
- When toolcalling is used, calculate cumulative usage from the preceding ChatResponses
- Fix OpenAI ChatModel's stream() operation
- Make sure that cumulative usage is calculated from the ChatResponse which has a valid usage
- Use overlapping buffer to check and store the usage from the response that holds the usage.
- Add tests for both call() and stream()
The Document class previously allowed multiple media entries while also having a
text field, leading to ambiguity in content handling. This change enforces a
clear separation between text and media documents to prevent content type
confusion and simplify document processing.
A Document now must contain either text content or a single media entry, but
never both. This aligns with the class's primary use in ETL pipelines where
clear content type boundaries are essential for proper embedding generation and
vector database storage.
Additional architectural changes:
- Document now implements a cleaner API by removing deprecated methods
- Removed MediaContent interface implementation from Document class
- Document.getMedia() now returns a single Media object instead of Collection
- Removed EMPTY_TEXT constant in favor of proper null handling
- Constructor signatures simplified and streamlined
- Builder pattern improved to enforce single content type constraint
The breaking changes include:
- Media is now a single entry instead of a collection
- Content field renamed to text for clarity
- Removed support for mixed content types
- Simplified builder API to prevent ambiguous construction
Prefer using text-related methods over deprecated content methods to
better reflect the actual content type being handled and improve API clarity.
- Add BedrockMediaFormat class to handle media format conversions for documents, images and videos
- Enhance Media class with builder pattern and comprehensive format constants
- Refactor BedrockProxyChatModel to support multimodal content handling
- Add integration tests for PDF, image and video processing
- Add unit tests for Media and BedrockMediaFormat classes
- Upgrade AWS SDK version from 2.26.7 to 2.29.29
- Remove redundant aws.sdk.version property in favor of awssdk.version
Documentation updates for Bedrock Converse API
- Added multimodal support documentation (images, video, documents)
- Added deprecation notices for existing Bedrock model implementations
- Updated feature comparison table
- Added warning notes about transitioning to Converse API
- Upgrade Ollama version used for IT from 0.3.14 to 0.5.1
- Fix string comparison pattern in OllamaModelManager
- Re-enable stream function calling test
- Improve multimodal test assertions
- Implement streaming tool call support in OllamaApi and OllamaChatModel
- Add OllamaApiHelper to manage merging of streaming chat response chunks
- Remove @Disabled annotations for streaming function call tests
- Update documentation to reflect new streaming function call capabilities
- Add a new default constructor for ChatResponse
- Update Ollama chat documentation to clarify streaming support requirements
- Deprecated withContent(), withImages(), and withToolCalls() methods
- Replaced with content(), images(), and toolCalls() methods
Add token and duration aggregation for Ollama chat responses
- Modify OllamaChatModel to support accumulating tokens and durations across multiple responses
- Update ChatResponse metadata generation to aggregate usage and duration metrics
- Add tests to verify metadata aggregation behavior
Refactor Ollama duration fields and tests
- Replace Duration fields in OllamaApi.ChatResponse with Long to represent durations in nanoseconds, ensuring precision and compatibility.
- Update methods to convert Long nanoseconds to Duration objects (getTotalDuration, getLoadDuration, getEvalDuration, getPromptEvalDuration).
- Adjust merge logic in OllamaApiHelper to sum Long values for duration fields.
- Modify test cases in OllamaChatModelTests to align with Long duration representation and Duration.ofNanos conversions.
- Add new test class OllamaDurationFieldsTests to validate JSON deserialization and Duration conversion for duration fields.
Resolves#1847
Related to #1800Resolves#1796
Related to #1307
test: Update OllamaWithOpenAiChatModelIT integration tests
- Remove @Disabled annotation for streamFunctionCallTest
- Add inputType for function callback in stream function call test
- Since the Document object's reference to the `embedding` is deprecated and will be removed, the VectorStore implementations require a way to store the embedding of the corresponding Document objects
- One way to fix this is, to have the EmbeddingModel#embed to return the embeddings in the same order as that of the Documents passed to it.
- Since both the Document and embedding collections use the List object, their iteration operation will make sure to keep them in line with the same order.
- A fix is required to preserve the order when batching strategy is applied.
- Updated the Javadoc for BatchingStrategy
- Fixed the Document List order in TokenCountBatchingStrategy
- Refactored the vector store implementations to update this change
Resolves #GH-1826
- Introduced new options for audio output modalities in ChatCompletionRequest
- Added AudioParameters configuration for voice and audio format selection
- Enhanced OpenAiChatModel to handle audio generation and embedding
- Updated AssistantMessage and Media classes to support audio media
- Added integration tests for audio output functionality
- Implemented support for text and audio multi-modal responses
- Updated Spring AI's chat model comparison table to clarify OpenAI's input/output modalities
- Added new configuration properties for audio output:
* spring.ai.openai.chat.options.output-modalities
* spring.ai.openai.chat.options.output-audio
- Extended documentation to explain audio output generation with the gpt-4o-audio-preview model
- Updated Spring Boot configuration metadata to support new audio-related properties
- Included auto-configuration integration test for chat model with audio response generation
Resolves#1841
Extracts shared function callback builder functionality into DefaultCommonCallbackInvokingSpec
base class, reducing code duplication across builder implementations.
Makes FunctionInvokingSpec and MethodInvokingSpec extend CommonCallbackInvokingSpec for better
code organization. Also fixes function/description builder order in Anthropic tests.
- Introduced a common base class for function callback builders to centralize shared logic
- Standardized the order of method chaining for function and description in multiple AI model test classes
- Refactored test cases across various AI model integrations
- Corrected builder method order from .description().function() to .function().description()
and .description().method() to .method().description()
- Updated multiple test files to consistently use .function() before .description()
- Updated documentation examples to reflect new builder method order
- Modified DefaultFunctionCallbackResolver to maintain new builder method order
- Updated DefaultChatClient and ChatClient test classes to reflect new builder pattern
- Simplified callback specification by removing parent spec reference
- Removed cascading getter logic for description, schema type, and other properties
- Minor adjustments to function callback builder and invoking specs
- Introduced `PerplexityWithOpenAiChatModelIT` integration test for Perplexity AI with OpenAI Chat Model.
- Includes various test cases for role-based prompts, streaming responses, token usage validation, and output converters.
- Added tests for function calls and metadata validation.
- Updated Antora navigation (`nav.adoc`) to include Perplexity AI documentation link.
- Enhanced chat model comparison documentation to highlight Perplexity AI integration.
- Added a dedicated `perplexity-chat.adoc` page under `spring-ai-docs` to provide detailed documentation for integrating Perplexity AI.
- Covers API prerequisites, auto-configuration, and runtime options.
- Explains configuration properties such as `spring.ai.openai.base-url`, `spring.ai.openai.chat.model`, and `spring.ai.openai.chat.options.*`.
- Provides examples for environment variable setup and runtime overrides.
- Highlights limitations like lack of multimodal support and explicit function calling.
- Includes a sample Spring Boot controller demonstrating integration usage.
- Links to Perplexity documentation for further reference.
When calling the OpenAiChatModel with ChatOptions, they are always ignored. Options are considered only when using OpenAiChatOptions or FunctionCallingOptions.
Signed-off-by: Thomas Vitale <ThomasVitale@users.noreply.github.com>
- Added support for audio input and output in OpenAI Chat Completion API
- Introduced new audio-related parameters, enums, and record types
- Updated ChatCompletionMessage, ChatCompletionChunk, and related classes
- Added new AudioParameters, AudioOutput, and InputAudio record types
- Implemented method to handle audio media content conversion
- Included new model enum for GPT-4o audio preview
- Extended existing API classes to accommodate audio modalities
- Modified usage tracking and metadata classes to handle audio-specific token details
- Improved ModelOptionsUtils with additional JSON utility methods
Tests:
- Updated test classes to validate audio input and output functionality
- Added integration tests for multimodal audio input with streaming and non-streaming methods
- Created parameterized tests for audio-enabled models
- Enhanced OpenAI API integration tests to cover audio-related scenarios
Docs:
- Updated documentation in spring-ai-docs to explain audio multimodality support
Resolves#1560
- Introduces a new FunctionCallbackResolver interface to define the strategy for
resolving FunctionCallback instances.
- Renames FunctionCallbackContext to DefaultFunctionCallbackResolver to better reflect its
implementation role. Updates all related components to use the new interface.
- Update the affected AI model implementations
- Replaces FunctionCallbackContext parameter with FunctionCallbackResolver in all
model constructors
- Updates builder patterns to use functionCallbackResolver() method instead of
withFunctionCallbackContext()
- Deprecates old withFunctionCallbackContext() methods in builders to guide
migration
- Updates integration tests to use DefaultFunctionCallbackResolver
- Improves documentation to clarify the resolver's role in function callbacks
- Moves SchemaType enum from FunctionCallbackContext to FunctionCallback
Resolves#758
- ChatGenerationMetadata provides Map for additional metadata
- Add Builder interface and DefaultChatGenerationMetadataBuilder
- Add DefaultChatGenerationMetadata implementation
- Update all AI model implementations to use the new builder pattern
- Deprecate ChatGenerationMetadata.from() factory method
- Remove commented out legacy code from ChatGenerationMetadata interface
- Make metadata and content filter collections immutable via Collections.unmodifiableSet()
- Add null validation in DefaultChatGenerationMetadata constructor
- Improve toString() method with more informative output
The commit restructures OpenAI token usage tracking by:
- Adding audio_tokens support in PromptTokensDetails
- Deprecating individual token getter methods in favor of consolidated records
- Introducing new PromptTokensDetails and CompletionTokenDetails records
- Updating tests to reflect the new structure
Resolves#1369 , #1720
Add support for no-argument Supplier and single-argument Consumer function
callbacks in the Spring AI core module. This enhancement allows:
- Registration of Supplier<O> callbacks with no input (Void) type
- Registration of Consumer<I> callbacks with no output (Void) type
- Support for Kotlin Function0 (equivalent to Java Supplier)
- Handle empty properties for Void input types in schema generation
- Enhance FunctionCallback builder to support Supplier/Consumer patterns
Additional changes:
- Add test coverage for both Supplier and Consumer callbacks in various scenarios
- Enhance TypeResolverHelper to support Consumer input type resolution
- Support lambda-style function declarations for improved ergonomics
- Add test cases for void input/output handling in OpenAI chat model
- Include examples of function calls without return values
- Add support for parameterless functions through Supplier interface
Add comprehensive documentation for the FunctionCallback API:
- Overview of the interface and its key methods
- Builder pattern usage with function and method invocation approaches
- Examples for different function types (Function, BiFunction, Supplier, Consumer)
- Best practices and common pitfalls
- Schema generation and customization options
Resolves#1718 , #1277 , #1118, #860
Introduces a simplified, type-safe builder pattern for function callbacks to
improve developer experience and code reliability. The new hierarchical API
separates concerns between direct function invocation and method reflection,
while providing better compile-time safety.
This change deprecates the older FunctionCallbackWrapper in favor of a more
intuitive FunctionCallback.Builder that better handles generic types via
ParameterizedTypeReference. It also adds automatic function description
generation as a fallback when none is provided, though explicit descriptions
are still recommended.
The update standardizes function callback handling across all AI model
implementations (OpenAI, Ollama, Minimax, etc.) and improves response
handling with configurable converters.
Core API Enhancements:
- New Builder Interface: Replaced FunctionCallbackWrapper.builder() with
FunctionCallback.builder(), introducing a hierarchical approach that improves
customization and type safety.
- Specialized Builders: Introduced FunctionInvokerBuilder for direct Function/BiFunction
implementations and MethodInvokerBuilder for reflection-based invocations.
- Generic Type Support: Added ParameterizedTypeReference for better handling of generic parameters.
- Unified Method Definition: Merged method() and argumentTypes() into a single method() call
for simplicity and type safety.
- Automatic Descriptions: Implemented auto-generation of function descriptions, with warnings
to encourage explicit descriptions.
- Configurable Response Converters: Enhanced response handling with support for custom
converters, reducing unnecessary JSON conversions.
Architecture Improvements:
- Established common Builder interface for shared properties
- Separated function object handling from constructor
- Added method-specific configuration (name, arg types, target)
- Added JSON schema generation support for ResolvableType
- Moved to standardized schema types across AI providers
- Set OPEN_API_SCHEMA as default for Vertex AI Gemini
Builder Pattern Standardization:
- Standardized builder method ordering across implementations
- Moved function() call after description() for consistency
- Improved function callback configuration with unified patterns
- Enhanced error handling and validation in DefaultFunctionCallbackBuilder
Deprecations:
- FunctionCallbackWrapper.Builder replaced by DefaultFunctionCallbackBuilder
- Removed CustomizedTypeReference in favor of ParameterizedTypeReference
- Deprecated older ChatClient API methods for function handling
Testing & Documentation:
- Updated all AI model implementations (OpenAI, Ollama, Minimax, Moonshot, ZhiPuAI)
- Added comprehensive integration tests for static/instance methods
- Added integration tests for auto-generated descriptions
- Updated documentation to reflect new builder pattern usage
- Added Kotlin extension for inputType() support
Co-authored-by: Sébastien Deleuze <sebastien.deleuze@broadcom.com>
- Remove deprecated types,methods and references
- Remove usage of "Generation(String text)" and replace with
"Generation(AssistantMessage)"
- Remove MiniMaxApi,MootShotApi,OpenAiApi,ZhiPuAiApi
ChatCompletionFinishReason's FUNCTION_CALL
- Remove OllamaApi's deprecated types
- Remove deprecated constructors from PostgresMlEmbeddingModel
- Remove deprecated constructor from Media
- Remove tokenNames usage from antlr4 FiltersLexer and FiltersParser
- Remove deprecated methods from CassandraVectorStoreProperties
- Remove deprecated constructor and static config class from QdrantVectorStore
- Minor cleanups on removing deprecated models and versions
- Remove old name for mixtral models
Resolves#1599
Query Analysis
* Introduce Query Analysis Module
* Define QueryTransformer API and TranslationQueryTransformer implementation
* Define QueryExpander API and MultiQueryExpander implementation
* Support QueryTransformer in RetrievalAugmentationAdvisor (support for QueryExpander will be in the next PR together with the needed DocumentFuser API).
Improvements
* Refine Retrieval and Augmentation Modules for increased robustness
* Expand test coverage for both modules
* Define clone() method for ChatClient.Builder
Tests
* Introduce “spring-ai-integration-tests” for full-fledged integration tests
* Add integration tests for RAG modules
* Add integration tests for RAG advisor
Query Analysis
* Introduce Query Analysis Module
* Define QueryTransformer API and TranslationQueryTransformer implementation
* Define QueryExpander API and MultiQueryExpander implementation
* Support QueryTransformer in RetrievalAugmentationAdvisor (support for QueryExpander will be in the next PR together with the needed DocumentFuser API).
Improvements
* Refine Retrieval and Augmentation Modules for increased robustness
* Expand test coverage for both modules
* Define clone() method for ChatClient.Builder
Tests
* Introduce “spring-ai-integration-tests” for full-fledged integration tests
* Add integration tests for RAG modules
* Add integration tests for RAG advisor
Relates to #gh-1603
Signed-off-by: Thomas Vitale <ThomasVitale@users.noreply.github.com>
This change enables more flexible integration between Spring AI and LLM function
calling capabilities while maintaining type safety and ease of use.
- Add new MethodFunctionCallback class to support method invocation via reflection
- Supports both static and non-static method calls
- Handles multiple parameter types including primitives, objects, collections
- Supports empty parameters and empty response
- Auto-generates JSON schema from method parameters
- Special handling for ToolContext parameters
- Builder pattern for easy configuration
- Add comprehensive unit tests for MethodFunctionCallback
- Add integration tests for MethodFunctionCallback with both Anthropic and OpenAI clients
- Add jackson-module-jsonSchema dependency
- Modify FunctionCallback to check for empty tool context
Testing coverage includes:
- Static method invocation scenarios
- Non-static method calls with various parameter types
- Void return type methods
- Complex parameter types (enums, records, lists)
- Tool context handling
- Error cases and validation
Add MethodFunctionCallback reference docs
- Verify that the BeanOutputConverter converts with the right order as specified in the JSON schema
Update the BeanOutputConverterTest's test case to tweak the order for validation
XFY Spark and possibly other vendors implement the OpenAI interface but return
tool_calls as a single object rather than an array. Add @JsonFormat annotation
to handle both array and single object responses in ChatCompletionMessage,
making Spring AI more broadly compatible with OpenAI API implementations.
Only ChatCompletionMessage.java required modification to support this alternate
format while maintaining backward compatibility with the standard OpenAI API
response structure.
Co-authored-by: jito <pigberger70@gmail.com>
- Extracts ResponseFormat from being a nested record in OpenAiApi to
a dedicated class with builder pattern support.
- Resolve the issue with constructor bindings for the Boog property
binding.
- Re-enables previously disabled response format integration tests.
- Add checkstyle changes
- Add schema field in ResponseFormat and set jsonSchema via the setter for schema,
this way schema set via a Boot property also sets the correct JsonSchema
- Add default constructors in ResponseFormat and JsonSchema
Resolves#1681
OpenAI's API returns additional token usage metrics that provide deeper
insight into API consumption. This adds support for:
- acceptedPredictionTokens: Tokens from accepted model predictions
- audioTokens: Tokens used for audio processing
- rejectedPredictionTokens: Tokens from rejected model predictions
These fields help track resource utilization and costs more accurately
by breaking down token usage by type. Added @JsonIgnoreProperties to
maintain compatibility with future OpenAI API additions.
Fixes warning logging in RetryUtils.SHORT_RETRY_TEMPLATE to reduce noise
in test output.
The DEFAULT_RETRY_TEMPLATE uses exponential backoff starting at 2s with max 3min
delay between retries, making tests run unnecessarily long. This change
introduces SHORT_RETRY_TEMPLATE with fixed 100ms backoff to speed up test
execution while preserving the same retry behavior and error handling.
- Add a new test case for testing function call with advisor in BedrockConverseChatClientIT
- Add a new test case for testing tool proxy function call in OpenAiChatClientProxyFunctionCallsIT
- Remove unused model property from BedrockConverseProxyChatProperties
