Commit Graph

82 Commits

Author SHA1 Message Date
Eddú Meléndez
8503078088 Add ServiceConnection support for
* ChromaDBContainer
 * MilvusContainer
 * QdrantContainer
 * RedisStackContainer
 * WeaviateContainer
 * OllamaContainer

Add docs
2024-03-18 23:29:55 -04:00
Christian Tzolov
666474caff Minor improvements of the MongoDB vector store impl. 2024-03-18 19:51:54 +01:00
Eddú Meléndez
150b268415 Add MongoDB Atlas vector store auto-configuration
It also provides a starter.
2024-03-18 15:07:34 +01:00
Eddú Meléndez
c5f35029f0 Remove withReuse in MongoDBAtlasContainer
`Reuse Container` is a Testcontainers experimental feature. It requires
`testcontainers.reuse.enable=true` in `~/.testcontainers.properties` in
order to take effect but in order to avoid surprises, this commit remove it.

See https://java.testcontainers.org/features/reuse/
2024-03-16 23:54:59 +01:00
Chris Smith
5f0123cc92 Add MongoDB Atlas Vector store
- Add VectorSearchAggregation used to actually preform the search
   on a given collection with embeddings.
 - add MongoDBVectorStore
 - Add MongoDBVectorStoreIT.  Integration test runs fine given...
   - You have a mongo atlas cluster to connect to (local or remote)
   - You have the search index "spring_ai_vector_search" setup correctly
   - Need to explore getting around this
   - Need to filter results using threshold
 - Add postfilter for threshold values - While a post filter is not ideal,
   it gets the job done. The mongo team seems to be working on having
   it availible as a prefilter option, in which this implementation
   can be updated to use later.
 - implement filtering threshold
 - fix a few sonar issues
 - formatting
 - use higher default num_candidates
 - use builder for configuration
 - add documentation and some refactor
 - use consistent property in integration test
 - finish implementing filter support
 - add documentation to filter converter
 - add vector search index auto creation

 - Add to BOM.
 - Fix version to 1.0.0-SN.
 - Move expresion converter from core to models/mongodb.
 - Fix style and license headers
2024-03-16 12:13:02 +01:00
Gerrit Meier
7c6bbef3ec Allow more customization for Neo4j store (id and constraint).
Unrelated to this change, the Neo4j test version increased to be current.
2024-03-15 12:26:20 +01:00
Mark Pollack
4c617e16b4 Prepare next development iteration 2024-03-12 14:33:45 -04:00
Mark Pollack
490f3cd1cb Milestone Release 0.8.1 2024-03-12 14:28:28 -04:00
Bragolgirith
777b79e1c5 Fix RedisVectorStore not closing Jedis pipelines 2024-03-12 15:15:05 +01:00
Christian Tzolov
ba940395b6 Use platform independent line separators 2024-03-08 09:43:15 +01:00
Mark Pollack
bcb559a82c add license header plugin and update all java files 2024-03-06 17:39:37 -05:00
Christian Tzolov
cbb59ce616 Improve readability of the PgVector Store code
Related to #461
2024-03-03 08:54:21 +01:00
Anush008
0995515f2f Minor code and doc adjustments
- chore: renamed test file
 - fix: BOOL_VALUE ObjectFactory
2024-03-01 11:49:01 +01:00
Christian Tzolov
a8110bb3da Qdrant improvements
- Update docs
 - Allow colleciton auto-creation if missing
 - Add cloud IT : QdrantVectorStoreCloudAutoConfigurationIT
 - Add auto-configuration properties tests: PgVectorStorePropertiesTests
2024-02-29 09:27:19 +01:00
Anush008
ea0b439dac Implement Qdrant vector store
- Implement QdrantVectorStore.
   Uses a custom parser for converting Spring AI metadata(Map<String, Object>) to Qdrant GRPC payload.
 - Implement Qdrant Expression Filter support.
   Uses a custom parser for converting Spring AI filters to Qdrant-compatible GRPC filters.
 - Add ITs using testcontainers.
 - Add antora docs adrant.adoc.
 - Add Qdrant vector store auto-configuraton and boot starter.

Additional (review) change:

 - Fix poms parent to 0.8.1-SNAPSHOT.
 - Rename ObjectFactory into QdrantObjectFactor.
 - Rename ValueFactory into QdrantValueFactory.
 - Move the org.springframework.ai.vectorstore package into org.springframework.ai.vectorstore.qdrant.
 - Add missing Autoconfigure definition.
 - Add missing license and JavaDocs.
 - Minor code style improvmentes.
 - Move the qdrant version to the main pom
 - Add QdrantVectorStoreAutoConfigurationIT
 - Remove guava dependency
 - Improve gdrant.adoc conent and structure.
 - Remove the grpc-protobuf dependency

Resolves #331
2024-02-28 19:31:54 +01:00
Christian Tzolov
65d42c9d4f Add full support or Vertex AI Gemini and Azure OpenAI function calling
- Extend the Spring AI Message with getMediaData() : List<MediaData>
   MediaData is a pair of MimeType and data of type Object.
   Message#getContent() return text only.

 - VertexAI Gemini Support
   - implement VertexAiGeiminChatClient for ChatClient and StreamingChat client and support for MediaData content.
     add IT tests for Chat, Streaming and Multimodality
   - add Auto-configuration + ITs
   - add Gemini Spring Boot starter.
   - add clients and boot starters to the Spring AI BOM.
   - add Anotra documentation for the Gemini chat client.
   - update gemini to latest 26.33.0 BOM.
   - add vertex ai gemini dependencies to the BOM.
   - add Vertex AI Gemini API Function Calling support
   - add Gemini API Function Calling Streaming support
   - add vertex ai gemini function calling documentation
   - factor out the Function Calling functionality into common abstraction used by OpenAI, Azure and Gemini.
   - group the VertexAI documentation under a common parent
   - add PortableFunctionCallingOption that implements FunctionCallingOptions and ChatOptions and provide builder for it.
   - remove some deprecated code.
   - allow authorization with GoogleCredentials form json file.
   - add AOT support for VertexAI Gemini.
   - move legacy Vertex AI into VertexAI PaLM2.
   - better handling for empty chat responses.
   - update the Gemini version to latest 26.33.0. This required lifting the protobuf-java to 3.25.2 as well.
   - fix a bug for handling System messages with Gemini.

- Implement Azure OpenAI Function Calling
  Uses the same the common abstractions used by OpenAI and Gemini: FunctionCallingOptions and AbstractFunctionCallSupport
2024-02-28 15:41:37 +01:00
Eddú Meléndez
6de57d5f37 Use Testcontainers Vector Database modules
- Update testcontainers version to 1.19.6
 - Use ChromaDBContainer
 - Use MilvusContainer
 - Use WeaviateContainer
 - Fix format
2024-02-27 17:30:43 +01:00
Mark Pollack
8532cb7bcc Bump to 0.8.1-SNAPSHOT 2024-02-23 11:11:14 -05:00
Mark Pollack
bc5c4e2c4f add jdbc starter to pgvector starter 2024-02-22 15:48:58 -05:00
Mark Pollack
a107e313fc Fixes #219 - Remove dependency cycles 2024-02-19 12:24:16 -05:00
Christian Tzolov
fdd3fbb2f9 Revert default OpenAI embedding to text-embedding-ada-002 as some vector stores have problems with 3-small 2024-02-16 11:19:34 +01:00
Christian Tzolov
7ef49b8cef Set the default OpenAI Embedding model to text-embedding-3-small
The text-embedding-3-small has the same dimensions as previous text-embedding-ada-002.
  The text-embedding-3-large has higher dimensionality not supported by some Vector Stores.
2024-02-16 10:27:13 +01:00
Gerrit Meier
e62543c761 Use the new vector index creation syntax for Neo4j.
To make the Neo4j module more future-proof, this commit
replaces the old vector index creation syntax with the new style.
Also, the new pattern is in line with the standard Neo4j index creation
and supports the _IF NOT EXISTS_ clause to run idempotent.
This allows us to remove the preceding call to check if the index exists.

As a consequent, the module will require Neo4j to be at least on version 5.15.
2024-02-16 09:40:36 +01:00
Adam Bchouti
36278afdb3 Add Pinecone namespace support for add, search and delete operations 2024-02-14 14:55:21 +01:00
Adam Bchouti
41885a8e62 Enable updating the pinecone vector store namespace 2024-02-14 14:41:09 +01:00
Christian Tzolov
57c66b91d7 Fix a broken test 2024-02-14 09:54:20 +01:00
Gerrit Meier
408337e327 Support filter expressions in Neo4j vector store.
- Move Neo4j filter converters from core to neoj4 vector store project.
 - Update neo4j adoc.

 Resolves #318
2024-02-14 09:23:44 +01:00
Christian Tzolov
f6447f420e Clean obsolate README files 2024-02-14 06:40:35 +01:00
sunshiningsoo
c0a341c300 Remove slash according to api 2024-02-12 16:32:09 +01:00
Gerrit Meier
254b8632cd Allow configuration of vector index name.
At the moment, it is not possible to configure SpringAI
to use an existing index in the database.
This commit enables the user to provide the index name
for auto configuration or builder usage.
2024-02-11 20:35:46 +01:00
Christian Tzolov
0abf1528dd Fix for Redis NPE when using FilterExpression #265
- Part of #265
2024-01-31 11:17:29 +01:00
Christian Tzolov
8f9a1475c1 Re-align the grpc dependecies and exclusions so that Milvus and Pinecone autoconfigurations ITs work 2024-01-26 18:32:55 +01:00
Christian Tzolov
aa8c3856ed Fix Chroma API 0.4.22+ compatiblity issues
- Makes upsert a void metod.
 - Fix affected test.
 - Replac getEmebedding with complex filters test by query Embedding.

 Resolves #224
2024-01-26 12:05:25 +01:00
Christian Tzolov
7262afb14d Capitalize the enumeration names
Resolves #208
2024-01-14 20:36:14 +01:00
Mark Pollack
85e3e0015e Remove client and embedding from package name for ChatClient and EmbeddingClient implementations
Moved up one package level the following classes
* org.springframework.ai.huggingface.client.HuggingfaceChatClient
* org.springframework.ai.openai.client.OpenAiChatClient and org.springframework.ai.openai.embedding.OpenAiEmbeddingClient
* org.springframework.ai.vertex.generation.VertexAiChatClient and org.springframework.ai.vertex.embedding.VertexAiEmbeddingClient

Fixes #211
2024-01-11 12:00:36 -05:00
Mark Pollack
a48a6d9611 Change package name for PostgresML and Transformers
Fixes #133
2024-01-11 10:11:53 -05:00
Giani Segatto
2a604262d8 Use jdbcTemplate.batchUpdate for PgVectorStore add() implementation
- remove unused variable
2024-01-02 13:20:23 -05:00
oujingzhou
b625ab1c15 Add Spring Boot auto-configuration for Neo4j vector store
- resolve Neo4j auto-configuraion property expossing external API.
 - move neo4j auto-conf under the vectorstore parent package.
2023-12-24 09:11:52 +01:00
Omkar Shetkar
dd64a50d2f Merge SimplePersistentVectorStore and InMemoryVector store to be SimpleVectorStore
- Doc updates

Fixes  #146
2023-12-20 15:48:34 -05:00
Mark Pollack
1a8fcac206 fix failing tests due to directory structure changes 2023-12-19 12:46:43 -05:00
Mark Pollack
82fe510b39 Moved Maven modules from top level directory and embedding-clients subdirectory to all be under a single models directory.
Rename artifact ID of

* `transformers-embedding` to `spring-ai-transformers`
2023-12-19 12:22:02 -05:00
Christian Tzolov
0e3192a8df Add AWS Bedrock AI support
- Add spring-ai-bedrock project with support for Cohere, Llama2, Ai21 Jurassic 2, Titan and Anthropic LLM models.
 - Add native API clients for CohereChat CohereEmbedding , Llama2Chat, JurassicChat, TitanChat and TitanEmbedding models, supporting both single shot and streaming completions (for the models that allows it)
 - Add ITs tests for the native API clients.
 - Implement Chat (AiClient) and ChatStreaming (AiStreamingClient) and EmbeddingClients (according to the models’ support for those) for Cohere, Llama2, and Anthropica. Titan and Jurassic2 are WIP.
 - Add ITs for the ChatClient, ChatStreamingClient and EmbeddingClient implementations.
 - Add Spring Boot Auto-configurations with flexible properties for the Llama2, Anthropic and Cohere modes + ITs
 - Add Spring Boot Starter configurations for all Bedrock models.
 - Add README documentations for all models.
 - Add BedrockAi APIs AOT hints
 - Add Ai21Jurassic2ChatBedrockApi, TitanEmbeddingBedrockApi, TitanChatBedrockApi
 - Add TitanChatBedrockApi

Resolves #66
2023-12-18 18:16:43 -05:00
jruaux
a232166cf3 Add support for Redis vector store
- Added autoconfiguration for Redis vector store
- Added spring boot starter for Redis vector store
- Supports portable metadata filter expressions

Fixes #11
2023-12-18 11:50:03 -05:00
Gerrit Meier
e3a1451f19 Replace Neo4j embedding storing with new function.
The old function got replaced and might get removed in future versions.
To ensure that the store is compatible with Neo4j 5.LATEST, this commit
changes the store function for embeddings to use the 5.13+ function.

The correct baseline database version is already mentioned in the README.
2023-12-13 17:55:59 +01:00
Christian Tzolov
e30be94a56 Replace the theokanning with custom OpenAi API client
- Introducing a new OpenAiApi native client for OpenAI API and get rid of the theokanning library.
   Amongst others the OpenAiApi allows:
    - easy base-url configuration (e.g. TAS-AI)
    - Flux response for streaming OpenAI results.
    - Exposes the http headers containing important metadata
    - Pure Spring ecosystem, making it easier for  Graal VM
 - Define a new  AiStreamClient interface returning Flux<AiResponse>
 - Refactor OpenAiClient and to use the new OpenAiApi and implement the AiStreamClient.
 - Use spring-retry to improve the OpenAI EmbeddingClient stability on 503 error.

 - Remove the OpenAI http header interceptor as the OpenAiApi returns ResponseEntity<T> that provides direct access to the headers.
 - Refactor the metadata headers and usage extraction.
 - Remove redundant and obsolete classes.
 - Fix dependency issue with Pinecone, netty-codec-http2 and Spring Boot 3.2
2023-12-13 15:52:03 +01:00
OğuzhanSarisakaloglu
7aad51da7f Fix typo errors in code and readme file. 2023-12-12 18:40:07 -05:00
Eddú Meléndez
f3614268b7 Fix typo 2023-12-12 15:49:49 -05:00
Christian Tzolov
6d3c429e95 remove the occurances of 'experimenta' in the git urls and docs 2023-12-12 17:48:11 +01:00
Christian Tzolov
75cf8bee8c Fix Milvus dependecy issue due to Boot 3.2 upgrade 2023-12-12 13:39:13 +01:00
Mark Pollack
79a0c97e83 update Java package names in remove 'experimental' 2023-12-11 12:25:02 -05:00