Commit Graph

122 Commits

Author SHA1 Message Date
Josh Long
2d43e40024 Add property to initialize schema for vector stores
* Default is fale
* Update docs
2024-05-26 00:31:19 -04:00
skewgod
593e5fd747 docs(qdrant): added additional info to setup api key for qdrant (#736) 2024-05-24 11:36:26 -04:00
Mark Pollack
4a2ad60059 Package refactoring
* Move ChatClient and related classes into the chat.client package
* Move ChatModel and related class into the chat.model package
* Smaller refactorings to remove DSM cycles
* Update README.md
2024-05-23 17:52:27 -04:00
Josh Long
fbfc87e814 Refactoring of ChatClient to add fluent API and introduce Model as dependent object
* Rename the ModelClient class hierarchy into Model:
  - Rename ModelClient into Model. Update all code and doc references.
  - Rename ChatClient to ChatModel. Update all ChatClient suffixes and chatClient fields and variables in code and doc.
  - Rename EmbeddingClient into EmbeddingModel. Update the XxxEmbeddingClient class and variable suffixes and embeddingClient variables and fields in code and docs.
  - Rename ImageClient into ImageModel.
  - Rename SpeechClient into SpeechModel.
  - Rename TranscriptionClient into TranscriptionModel.
  - Update all javadocs and antora pages. Update the related diagrams.

* Create fluent API in ChatClient interface that now includes streaming support
* Add OpenAI FunctionCallbackWrapper2IT auto-config tests.
* Add ChatClientTest mockito testing.
* Add ChatModel#getDefaultOptions(), and remove @FunctionalInterface

* ChatModel enums extend the new ModelDescription interface.
* Implement fromOptions copy method in every ChatOptions implementation.
* Extend ChatClient to use the model default options if not provided explicitly.

* Update readme to provide guidance on how to adapt to breaking changes.

Co-authored-by: Christian Tzolov <ctzolov@vmware.com>
Co-authored-by: Mark Pollack <mpollack@vmware.com>
2024-05-22 16:07:12 -04:00
Dimitrios Begnis
bce45c2d2f Use <> for inequality operator in Neo4j filter expression implementation 2024-05-21 17:17:32 -04:00
Christian Tzolov
3475f17e98 Unify the vector store module and pom names
spring-ai-qdrant -> spring-ai-qdrant-store
 spring-ai-cassandra -> spring-ai-cassandra-store
 spring-ai-pinecone -> spring-ai-pinecone-store
 spring-ai-redis -> spring-ai-redis-store
 spring-ai-qdrant -> spring-ai-qdrant-store
 spring-ai-gemfire -> spring-ai-gemfire-store
 spring-ai-azure-vector-store-spring-boot-starter -> spring-ai-azure-store-spring-boot-starter
 spring-ai-redis-spring-boot-starter -> spring-ai-redis-store-spring-boot-starter
2024-05-17 17:05:09 +02:00
Christian Tzolov
1985824fa9 clean code imports 2024-05-17 15:32:46 +02:00
Christian Tzolov
3ee04ed8da fix code style and missing dependency renaming 2024-05-17 15:31:58 +02:00
Josh Long
04d854cc49 make project names consistent 2024-05-17 15:31:58 +02:00
Josh Long
c5b7dd9c46 i ran the spring javaformat maven plugin to clean up any source code incompatabilities. 2024-05-17 15:31:57 +02:00
Josh Long
8c758617f3 moving to ApplicationListener<ApplicationReadyEvent> to avoid slowing down the application on startup. 2024-05-17 15:31:53 +02:00
Eddú Meléndez
d610dd6f1d Add default collection name to Qdrant
Add a default collection name similar to other vector store
implementations. Currently, when using starters, qdrant requires
a collection name. Otherwise, it fails.
2024-05-15 12:02:05 +02:00
Christian Tzolov
b0799babf4 Update Qdrant client version from 1.7.1 to 1.9.1
- Also update the qdrant/qdrant docker image version from v1.7.4 to v1.9.2
2024-05-15 06:19:46 +02:00
Eddú Meléndez
cdb80e83e9 Fix ChromaDB basic authentication
Currently, `username` is added as a password in `BasicAuthenticationInterceptor`.
This commit fixes the issue and also make sure the integration test
setup is correct.
2024-05-11 14:51:28 +02:00
Eddú Meléndez
5beef21a4e Expose QdrantClient and WeaviateClient as beans
Currently, `QdrantClient` and `WeaviateClient` are not exposed as
  beans. Having access to those would benefit to perform operations
  with an already configured client.

  - Deprecate  QdrantVectorStoreConfig.
  - Update Qdrant manual config adoc.
  - Improve Qdrant adoc.
  - Update Weaviate docs.
2024-05-10 15:15:14 +03:00
Christian Tzolov
6154fa663f Disable MondoDB IT due to #698 2024-05-09 13:27:57 +03:00
Mark Pollack
dfb8bf6a44 Add a new abstraction to simplify implementation of common ChatBot use cases
* Add ChatBot and basic DefaultChatBot
* Add streaming ChatBot support.
* Add Evaluator interface and RelevancyEvaluator implementation
* Add Content data type abstraction for Document and Message
* Renaming and package refactoring
* update .gitignore to allow node package name
* Add List<Media> to node and move ai.transformer package to ai.prompt.transformer
* Add Short/Long term memory support.
* Add mixing transformers support

Docs TBD
2024-05-08 15:14:50 -04:00
Christian Tzolov
670bf8e259 Adhere to Spring Framework code sytle
- Wildcard imports such as import java.util.* or import static org.assertj.core.api.Assertions.* are forbidden, even in test code.
   https://github.com/spring-projects/spring-framework/wiki/Code-Style#import-statements
2024-05-04 20:32:37 +03:00
mck
8a5f9dfb22 Add option to CassandraVectorStore to return embeddings in documents from similarity searches 2024-05-04 16:15:13 +02:00
mck
ac1a0d67e9 Avoid calling CREATE KEYSPACE|TABLE IF NOT EXISTS … unnecessarily
This also is a fix for AstraDB, which throws an exception if you try to `CREATE KEYSPACE IF NOT EXISTS …`
And use more compatible `USING 'StorageAttachedIndex'` index creation syntax.
2024-05-04 15:36:24 +02:00
wmz7year
f90c51a729 Improve ElasticsearchVectorStore,
- add index configuration and add support for ES response error handling.
 - rename dims to dimension propety.
 - add property javadocs
 - improve the elasticsearch javadoc.
2024-04-30 09:58:21 +03:00
Vaclav Bartacek
b9ba62507d Fix Neo4jVectorStore max allowed dimension size to 4096
- The Neo4j "vector-2.0" index provider allows up to 4096 dimensions.
  - Remove upper dimension boundry check
2024-04-28 20:30:06 +03:00
Christian Tzolov
9026263fd0 Fix javadoc after daf131be2c 2024-04-28 20:28:15 +03:00
mck
daf131be2c Fix column creation, when adding additional normal and embedding colums), and make index name unique (for when there are multiple vector indexes in the same keyspace)
And change stream to for-loop when converting List<Double> to Float[] for performance
2024-04-28 18:34:15 +03:00
Sagar Bhat
773b7bd3ca PgVectorStore: creating an index only if missing
- Resolve an issue where a new index keeps getting created during application start up.
 - Solution is is to create an index only if an index on the embedding column does not exist.
 - Add missing index name.
2024-04-26 19:03:45 +02:00
Ikko Eltociear Ashimine
db6f7cda3e Update AzureVectorStore.java
minor fix
2024-04-26 15:55:35 +02:00
omar.mahamid
6178b3dcc4 Fix CassandraVectorStoreConfig logger name
aligne logger name with the className
2024-04-20 07:40:01 +02:00
mck
0eaf7d05c9 Cassandra Vector Store initial impl follow up
- add concurrency to store.add(..) (bc embeddingClient is slow)
- CassandraVectorStoreAutoConfiguration uses CassandraAutoConfiguration
- driver profiles for production stability+performance,
- small cleanups and naming fixes,
- main doc tidy-up
- astradb compatibility (protocol V4)
– don't create embeddings again for documents that already have them
  similar to https://github.com/spring-projects/spring-ai/pull/413
2024-04-19 18:13:11 +02:00
Eddú Meléndez
eea3801fbd Remove @Testcontainers from HanaDB Tests
HanaDB tests don't rely on Testcontainers.
2024-04-10 17:31:28 +02:00
mck
656d238285 Implement Apache Cassandra vector store
The CassandraVectorStore is for managing and querying vector data in an Apache Cassandra db.
  It offers functionalities like adding, deleting, and performing similarity searches on documents.

  The store utilizes CQL to index and search vector data. It allows for custom metadata fields in
  the documents to be stored alongside the vector and content data.

  This class requires a CassandraVectorStoreConfig configuration object for initialization, which
  includes settings like connection details, index name, field names, etc. It also requires an
  EmbeddingClient to convert documents into embeddings before storing them.

  A schema matching the configuration is automatically created if it doesn't exist. Missing columns
  and indexes in existing tables will also be automatically created. Disable this with the disallowSchemaCreation.

  This class is designed to work with brand new tables that it creates for you, or on top of existing
  Cassandra tables. The latter is appropriate when wanting to keep data in place, creating embeddings
  next to it, and performing vector similarity searches in-situ.

  Instances of this class are not dynamic against server-side schema changes. If you change the schema
  server-side you need a new CassandraVectorStore instance.

 - Add auto-configure with tests.
 - reformat code style
 - Change field terminology to column (as appropriate for cassandra and cql)
 - Add doc page with an advanced example.
 - Add the dependencies to Spring AI BOM
 – add to `AutoConfiguration.imports`

 - Add @since annotation
 - Fix javadoc issue
 - Streamline the adoc content and layout
2024-04-10 16:51:47 +02:00
Christian Tzolov
0b04c8928a Update vector-stores README links 2024-04-09 20:55:39 +02:00
Rahul
466b824840 Add SAP HanaDB vector store integration
- Implement a HanaCloudVectorStore and tests
 - Implement Autoconfiguraiton + properties
 - Add boot starter
 - Update BOM with vector store and boot dependencies.
 - Add antora docuementation
 - added junit for HanaCloudVectorStoreProperties.java and documentation
   to create a BTP trial account and provision an instance for SAP Hana Cloud db
 - updated license, formatting and javadoc
 - IT for HanaCloudVectorStoreAutoConfiguration
 - IT for HanaCloudVectorStoreAutoConfiguration

Additional
 - add @AutoConfiguration(after = { JpaRepositoriesAutoConfiguration.class })
 - update the handa docs structure.
2024-04-09 12:36:42 +02:00
Christian Tzolov
8e0fbe49ee Remove the spring-boot-starter-jdbc dependency from PGVector project 2024-04-09 10:57:17 +02:00
Christian Tzolov
7c09c0963b Fix Chroma package name spelling error
- Rename org.springframework.ai.vectorsore into org.springframework.ai.vectorstore
 - Resolves #540
2024-04-07 13:49:58 +02:00
Eddú Meléndez
50f549d960 Update testcontainers redis version to 2.2.0 2024-04-04 13:43:01 +02:00
Eddú Meléndez
8fa675b145 Update spring boot version to 3.2.4 2024-04-04 13:43:01 +02:00
Sagar Bhat
ed68dfb00c Correct grammar for a message to be thrown as part of an exception when the document was not uploaded successfully (part of the 'add' method). 2024-03-27 09:15:08 +01:00
Christian Tzolov
50c464fd16 Exlucde, a parasite, lombok dependency 2024-03-22 09:23:35 +01:00
Jemin Huh
9334d7d540 Add Elasticsearch vector store integration
- Implement ElasticsearchVectoSotore and IT.
 - Add ElasticsearchAiSearchFilterExpressionConverter.
 - Add dependency to BOM and module to parent pom.
 - Fix ElasticsearchVectorStoreIT FilterExpression with
   Date type requires the use of epoch milliseconds.
 - Add license formatting.
2024-03-20 07:41:05 +01:00
geetrawat
7634c6b780 Add Gemfire vector store
- Implement a GemFireVectorStore implementing the VectorStore interface.
 - Add unit and integration test.
 - Add antora documentation.
 - Add to BOM.
2024-03-19 07:41:17 +01:00
Eddú Meléndez
8503078088 Add ServiceConnection support for
* ChromaDBContainer
 * MilvusContainer
 * QdrantContainer
 * RedisStackContainer
 * WeaviateContainer
 * OllamaContainer

Add docs
2024-03-18 23:29:55 -04:00
Christian Tzolov
666474caff Minor improvements of the MongoDB vector store impl. 2024-03-18 19:51:54 +01:00
Eddú Meléndez
150b268415 Add MongoDB Atlas vector store auto-configuration
It also provides a starter.
2024-03-18 15:07:34 +01:00
Eddú Meléndez
c5f35029f0 Remove withReuse in MongoDBAtlasContainer
`Reuse Container` is a Testcontainers experimental feature. It requires
`testcontainers.reuse.enable=true` in `~/.testcontainers.properties` in
order to take effect but in order to avoid surprises, this commit remove it.

See https://java.testcontainers.org/features/reuse/
2024-03-16 23:54:59 +01:00
Chris Smith
5f0123cc92 Add MongoDB Atlas Vector store
- Add VectorSearchAggregation used to actually preform the search
   on a given collection with embeddings.
 - add MongoDBVectorStore
 - Add MongoDBVectorStoreIT.  Integration test runs fine given...
   - You have a mongo atlas cluster to connect to (local or remote)
   - You have the search index "spring_ai_vector_search" setup correctly
   - Need to explore getting around this
   - Need to filter results using threshold
 - Add postfilter for threshold values - While a post filter is not ideal,
   it gets the job done. The mongo team seems to be working on having
   it availible as a prefilter option, in which this implementation
   can be updated to use later.
 - implement filtering threshold
 - fix a few sonar issues
 - formatting
 - use higher default num_candidates
 - use builder for configuration
 - add documentation and some refactor
 - use consistent property in integration test
 - finish implementing filter support
 - add documentation to filter converter
 - add vector search index auto creation

 - Add to BOM.
 - Fix version to 1.0.0-SN.
 - Move expresion converter from core to models/mongodb.
 - Fix style and license headers
2024-03-16 12:13:02 +01:00
Gerrit Meier
7c6bbef3ec Allow more customization for Neo4j store (id and constraint).
Unrelated to this change, the Neo4j test version increased to be current.
2024-03-15 12:26:20 +01:00
Mark Pollack
4c617e16b4 Prepare next development iteration 2024-03-12 14:33:45 -04:00
Mark Pollack
490f3cd1cb Milestone Release 0.8.1 2024-03-12 14:28:28 -04:00
Bragolgirith
777b79e1c5 Fix RedisVectorStore not closing Jedis pipelines 2024-03-12 15:15:05 +01:00
Christian Tzolov
ba940395b6 Use platform independent line separators 2024-03-08 09:43:15 +01:00