Improved GemFire support
- Adds spring boot auto-configuration support for GemFireVectorStore - Adds integration test GemFireVectorStoreAutoConfigurationIT - Includes gemfire-testcontainers in integration tests - Adds unit test GemFireVectorStorePropertiesTests - Refactors GemFireVectorStore.java extracting GemFireVectorStoreConfig.java - Renames spring-ai-gemfire to spring-ai-gemfire-store - Adds GemFireConnectionDetails - Adds GemFireVectorStoreProperties with default values - Remove gemfire-release-repo maven repository Co-authored-by: Louis Jacome <louis.jacome@broadcom.com> Co-authored-by: Jason Huyn <jason.huynh@broadcom.com>
This commit is contained in:
2
pom.xml
2
pom.xml
@@ -48,6 +48,7 @@
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-cassandra-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-chroma-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-elasticsearch-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-gemfire-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-hanadb-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-milvus-store</module>
|
||||
<module>spring-ai-spring-boot-starters/spring-ai-starter-mongodb-atlas-store</module>
|
||||
@@ -166,6 +167,7 @@
|
||||
<postgresql.version>42.7.2</postgresql.version>
|
||||
<elasticsearch-java.version>8.13.3</elasticsearch-java.version>
|
||||
<milvus.version>2.3.4</milvus.version>
|
||||
<gemfire.testcontainers.version>2.3.0</gemfire.testcontainers.version>
|
||||
<pinecone.version>0.8.0</pinecone.version>
|
||||
<fastjson.version>2.0.46</fastjson.version>
|
||||
<azure-search.version>11.6.1</azure-search.version>
|
||||
|
||||
@@ -446,6 +446,12 @@
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-gemfire-store-spring-boot-starter</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-zhipuai-spring-boot-starter</artifactId>
|
||||
|
||||
@@ -1,39 +1,63 @@
|
||||
= GemFire Vector Store
|
||||
|
||||
This section walks you through setting up the GemFire VectorStore to store document embeddings and perform similarity searches.
|
||||
This section walks you through setting up the `GemFireVectorStore` to store document embeddings and perform similarity searches.
|
||||
|
||||
link:https://tanzu.vmware.com/gemfire[GemFire] is an ultra high speed in-memory data and compute grid, with vector extensions to store and search vectors efficiently.
|
||||
link:https://tanzu.vmware.com/gemfire[GemFire] is a distributed, in-memory, key-value store performing read and write operations at blazingly fast speeds. It offers highly available parallel message queues, continuous availability, and an event-driven architecture you can scale dynamically without downtime. As your data size requirements increase to support high-performance, real-time apps, GemFire can easily scale linearly.
|
||||
|
||||
link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/overview.html[GemFire VectorDB] extends GemFire's capabilities, serving as a versatile vector database that efficiently stores, retrieves, and performs vector searches through a distributed and resilient infrastructure:
|
||||
|
||||
Capabilities:
|
||||
- Create Indexes
|
||||
- Store vectors and the associated metadata
|
||||
- Perform vector searches based on similarity
|
||||
link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/overview.html[GemFire VectorDB] extends GemFire's capabilities, serving as a versatile vector database that efficiently stores, retrieves, and performs vector similarity searches.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
Access to a GemFire cluster with the link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/install.html[GemFire Vector Database] extension installed.
|
||||
You can download the GemFire VectorDB extension from the link:https://network.pivotal.io/products/gemfire-vectordb/[VMware Tanzu Network] after signing in.
|
||||
1. A GemFire cluster with the GemFire VectorDB extension enabled
|
||||
- link:https://docs.vmware.com/en/VMware-GemFire-VectorDB/1.0/gemfire-vectordb/install.html[Install GemFire VectorDB extension]
|
||||
|
||||
== Dependencies
|
||||
2. An `EmbeddingModel` bean to compute the document embeddings. Refer to the xref:api/embeddings.adoc#available-implementations[EmbeddingModel] section for more information.
|
||||
An option that runs locally on your machine is xref:api/embeddings/onnx.adoc[ONNX] and the all-MiniLM-L6-v2 Sentence Transformers.
|
||||
|
||||
Add these dependencies to your project:
|
||||
== Auto-configuration
|
||||
|
||||
- Embedding Model boot starter, required for calculating embeddings.
|
||||
- Transformers Embedding (Local) and follow the ONNX Transformers Embedding instructions.
|
||||
Add the GemFire VectorStore Spring Boot starter to you project's Maven build file `pom.xml`:
|
||||
|
||||
[source,xml]
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-transformers</artifactId>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-gemfire-store-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
- Add the GemFire VectorDB dependencies
|
||||
or to your Gradle `build.gradle` file
|
||||
|
||||
[source,xml]
|
||||
[source, xml]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-gemfire-store-spring-boot-starter'
|
||||
}
|
||||
----
|
||||
|
||||
=== Configuration properties
|
||||
|
||||
You can use the following properties in your Spring Boot configuration to further configure the `GemFireVectorStore`.
|
||||
|
||||
|===
|
||||
|Property|Default value
|
||||
|
||||
|`spring.ai.vectorstore.gemfire.host`|localhost
|
||||
|`spring.ai.vectorstore.gemfire.port`|8080
|
||||
|`spring.ai.vectorstore.gemfire.index-name`|spring-ai-gemfire-store
|
||||
|`spring.ai.vectorstore.gemfire.beam-width`|100
|
||||
|`spring.ai.vectorstore.gemfire.max-connections`|16
|
||||
|`spring.ai.vectorstore.gemfire.vector-similarity-function`|COSINE
|
||||
|`spring.ai.vectorstore.gemfire.fields`|[]
|
||||
|`spring.ai.vectorstore.gemfire.buckets`|0
|
||||
|===
|
||||
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
To use just the `GemFireVectorStore`, without Spring Boot's Auto-configuration add the following dependency to your project’s Maven `pom.xml`:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
@@ -41,82 +65,71 @@ Add these dependencies to your project:
|
||||
</dependency>
|
||||
----
|
||||
|
||||
For Gradle users, add the following to your `build.gradle` file under the dependencies block to use just the `GemFireVectorStore`:
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
[souce, xml]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-gemfire-store'
|
||||
}
|
||||
----
|
||||
|
||||
== Usage
|
||||
|
||||
== Sample Code
|
||||
|
||||
- To configure GemFire in your application, use the following setup:
|
||||
Here is a sample that creates an instance of the `GemfireVectorStore` instead of using AutoConfiguration
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@Bean
|
||||
public GemFireVectorStoreConfig gemFireVectorStoreConfig() {
|
||||
return GemFireVectorStoreConfig.builder()
|
||||
.withUrl("http://localhost:8080")
|
||||
.withIndexName("spring-ai-test-index")
|
||||
.build();
|
||||
public VectorStore vectorStore(EmbeddingModel embeddingModel) {
|
||||
return new GemFireVectorStore(new GemFireVectorStoreConfig()
|
||||
.setIndexName("my-vector-index")
|
||||
.setPort(7071), embeddingClient);
|
||||
}
|
||||
----
|
||||
|
||||
- Create a GemFireVectorStore instance connected to your GemFire VectorDB:
|
||||
[NOTE]
|
||||
====
|
||||
The GemFire VectorStore does not yet support xref:api/vectordbs.adoc#metadata-filters[metadata filters].
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
The default configuration connects to a GemFire cluster at `localhost:8080`
|
||||
====
|
||||
|
||||
- In your application, create a few documents:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@Bean
|
||||
public VectorStore vectorStore(GemFireVectorStoreConfig config, EmbeddingModel embeddingModel) {
|
||||
return new GemFireVectorStore(config, embeddingModel);
|
||||
}
|
||||
List<Document> documents = List.of(
|
||||
new Document("Spring AI rocks!! Spring AI rocks!! Spring AI rocks!! Spring AI rocks!! Spring AI rocks!!", Map.of("country", "UK", "year", 2020)),
|
||||
new Document("The World is Big and Salvation Lurks Around the Corner", Map.of()),
|
||||
new Document("You walk forward facing the past and you turn back toward the future.", Map.of("country", "NL", "year", 2023)));
|
||||
----
|
||||
- Create a Vector Index which will configure GemFire region.
|
||||
|
||||
- Add the documents to the vector store:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
public void createIndex() {
|
||||
try {
|
||||
CreateRequest createRequest = new CreateRequest();
|
||||
createRequest.setName(INDEX_NAME);
|
||||
createRequest.setBeamWidth(20);
|
||||
createRequest.setMaxConnections(16);
|
||||
ObjectMapper objectMapper = new ObjectMapper();
|
||||
String index = objectMapper.writeValueAsString(createRequest);
|
||||
client.post()
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.bodyValue(index)
|
||||
.retrieve()
|
||||
.bodyToMono(Void.class)
|
||||
.block();
|
||||
}
|
||||
catch (Exception e) {
|
||||
logger.warn("An unexpected error occurred while creating the index");
|
||||
}
|
||||
}
|
||||
vectorStore.add(documents);
|
||||
----
|
||||
|
||||
- Create some documents:
|
||||
- And to retrieve documents using similarity search:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
List<Document> documents = List.of(
|
||||
new Document("1", getText("classpath:/test/data/spring.ai.txt"), Map.of("meta1", "meta1")),
|
||||
new Document("2", getText("classpath:/test/data/time.shelter.txt"), Map.of()),
|
||||
new Document("3", getText("classpath:/test/data/great.depression.txt"), Map.of("meta2", "meta2")));
|
||||
List<Document> results = vectorStore.similaritySearch(
|
||||
SearchRequest.query("Spring").withTopK(5));
|
||||
----
|
||||
|
||||
- Add the documents to GemFire VectorDB:
|
||||
You should retrieve the document containing the text "Spring AI rocks!!".
|
||||
|
||||
You can also limit the number of results using a similarity threshold:
|
||||
[source,java]
|
||||
----
|
||||
vectorStore.add(List.of(document));
|
||||
List<Document> results = vectorStore.similaritySearch(
|
||||
SearchRequest.query("Spring").withTopK(5)
|
||||
.withSimilarityThreshold(0.5d));
|
||||
----
|
||||
|
||||
- And finally, retrieve documents similar to a query:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
List<Document> results = vectorStore.similaritySearch("Spring", 5);
|
||||
----
|
||||
|
||||
If all goes well, you should retrieve the document containing the text "Spring AI rocks!!".
|
||||
|
||||
|
||||
@@ -297,6 +297,13 @@
|
||||
<optional>true</optional>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-gemfire-store</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
<optional>true</optional>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-minimax</artifactId>
|
||||
@@ -458,6 +465,13 @@
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>dev.gemfire</groupId>
|
||||
<artifactId>gemfire-testcontainers</artifactId>
|
||||
<version>2.3.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
</project>
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
/*
|
||||
* Copyright 2024 - 2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.springframework.ai.autoconfigure.vectorstore.gemfire;
|
||||
|
||||
import org.springframework.boot.autoconfigure.service.connection.ConnectionDetails;
|
||||
|
||||
/**
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
public interface GemFireConnectionDetails extends ConnectionDetails {
|
||||
|
||||
String getHost();
|
||||
|
||||
int getPort();
|
||||
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.autoconfigure.vectorstore.gemfire;
|
||||
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStore;
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStoreConfig;
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
/**
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
@AutoConfiguration
|
||||
@ConditionalOnClass({ GemFireVectorStore.class, EmbeddingModel.class })
|
||||
@EnableConfigurationProperties(GemFireVectorStoreProperties.class)
|
||||
@ConditionalOnProperty(prefix = "spring.ai.vectorstore.gemfire", value = { "index-name" })
|
||||
public class GemFireVectorStoreAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean(GemFireConnectionDetails.class)
|
||||
GemFireVectorStoreAutoConfiguration.PropertiesGemFireConnectionDetails gemfireConnectionDetails(
|
||||
GemFireVectorStoreProperties properties) {
|
||||
return new GemFireVectorStoreAutoConfiguration.PropertiesGemFireConnectionDetails(properties);
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public GemFireVectorStore gemfireVectorStore(EmbeddingModel embeddingModel, GemFireVectorStoreProperties properties,
|
||||
GemFireConnectionDetails gemFireConnectionDetails) {
|
||||
var config = new GemFireVectorStoreConfig();
|
||||
|
||||
config.setHost(gemFireConnectionDetails.getHost())
|
||||
.setPort(gemFireConnectionDetails.getPort())
|
||||
.setIndexName(properties.getIndexName())
|
||||
.setBeamWidth(properties.getBeamWidth())
|
||||
.setMaxConnections(properties.getMaxConnections())
|
||||
.setBuckets(properties.getBuckets())
|
||||
.setVectorSimilarityFunction(properties.getVectorSimilarityFunction())
|
||||
.setFields(properties.getFields())
|
||||
.setSslEnabled(properties.isSslEnabled());
|
||||
return new GemFireVectorStore(config, embeddingModel);
|
||||
}
|
||||
|
||||
private static class PropertiesGemFireConnectionDetails implements GemFireConnectionDetails {
|
||||
|
||||
private final GemFireVectorStoreProperties properties;
|
||||
|
||||
PropertiesGemFireConnectionDetails(GemFireVectorStoreProperties properties) {
|
||||
this.properties = properties;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getHost() {
|
||||
return this.properties.getHost();
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getPort() {
|
||||
return this.properties.getPort();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.autoconfigure.vectorstore.gemfire;
|
||||
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStoreConfig;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
|
||||
/**
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
@ConfigurationProperties(GemFireVectorStoreProperties.CONFIG_PREFIX)
|
||||
public class GemFireVectorStoreProperties {
|
||||
|
||||
/**
|
||||
* Configuration prefix for Spring AI VectorStore GemFire.
|
||||
*/
|
||||
public static final String CONFIG_PREFIX = "spring.ai.vectorstore.gemfire";
|
||||
|
||||
/**
|
||||
* The host of the GemFire to connect to. To specify a custom host, use
|
||||
* "spring.ai.vectorstore.gemfire.host";
|
||||
*
|
||||
*/
|
||||
private String host = GemFireVectorStoreConfig.DEFAULT_HOST;
|
||||
|
||||
/**
|
||||
* The port of the GemFire to connect to. To specify a custom port, use
|
||||
* "spring.ai.vectorstore.gemfire.port";
|
||||
*/
|
||||
private int port = GemFireVectorStoreConfig.DEFAULT_PORT;
|
||||
|
||||
/**
|
||||
* The name of the index in the GemFire. To specify a custom index, use
|
||||
* "spring.ai.vectorstore.gemfire.index-name";
|
||||
*/
|
||||
private String indexName = GemFireVectorStoreConfig.DEFAULT_INDEX_NAME;
|
||||
|
||||
/**
|
||||
* The beam width for similarity queries. Default value is {@code 100}. To specify a
|
||||
* custom beam width, use "spring.ai.vectorstore.gemfire.beam-width";
|
||||
*/
|
||||
private int beamWidth = GemFireVectorStoreConfig.DEFAULT_BEAM_WIDTH;
|
||||
|
||||
/**
|
||||
* The maximum number of connections allowed. Default value is {@code 16}. To specify
|
||||
* custom number of connections, use "spring.ai.vectorstore.gemfire.max-connections";
|
||||
*/
|
||||
private int maxConnections = GemFireVectorStoreConfig.DEFAULT_MAX_CONNECTIONS;
|
||||
|
||||
/**
|
||||
* The similarity function to be used for vector comparisons. Default value is
|
||||
* {@code "COSINE"}. To specify custom vectorSimilarityFunction, use
|
||||
* "spring.ai.vectorstore.gemfire.vector-similarity-function";
|
||||
*
|
||||
*/
|
||||
private String vectorSimilarityFunction = GemFireVectorStoreConfig.DEFAULT_SIMILARITY_FUNCTION;
|
||||
|
||||
/**
|
||||
* The fields to be used for queries. Default value is an array containing
|
||||
* {@code "vector"}. To specify custom fields, use
|
||||
* "spring.ai.vectorstore.gemfire.fields"
|
||||
*/
|
||||
private String[] fields = GemFireVectorStoreConfig.DEFAULT_FIELDS;
|
||||
|
||||
/**
|
||||
* The number of buckets to use for partitioning the data. Default value is {@code 0}.
|
||||
*
|
||||
* To specify custom buckets, use "spring.ai.vectorstore.gemfire.buckets";
|
||||
*
|
||||
*/
|
||||
private int buckets = GemFireVectorStoreConfig.DEFAULT_BUCKETS;
|
||||
|
||||
/**
|
||||
* Set to true if GemFire cluster is ssl enabled
|
||||
*
|
||||
* To specify sslEnabled, use "spring.ai.vectorstore.gemfire.ssl-enabled";
|
||||
*/
|
||||
private boolean sslEnabled = GemFireVectorStoreConfig.DEFAULT_SSL_ENABLED;
|
||||
|
||||
public int getBeamWidth() {
|
||||
return beamWidth;
|
||||
}
|
||||
|
||||
public void setBeamWidth(int beamWidth) {
|
||||
this.beamWidth = beamWidth;
|
||||
}
|
||||
|
||||
public int getPort() {
|
||||
return port;
|
||||
}
|
||||
|
||||
public void setPort(int port) {
|
||||
this.port = port;
|
||||
}
|
||||
|
||||
public String getHost() {
|
||||
return host;
|
||||
}
|
||||
|
||||
public void setHost(String host) {
|
||||
this.host = host;
|
||||
}
|
||||
|
||||
public String getIndexName() {
|
||||
return indexName;
|
||||
}
|
||||
|
||||
public void setIndexName(String indexName) {
|
||||
this.indexName = indexName;
|
||||
}
|
||||
|
||||
public int getMaxConnections() {
|
||||
return maxConnections;
|
||||
}
|
||||
|
||||
public void setMaxConnections(int maxConnections) {
|
||||
this.maxConnections = maxConnections;
|
||||
}
|
||||
|
||||
public String getVectorSimilarityFunction() {
|
||||
return vectorSimilarityFunction;
|
||||
}
|
||||
|
||||
public void setVectorSimilarityFunction(String vectorSimilarityFunction) {
|
||||
this.vectorSimilarityFunction = vectorSimilarityFunction;
|
||||
}
|
||||
|
||||
public String[] getFields() {
|
||||
return fields;
|
||||
}
|
||||
|
||||
public void setFields(String[] fields) {
|
||||
this.fields = fields;
|
||||
}
|
||||
|
||||
public int getBuckets() {
|
||||
return buckets;
|
||||
}
|
||||
|
||||
public void setBuckets(int buckets) {
|
||||
this.buckets = buckets;
|
||||
}
|
||||
|
||||
public boolean isSslEnabled() {
|
||||
return sslEnabled;
|
||||
}
|
||||
|
||||
public void setSslEnabled(boolean sslEnabled) {
|
||||
this.sslEnabled = sslEnabled;
|
||||
}
|
||||
|
||||
}
|
||||
@@ -32,6 +32,7 @@ org.springframework.ai.autoconfigure.vectorstore.mongo.MongoDBAtlasVectorStoreAu
|
||||
org.springframework.ai.autoconfigure.anthropic.AnthropicAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.watsonxai.WatsonxAiAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.vectorstore.elasticsearch.ElasticsearchVectorStoreAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.vectorstore.gemfire.GemFireVectorStoreAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.vectorstore.cassandra.CassandraVectorStoreAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.zhipuai.ZhiPuAiAutoConfiguration
|
||||
org.springframework.ai.autoconfigure.chat.client.ChatClientAutoConfiguration
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.autoconfigure.vectorstore.gemfire;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.hamcrest.Matchers.hasSize;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.github.dockerjava.api.model.ExposedPort;
|
||||
import com.github.dockerjava.api.model.PortBinding;
|
||||
import com.github.dockerjava.api.model.Ports;
|
||||
import com.vmware.gemfire.testcontainers.GemFireCluster;
|
||||
import org.awaitility.Awaitility;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.Assertions;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.ai.ResourceUtils;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.transformers.TransformersEmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStore;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
/**
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
|
||||
class GemFireVectorStoreAutoConfigurationIT {
|
||||
|
||||
private static GemFireCluster gemFireCluster;
|
||||
|
||||
private static final String INDEX_NAME = "spring-ai-index";
|
||||
|
||||
private static final int BEAM_WIDTH = 50;
|
||||
|
||||
private static final int MAX_CONNECTIONS = 8;
|
||||
|
||||
private static final String SIMILARITY_FUNCTION = "DOT_PRODUCT";
|
||||
|
||||
private static final String[] FIELDS = { "someField1", "someField2" };
|
||||
|
||||
private static final int BUCKET_COUNT = 2;
|
||||
|
||||
private static final int HTTP_SERVICE_PORT = 9090;
|
||||
|
||||
private static final int LOCATOR_COUNT = 1;
|
||||
|
||||
private static final int SERVER_COUNT = 1;
|
||||
|
||||
@AfterAll
|
||||
public static void stopGemFireCluster() {
|
||||
gemFireCluster.close();
|
||||
}
|
||||
|
||||
List<Document> documents = List.of(
|
||||
new Document(ResourceUtils.getText("classpath:/test/data/spring.ai.txt"), Map.of("spring", "great")),
|
||||
new Document(ResourceUtils.getText("classpath:/test/data/time.shelter.txt")), new Document(
|
||||
ResourceUtils.getText("classpath:/test/data/great.depression.txt"), Map.of("depression", "bad")));
|
||||
|
||||
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
|
||||
.withConfiguration(AutoConfigurations.of(GemFireVectorStoreAutoConfiguration.class))
|
||||
.withUserConfiguration(Config.class)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.index-name=" + INDEX_NAME)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.beam-width=" + BEAM_WIDTH)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.max-connections=" + MAX_CONNECTIONS)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.vector-similarity-function=" + SIMILARITY_FUNCTION)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.buckets=" + BUCKET_COUNT)
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.fields=someField1,someField2")
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.host=localhost")
|
||||
.withPropertyValues("spring.ai.vectorstore.gemfire.port=" + HTTP_SERVICE_PORT);
|
||||
|
||||
@BeforeAll
|
||||
public static void startGemFireCluster() {
|
||||
Ports.Binding hostPort = Ports.Binding.bindPort(HTTP_SERVICE_PORT);
|
||||
ExposedPort exposedPort = new ExposedPort(HTTP_SERVICE_PORT);
|
||||
PortBinding mappedPort = new PortBinding(hostPort, exposedPort);
|
||||
gemFireCluster = new GemFireCluster("gemfire/gemfire-all:10.1-jdk17", LOCATOR_COUNT, SERVER_COUNT);
|
||||
gemFireCluster.withConfiguration(GemFireCluster.SERVER_GLOB,
|
||||
container -> container.withExposedPorts(HTTP_SERVICE_PORT)
|
||||
.withCreateContainerCmdModifier(cmd -> cmd.getHostConfig().withPortBindings(mappedPort)));
|
||||
gemFireCluster.withGemFireProperty(GemFireCluster.SERVER_GLOB, "http-service-port",
|
||||
Integer.toString(HTTP_SERVICE_PORT));
|
||||
gemFireCluster.acceptLicense().start();
|
||||
|
||||
System.setProperty("spring.data.gemfire.pool.locators",
|
||||
String.format("localhost[%d]", gemFireCluster.getLocatorPort()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void ensureGemFireVectorStoreCustomConfiguration() {
|
||||
this.contextRunner.run(context -> {
|
||||
GemFireVectorStore store = context.getBean(GemFireVectorStore.class);
|
||||
Assertions.assertNotNull(store);
|
||||
assertThat(store.getIndexName()).isEqualTo(INDEX_NAME);
|
||||
assertThat(store.getBeamWidth()).isEqualTo(BEAM_WIDTH);
|
||||
assertThat(store.getMaxConnections()).isEqualTo(MAX_CONNECTIONS);
|
||||
assertThat(store.getVectorSimilarityFunction()).isEqualTo(SIMILARITY_FUNCTION);
|
||||
assertThat(store.getFields()).isEqualTo(FIELDS);
|
||||
|
||||
String indexJson = store.getIndex();
|
||||
Map<String, Object> index = parseIndex(indexJson);
|
||||
assertThat(index.get("name")).isEqualTo(INDEX_NAME);
|
||||
assertThat(index.get("beam-width")).isEqualTo(BEAM_WIDTH);
|
||||
assertThat(index.get("max-connections")).isEqualTo(MAX_CONNECTIONS);
|
||||
assertThat(index.get("vector-similarity-function")).isEqualTo(SIMILARITY_FUNCTION);
|
||||
assertThat(index.get("buckets")).isEqualTo(BUCKET_COUNT);
|
||||
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
public void addAndSearchTest() {
|
||||
contextRunner.run(context -> {
|
||||
VectorStore vectorStore = context.getBean(VectorStore.class);
|
||||
vectorStore.add(documents);
|
||||
|
||||
Awaitility.await().until(() -> {
|
||||
return vectorStore.similaritySearch(SearchRequest.query("Spring").withTopK(1));
|
||||
}, hasSize(1));
|
||||
|
||||
List<Document> results = vectorStore.similaritySearch(SearchRequest.query("Spring").withTopK(1));
|
||||
|
||||
assertThat(results).hasSize(1);
|
||||
Document resultDoc = results.get(0);
|
||||
assertThat(resultDoc.getId()).isEqualTo(documents.get(0).getId());
|
||||
assertThat(resultDoc.getContent()).contains(
|
||||
"Spring AI provides abstractions that serve as the foundation for developing AI applications.");
|
||||
assertThat(resultDoc.getMetadata()).hasSize(2);
|
||||
assertThat(resultDoc.getMetadata()).containsKeys("spring", "distance");
|
||||
|
||||
// Remove all documents from the store
|
||||
vectorStore.delete(documents.stream().map(doc -> doc.getId()).toList());
|
||||
|
||||
Awaitility.await().until(() -> {
|
||||
return vectorStore.similaritySearch(SearchRequest.query("Spring").withTopK(1));
|
||||
}, hasSize(0));
|
||||
});
|
||||
}
|
||||
|
||||
private Map<String, Object> parseIndex(String json) {
|
||||
try {
|
||||
JsonNode rootNode = new ObjectMapper().readTree(json);
|
||||
Map<String, Object> indexDetails = new HashMap<>();
|
||||
if (rootNode.isObject()) {
|
||||
if (rootNode.has("name"))
|
||||
indexDetails.put("name", rootNode.get("name").asText());
|
||||
if (rootNode.has("beam-width"))
|
||||
indexDetails.put("beam-width", rootNode.get("beam-width").asInt());
|
||||
if (rootNode.has("max-connections"))
|
||||
indexDetails.put("max-connections", rootNode.get("max-connections").asInt());
|
||||
if (rootNode.has("vector-similarity-function"))
|
||||
indexDetails.put("vector-similarity-function", rootNode.get("vector-similarity-function").asText());
|
||||
if (rootNode.has("buckets"))
|
||||
indexDetails.put("buckets", rootNode.get("buckets").asInt());
|
||||
if (rootNode.has("number-of-embeddings"))
|
||||
indexDetails.put("number-of-embeddings", rootNode.get("number-of-embeddings").asInt());
|
||||
}
|
||||
return indexDetails;
|
||||
}
|
||||
catch (Exception e) {
|
||||
return new HashMap<>();
|
||||
}
|
||||
}
|
||||
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
static class Config {
|
||||
|
||||
@Bean
|
||||
public EmbeddingModel embeddingModel() {
|
||||
return new TransformersEmbeddingModel();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.autoconfigure.vectorstore.gemfire;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStoreConfig;
|
||||
|
||||
/**
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
class GemFireVectorStorePropertiesTests {
|
||||
|
||||
@Test
|
||||
void defaultValues() {
|
||||
var props = new GemFireVectorStoreProperties();
|
||||
assertThat(props.getIndexName()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_INDEX_NAME);
|
||||
assertThat(props.getHost()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_HOST);
|
||||
assertThat(props.getPort()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_PORT);
|
||||
assertThat(props.getBeamWidth()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_BEAM_WIDTH);
|
||||
assertThat(props.getMaxConnections()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_MAX_CONNECTIONS);
|
||||
assertThat(props.getFields()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_FIELDS);
|
||||
assertThat(props.getBuckets()).isEqualTo(GemFireVectorStoreConfig.DEFAULT_BUCKETS);
|
||||
}
|
||||
|
||||
@Test
|
||||
void customValues() {
|
||||
var props = new GemFireVectorStoreProperties();
|
||||
props.setIndexName("spring-ai-index");
|
||||
props.setHost("localhost");
|
||||
props.setPort(9090);
|
||||
props.setBeamWidth(10);
|
||||
props.setMaxConnections(10);
|
||||
props.setFields(new String[] { "test" });
|
||||
props.setBuckets(10);
|
||||
|
||||
assertThat(props.getIndexName()).isEqualTo("spring-ai-index");
|
||||
assertThat(props.getHost()).isEqualTo("localhost");
|
||||
assertThat(props.getPort()).isEqualTo(9090);
|
||||
assertThat(props.getBeamWidth()).isEqualTo(10);
|
||||
assertThat(props.getMaxConnections()).isEqualTo(10);
|
||||
assertThat(props.getFields()).isEqualTo(new String[] { "test" });
|
||||
assertThat(props.getBuckets()).isEqualTo(10);
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>spring-ai-gemfire-store-spring-boot-starter</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
<name>Spring AI Starter - GemFire Vector Store</name>
|
||||
<description>Spring AI GemFire Vector Store Auto Configuration</description>
|
||||
<url>https://github.com/spring-projects/spring-ai</url>
|
||||
|
||||
<scm>
|
||||
<url>https://github.com/spring-projects/spring-ai</url>
|
||||
<connection>git://github.com/spring-projects/spring-ai.git</connection>
|
||||
<developerConnection>git@github.com:spring-projects/spring-ai.git</developerConnection>
|
||||
</scm>
|
||||
|
||||
<dependencies>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-spring-boot-autoconfigure</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-gemfire-store</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
</project>
|
||||
@@ -38,6 +38,13 @@
|
||||
</dependency>
|
||||
|
||||
<!-- TESTING -->
|
||||
<dependency>
|
||||
<groupId>dev.gemfire</groupId>
|
||||
<artifactId>gemfire-testcontainers</artifactId>
|
||||
<version>${gemfire.testcontainers.version}</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-openai</artifactId>
|
||||
@@ -71,10 +78,6 @@
|
||||
<version>3.0.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.logging.log4j</groupId>
|
||||
<artifactId>log4j-core</artifactId>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright 2023 - 2024 the original author or authors.
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
@@ -32,6 +32,7 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.beans.factory.InitializingBean;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.util.Assert;
|
||||
@@ -48,13 +49,11 @@ import reactor.util.annotation.NonNull;
|
||||
*
|
||||
* @author Geet Rawat
|
||||
*/
|
||||
public class GemFireVectorStore implements VectorStore {
|
||||
|
||||
public static final String QUERY = "/query";
|
||||
public class GemFireVectorStore implements VectorStore, InitializingBean {
|
||||
|
||||
private static final Logger logger = LoggerFactory.getLogger(GemFireVectorStore.class);
|
||||
|
||||
private static final String DISTANCE_METADATA_FIELD_NAME = "distance";
|
||||
private static final String DEFAULT_URI = "http{ssl}://{host}:{port}/gemfire-vectordb/v1/indexes";
|
||||
|
||||
private static final String EMBEDDINGS = "/embeddings";
|
||||
|
||||
@@ -62,179 +61,135 @@ public class GemFireVectorStore implements VectorStore {
|
||||
|
||||
private final EmbeddingModel embeddingModel;
|
||||
|
||||
private final int topKPerBucket;
|
||||
private static final String DOCUMENT_FIELD = "document";
|
||||
|
||||
private final int topK;
|
||||
// Create Index Parameters
|
||||
|
||||
private final String documentField;
|
||||
private String indexName;
|
||||
|
||||
public static final class GemFireVectorStoreConfig {
|
||||
public String getIndexName() {
|
||||
return indexName;
|
||||
}
|
||||
|
||||
private final WebClient client;
|
||||
private int beamWidth;
|
||||
|
||||
private final String index;
|
||||
public int getBeamWidth() {
|
||||
return beamWidth;
|
||||
}
|
||||
|
||||
private final int topKPerBucket;
|
||||
private int maxConnections;
|
||||
|
||||
public final int topK;
|
||||
public int getMaxConnections() {
|
||||
return maxConnections;
|
||||
}
|
||||
|
||||
private final String documentField;
|
||||
|
||||
public static Builder builder() {
|
||||
return new Builder();
|
||||
}
|
||||
|
||||
private GemFireVectorStoreConfig(Builder builder) {
|
||||
String base = UriComponentsBuilder.fromUriString(DEFAULT_URI)
|
||||
.build(builder.sslEnabled ? "s" : "", builder.host, builder.port)
|
||||
.toString();
|
||||
this.index = builder.index;
|
||||
this.client = WebClient.create(base);
|
||||
this.topKPerBucket = builder.topKPerBucket;
|
||||
this.topK = builder.topK;
|
||||
this.documentField = builder.documentField;
|
||||
}
|
||||
|
||||
public static class Builder {
|
||||
|
||||
private String host;
|
||||
|
||||
private int port = DEFAULT_PORT;
|
||||
|
||||
private boolean sslEnabled;
|
||||
|
||||
private long connectionTimeout;
|
||||
|
||||
private long requestTimeout;
|
||||
|
||||
private String index;
|
||||
|
||||
private int topKPerBucket = DEFAULT_TOP_K_PER_BUCKET;
|
||||
|
||||
private int topK = DEFAULT_TOP_K;
|
||||
|
||||
private String documentField = DEFAULT_DOCUMENT_FIELD;
|
||||
|
||||
public Builder withHost(String host) {
|
||||
Assert.hasText(host, "host must have a value");
|
||||
this.host = host;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withPort(int port) {
|
||||
Assert.isTrue(port > 0, "port must be positive");
|
||||
this.port = port;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withSslEnabled(boolean sslEnabled) {
|
||||
this.sslEnabled = sslEnabled;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withConnectionTimeout(long timeout) {
|
||||
Assert.isTrue(timeout >= 0, "timeout must be >= 0");
|
||||
this.connectionTimeout = timeout;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withRequestTimeout(long timeout) {
|
||||
Assert.isTrue(timeout >= 0, "timeout must be >= 0");
|
||||
this.requestTimeout = timeout;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withIndex(String index) {
|
||||
Assert.hasText(index, "index must have a value");
|
||||
this.index = index;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withTopKPerBucket(int topKPerBucket) {
|
||||
Assert.isTrue(topKPerBucket > 0, "topKPerBucket must be positive");
|
||||
this.topKPerBucket = topKPerBucket;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withTopK(int topK) {
|
||||
Assert.isTrue(topK > 0, "topK must be positive");
|
||||
this.topK = topK;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder withDocumentField(String documentField) {
|
||||
Assert.hasText(documentField, "documentField must have a value");
|
||||
this.documentField = documentField;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig build() {
|
||||
return new GemFireVectorStoreConfig(this);
|
||||
}
|
||||
private int buckets;
|
||||
|
||||
public int getBuckets() {
|
||||
return buckets;
|
||||
}
|
||||
|
||||
private String vectorSimilarityFunction;
|
||||
|
||||
public String getVectorSimilarityFunction() {
|
||||
return vectorSimilarityFunction;
|
||||
}
|
||||
|
||||
private String[] fields;
|
||||
|
||||
public String[] getFields() {
|
||||
return fields;
|
||||
}
|
||||
|
||||
// Query Defaults
|
||||
private static final String QUERY = "/query";
|
||||
|
||||
private static final String DISTANCE_METADATA_FIELD_NAME = "distance";
|
||||
|
||||
/**
|
||||
* Initializes the GemFireVectorStore after properties are set. This method is called
|
||||
* after all bean properties have been set and allows the bean to perform any
|
||||
* initialization it requires.
|
||||
*/
|
||||
@Override
|
||||
public void afterPropertiesSet() throws Exception {
|
||||
if (indexExists()) {
|
||||
deleteIndex();
|
||||
}
|
||||
createIndex();
|
||||
|
||||
}
|
||||
|
||||
private static final int DEFAULT_PORT = 9090;
|
||||
|
||||
public static final String DEFAULT_URI = "http{ssl}://{host}:{port}/gemfire-vectordb/v1/indexes";
|
||||
|
||||
private static final int DEFAULT_TOP_K_PER_BUCKET = 10;
|
||||
|
||||
private static final int DEFAULT_TOP_K = 10;
|
||||
|
||||
private static final String DEFAULT_DOCUMENT_FIELD = "document";
|
||||
|
||||
public String indexName;
|
||||
|
||||
public void setIndexName(String indexName) {
|
||||
this.indexName = indexName;
|
||||
/**
|
||||
* Checks if the index exists in the GemFireVectorStore.
|
||||
* @return {@code true} if the index exists, {@code false} otherwise
|
||||
*/
|
||||
public boolean indexExists() {
|
||||
String indexResponse = getIndex();
|
||||
return !indexResponse.isEmpty();
|
||||
}
|
||||
|
||||
public GemFireVectorStore(GemFireVectorStoreConfig config, EmbeddingModel embedding) {
|
||||
public String getIndex() {
|
||||
return client.get().uri("/" + indexName).retrieve().bodyToMono(String.class).onErrorReturn("").block();
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures and initializes a GemFireVectorStore instance based on the provided
|
||||
* configuration.
|
||||
* @param config the configuration for the GemFireVectorStore
|
||||
* @param embeddingModel the embedding client used for generating embeddings
|
||||
*/
|
||||
|
||||
public GemFireVectorStore(GemFireVectorStoreConfig config, EmbeddingModel embeddingModel) {
|
||||
Assert.notNull(config, "GemFireVectorStoreConfig must not be null");
|
||||
Assert.notNull(embedding, "EmbeddingModel must not be null");
|
||||
this.client = config.client;
|
||||
this.embeddingModel = embedding;
|
||||
this.topKPerBucket = config.topKPerBucket;
|
||||
this.topK = config.topK;
|
||||
this.documentField = config.documentField;
|
||||
Assert.notNull(embeddingModel, "EmbeddingModel must not be null");
|
||||
this.indexName = config.indexName;
|
||||
this.embeddingModel = embeddingModel;
|
||||
this.beamWidth = config.beamWidth;
|
||||
this.maxConnections = config.maxConnections;
|
||||
this.buckets = config.buckets;
|
||||
this.vectorSimilarityFunction = config.vectorSimilarityFunction;
|
||||
this.fields = config.fields;
|
||||
|
||||
String base = UriComponentsBuilder.fromUriString(DEFAULT_URI)
|
||||
.build(config.sslEnabled ? "s" : "", config.host, config.port)
|
||||
.toString();
|
||||
this.client = WebClient.create(base);
|
||||
}
|
||||
|
||||
private static final class CreateRequest {
|
||||
public static class CreateRequest {
|
||||
|
||||
@JsonProperty("name")
|
||||
private String name;
|
||||
private String indexName;
|
||||
|
||||
@JsonProperty("beam-width")
|
||||
private int beamWidth = 100;
|
||||
private int beamWidth;
|
||||
|
||||
@JsonProperty("max-connections")
|
||||
private int maxConnections = 16;
|
||||
private int maxConnections;
|
||||
|
||||
@JsonProperty("vector-similarity-function")
|
||||
private String vectorSimilarityFunction = "COSINE";
|
||||
private String vectorSimilarityFunction;
|
||||
|
||||
@JsonProperty("fields")
|
||||
private String[] fields = new String[] { "vector" };
|
||||
private String[] fields;
|
||||
|
||||
@JsonProperty("buckets")
|
||||
private int buckets = 0;
|
||||
private int buckets;
|
||||
|
||||
public CreateRequest() {
|
||||
}
|
||||
|
||||
public CreateRequest(String name) {
|
||||
this.name = name;
|
||||
public CreateRequest(String indexName) {
|
||||
this.indexName = indexName;
|
||||
}
|
||||
|
||||
public String getName() {
|
||||
return name;
|
||||
public String getIndexName() {
|
||||
return indexName;
|
||||
}
|
||||
|
||||
public void setName(String name) {
|
||||
this.name = name;
|
||||
public void setIndexName(String indexName) {
|
||||
this.indexName = indexName;
|
||||
}
|
||||
|
||||
public int getBeamWidth() {
|
||||
@@ -419,7 +374,7 @@ public class GemFireVectorStore implements VectorStore {
|
||||
// Compute and assign an embedding to the document.
|
||||
document.setEmbedding(this.embeddingModel.embed(document));
|
||||
List<Float> floatVector = document.getEmbedding().stream().map(Double::floatValue).toList();
|
||||
return new UploadRequest.Embedding(document.getId(), floatVector, documentField, document.getContent(),
|
||||
return new UploadRequest.Embedding(document.getId(), floatVector, DOCUMENT_FIELD, document.getContent(),
|
||||
document.getMetadata());
|
||||
}).toList());
|
||||
|
||||
@@ -454,7 +409,7 @@ public class GemFireVectorStore implements VectorStore {
|
||||
.block();
|
||||
}
|
||||
catch (Exception e) {
|
||||
logger.warn("Error removing embedding: " + e);
|
||||
logger.warn("Error removing embedding: {}", e.getMessage(), e);
|
||||
return Optional.of(false);
|
||||
}
|
||||
return Optional.of(true);
|
||||
@@ -463,22 +418,26 @@ public class GemFireVectorStore implements VectorStore {
|
||||
@Override
|
||||
public List<Document> similaritySearch(SearchRequest request) {
|
||||
if (request.hasFilterExpression()) {
|
||||
throw new UnsupportedOperationException("Gemfire does not support metadata filter expressions yet.");
|
||||
throw new UnsupportedOperationException("GemFire currently does not support metadata filter expressions.");
|
||||
}
|
||||
List<Double> vector = this.embeddingModel.embed(request.getQuery());
|
||||
List<Float> floatVector = vector.stream().map(Double::floatValue).toList();
|
||||
|
||||
return client.post()
|
||||
.uri("/" + indexName + QUERY)
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.bodyValue(new QueryRequest(floatVector, request.getTopK(), topKPerBucket, true))
|
||||
.bodyValue(new QueryRequest(floatVector, request.getTopK(), request.getTopK(), // TopKPerBucket
|
||||
true))
|
||||
.retrieve()
|
||||
.bodyToFlux(QueryResponse.class)
|
||||
.filter(r -> r.score >= request.getSimilarityThreshold())
|
||||
.map(r -> {
|
||||
Map<String, Object> metadata = r.metadata;
|
||||
if (r.metadata == null) {
|
||||
metadata = new HashMap<>();
|
||||
metadata.put(DOCUMENT_FIELD, "--Deleted--");
|
||||
}
|
||||
metadata.put(DISTANCE_METADATA_FIELD_NAME, 1 - r.score);
|
||||
String content = (String) metadata.remove(documentField);
|
||||
String content = (String) metadata.remove(DOCUMENT_FIELD);
|
||||
return new Document(r.key, content, metadata);
|
||||
})
|
||||
.collectList()
|
||||
@@ -486,10 +445,22 @@ public class GemFireVectorStore implements VectorStore {
|
||||
.block();
|
||||
}
|
||||
|
||||
public void createIndex(String indexName) throws JsonProcessingException {
|
||||
/**
|
||||
* Creates a new index in the GemFireVectorStore using specified parameters. This
|
||||
* method is invoked during initialization.
|
||||
* @throws JsonProcessingException if an error occurs during JSON processing
|
||||
*/
|
||||
public void createIndex() throws JsonProcessingException {
|
||||
CreateRequest createRequest = new CreateRequest(indexName);
|
||||
createRequest.setBeamWidth(beamWidth);
|
||||
createRequest.setMaxConnections(maxConnections);
|
||||
createRequest.setBuckets(buckets);
|
||||
createRequest.setVectorSimilarityFunction(vectorSimilarityFunction);
|
||||
createRequest.setFields(fields);
|
||||
|
||||
ObjectMapper objectMapper = new ObjectMapper();
|
||||
String index = objectMapper.writeValueAsString(createRequest);
|
||||
|
||||
client.post()
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.bodyValue(index)
|
||||
@@ -499,9 +470,8 @@ public class GemFireVectorStore implements VectorStore {
|
||||
.block();
|
||||
}
|
||||
|
||||
public void deleteIndex(String indexName) {
|
||||
public void deleteIndex() {
|
||||
DeleteRequest deleteRequest = new DeleteRequest();
|
||||
deleteRequest.setDeleteData(true);
|
||||
client.method(HttpMethod.DELETE)
|
||||
.uri("/" + indexName)
|
||||
.body(BodyInserters.fromValue(deleteRequest))
|
||||
@@ -511,6 +481,12 @@ public class GemFireVectorStore implements VectorStore {
|
||||
.block();
|
||||
}
|
||||
|
||||
/**
|
||||
* Handles exceptions that occur during HTTP client operations and maps them to
|
||||
* appropriate runtime exceptions.
|
||||
* @param ex the exception that occurred during HTTP client operation
|
||||
* @return a mapped runtime exception corresponding to the HTTP client exception
|
||||
*/
|
||||
private Throwable handleHttpClientException(Throwable ex) {
|
||||
if (!(ex instanceof WebClientResponseException clientException)) {
|
||||
throw new RuntimeException(String.format("Got an unexpected error: %s", ex));
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
package org.springframework.ai.vectorstore;
|
||||
|
||||
import org.springframework.util.Assert;
|
||||
|
||||
public final class GemFireVectorStoreConfig {
|
||||
|
||||
// Create Index DEFAULT Values
|
||||
public static final String DEFAULT_HOST = "localhost";
|
||||
|
||||
public static final int DEFAULT_PORT = 8080;
|
||||
|
||||
public static final String DEFAULT_INDEX_NAME = "spring-ai-gemfire-index";
|
||||
|
||||
public static final int UPPER_BOUND_BEAM_WIDTH = 3200;
|
||||
|
||||
public static final int DEFAULT_BEAM_WIDTH = 100;
|
||||
|
||||
private static final int UPPER_BOUND_MAX_CONNECTIONS = 512;
|
||||
|
||||
public static final int DEFAULT_MAX_CONNECTIONS = 16;
|
||||
|
||||
public static final String DEFAULT_SIMILARITY_FUNCTION = "COSINE";
|
||||
|
||||
public static final String[] DEFAULT_FIELDS = new String[] {};
|
||||
|
||||
public static final int DEFAULT_BUCKETS = 0;
|
||||
|
||||
public static final boolean DEFAULT_SSL_ENABLED = false;
|
||||
|
||||
String host = GemFireVectorStoreConfig.DEFAULT_HOST;
|
||||
|
||||
int port = DEFAULT_PORT;
|
||||
|
||||
String indexName = DEFAULT_INDEX_NAME;
|
||||
|
||||
int beamWidth = DEFAULT_BEAM_WIDTH;
|
||||
|
||||
int maxConnections = DEFAULT_MAX_CONNECTIONS;
|
||||
|
||||
String vectorSimilarityFunction = DEFAULT_SIMILARITY_FUNCTION;
|
||||
|
||||
String[] fields = DEFAULT_FIELDS;
|
||||
|
||||
int buckets = DEFAULT_BUCKETS;
|
||||
|
||||
boolean sslEnabled = DEFAULT_SSL_ENABLED;
|
||||
|
||||
public GemFireVectorStoreConfig() {
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setHost(String host) {
|
||||
Assert.hasText(host, "host must have a value");
|
||||
this.host = host;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setPort(int port) {
|
||||
Assert.isTrue(port > 0, "port must be positive");
|
||||
this.port = port;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setSslEnabled(boolean sslEnabled) {
|
||||
this.sslEnabled = sslEnabled;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setIndexName(String indexName) {
|
||||
Assert.hasText(indexName, "indexName must have a value");
|
||||
this.indexName = indexName;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setBeamWidth(int beamWidth) {
|
||||
Assert.isTrue(beamWidth > 0, "beamWidth must be positive");
|
||||
Assert.isTrue(beamWidth <= GemFireVectorStoreConfig.UPPER_BOUND_BEAM_WIDTH,
|
||||
"beamWidth must be less than or equal to " + GemFireVectorStoreConfig.UPPER_BOUND_BEAM_WIDTH);
|
||||
this.beamWidth = beamWidth;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setMaxConnections(int maxConnections) {
|
||||
Assert.isTrue(maxConnections > 0, "maxConnections must be positive");
|
||||
Assert.isTrue(maxConnections <= GemFireVectorStoreConfig.UPPER_BOUND_MAX_CONNECTIONS,
|
||||
"maxConnections must be less than or equal to " + GemFireVectorStoreConfig.UPPER_BOUND_MAX_CONNECTIONS);
|
||||
this.maxConnections = maxConnections;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setBuckets(int buckets) {
|
||||
Assert.isTrue(buckets >= 0, "bucket must be 1 or more");
|
||||
this.buckets = buckets;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setVectorSimilarityFunction(String vectorSimilarityFunction) {
|
||||
Assert.hasText(vectorSimilarityFunction, "vectorSimilarityFunction must have a value");
|
||||
this.vectorSimilarityFunction = vectorSimilarityFunction;
|
||||
return this;
|
||||
}
|
||||
|
||||
public GemFireVectorStoreConfig setFields(String[] fields) {
|
||||
this.fields = fields;
|
||||
return this;
|
||||
}
|
||||
|
||||
}
|
||||
@@ -26,16 +26,17 @@ import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
import com.github.dockerjava.api.model.ExposedPort;
|
||||
import com.github.dockerjava.api.model.PortBinding;
|
||||
import com.github.dockerjava.api.model.Ports;
|
||||
import com.vmware.gemfire.testcontainers.GemFireCluster;
|
||||
import org.awaitility.Awaitility;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.transformers.TransformersEmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.GemFireVectorStore.GemFireVectorStoreConfig;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
@@ -46,11 +47,40 @@ import org.springframework.core.io.DefaultResourceLoader;
|
||||
* @author Geet Rawat
|
||||
* @since 1.0.0
|
||||
*/
|
||||
@EnabledIfEnvironmentVariable(named = "GEMFIRE_HOST", matches = ".+")
|
||||
public class GemFireVectorStoreIT {
|
||||
|
||||
public static final String INDEX_NAME = "spring-ai-index1";
|
||||
|
||||
private static GemFireCluster gemFireCluster;
|
||||
|
||||
private static final int HTTP_SERVICE_PORT = 9090;
|
||||
|
||||
private static final int LOCATOR_COUNT = 1;
|
||||
|
||||
private static final int SERVER_COUNT = 1;
|
||||
|
||||
@AfterAll
|
||||
public static void stopGemFireCluster() {
|
||||
gemFireCluster.close();
|
||||
}
|
||||
|
||||
@BeforeAll
|
||||
public static void startGemFireCluster() {
|
||||
Ports.Binding hostPort = Ports.Binding.bindPort(HTTP_SERVICE_PORT);
|
||||
ExposedPort exposedPort = new ExposedPort(HTTP_SERVICE_PORT);
|
||||
PortBinding mappedPort = new PortBinding(hostPort, exposedPort);
|
||||
gemFireCluster = new GemFireCluster("gemfire/gemfire-all:10.1-jdk17", LOCATOR_COUNT, SERVER_COUNT);
|
||||
gemFireCluster.withConfiguration(GemFireCluster.SERVER_GLOB,
|
||||
container -> container.withExposedPorts(HTTP_SERVICE_PORT)
|
||||
.withCreateContainerCmdModifier(cmd -> cmd.getHostConfig().withPortBindings(mappedPort)));
|
||||
gemFireCluster.withGemFireProperty(GemFireCluster.SERVER_GLOB, "http-service-port",
|
||||
Integer.toString(HTTP_SERVICE_PORT));
|
||||
gemFireCluster.acceptLicense().start();
|
||||
|
||||
System.setProperty("spring.data.gemfire.pool.locators",
|
||||
String.format("localhost[%d]", gemFireCluster.getLocatorPort()));
|
||||
}
|
||||
|
||||
List<Document> documents = List.of(
|
||||
new Document("1", getText("classpath:/test/data/spring.ai.txt"), Map.of("meta1", "meta1")),
|
||||
new Document("2", getText("classpath:/test/data/time.shelter.txt"), Map.of()),
|
||||
@@ -69,25 +99,16 @@ public class GemFireVectorStoreIT {
|
||||
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
|
||||
.withUserConfiguration(TestApplication.class);
|
||||
|
||||
@BeforeEach
|
||||
public void createIndex() {
|
||||
contextRunner.run(c -> c.getBean(GemFireVectorStore.class).createIndex(INDEX_NAME));
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
public void deleteIndex() {
|
||||
contextRunner.run(c -> c.getBean(GemFireVectorStore.class).deleteIndex(INDEX_NAME));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void addAndDeleteEmbeddingTest() {
|
||||
contextRunner.run(context -> {
|
||||
VectorStore vectorStore = context.getBean(VectorStore.class);
|
||||
vectorStore.add(documents);
|
||||
vectorStore.delete(documents.stream().map(doc -> doc.getId()).toList());
|
||||
Awaitility.await().atMost(1, MINUTES).until(() -> {
|
||||
return vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(3));
|
||||
}, hasSize(0));
|
||||
Awaitility.await()
|
||||
.atMost(1, MINUTES)
|
||||
.until(() -> vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(3)),
|
||||
hasSize(0));
|
||||
});
|
||||
}
|
||||
|
||||
@@ -97,14 +118,15 @@ public class GemFireVectorStoreIT {
|
||||
VectorStore vectorStore = context.getBean(VectorStore.class);
|
||||
vectorStore.add(documents);
|
||||
|
||||
Awaitility.await().atMost(1, MINUTES).until(() -> {
|
||||
return vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(1));
|
||||
}, hasSize(1));
|
||||
Awaitility.await()
|
||||
.atMost(1, MINUTES)
|
||||
.until(() -> vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(1)),
|
||||
hasSize(1));
|
||||
|
||||
List<Document> results = vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(5));
|
||||
Document resultDoc = results.get(0);
|
||||
assertThat(resultDoc.getId()).isEqualTo(documents.get(2).getId());
|
||||
assertThat(resultDoc.getContent()).contains("The Great Depression (1929–1939) was an economic shock");
|
||||
assertThat(resultDoc.getContent()).contains("The Great Depression (1929–1939)" + " was an economic shock");
|
||||
assertThat(resultDoc.getMetadata()).hasSize(2);
|
||||
assertThat(resultDoc.getMetadata()).containsKey("meta2");
|
||||
assertThat(resultDoc.getMetadata()).containsKey("distance");
|
||||
@@ -120,9 +142,10 @@ public class GemFireVectorStoreIT {
|
||||
Collections.singletonMap("meta1", "meta1"));
|
||||
vectorStore.add(List.of(document));
|
||||
SearchRequest springSearchRequest = SearchRequest.query("Spring").withTopK(5);
|
||||
Awaitility.await().atMost(1, MINUTES).until(() -> {
|
||||
return vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(1));
|
||||
}, hasSize(1));
|
||||
Awaitility.await()
|
||||
.atMost(1, MINUTES)
|
||||
.until(() -> vectorStore.similaritySearch(SearchRequest.query("Great Depression").withTopK(1)),
|
||||
hasSize(1));
|
||||
List<Document> results = vectorStore.similaritySearch(springSearchRequest);
|
||||
Document resultDoc = results.get(0);
|
||||
assertThat(resultDoc.getId()).isEqualTo(document.getId());
|
||||
@@ -131,7 +154,7 @@ public class GemFireVectorStoreIT {
|
||||
assertThat(resultDoc.getMetadata()).containsKey("distance");
|
||||
|
||||
Document sameIdDocument = new Document(document.getId(),
|
||||
"The World is Big and Salvation Lurks Around the Corner",
|
||||
"The World is Big and Salvation Lurks " + "Around the Corner",
|
||||
Collections.singletonMap("meta2", "meta2"));
|
||||
|
||||
vectorStore.add(List.of(sameIdDocument));
|
||||
@@ -141,7 +164,7 @@ public class GemFireVectorStoreIT {
|
||||
assertThat(results).hasSize(1);
|
||||
resultDoc = results.get(0);
|
||||
assertThat(resultDoc.getId()).isEqualTo(document.getId());
|
||||
assertThat(resultDoc.getContent()).isEqualTo("The World is Big and Salvation Lurks Around the Corner");
|
||||
assertThat(resultDoc.getContent()).isEqualTo("The World is Big and Salvation" + " Lurks Around the Corner");
|
||||
assertThat(resultDoc.getMetadata()).containsKey("meta2");
|
||||
assertThat(resultDoc.getMetadata()).containsKey("distance");
|
||||
});
|
||||
@@ -154,10 +177,11 @@ public class GemFireVectorStoreIT {
|
||||
VectorStore vectorStore = context.getBean(VectorStore.class);
|
||||
vectorStore.add(documents);
|
||||
|
||||
Awaitility.await().atMost(1, MINUTES).until(() -> {
|
||||
return vectorStore
|
||||
.similaritySearch(SearchRequest.query("Great Depression").withTopK(5).withSimilarityThresholdAll());
|
||||
}, hasSize(3));
|
||||
Awaitility.await()
|
||||
.atMost(1, MINUTES)
|
||||
.until(() -> vectorStore
|
||||
.similaritySearch(SearchRequest.query("Great Depression").withTopK(5).withSimilarityThresholdAll()),
|
||||
hasSize(3));
|
||||
|
||||
List<Document> fullResult = vectorStore
|
||||
.similaritySearch(SearchRequest.query("Depression").withTopK(5).withSimilarityThresholdAll());
|
||||
@@ -173,7 +197,7 @@ public class GemFireVectorStoreIT {
|
||||
|
||||
Document resultDoc = results.get(0);
|
||||
assertThat(resultDoc.getId()).isEqualTo(documents.get(2).getId());
|
||||
assertThat(resultDoc.getContent()).contains("The Great Depression (1929–1939) was an economic shock");
|
||||
assertThat(resultDoc.getContent()).contains("The Great Depression " + "(1929–1939) was an economic shock");
|
||||
assertThat(resultDoc.getMetadata()).containsKey("meta2");
|
||||
assertThat(resultDoc.getMetadata()).containsKey("distance");
|
||||
});
|
||||
@@ -185,14 +209,14 @@ public class GemFireVectorStoreIT {
|
||||
|
||||
@Bean
|
||||
public GemFireVectorStoreConfig gemfireVectorStoreConfig() {
|
||||
return GemFireVectorStoreConfig.builder().withHost("localhost").build();
|
||||
return new GemFireVectorStoreConfig().setHost("localhost")
|
||||
.setPort(HTTP_SERVICE_PORT)
|
||||
.setIndexName(INDEX_NAME);
|
||||
}
|
||||
|
||||
@Bean
|
||||
public GemFireVectorStore vectorStore(GemFireVectorStoreConfig config, EmbeddingModel embeddingModel) {
|
||||
GemFireVectorStore gemFireVectorStore = new GemFireVectorStore(config, embeddingModel);
|
||||
gemFireVectorStore.setIndexName(INDEX_NAME);
|
||||
return gemFireVectorStore;
|
||||
return new GemFireVectorStore(config, embeddingModel);
|
||||
}
|
||||
|
||||
@Bean
|
||||
@@ -202,4 +226,4 @@ public class GemFireVectorStoreIT {
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user