PgVectorStore package rename and builder support
- Move PgVectorStore and related classes to org.springframework.ai.pg.vectorstore package - Update builder pattern to use more idiomatic method names (e.g. withSchemaName -> schemaName) - Deprecate existing constructors and old Builder class in favor of new static builder() method - Update tests to reflect the new builder style usage - Update docs
This commit is contained in:
committed by
Mark Pollack
parent
9e18652bb8
commit
257b963da9
@@ -100,6 +100,8 @@ spring:
|
||||
index-type: HNSW
|
||||
distance-type: COSINE_DISTANCE
|
||||
dimensions: 1536
|
||||
batching-strategy: TOKEN_COUNT # Optional: Controls how documents are batched for embedding
|
||||
max-document-batch-size: 10000 # Optional: Maximum number of documents per batch
|
||||
----
|
||||
|
||||
TIP: If you run PGvector as a Spring Boot dev service via link:https://docs.spring.io/spring-boot/reference/features/dev-services.html#features.dev-services.docker-compose[Docker Compose]
|
||||
@@ -108,7 +110,7 @@ you don't need to configure URL, username and password since they are autoconfig
|
||||
|
||||
TIP: Check the list of xref:#pgvector-properties[configuration parameters] to learn about the default values and configuration options.
|
||||
|
||||
Now you can auto-wire the `PgVectorStore` in your application and use it
|
||||
Now you can auto-wire the `VectorStore` in your application and use it
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@@ -137,7 +139,7 @@ You can use the following properties in your Spring Boot configuration to custom
|
||||
|===
|
||||
|Property| Description | Default value
|
||||
|
||||
|`spring.ai.vectorstore.pgvector.index-type`| Nearest neighbor search index type. Options are `NONE` - exact nearest neighbor search, `IVFFlat` - index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff). `HNSW` - creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.| HNSW
|
||||
|`spring.ai.vectorstore.pgvector.index-type`| Nearest neighbor search index type. Options are `NONE` - exact nearest neighbor search, `IVFFlat` - index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff). `HNSW` - creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There's no training step like IVFFlat, so the index can be created without any data in the table.| HNSW
|
||||
|`spring.ai.vectorstore.pgvector.distance-type`| Search distance type. Defaults to `COSINE_DISTANCE`. But if vectors are normalized to length 1, you can use `EUCLIDEAN_DISTANCE` or `NEGATIVE_INNER_PRODUCT` for best performance.| COSINE_DISTANCE
|
||||
|`spring.ai.vectorstore.pgvector.dimensions`| Embeddings dimension. If not specified explicitly the PgVectorStore will retrieve the dimensions form the provided `EmbeddingModel`. Dimensions are set to the embedding column the on table creation. If you change the dimensions your would have to re-create the vector_store table as well. | -
|
||||
|`spring.ai.vectorstore.pgvector.remove-existing-vector-store-table` | Deletes the existing `vector_store` table on start up. | false
|
||||
@@ -145,6 +147,8 @@ You can use the following properties in your Spring Boot configuration to custom
|
||||
|`spring.ai.vectorstore.pgvector.schema-name` | Vector store schema name | `public`
|
||||
|`spring.ai.vectorstore.pgvector.table-name` | Vector store table name | `vector_store`
|
||||
|`spring.ai.vectorstore.pgvector.schema-validation` | Enables schema and table name validation to ensure they are valid and existing objects. | false
|
||||
|`spring.ai.vectorstore.pgvector.batching-strategy` | Strategy for batching documents when calculating embeddings. Options are `TOKEN_COUNT` or `FIXED_SIZE`. | TOKEN_COUNT
|
||||
|`spring.ai.vectorstore.pgvector.max-document-batch-size` | Maximum number of documents to process in a single batch. | 10000
|
||||
|
||||
|===
|
||||
|
||||
@@ -182,7 +186,7 @@ vectorStore.similaritySearch(SearchRequest.defaults()
|
||||
b.eq("article_type", "blog")).build()));
|
||||
----
|
||||
|
||||
NOTE: These filter expressions are converted into the equivalent PgVector filters.
|
||||
NOTE: These filter expressions are converted into PostgreSQL JSON path expressions for efficient metadata filtering.
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
@@ -216,7 +220,17 @@ To configure PgVector in your application, you can use the following setup:
|
||||
----
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
return new PgVectorStore(jdbcTemplate, embeddingModel);
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.dimensions(1536) // Optional: defaults to model dimensions or 1536
|
||||
.distanceType(COSINE_DISTANCE) // Optional: defaults to COSINE_DISTANCE
|
||||
.indexType(HNSW) // Optional: defaults to HNSW
|
||||
.initializeSchema(true) // Optional: defaults to false
|
||||
.schemaName("public") // Optional: defaults to "public"
|
||||
.vectorTableName("vector_store") // Optional: defaults to "vector_store"
|
||||
.maxDocumentBatchSize(10000) // Optional: defaults to 10000
|
||||
.build();
|
||||
}
|
||||
----
|
||||
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
[[upgrade-notes]]
|
||||
= Upgrading Notes
|
||||
|
||||
== Upgrading to 1.0.0.RC1
|
||||
|
||||
== Upgrading to 1.0.0.M5
|
||||
|
||||
* Vector Builders have been refactored for consistency.
|
||||
* Current VectorStore implementation constructors have been deprecated, use the builder pattern.
|
||||
* VectorStore implementation packages have been moved into unique package names, avoiding conflicts across artifact. For example `org.springframework.ai.vectorstore` to `org.springframework.ai.pgvector.vectorstore`.
|
||||
|
||||
== Upgrading to 1.0.0.RC3
|
||||
|
||||
* The type of the portable chat options (`frequencyPenalty`, `presencePenalty`, `temperature`, `topP`) has been changed from `Float` to `Double`.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ import org.springframework.ai.rag.preretrieval.query.transformation.TranslationQ
|
||||
import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever;
|
||||
import org.springframework.ai.reader.markdown.MarkdownDocumentReader;
|
||||
import org.springframework.ai.reader.markdown.config.MarkdownDocumentReaderConfig;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
@@ -29,7 +29,7 @@ import org.springframework.ai.integration.tests.TestApplication;
|
||||
import org.springframework.ai.rag.Query;
|
||||
import org.springframework.ai.rag.retrieval.search.DocumentRetriever;
|
||||
import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.Filter;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
@@ -23,7 +23,7 @@ import io.micrometer.observation.ObservationRegistry;
|
||||
import org.springframework.ai.embedding.BatchingStrategy;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.embedding.TokenCountBatchingStrategy;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.vectorstore.observation.VectorStoreObservationConvention;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
@@ -62,18 +62,21 @@ public class PgVectorStoreAutoConfiguration {
|
||||
|
||||
var initializeSchema = properties.isInitializeSchema();
|
||||
|
||||
return new PgVectorStore.Builder(jdbcTemplate, embeddingModel).withSchemaName(properties.getSchemaName())
|
||||
.withVectorTableName(properties.getTableName())
|
||||
.withVectorTableValidationsEnabled(properties.isSchemaValidation())
|
||||
.withDimensions(properties.getDimensions())
|
||||
.withDistanceType(properties.getDistanceType())
|
||||
.withRemoveExistingVectorStoreTable(properties.isRemoveExistingVectorStoreTable())
|
||||
.withIndexType(properties.getIndexType())
|
||||
.withInitializeSchema(initializeSchema)
|
||||
.withObservationRegistry(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
|
||||
.withSearchObservationConvention(customObservationConvention.getIfAvailable(() -> null))
|
||||
.withBatchingStrategy(batchingStrategy)
|
||||
.withMaxDocumentBatchSize(properties.getMaxDocumentBatchSize())
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName(properties.getSchemaName())
|
||||
.vectorTableName(properties.getTableName())
|
||||
.vectorTableValidationsEnabled(properties.isSchemaValidation())
|
||||
.dimensions(properties.getDimensions())
|
||||
.distanceType(properties.getDistanceType())
|
||||
.removeExistingVectorStoreTable(properties.isRemoveExistingVectorStoreTable())
|
||||
.indexType(properties.getIndexType())
|
||||
.initializeSchema(initializeSchema)
|
||||
.observationRegistry(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
|
||||
.customObservationConvention(customObservationConvention.getIfAvailable(() -> null))
|
||||
.batchingStrategy(batchingStrategy)
|
||||
.maxDocumentBatchSize(properties.getMaxDocumentBatchSize())
|
||||
.build();
|
||||
}
|
||||
|
||||
|
||||
@@ -17,9 +17,9 @@
|
||||
package org.springframework.ai.autoconfigure.vectorstore.pgvector;
|
||||
|
||||
import org.springframework.ai.autoconfigure.vectorstore.CommonVectorStoreProperties;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgDistanceType;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgDistanceType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
|
||||
/**
|
||||
|
||||
@@ -33,7 +33,7 @@ import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.observation.conventions.VectorStoreProvider;
|
||||
import org.springframework.ai.transformers.TransformersEmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.observation.VectorStoreObservationContext;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
|
||||
@@ -18,9 +18,9 @@ package org.springframework.ai.autoconfigure.vectorstore.pgvector;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import org.springframework.ai.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgDistanceType;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgDistanceType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgIndexType;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
@@ -43,6 +43,9 @@ import org.springframework.ai.embedding.TokenCountBatchingStrategy;
|
||||
import org.springframework.ai.observation.conventions.VectorStoreProvider;
|
||||
import org.springframework.ai.observation.conventions.VectorStoreSimilarityMetric;
|
||||
import org.springframework.ai.util.JacksonUtils;
|
||||
import org.springframework.ai.vectorstore.AbstractVectorStoreBuilder;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionConverter;
|
||||
import org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore;
|
||||
import org.springframework.ai.vectorstore.observation.VectorStoreObservationContext;
|
||||
@@ -54,6 +57,7 @@ import org.springframework.jdbc.core.RowMapper;
|
||||
import org.springframework.jdbc.core.SqlTypeValue;
|
||||
import org.springframework.jdbc.core.StatementCreatorUtils;
|
||||
import org.springframework.lang.Nullable;
|
||||
import org.springframework.util.Assert;
|
||||
import org.springframework.util.StringUtils;
|
||||
|
||||
/**
|
||||
@@ -99,8 +103,6 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
|
||||
private final JdbcTemplate jdbcTemplate;
|
||||
|
||||
private final EmbeddingModel embeddingModel;
|
||||
|
||||
private final String schemaName;
|
||||
|
||||
private final boolean schemaValidation;
|
||||
@@ -123,15 +125,18 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
|
||||
private final int maxDocumentBatchSize;
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public PgVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
this(jdbcTemplate, embeddingModel, INVALID_EMBEDDING_DIMENSION, PgDistanceType.COSINE_DISTANCE, false,
|
||||
PgIndexType.NONE, false);
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public PgVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions) {
|
||||
this(jdbcTemplate, embeddingModel, dimensions, PgDistanceType.COSINE_DISTANCE, false, PgIndexType.NONE, false);
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public PgVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions,
|
||||
PgDistanceType distanceType, boolean removeExistingVectorStoreTable, PgIndexType createIndexMethod,
|
||||
boolean initializeSchema) {
|
||||
@@ -140,60 +145,62 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
createIndexMethod, initializeSchema);
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public PgVectorStore(String vectorTableName, JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel,
|
||||
int dimensions, PgDistanceType distanceType, boolean removeExistingVectorStoreTable,
|
||||
PgIndexType createIndexMethod, boolean initializeSchema) {
|
||||
|
||||
this(DEFAULT_SCHEMA_NAME, vectorTableName, DEFAULT_SCHEMA_VALIDATION, jdbcTemplate, embeddingModel, dimensions,
|
||||
distanceType, removeExistingVectorStoreTable, createIndexMethod, initializeSchema);
|
||||
this(builder().jdbcTemplate(jdbcTemplate)
|
||||
.schemaName(DEFAULT_SCHEMA_NAME)
|
||||
.vectorTableName(vectorTableName)
|
||||
.vectorTableValidationsEnabled(DEFAULT_SCHEMA_VALIDATION)
|
||||
.dimensions(dimensions)
|
||||
.distanceType(distanceType)
|
||||
.removeExistingVectorStoreTable(removeExistingVectorStoreTable)
|
||||
.indexType(createIndexMethod)
|
||||
.initializeSchema(initializeSchema));
|
||||
}
|
||||
|
||||
private PgVectorStore(String schemaName, String vectorTableName, boolean vectorTableValidationsEnabled,
|
||||
JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions, PgDistanceType distanceType,
|
||||
boolean removeExistingVectorStoreTable, PgIndexType createIndexMethod, boolean initializeSchema) {
|
||||
/**
|
||||
* @param builder {@link VectorStore.Builder} for pg vector store
|
||||
*/
|
||||
protected PgVectorStore(PgVectorStoreBuilder builder) {
|
||||
super(builder);
|
||||
|
||||
this(schemaName, vectorTableName, vectorTableValidationsEnabled, jdbcTemplate, embeddingModel, dimensions,
|
||||
distanceType, removeExistingVectorStoreTable, createIndexMethod, initializeSchema,
|
||||
ObservationRegistry.NOOP, null, new TokenCountBatchingStrategy(), MAX_DOCUMENT_BATCH_SIZE);
|
||||
}
|
||||
|
||||
private PgVectorStore(String schemaName, String vectorTableName, boolean vectorTableValidationsEnabled,
|
||||
JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions, PgDistanceType distanceType,
|
||||
boolean removeExistingVectorStoreTable, PgIndexType createIndexMethod, boolean initializeSchema,
|
||||
ObservationRegistry observationRegistry, VectorStoreObservationConvention customObservationConvention,
|
||||
BatchingStrategy batchingStrategy, int maxDocumentBatchSize) {
|
||||
|
||||
super(observationRegistry, customObservationConvention);
|
||||
Assert.notNull(builder.jdbcTemplate, "JdbcTemplate must not be null");
|
||||
|
||||
this.objectMapper = JsonMapper.builder().addModules(JacksonUtils.instantiateAvailableModules()).build();
|
||||
|
||||
this.vectorTableName = (null == vectorTableName || vectorTableName.isEmpty()) ? DEFAULT_TABLE_NAME
|
||||
: vectorTableName.trim();
|
||||
String vectorTable = builder.vectorTableName;
|
||||
this.vectorTableName = (null == vectorTable || vectorTable.isEmpty()) ? DEFAULT_TABLE_NAME : vectorTable.trim();
|
||||
logger.info("Using the vector table name: {}. Is empty: {}", this.vectorTableName,
|
||||
(vectorTableName == null || vectorTableName.isEmpty()));
|
||||
(this.vectorTableName == null || this.vectorTableName.isEmpty()));
|
||||
|
||||
this.vectorIndexName = this.vectorTableName.equals(DEFAULT_TABLE_NAME) ? DEFAULT_VECTOR_INDEX_NAME
|
||||
: this.vectorTableName + "_index";
|
||||
|
||||
this.schemaName = schemaName;
|
||||
this.schemaValidation = vectorTableValidationsEnabled;
|
||||
this.schemaName = builder.schemaName;
|
||||
this.schemaValidation = builder.vectorTableValidationsEnabled;
|
||||
|
||||
this.jdbcTemplate = jdbcTemplate;
|
||||
this.embeddingModel = embeddingModel;
|
||||
this.dimensions = dimensions;
|
||||
this.distanceType = distanceType;
|
||||
this.removeExistingVectorStoreTable = removeExistingVectorStoreTable;
|
||||
this.createIndexMethod = createIndexMethod;
|
||||
this.initializeSchema = initializeSchema;
|
||||
this.schemaValidator = new PgVectorSchemaValidator(jdbcTemplate);
|
||||
this.batchingStrategy = batchingStrategy;
|
||||
this.maxDocumentBatchSize = maxDocumentBatchSize;
|
||||
this.jdbcTemplate = builder.jdbcTemplate;
|
||||
this.dimensions = builder.dimensions;
|
||||
this.distanceType = builder.distanceType;
|
||||
this.removeExistingVectorStoreTable = builder.removeExistingVectorStoreTable;
|
||||
this.createIndexMethod = builder.indexType;
|
||||
this.initializeSchema = builder.initializeSchema;
|
||||
this.schemaValidator = new PgVectorSchemaValidator(this.jdbcTemplate);
|
||||
this.batchingStrategy = builder.batchingStrategy;
|
||||
this.maxDocumentBatchSize = builder.maxDocumentBatchSize;
|
||||
}
|
||||
|
||||
public PgDistanceType getDistanceType() {
|
||||
return this.distanceType;
|
||||
}
|
||||
|
||||
public static PgVectorStoreBuilder builder() {
|
||||
return new PgVectorStoreBuilder();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void doAdd(List<Document> documents) {
|
||||
List<float[]> embeddings = this.embeddingModel.embed(documents, EmbeddingOptionsBuilder.builder().build(),
|
||||
@@ -527,6 +534,94 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
|
||||
}
|
||||
|
||||
public static class PgVectorStoreBuilder extends AbstractVectorStoreBuilder<PgVectorStoreBuilder> {
|
||||
|
||||
private JdbcTemplate jdbcTemplate;
|
||||
|
||||
private String schemaName = PgVectorStore.DEFAULT_SCHEMA_NAME;
|
||||
|
||||
private String vectorTableName = PgVectorStore.DEFAULT_TABLE_NAME;
|
||||
|
||||
private boolean vectorTableValidationsEnabled = PgVectorStore.DEFAULT_SCHEMA_VALIDATION;
|
||||
|
||||
private int dimensions = PgVectorStore.INVALID_EMBEDDING_DIMENSION;
|
||||
|
||||
private PgDistanceType distanceType = PgDistanceType.COSINE_DISTANCE;
|
||||
|
||||
private boolean removeExistingVectorStoreTable = false;
|
||||
|
||||
private PgIndexType indexType = PgIndexType.HNSW;
|
||||
|
||||
private boolean initializeSchema;
|
||||
|
||||
private BatchingStrategy batchingStrategy = new TokenCountBatchingStrategy();
|
||||
|
||||
private int maxDocumentBatchSize = MAX_DOCUMENT_BATCH_SIZE;
|
||||
|
||||
public PgVectorStoreBuilder jdbcTemplate(JdbcTemplate jdbcTemplate) {
|
||||
Assert.notNull(jdbcTemplate, "JdbcTemplate must not be null");
|
||||
this.jdbcTemplate = jdbcTemplate;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder schemaName(String schemaName) {
|
||||
this.schemaName = schemaName;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder vectorTableName(String vectorTableName) {
|
||||
this.vectorTableName = vectorTableName;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder vectorTableValidationsEnabled(boolean vectorTableValidationsEnabled) {
|
||||
this.vectorTableValidationsEnabled = vectorTableValidationsEnabled;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder dimensions(int dimensions) {
|
||||
this.dimensions = dimensions;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder distanceType(PgDistanceType distanceType) {
|
||||
this.distanceType = distanceType;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder removeExistingVectorStoreTable(boolean removeExistingVectorStoreTable) {
|
||||
this.removeExistingVectorStoreTable = removeExistingVectorStoreTable;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder indexType(PgIndexType indexType) {
|
||||
this.indexType = indexType;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder initializeSchema(boolean initializeSchema) {
|
||||
this.initializeSchema = initializeSchema;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder batchingStrategy(BatchingStrategy batchingStrategy) {
|
||||
this.batchingStrategy = batchingStrategy;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStoreBuilder maxDocumentBatchSize(int maxDocumentBatchSize) {
|
||||
this.maxDocumentBatchSize = maxDocumentBatchSize;
|
||||
return this;
|
||||
}
|
||||
|
||||
public PgVectorStore build() {
|
||||
validate();
|
||||
return new PgVectorStore(this);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public static class Builder {
|
||||
|
||||
private final JdbcTemplate jdbcTemplate;
|
||||
@@ -558,7 +653,6 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
@Nullable
|
||||
private VectorStoreObservationConvention searchObservationConvention;
|
||||
|
||||
// Builder constructor with mandatory parameters
|
||||
public Builder(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
if (jdbcTemplate == null || embeddingModel == null) {
|
||||
throw new IllegalArgumentException("JdbcTemplate and EmbeddingModel must not be null");
|
||||
@@ -628,11 +722,20 @@ public class PgVectorStore extends AbstractObservationVectorStore implements Ini
|
||||
}
|
||||
|
||||
public PgVectorStore build() {
|
||||
return new PgVectorStore(this.schemaName, this.vectorTableName, this.vectorTableValidationsEnabled,
|
||||
this.jdbcTemplate, this.embeddingModel, this.dimensions, this.distanceType,
|
||||
this.removeExistingVectorStoreTable, this.indexType, this.initializeSchema,
|
||||
this.observationRegistry, this.searchObservationConvention, this.batchingStrategy,
|
||||
this.maxDocumentBatchSize);
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.schemaName(this.schemaName)
|
||||
.vectorTableName(this.vectorTableName)
|
||||
.vectorTableValidationsEnabled(this.vectorTableValidationsEnabled)
|
||||
.dimensions(this.dimensions)
|
||||
.distanceType(this.distanceType)
|
||||
.removeExistingVectorStoreTable(this.removeExistingVectorStoreTable)
|
||||
.indexType(this.indexType)
|
||||
.initializeSchema(this.initializeSchema)
|
||||
.batchingStrategy(this.batchingStrategy)
|
||||
.maxDocumentBatchSize(this.maxDocumentBatchSize)
|
||||
.build();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Provides the API for embedding observations.
|
||||
*/
|
||||
@NonNullApi
|
||||
@NonNullFields
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import org.springframework.lang.NonNullApi;
|
||||
import org.springframework.lang.NonNullFields;
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
@@ -47,7 +47,12 @@ public class PgVectorEmbeddingDimensionsTests {
|
||||
|
||||
final int explicitDimensions = 696;
|
||||
|
||||
var dim = new PgVectorStore(this.jdbcTemplate, this.embeddingModel, explicitDimensions).embeddingDimensions();
|
||||
PgVectorStore pgVectorStore = PgVectorStore.builder()
|
||||
.jdbcTemplate(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.dimensions(explicitDimensions)
|
||||
.build();
|
||||
var dim = pgVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(explicitDimensions);
|
||||
verify(this.embeddingModel, never()).dimensions();
|
||||
@@ -57,7 +62,11 @@ public class PgVectorEmbeddingDimensionsTests {
|
||||
public void embeddingModelDimensions() {
|
||||
given(this.embeddingModel.dimensions()).willReturn(969);
|
||||
|
||||
var dim = new PgVectorStore(this.jdbcTemplate, this.embeddingModel).embeddingDimensions();
|
||||
PgVectorStore pgVectorStore = PgVectorStore.builder()
|
||||
.jdbcTemplate(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.build();
|
||||
var dim = pgVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(969);
|
||||
|
||||
@@ -69,7 +78,11 @@ public class PgVectorEmbeddingDimensionsTests {
|
||||
|
||||
given(this.embeddingModel.dimensions()).willThrow(new RuntimeException());
|
||||
|
||||
var dim = new PgVectorStore(this.jdbcTemplate, this.embeddingModel).embeddingDimensions();
|
||||
PgVectorStore pgVectorStore = PgVectorStore.builder()
|
||||
.jdbcTemplate(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.build();
|
||||
var dim = pgVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(PgVectorStore.OPENAI_EMBEDDING_DIMENSION_SIZE);
|
||||
verify(this.embeddingModel, only()).dimensions();
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import org.testcontainers.utility.DockerImageName;
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.Random;
|
||||
|
||||
@@ -30,7 +30,8 @@ import org.testcontainers.junit.jupiter.Testcontainers;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.openai.OpenAiEmbeddingModel;
|
||||
import org.springframework.ai.openai.api.OpenAiApi;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
@@ -195,14 +196,17 @@ public class PgVectorStoreCustomNamesIT {
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
|
||||
return new PgVectorStore.Builder(jdbcTemplate, embeddingModel).withSchemaName(this.schemaName)
|
||||
.withVectorTableName(this.vectorTableName)
|
||||
.withVectorTableValidationsEnabled(this.schemaValidation)
|
||||
.withDimensions(this.dimensions)
|
||||
.withDistanceType(PgVectorStore.PgDistanceType.COSINE_DISTANCE)
|
||||
.withRemoveExistingVectorStoreTable(true)
|
||||
.withIndexType(PgIndexType.HNSW)
|
||||
.withInitializeSchema(true)
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName(this.schemaName)
|
||||
.vectorTableName(this.vectorTableName)
|
||||
.vectorTableValidationsEnabled(this.schemaValidation)
|
||||
.dimensions(this.dimensions)
|
||||
.distanceType(PgVectorStore.PgDistanceType.COSINE_DISTANCE)
|
||||
.removeExistingVectorStoreTable(true)
|
||||
.indexType(PgIndexType.HNSW)
|
||||
.initializeSchema(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -43,7 +43,9 @@ import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.openai.OpenAiEmbeddingModel;
|
||||
import org.springframework.ai.openai.api.OpenAiApi;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionTextParser.FilterExpressionParseException;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
@@ -354,8 +356,15 @@ public class PgVectorStoreIT {
|
||||
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
return new PgVectorStore(jdbcTemplate, embeddingModel, PgVectorStore.INVALID_EMBEDDING_DIMENSION,
|
||||
this.distanceType, true, PgIndexType.HNSW, true);
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.dimensions(PgVectorStore.INVALID_EMBEDDING_DIMENSION)
|
||||
.distanceType(this.distanceType)
|
||||
.initializeSchema(true)
|
||||
.indexType(PgIndexType.HNSW)
|
||||
.removeExistingVectorStoreTable(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
@@ -41,7 +41,9 @@ import org.springframework.ai.observation.conventions.VectorStoreSimilarityMetri
|
||||
import org.springframework.ai.openai.OpenAiChatModel;
|
||||
import org.springframework.ai.openai.OpenAiEmbeddingModel;
|
||||
import org.springframework.ai.openai.api.OpenAiApi;
|
||||
import org.springframework.ai.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.pgvector.vectorstore.PgVectorStore.PgIndexType;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.observation.DefaultVectorStoreObservationConvention;
|
||||
import org.springframework.ai.vectorstore.observation.VectorStoreObservationDocumentation.HighCardinalityKeyNames;
|
||||
import org.springframework.ai.vectorstore.observation.VectorStoreObservationDocumentation.LowCardinalityKeyNames;
|
||||
@@ -185,11 +187,13 @@ public class PgVectorStoreObservationIT {
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel,
|
||||
ObservationRegistry observationRegistry) {
|
||||
return new PgVectorStore.Builder(jdbcTemplate, embeddingModel)
|
||||
.withDistanceType(PgVectorStore.PgDistanceType.COSINE_DISTANCE)
|
||||
.withIndexType(PgIndexType.HNSW)
|
||||
.withObservationRegistry(observationRegistry)
|
||||
.withInitializeSchema(true)
|
||||
return PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.distanceType(PgVectorStore.PgDistanceType.COSINE_DISTANCE)
|
||||
.indexType(PgIndexType.HNSW)
|
||||
.observationRegistry(observationRegistry)
|
||||
.initializeSchema(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.Collections;
|
||||
|
||||
@@ -79,7 +79,10 @@ public class PgVectorStoreTests {
|
||||
// Given
|
||||
var jdbcTemplate = mock(JdbcTemplate.class);
|
||||
var embeddingModel = mock(EmbeddingModel.class);
|
||||
var pgVectorStore = new PgVectorStore.Builder(jdbcTemplate, embeddingModel).withMaxDocumentBatchSize(1000)
|
||||
var pgVectorStore = PgVectorStore.builder()
|
||||
.jdbcTemplate(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.maxDocumentBatchSize(1000)
|
||||
.build();
|
||||
|
||||
// Testing with 9989 documents
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore;
|
||||
package org.springframework.ai.pgvector.vectorstore;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
Reference in New Issue
Block a user