Add builder pattern to MariaDBVectorStore and refactor package name
Introduces a builder pattern for configuring MariaDBVectorStore instances and improves the overall implementation. This change: - Makes configuration more flexible and type-safe through builder methods - Deprecates old constructors and builder in favor of the new builder pattern - Adds comprehensive validation of configuration options - Improves documentation with clear examples and better structure - Updates all test classes to use the new builder pattern - Adds comprehensive builder tests - Updated reference documentation The builder pattern provides a more maintainable and user-friendly way to configure vector stores while ensuring configuration validity at compile time. This aligns with the project's move towards using builder patterns across all vector store implementations.
This commit is contained in:
committed by
Mark Pollack
parent
8877672846
commit
8edf1bea47
@@ -1,18 +1,28 @@
|
||||
= MariaDB Vector
|
||||
= MariaDB Vector Store
|
||||
|
||||
This section walks you through setting up the MariaDB `VectorStore` to store document embeddings and perform similarity searches.
|
||||
This section walks you through setting up `MariaDBVectorStore` to store document embeddings and perform similarity searches.
|
||||
|
||||
link:https://mariadb.org/projects/mariadb-vector/[MariaDB vector] is part of MariaDB 11.7 and enables storing and searching over machine learning-generated embeddings.
|
||||
link:https://mariadb.org/projects/mariadb-vector/[MariaDB Vector] is part of MariaDB 11.7 and enables storing and searching over machine learning-generated embeddings.
|
||||
It provides efficient vector similarity search capabilities using vector indexes, supporting both cosine similarity and Euclidean distance metrics.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
* A running MariaDB (11.7+) instance. The following options are available:
|
||||
** link:https://hub.docker.com/_/mariadb[Docker] image
|
||||
** link:https://mariadb.org/download/[MariaDB Server]
|
||||
** link:https://mariadb.com/products/skysql/[MariaDB SkySQL]
|
||||
* If required, an API key for the xref:api/embeddings.adoc#available-implementations[EmbeddingModel] to generate the embeddings stored by the `MariaDBVectorStore`.
|
||||
|
||||
== Auto-Configuration
|
||||
|
||||
Add the MariaDBVectorStore boot starter dependency to your project:
|
||||
Spring AI provides Spring Boot auto-configuration for the MariaDB Vector Store.
|
||||
To enable it, add the following dependency to your project's Maven `pom.xml` file:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-mariadb-store-spring-boot-starter</artifactId>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-mariadb-store-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
@@ -25,57 +35,27 @@ dependencies {
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
|
||||
The vector store implementation can initialize the required schema for you, but you must opt-in by specifying the `initializeSchema` boolean in the appropriate constructor or by setting `...initialize-schema=true` in the `application.properties` file.
|
||||
|
||||
The Vector Store also requires an `EmbeddingModel` instance to calculate embeddings for the documents.
|
||||
You can pick one of the available xref:api/embeddings.adoc#available-implementations[EmbeddingModel Implementations].
|
||||
NOTE: This is a breaking change! In earlier versions of Spring AI, this schema initialization happened by default.
|
||||
|
||||
For example, to use the xref:api/embeddings/openai-embeddings.adoc[OpenAI EmbeddingModel], add the following dependency to your project:
|
||||
Additionally, you will need a configured `EmbeddingModel` bean. Refer to the xref:api/embeddings.adoc#available-implementations[EmbeddingModel] section for more information.
|
||||
|
||||
For example, to use the xref:api/embeddings/openai-embeddings.adoc[OpenAI EmbeddingModel], add the following dependency:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or to your Gradle `build.gradle` build file.
|
||||
TIP: Refer to the xref:getting-started.adoc#repositories[Repositories] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
[source,groovy]
|
||||
----
|
||||
dependencies {
|
||||
implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter'
|
||||
}
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
Refer to the xref:getting-started.adoc#repositories[Repositories] section to add Milestone and/or Snapshot Repositories to your build file.
|
||||
|
||||
To connect to and configure the `MariaDBVectorStore`, you need to provide access details for your instance.
|
||||
A simple configuration can be provided via Spring Boot's `application.yml`.
|
||||
|
||||
[yml]
|
||||
----
|
||||
spring:
|
||||
datasource:
|
||||
url: jdbc:mariadb://localhost/db
|
||||
username: myUser
|
||||
password: myPassword
|
||||
ai:
|
||||
vectorstore:
|
||||
mariadbvector:
|
||||
distance-type: COSINE
|
||||
dimensions: 1536
|
||||
----
|
||||
|
||||
TIP: If you run MariaDBvector as a Spring Boot dev service via link:https://docs.spring.io/spring-boot/reference/features/dev-services.html#features.dev-services.docker-compose[Docker Compose]
|
||||
or link:https://docs.spring.io/spring-boot/reference/features/dev-services.html#features.dev-services.testcontainers[Testcontainers],
|
||||
you don't need to configure URL, username and password since they are autoconfigured by Spring Boot.
|
||||
|
||||
TIP: Check the list of xref:#mariadbvector-properties[configuration parameters] to learn about the default values and configuration options.
|
||||
|
||||
Now you can auto-wire the `MariaDBVectorStore` in your application and use it
|
||||
Now you can auto-wire the `MariaDBVectorStore` in your application:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@@ -88,38 +68,114 @@ List<Document> documents = List.of(
|
||||
new Document("The World is Big and Salvation Lurks Around the Corner"),
|
||||
new Document("You walk forward facing the past and you turn back toward the future.", Map.of("meta2", "meta2")));
|
||||
|
||||
// Add the documents to PGVector
|
||||
// Add the documents to MariaDB
|
||||
vectorStore.add(documents);
|
||||
|
||||
// Retrieve documents similar to a query
|
||||
List<Document> results = this.vectorStore.similaritySearch(SearchRequest.query("Spring").withTopK(5));
|
||||
List<Document> results = vectorStore.similaritySearch(SearchRequest.query("Spring").withTopK(5));
|
||||
----
|
||||
|
||||
[[mariadbvector-properties]]
|
||||
=== Configuration properties
|
||||
=== Configuration Properties
|
||||
|
||||
You can use the following properties in your Spring Boot configuration to customize the MariaDB vector store.
|
||||
To connect to MariaDB and use the `MariaDBVectorStore`, you need to provide access details for your instance.
|
||||
A simple configuration can be provided via Spring Boot's `application.yml`:
|
||||
|
||||
[source,yaml]
|
||||
----
|
||||
spring:
|
||||
datasource:
|
||||
url: jdbc:mariadb://localhost/db
|
||||
username: myUser
|
||||
password: myPassword
|
||||
ai:
|
||||
vectorstore:
|
||||
mariadb:
|
||||
initialize-schema: true
|
||||
distance-type: COSINE
|
||||
dimensions: 1536
|
||||
----
|
||||
|
||||
TIP: If you run MariaDB Vector as a Spring Boot dev service via link:https://docs.spring.io/spring-boot/reference/features/dev-services.html#features.dev-services.docker-compose[Docker Compose]
|
||||
or link:https://docs.spring.io/spring-boot/reference/features/dev-services.html#features.dev-services.testcontainers[Testcontainers],
|
||||
you don't need to configure URL, username and password since they are autoconfigured by Spring Boot.
|
||||
|
||||
Properties starting with `spring.ai.vectorstore.mariadb.*` are used to configure the `MariaDBVectorStore`:
|
||||
|
||||
[cols="2,5,1",stripes=even]
|
||||
|===
|
||||
|Property| Description | Default value
|
||||
|Property | Description | Default Value
|
||||
|
||||
|`spring.ai.vectorstore.mariadb.distance-type`| Search distance type. Defaults to `COSINE`. But if vectors are normalized to length 1, you can use `EUCLIDEAN` for best performance.| COSINE
|
||||
|`spring.ai.vectorstore.mariadb.dimensions`| Embeddings dimension. If not specified explicitly the PgVectorStore will retrieve the dimensions form the provided `EmbeddingModel`. Dimensions are set to the embedding column the on table creation. If you change the dimensions your would have to re-create the vector_store table as well. | -
|
||||
|`spring.ai.vectorstore.mariadb.remove-existing-vector-store-table` | Deletes the existing `vector_store` table on start up. | false
|
||||
|`spring.ai.vectorstore.mariadb.initialize-schema` | Whether to initialize the required schema | false
|
||||
|`spring.ai.vectorstore.mariadb.schema-name` | Vector store schema name | null
|
||||
|`spring.ai.vectorstore.mariadb.initialize-schema`| Whether to initialize the required schema | `false`
|
||||
|`spring.ai.vectorstore.mariadb.distance-type`| Search distance type. Use `COSINE` (default) or `EUCLIDEAN`. If vectors are normalized to length 1, you can use `EUCLIDEAN` for best performance.| `COSINE`
|
||||
|`spring.ai.vectorstore.mariadb.dimensions`| Embeddings dimension. If not specified explicitly, will retrieve dimensions from the provided `EmbeddingModel`. | `1536`
|
||||
|`spring.ai.vectorstore.mariadb.remove-existing-vector-store-table` | Deletes the existing vector store table on startup. | `false`
|
||||
|`spring.ai.vectorstore.mariadb.schema-name` | Vector store schema name | `null`
|
||||
|`spring.ai.vectorstore.mariadb.table-name` | Vector store table name | `vector_store`
|
||||
|`spring.ai.vectorstore.mariadb.schema-validation` | Enables schema and table name validation to ensure they are valid and existing objects. | false
|
||||
|
||||
|`spring.ai.vectorstore.mariadb.schema-validation` | Enables schema and table name validation to ensure they are valid and existing objects. | `false`
|
||||
|===
|
||||
|
||||
TIP: If you configure a custom schema and/or table name, consider enabling schema validation by setting `spring.ai.vectorstore.mariadb.schema-validation=true`.
|
||||
This ensures the correctness of the names and reduces the risk of SQL injection attacks.
|
||||
|
||||
== Metadata filtering
|
||||
== Manual Configuration
|
||||
|
||||
You can leverage the generic, portable link:https://docs.spring.io/spring-ai/reference/api/vectordbs.html#_metadata_filters[metadata filters] with the MariaDB Vector store.
|
||||
Instead of using the Spring Boot auto-configuration, you can manually configure the MariaDB vector store. For this you need to add the following dependencies to your project:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-jdbc</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mariadb.jdbc</groupId>
|
||||
<artifactId>mariadb-java-client</artifactId>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-mariadb-store</artifactId>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
|
||||
Then create the `MariaDBVectorStore` bean using the builder pattern:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
return MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.dimensions(1536) // Optional: defaults to 1536
|
||||
.distanceType(MariaDBDistanceType.COSINE) // Optional: defaults to COSINE
|
||||
.schemaName("mydb") // Optional: defaults to null
|
||||
.vectorTableName("custom_vectors") // Optional: defaults to "vector_store"
|
||||
.contentFieldName("text") // Optional: defaults to "content"
|
||||
.embeddingFieldName("embedding") // Optional: defaults to "embedding"
|
||||
.idFieldName("doc_id") // Optional: defaults to "id"
|
||||
.metadataFieldName("meta") // Optional: defaults to "metadata"
|
||||
.initializeSchema(true) // Optional: defaults to false
|
||||
.schemaValidation(true) // Optional: defaults to false
|
||||
.removeExistingVectorStoreTable(false) // Optional: defaults to false
|
||||
.maxDocumentBatchSize(10000) // Optional: defaults to 10000
|
||||
.build();
|
||||
}
|
||||
|
||||
// This can be any EmbeddingModel implementation
|
||||
@Bean
|
||||
public EmbeddingModel embeddingModel() {
|
||||
return new OpenAiEmbeddingModel(new OpenAiApi(System.getenv("OPENAI_API_KEY")));
|
||||
}
|
||||
----
|
||||
|
||||
== Metadata Filtering
|
||||
|
||||
You can leverage the generic, portable xref:api/vectordbs.adoc#metadata-filters[metadata filters] with MariaDB Vector store.
|
||||
|
||||
For example, you can use either the text expression language:
|
||||
|
||||
@@ -127,10 +183,10 @@ For example, you can use either the text expression language:
|
||||
----
|
||||
vectorStore.similaritySearch(
|
||||
SearchRequest.defaults()
|
||||
.withQuery("The World")
|
||||
.withTopK(TOP_K)
|
||||
.withSimilarityThreshold(SIMILARITY_THRESHOLD)
|
||||
.withFilterExpression("author in ['john', 'jill'] && article_type == 'blog'"));
|
||||
.withQuery("The World")
|
||||
.withTopK(TOP_K)
|
||||
.withSimilarityThreshold(SIMILARITY_THRESHOLD)
|
||||
.withFilterExpression("author in ['john', 'jill'] && article_type == 'blog'"));
|
||||
----
|
||||
|
||||
or programmatically using the `Filter.Expression` DSL:
|
||||
@@ -144,44 +200,8 @@ vectorStore.similaritySearch(SearchRequest.defaults()
|
||||
.withTopK(TOP_K)
|
||||
.withSimilarityThreshold(SIMILARITY_THRESHOLD)
|
||||
.withFilterExpression(b.and(
|
||||
b.in("author","john", "jill"),
|
||||
b.in("author", "john", "jill"),
|
||||
b.eq("article_type", "blog")).build()));
|
||||
----
|
||||
|
||||
NOTE: These filter expressions are converted into the equivalent PgVector filters.
|
||||
|
||||
== Manual Configuration
|
||||
|
||||
Instead of using the Spring Boot auto-configuration, you can manually configure the `MariaDBVectorStore`.
|
||||
For this you need to add the MariaDB connector and `JdbcTemplate` auto-configuration dependencies to your project:
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-jdbc</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mariadb.jdbc</groupId>
|
||||
<artifactId>mariadb-java-client</artifactId>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-mariadb-store</artifactId>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
TIP: Refer to the xref:getting-started.adoc#dependency-management[Dependency Management] section to add the Spring AI BOM to your build file.
|
||||
|
||||
To configure MariaDB Vector in your application, you can use the following setup:
|
||||
|
||||
[source,java]
|
||||
----
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
return new MariaDBVectorStore(jdbcTemplate, embeddingModel);
|
||||
}
|
||||
----
|
||||
NOTE: These filter expressions are automatically converted into the equivalent MariaDB JSON path expressions.
|
||||
|
||||
@@ -57,21 +57,23 @@ public class MariaDbStoreAutoConfiguration {
|
||||
|
||||
var initializeSchema = properties.isInitializeSchema();
|
||||
|
||||
return new MariaDBVectorStore.Builder(jdbcTemplate, embeddingModel).withSchemaName(properties.getSchemaName())
|
||||
.withVectorTableName(properties.getTableName())
|
||||
.withVectorTableValidationsEnabled(properties.isSchemaValidation())
|
||||
.withDimensions(properties.getDimensions())
|
||||
.withDistanceType(properties.getDistanceType())
|
||||
.withContentFieldName(properties.getContentFieldName())
|
||||
.withEmbeddingFieldName(properties.getEmbeddingFieldName())
|
||||
.withIdFieldName(properties.getIdFieldName())
|
||||
.withMetadataFieldName(properties.getMetadataFieldName())
|
||||
.withRemoveExistingVectorStoreTable(properties.isRemoveExistingVectorStoreTable())
|
||||
.withInitializeSchema(initializeSchema)
|
||||
.withObservationRegistry(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
|
||||
.withSearchObservationConvention(customObservationConvention.getIfAvailable(() -> null))
|
||||
.withBatchingStrategy(batchingStrategy)
|
||||
.withMaxDocumentBatchSize(properties.getMaxDocumentBatchSize())
|
||||
return MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName(properties.getSchemaName())
|
||||
.vectorTableName(properties.getTableName())
|
||||
.schemaValidation(properties.isSchemaValidation())
|
||||
.dimensions(properties.getDimensions())
|
||||
.distanceType(properties.getDistanceType())
|
||||
.contentFieldName(properties.getContentFieldName())
|
||||
.embeddingFieldName(properties.getEmbeddingFieldName())
|
||||
.idFieldName(properties.getIdFieldName())
|
||||
.metadataFieldName(properties.getMetadataFieldName())
|
||||
.removeExistingVectorStoreTable(properties.isRemoveExistingVectorStoreTable())
|
||||
.initializeSchema(initializeSchema)
|
||||
.observationRegistry(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
|
||||
.customObservationConvention(customObservationConvention.getIfAvailable(() -> null))
|
||||
.batchingStrategy(batchingStrategy)
|
||||
.maxDocumentBatchSize(properties.getMaxDocumentBatchSize())
|
||||
.build();
|
||||
}
|
||||
|
||||
|
||||
@@ -34,6 +34,7 @@ import org.springframework.ai.embedding.TokenCountBatchingStrategy;
|
||||
import org.springframework.ai.observation.conventions.VectorStoreProvider;
|
||||
import org.springframework.ai.observation.conventions.VectorStoreSimilarityMetric;
|
||||
import org.springframework.ai.util.JacksonUtils;
|
||||
import org.springframework.ai.vectorstore.AbstractVectorStoreBuilder;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionConverter;
|
||||
import org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore;
|
||||
@@ -44,11 +45,91 @@ import org.springframework.jdbc.core.BatchPreparedStatementSetter;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.core.RowMapper;
|
||||
import org.springframework.lang.Nullable;
|
||||
import org.springframework.util.Assert;
|
||||
import org.springframework.util.StringUtils;
|
||||
|
||||
/**
|
||||
* Uses the "vector_store" table to store the Spring AI vector data. The table and the
|
||||
* vector index will be auto-created if not available.
|
||||
* MariaDB-based vector store implementation using MariaDB's vector search capabilities.
|
||||
*
|
||||
* <p>
|
||||
* The store uses MariaDB's vector search functionality to persist and query vector
|
||||
* embeddings along with their associated document content and metadata. The
|
||||
* implementation leverages MariaDB's vector index for efficient k-NN search operations.
|
||||
* </p>
|
||||
*
|
||||
* <p>
|
||||
* Features:
|
||||
* </p>
|
||||
* <ul>
|
||||
* <li>Automatic schema initialization with configurable index creation</li>
|
||||
* <li>Support for multiple distance functions: Cosine and Euclidean</li>
|
||||
* <li>Metadata filtering using JSON path expressions</li>
|
||||
* <li>Configurable similarity thresholds for search results</li>
|
||||
* <li>Batch processing support with configurable strategies</li>
|
||||
* <li>Observation and metrics support through Micrometer</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>
|
||||
* Basic usage example:
|
||||
* </p>
|
||||
* <pre>{@code
|
||||
* MariaDBVectorStore vectorStore = MariaDBVectorStore.builder(jdbcTemplate)
|
||||
* .embeddingModel(embeddingModel)
|
||||
* .initializeSchema(true)
|
||||
* .build();
|
||||
*
|
||||
* // Add documents
|
||||
* vectorStore.add(List.of(
|
||||
* new Document("content1", Map.of("key1", "value1")),
|
||||
* new Document("content2", Map.of("key2", "value2"))
|
||||
* ));
|
||||
*
|
||||
* // Search with filters
|
||||
* List<Document> results = vectorStore.similaritySearch(
|
||||
* SearchRequest.query("search text")
|
||||
* .withTopK(5)
|
||||
* .withSimilarityThreshold(0.7)
|
||||
* .withFilterExpression("key1 == 'value1'")
|
||||
* );
|
||||
* }</pre>
|
||||
*
|
||||
* <p>
|
||||
* Advanced configuration example:
|
||||
* </p>
|
||||
* <pre>{@code
|
||||
* MariaDBVectorStore vectorStore = MariaDBVectorStore.builder(jdbcTemplate)
|
||||
* .embeddingModel(embeddingModel)
|
||||
* .schemaName("mydb")
|
||||
* .distanceType(MariaDBDistanceType.COSINE)
|
||||
* .dimensions(1536)
|
||||
* .vectorTableName("custom_vectors")
|
||||
* .contentFieldName("text")
|
||||
* .embeddingFieldName("embedding")
|
||||
* .idFieldName("doc_id")
|
||||
* .metadataFieldName("meta")
|
||||
* .initializeSchema(true)
|
||||
* .batchingStrategy(new TokenCountBatchingStrategy())
|
||||
* .build();
|
||||
* }</pre>
|
||||
*
|
||||
* <p>
|
||||
* Requirements:
|
||||
* </p>
|
||||
* <ul>
|
||||
* <li>MariaDB 11.3.0 or later</li>
|
||||
* <li>Table schema with id (UUID), text (TEXT), metadata (JSON), and embedding (VECTOR)
|
||||
* properties</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>
|
||||
* Distance Functions:
|
||||
* </p>
|
||||
* <ul>
|
||||
* <li>cosine: Default, suitable for most use cases. Measures cosine similarity between
|
||||
* vectors.</li>
|
||||
* <li>euclidean: Euclidean distance between vectors. Lower values indicate higher
|
||||
* similarity.</li>
|
||||
* </ul>
|
||||
*
|
||||
* @author Diego Dupin
|
||||
* @author Ilayaperumal Gopinathan
|
||||
@@ -76,7 +157,7 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
|
||||
public static final String DEFAULT_COLUMN_CONTENT = "content";
|
||||
|
||||
private static Map<MariaDBDistanceType, VectorStoreSimilarityMetric> SIMILARITY_TYPE_MAPPING = Map.of(
|
||||
private static final Map<MariaDBDistanceType, VectorStoreSimilarityMetric> SIMILARITY_TYPE_MAPPING = Map.of(
|
||||
MariaDBDistanceType.COSINE, VectorStoreSimilarityMetric.COSINE, MariaDBDistanceType.EUCLIDEAN,
|
||||
VectorStoreSimilarityMetric.EUCLIDEAN);
|
||||
|
||||
@@ -86,8 +167,6 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
|
||||
private final JdbcTemplate jdbcTemplate;
|
||||
|
||||
private final EmbeddingModel embeddingModel;
|
||||
|
||||
private final String schemaName;
|
||||
|
||||
private final boolean schemaValidation;
|
||||
@@ -116,39 +195,60 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
|
||||
private final int maxDocumentBatchSize;
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public MariaDBVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
this(jdbcTemplate, embeddingModel, INVALID_EMBEDDING_DIMENSION, MariaDBDistanceType.COSINE, false, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public MariaDBVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions) {
|
||||
this(jdbcTemplate, embeddingModel, dimensions, MariaDBDistanceType.COSINE, false, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public MariaDBVectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions,
|
||||
MariaDBDistanceType distanceType, boolean removeExistingVectorStoreTable, boolean initializeSchema) {
|
||||
|
||||
this(DEFAULT_TABLE_NAME, jdbcTemplate, embeddingModel, dimensions, distanceType, removeExistingVectorStoreTable,
|
||||
initializeSchema);
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public MariaDBVectorStore(String vectorTableName, JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel,
|
||||
int dimensions, MariaDBDistanceType distanceType, boolean removeExistingVectorStoreTable,
|
||||
boolean initializeSchema) {
|
||||
|
||||
this(null, vectorTableName, DEFAULT_SCHEMA_VALIDATION, jdbcTemplate, embeddingModel, dimensions, distanceType,
|
||||
removeExistingVectorStoreTable, initializeSchema);
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
private MariaDBVectorStore(String schemaName, String vectorTableName, boolean vectorTableValidationsEnabled,
|
||||
JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions, MariaDBDistanceType distanceType,
|
||||
boolean removeExistingVectorStoreTable, boolean initializeSchema) {
|
||||
|
||||
this(schemaName, vectorTableName, vectorTableValidationsEnabled, jdbcTemplate, embeddingModel, dimensions,
|
||||
distanceType, removeExistingVectorStoreTable, initializeSchema, ObservationRegistry.NOOP, null,
|
||||
new TokenCountBatchingStrategy(), MAX_DOCUMENT_BATCH_SIZE, DEFAULT_COLUMN_EMBEDDING,
|
||||
DEFAULT_COLUMN_METADATA, DEFAULT_COLUMN_ID, DEFAULT_COLUMN_CONTENT);
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Use {@link #builder(JdbcTemplate)} instead
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
private MariaDBVectorStore(String schemaName, String vectorTableName, boolean vectorTableValidationsEnabled,
|
||||
JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel, int dimensions, MariaDBDistanceType distanceType,
|
||||
boolean removeExistingVectorStoreTable, boolean initializeSchema, ObservationRegistry observationRegistry,
|
||||
@@ -156,36 +256,73 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
int maxDocumentBatchSize, String contentFieldName, String embeddingFieldName, String idFieldName,
|
||||
String metadataFieldName) {
|
||||
|
||||
super(observationRegistry, customObservationConvention);
|
||||
this(builder(jdbcTemplate).vectorTableName(vectorTableName)
|
||||
.embeddingModel(embeddingModel)
|
||||
.dimensions(dimensions)
|
||||
.distanceType(distanceType)
|
||||
.removeExistingVectorStoreTable(removeExistingVectorStoreTable)
|
||||
.initializeSchema(initializeSchema)
|
||||
.observationRegistry(observationRegistry)
|
||||
.customObservationConvention(customObservationConvention)
|
||||
.batchingStrategy(batchingStrategy)
|
||||
.maxDocumentBatchSize(maxDocumentBatchSize)
|
||||
.contentFieldName(contentFieldName)
|
||||
.embeddingFieldName(embeddingFieldName)
|
||||
.idFieldName(idFieldName)
|
||||
.metadataFieldName(metadataFieldName));
|
||||
}
|
||||
|
||||
/**
|
||||
* Protected constructor for creating a MariaDBVectorStore instance using the builder
|
||||
* pattern.
|
||||
* @param builder the {@link MariaDBBuilder} containing all configuration settings
|
||||
* @throws IllegalArgumentException if required parameters are missing or invalid
|
||||
* @see MariaDBBuilder
|
||||
* @since 1.0.0
|
||||
*/
|
||||
protected MariaDBVectorStore(MariaDBBuilder builder) {
|
||||
super(builder);
|
||||
|
||||
Assert.notNull(builder.jdbcTemplate, "JdbcTemplate must not be null");
|
||||
|
||||
this.objectMapper = JsonMapper.builder().addModules(JacksonUtils.instantiateAvailableModules()).build();
|
||||
|
||||
this.vectorTableName = (null == vectorTableName || vectorTableName.isEmpty()) ? DEFAULT_TABLE_NAME
|
||||
: MariaDBSchemaValidator.validateAndEnquoteIdentifier(vectorTableName.trim(), false);
|
||||
this.vectorTableName = (null == builder.vectorTableName || builder.vectorTableName.isEmpty())
|
||||
? DEFAULT_TABLE_NAME
|
||||
: MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.vectorTableName.trim(), false);
|
||||
|
||||
logger.info("Using the vector table name: {}. Is empty: {}", this.vectorTableName,
|
||||
(vectorTableName == null || vectorTableName.isEmpty()));
|
||||
|
||||
this.schemaName = schemaName == null ? null
|
||||
: MariaDBSchemaValidator.validateAndEnquoteIdentifier(schemaName, false);
|
||||
this.schemaValidation = vectorTableValidationsEnabled;
|
||||
|
||||
this.jdbcTemplate = jdbcTemplate;
|
||||
this.embeddingModel = embeddingModel;
|
||||
this.dimensions = dimensions;
|
||||
this.distanceType = distanceType;
|
||||
this.removeExistingVectorStoreTable = removeExistingVectorStoreTable;
|
||||
this.initializeSchema = initializeSchema;
|
||||
this.schemaName = builder.schemaName == null ? null
|
||||
: MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.schemaName, false);
|
||||
this.schemaValidation = builder.schemaValidation;
|
||||
this.jdbcTemplate = builder.jdbcTemplate;
|
||||
this.dimensions = builder.dimensions;
|
||||
this.distanceType = builder.distanceType;
|
||||
this.removeExistingVectorStoreTable = builder.removeExistingVectorStoreTable;
|
||||
this.initializeSchema = builder.initializeSchema;
|
||||
this.schemaValidator = new MariaDBSchemaValidator(jdbcTemplate);
|
||||
this.batchingStrategy = batchingStrategy;
|
||||
this.maxDocumentBatchSize = maxDocumentBatchSize;
|
||||
this.batchingStrategy = builder.batchingStrategy;
|
||||
this.maxDocumentBatchSize = builder.maxDocumentBatchSize;
|
||||
|
||||
this.contentFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(contentFieldName, false);
|
||||
this.embeddingFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(embeddingFieldName, false);
|
||||
this.idFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(idFieldName, false);
|
||||
this.metadataFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(metadataFieldName, false);
|
||||
this.contentFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.contentFieldName, false);
|
||||
this.embeddingFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.embeddingFieldName,
|
||||
false);
|
||||
this.idFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.idFieldName, false);
|
||||
this.metadataFieldName = MariaDBSchemaValidator.validateAndEnquoteIdentifier(builder.metadataFieldName, false);
|
||||
filterExpressionConverter = new MariaDBFilterExpressionConverter(this.metadataFieldName);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a new MariaDBBuilder instance. This is the recommended way to instantiate a
|
||||
* MariaDBVectorStore.
|
||||
* @return a new MariaDBBuilder instance
|
||||
*/
|
||||
public static MariaDBBuilder builder(JdbcTemplate jdbcTemplate) {
|
||||
return new MariaDBBuilder(jdbcTemplate);
|
||||
}
|
||||
|
||||
public MariaDBDistanceType getDistanceType() {
|
||||
return this.distanceType;
|
||||
}
|
||||
@@ -421,6 +558,217 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* Builder for creating instances of {@link MariaDBVectorStore}. This builder provides
|
||||
* a fluent API for configuring all aspects of the vector store.
|
||||
*
|
||||
* @since 1.0.0
|
||||
*/
|
||||
public static final class MariaDBBuilder extends AbstractVectorStoreBuilder<MariaDBBuilder> {
|
||||
|
||||
private String contentFieldName = DEFAULT_COLUMN_CONTENT;
|
||||
|
||||
private String embeddingFieldName = DEFAULT_COLUMN_EMBEDDING;
|
||||
|
||||
private String idFieldName = DEFAULT_COLUMN_ID;
|
||||
|
||||
private String metadataFieldName = DEFAULT_COLUMN_METADATA;
|
||||
|
||||
private final JdbcTemplate jdbcTemplate;
|
||||
|
||||
private String schemaName;
|
||||
|
||||
private String vectorTableName = DEFAULT_TABLE_NAME;
|
||||
|
||||
private boolean schemaValidation = DEFAULT_SCHEMA_VALIDATION;
|
||||
|
||||
private int dimensions = INVALID_EMBEDDING_DIMENSION;
|
||||
|
||||
private MariaDBDistanceType distanceType = MariaDBDistanceType.COSINE;
|
||||
|
||||
private boolean removeExistingVectorStoreTable = false;
|
||||
|
||||
private boolean initializeSchema = false;
|
||||
|
||||
private BatchingStrategy batchingStrategy = new TokenCountBatchingStrategy();
|
||||
|
||||
private int maxDocumentBatchSize = MAX_DOCUMENT_BATCH_SIZE;
|
||||
|
||||
/**
|
||||
* Creates a new builder instance with the required JDBC template.
|
||||
* @param jdbcTemplate the JDBC template for database operations
|
||||
* @throws IllegalArgumentException if jdbcTemplate is null
|
||||
*/
|
||||
MariaDBBuilder(JdbcTemplate jdbcTemplate) {
|
||||
Assert.notNull(jdbcTemplate, "JdbcTemplate must not be null");
|
||||
this.jdbcTemplate = jdbcTemplate;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the schema name for the vector store table.
|
||||
* @param schemaName the database schema name (can be null for default schema)
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder schemaName(String schemaName) {
|
||||
this.schemaName = schemaName;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the vector store table name.
|
||||
* @param vectorTableName the name for the vector store table (defaults to
|
||||
* {@value DEFAULT_TABLE_NAME})
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder vectorTableName(String vectorTableName) {
|
||||
this.vectorTableName = vectorTableName;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures whether schema validation should be performed.
|
||||
* @param schemaValidation true to enable schema validation, false to disable
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder schemaValidation(boolean schemaValidation) {
|
||||
this.schemaValidation = schemaValidation;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the dimension size of the embedding vectors.
|
||||
* @param dimensions the dimension of the embeddings
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder dimensions(int dimensions) {
|
||||
this.dimensions = dimensions;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the distance type used for similarity calculations.
|
||||
* @param distanceType the distance type to use
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if distanceType is null
|
||||
*/
|
||||
public MariaDBBuilder distanceType(MariaDBDistanceType distanceType) {
|
||||
Assert.notNull(distanceType, "DistanceType must not be null");
|
||||
this.distanceType = distanceType;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures whether to remove any existing vector store table.
|
||||
* @param removeExistingVectorStoreTable true to remove existing table, false to
|
||||
* keep it
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder removeExistingVectorStoreTable(boolean removeExistingVectorStoreTable) {
|
||||
this.removeExistingVectorStoreTable = removeExistingVectorStoreTable;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures whether to initialize the database schema.
|
||||
* @param initializeSchema true to initialize schema, false otherwise
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder initializeSchema(boolean initializeSchema) {
|
||||
this.initializeSchema = initializeSchema;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the strategy for batching operations.
|
||||
* @param batchingStrategy the batching strategy to use
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if batchingStrategy is null
|
||||
*/
|
||||
public MariaDBBuilder batchingStrategy(BatchingStrategy batchingStrategy) {
|
||||
Assert.notNull(batchingStrategy, "BatchingStrategy must not be null");
|
||||
this.batchingStrategy = batchingStrategy;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the maximum batch size for document operations.
|
||||
* @param maxDocumentBatchSize the maximum number of documents to process in a
|
||||
* batch
|
||||
* @return this builder instance
|
||||
*/
|
||||
public MariaDBBuilder maxDocumentBatchSize(int maxDocumentBatchSize) {
|
||||
Assert.isTrue(maxDocumentBatchSize > 0, "MaxDocumentBatchSize must be positive");
|
||||
this.maxDocumentBatchSize = maxDocumentBatchSize;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the name of the content field in the database.
|
||||
* @param name the field name for document content (defaults to
|
||||
* {@value DEFAULT_COLUMN_CONTENT})
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if name is null or empty
|
||||
*/
|
||||
public MariaDBBuilder contentFieldName(String name) {
|
||||
Assert.hasText(name, "ContentFieldName must not be empty");
|
||||
this.contentFieldName = name;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the name of the embedding field in the database.
|
||||
* @param name the field name for embeddings (defaults to
|
||||
* {@value DEFAULT_COLUMN_EMBEDDING})
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if name is null or empty
|
||||
*/
|
||||
public MariaDBBuilder embeddingFieldName(String name) {
|
||||
Assert.hasText(name, "EmbeddingFieldName must not be empty");
|
||||
this.embeddingFieldName = name;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the name of the ID field in the database.
|
||||
* @param name the field name for document IDs (defaults to
|
||||
* {@value DEFAULT_COLUMN_ID})
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if name is null or empty
|
||||
*/
|
||||
public MariaDBBuilder idFieldName(String name) {
|
||||
Assert.hasText(name, "IdFieldName must not be empty");
|
||||
this.idFieldName = name;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the name of the metadata field in the database.
|
||||
* @param name the field name for document metadata (defaults to
|
||||
* {@value DEFAULT_COLUMN_METADATA})
|
||||
* @return this builder instance
|
||||
* @throws IllegalArgumentException if name is null or empty
|
||||
*/
|
||||
public MariaDBBuilder metadataFieldName(String name) {
|
||||
Assert.hasText(name, "MetadataFieldName must not be empty");
|
||||
this.metadataFieldName = name;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds and returns a new MariaDBVectorStore instance with the configured
|
||||
* settings.
|
||||
* @return a new MariaDBVectorStore instance
|
||||
* @throws IllegalStateException if the builder configuration is invalid
|
||||
*/
|
||||
@Override
|
||||
public MariaDBVectorStore build() {
|
||||
validate();
|
||||
return new MariaDBVectorStore(this);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public static class Builder {
|
||||
|
||||
private String contentFieldName = DEFAULT_COLUMN_CONTENT;
|
||||
@@ -459,6 +807,7 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
private VectorStoreObservationConvention searchObservationConvention;
|
||||
|
||||
// Builder constructor with mandatory parameters
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
if (jdbcTemplate == null || embeddingModel == null) {
|
||||
throw new IllegalArgumentException("JdbcTemplate and EmbeddingModel must not be null");
|
||||
@@ -467,56 +816,67 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
this.embeddingModel = embeddingModel;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withSchemaName(String schemaName) {
|
||||
this.schemaName = schemaName;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withVectorTableName(String vectorTableName) {
|
||||
this.vectorTableName = vectorTableName;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withVectorTableValidationsEnabled(boolean vectorTableValidationsEnabled) {
|
||||
this.vectorTableValidationsEnabled = vectorTableValidationsEnabled;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withDimensions(int dimensions) {
|
||||
this.dimensions = dimensions;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withDistanceType(MariaDBDistanceType distanceType) {
|
||||
this.distanceType = distanceType;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withRemoveExistingVectorStoreTable(boolean removeExistingVectorStoreTable) {
|
||||
this.removeExistingVectorStoreTable = removeExistingVectorStoreTable;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withInitializeSchema(boolean initializeSchema) {
|
||||
this.initializeSchema = initializeSchema;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withObservationRegistry(ObservationRegistry observationRegistry) {
|
||||
this.observationRegistry = observationRegistry;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withSearchObservationConvention(VectorStoreObservationConvention customObservationConvention) {
|
||||
this.searchObservationConvention = customObservationConvention;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withBatchingStrategy(BatchingStrategy batchingStrategy) {
|
||||
this.batchingStrategy = batchingStrategy;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withMaxDocumentBatchSize(int maxDocumentBatchSize) {
|
||||
this.maxDocumentBatchSize = maxDocumentBatchSize;
|
||||
return this;
|
||||
@@ -527,6 +887,7 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
* @param name the content field name to use
|
||||
* @return this builder
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withContentFieldName(String name) {
|
||||
this.contentFieldName = name;
|
||||
return this;
|
||||
@@ -537,6 +898,7 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
* @param name the embedding field name to use
|
||||
* @return this builder
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withEmbeddingFieldName(String name) {
|
||||
this.embeddingFieldName = name;
|
||||
return this;
|
||||
@@ -547,6 +909,7 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
* @param name the id field name to use
|
||||
* @return this builder
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withIdFieldName(String name) {
|
||||
this.idFieldName = name;
|
||||
return this;
|
||||
@@ -557,11 +920,13 @@ public class MariaDBVectorStore extends AbstractObservationVectorStore implement
|
||||
* @param name the metadata field name to use
|
||||
* @return this builder
|
||||
*/
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public Builder withMetadataFieldName(String name) {
|
||||
this.metadataFieldName = name;
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated(forRemoval = true, since = "1.0.0-M5")
|
||||
public MariaDBVectorStore build() {
|
||||
return new MariaDBVectorStore(this.schemaName, this.vectorTableName, this.vectorTableValidationsEnabled,
|
||||
this.jdbcTemplate, this.embeddingModel, this.dimensions, this.distanceType,
|
||||
|
||||
@@ -47,8 +47,11 @@ public class MariaDBEmbeddingDimensionsTests {
|
||||
|
||||
final int explicitDimensions = 696;
|
||||
|
||||
var dim = new MariaDBVectorStore(this.jdbcTemplate, this.embeddingModel, explicitDimensions)
|
||||
.embeddingDimensions();
|
||||
MariaDBVectorStore mariaDBVectorStore = MariaDBVectorStore.builder(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.dimensions(explicitDimensions)
|
||||
.build();
|
||||
var dim = mariaDBVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(explicitDimensions);
|
||||
verify(this.embeddingModel, never()).dimensions();
|
||||
@@ -58,7 +61,10 @@ public class MariaDBEmbeddingDimensionsTests {
|
||||
public void embeddingModelDimensions() {
|
||||
when(this.embeddingModel.dimensions()).thenReturn(969);
|
||||
|
||||
var dim = new MariaDBVectorStore(this.jdbcTemplate, this.embeddingModel).embeddingDimensions();
|
||||
MariaDBVectorStore mariaDBVectorStore = MariaDBVectorStore.builder(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.build();
|
||||
var dim = mariaDBVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(969);
|
||||
|
||||
@@ -70,7 +76,10 @@ public class MariaDBEmbeddingDimensionsTests {
|
||||
|
||||
when(this.embeddingModel.dimensions()).thenThrow(new RuntimeException());
|
||||
|
||||
var dim = new MariaDBVectorStore(this.jdbcTemplate, this.embeddingModel).embeddingDimensions();
|
||||
MariaDBVectorStore mariaDBVectorStore = MariaDBVectorStore.builder(this.jdbcTemplate)
|
||||
.embeddingModel(this.embeddingModel)
|
||||
.build();
|
||||
var dim = mariaDBVectorStore.embeddingDimensions();
|
||||
|
||||
assertThat(dim).isEqualTo(MariaDBVectorStore.OPENAI_EMBEDDING_DIMENSION_SIZE);
|
||||
verify(this.embeddingModel, only()).dimensions();
|
||||
|
||||
@@ -217,13 +217,15 @@ public class MariaDBStoreCustomNamesIT {
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
|
||||
return new MariaDBVectorStore.Builder(jdbcTemplate, embeddingModel).withSchemaName(this.schemaName)
|
||||
.withVectorTableName(this.vectorTableName)
|
||||
.withVectorTableValidationsEnabled(this.schemaValidation)
|
||||
.withDimensions(this.dimensions)
|
||||
.withDistanceType(MariaDBVectorStore.MariaDBDistanceType.COSINE)
|
||||
.withRemoveExistingVectorStoreTable(true)
|
||||
.withInitializeSchema(true)
|
||||
return MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName(this.schemaName)
|
||||
.vectorTableName(this.vectorTableName)
|
||||
.schemaValidation(this.schemaValidation)
|
||||
.dimensions(this.dimensions)
|
||||
.distanceType(MariaDBVectorStore.MariaDBDistanceType.COSINE)
|
||||
.removeExistingVectorStoreTable(true)
|
||||
.initializeSchema(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
|
||||
@@ -347,8 +347,13 @@ public class MariaDBStoreIT {
|
||||
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel) {
|
||||
return new MariaDBVectorStore(jdbcTemplate, embeddingModel, MariaDBVectorStore.INVALID_EMBEDDING_DIMENSION,
|
||||
this.distanceType, true, true);
|
||||
return MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.dimensions(MariaDBVectorStore.INVALID_EMBEDDING_DIMENSION)
|
||||
.distanceType(this.distanceType)
|
||||
.removeExistingVectorStoreTable(true)
|
||||
.initializeSchema(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
|
||||
@@ -177,10 +177,12 @@ public class MariaDBStoreObservationIT {
|
||||
@Bean
|
||||
public VectorStore vectorStore(JdbcTemplate jdbcTemplate, EmbeddingModel embeddingModel,
|
||||
ObservationRegistry observationRegistry) {
|
||||
return new MariaDBVectorStore.Builder(jdbcTemplate, embeddingModel).withSchemaName(schemaName)
|
||||
.withDistanceType(MariaDBVectorStore.MariaDBDistanceType.COSINE)
|
||||
.withObservationRegistry(observationRegistry)
|
||||
.withInitializeSchema(true)
|
||||
return MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName(schemaName)
|
||||
.distanceType(MariaDBVectorStore.MariaDBDistanceType.COSINE)
|
||||
.observationRegistry(observationRegistry)
|
||||
.initializeSchema(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
|
||||
@@ -70,8 +70,9 @@ public class MariaDBStoreTests {
|
||||
// Given
|
||||
var jdbcTemplate = mock(JdbcTemplate.class);
|
||||
var embeddingModel = mock(EmbeddingModel.class);
|
||||
var mariadbVectorStore = new MariaDBVectorStore.Builder(jdbcTemplate, embeddingModel)
|
||||
.withMaxDocumentBatchSize(1000)
|
||||
var mariadbVectorStore = MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.maxDocumentBatchSize(1000)
|
||||
.build();
|
||||
|
||||
// Testing with 9989 documents
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
/*
|
||||
* Copyright 2023-2024 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.ai.vectorstore.mariadb;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.mariadb.MariaDBVectorStore.MariaDBDistanceType;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
/**
|
||||
* Unit tests for {@link MariaDBVectorStore.MariaDBBuilder}.
|
||||
*
|
||||
* @author Mark Pollack
|
||||
*/
|
||||
class MariaDBVectorStoreBuilderTests {
|
||||
|
||||
private final JdbcTemplate jdbcTemplate = mock(JdbcTemplate.class);
|
||||
|
||||
private final EmbeddingModel embeddingModel = mock(EmbeddingModel.class);
|
||||
|
||||
@Test
|
||||
void shouldFailOnMissingEmbeddingModel() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate).build())
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("EmbeddingModel must be configured");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldFailOnMissingJdbcTemplate() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(null).build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("JdbcTemplate must not be null");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldUseDefaultValues() {
|
||||
MariaDBVectorStore vectorStore = MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.build();
|
||||
|
||||
assertThat(vectorStore).hasFieldOrPropertyWithValue("vectorTableName", "vector_store")
|
||||
.hasFieldOrPropertyWithValue("schemaName", null)
|
||||
.hasFieldOrPropertyWithValue("schemaValidation", false)
|
||||
.hasFieldOrPropertyWithValue("dimensions", -1)
|
||||
.hasFieldOrPropertyWithValue("distanceType", MariaDBDistanceType.COSINE)
|
||||
.hasFieldOrPropertyWithValue("removeExistingVectorStoreTable", false)
|
||||
.hasFieldOrPropertyWithValue("initializeSchema", false)
|
||||
.hasFieldOrPropertyWithValue("maxDocumentBatchSize", 10000)
|
||||
.hasFieldOrPropertyWithValue("contentFieldName", "content")
|
||||
.hasFieldOrPropertyWithValue("embeddingFieldName", "embedding")
|
||||
.hasFieldOrPropertyWithValue("idFieldName", "id")
|
||||
.hasFieldOrPropertyWithValue("metadataFieldName", "metadata");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldConfigureCustomValues() {
|
||||
MariaDBVectorStore vectorStore = MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.schemaName("custom_schema")
|
||||
.vectorTableName("custom_vectors")
|
||||
.schemaValidation(true)
|
||||
.dimensions(512)
|
||||
.distanceType(MariaDBDistanceType.EUCLIDEAN)
|
||||
.removeExistingVectorStoreTable(true)
|
||||
.initializeSchema(true)
|
||||
.maxDocumentBatchSize(5000)
|
||||
.contentFieldName("text")
|
||||
.embeddingFieldName("vector")
|
||||
.idFieldName("doc_id")
|
||||
.metadataFieldName("meta")
|
||||
.build();
|
||||
|
||||
assertThat(vectorStore).hasFieldOrPropertyWithValue("vectorTableName", "custom_vectors")
|
||||
.hasFieldOrPropertyWithValue("schemaName", "custom_schema")
|
||||
.hasFieldOrPropertyWithValue("schemaValidation", true)
|
||||
.hasFieldOrPropertyWithValue("dimensions", 512)
|
||||
.hasFieldOrPropertyWithValue("distanceType", MariaDBDistanceType.EUCLIDEAN)
|
||||
.hasFieldOrPropertyWithValue("removeExistingVectorStoreTable", true)
|
||||
.hasFieldOrPropertyWithValue("initializeSchema", true)
|
||||
.hasFieldOrPropertyWithValue("maxDocumentBatchSize", 5000)
|
||||
.hasFieldOrPropertyWithValue("contentFieldName", "text")
|
||||
.hasFieldOrPropertyWithValue("embeddingFieldName", "vector")
|
||||
.hasFieldOrPropertyWithValue("idFieldName", "doc_id")
|
||||
.hasFieldOrPropertyWithValue("metadataFieldName", "meta");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldValidateFieldNames() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.contentFieldName("")
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("ContentFieldName must not be empty");
|
||||
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.embeddingFieldName("")
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("EmbeddingFieldName must not be empty");
|
||||
|
||||
assertThatThrownBy(
|
||||
() -> MariaDBVectorStore.builder(jdbcTemplate).embeddingModel(embeddingModel).idFieldName("").build())
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("IdFieldName must not be empty");
|
||||
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.metadataFieldName("")
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("MetadataFieldName must not be empty");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldValidateMaxDocumentBatchSize() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.maxDocumentBatchSize(0)
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("MaxDocumentBatchSize must be positive");
|
||||
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.maxDocumentBatchSize(-1)
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("MaxDocumentBatchSize must be positive");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldValidateDistanceType() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.distanceType(null)
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("DistanceType must not be null");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldValidateBatchingStrategy() {
|
||||
assertThatThrownBy(() -> MariaDBVectorStore.builder(jdbcTemplate)
|
||||
.embeddingModel(embeddingModel)
|
||||
.batchingStrategy(null)
|
||||
.build()).isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("BatchingStrategy must not be null");
|
||||
}
|
||||
|
||||
}
|
||||
Reference in New Issue
Block a user