diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc index 937051e7b..3fbfa4f3c 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/nav.adoc @@ -26,8 +26,8 @@ *** xref:api/clients/vertexai-chat.adoc[] ** xref:api/prompt.adoc[] ** xref:api/output-parser.adoc[] -** xref:api/etl-pipeline.adoc[] ** xref:api/vectordbs.adoc[] +** xref:api/etl-pipeline.adoc[] *** xref:api/vectordbs/azure.adoc[] *** xref:api/vectordbs/chroma.adoc[] *** xref:api/vectordbs/milvus.adoc[] @@ -35,6 +35,7 @@ *** xref:api/vectordbs/pgvector.adoc[] *** xref:api/vectordbs/weaviate.adoc[] *** xref:api/vectordbs/redis.adoc[] +*** xref:api/vectordbs/pinecone.adoc[] ** xref:api/testing.adoc[] * Appendices ** xref:glossary.adoc[] diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/index.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/index.adoc index 7b7abd07b..9ff0f5050 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/index.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/index.adoc @@ -9,6 +9,7 @@ The <> can help you understand the basic constructs used impl == APIs * xref:api/embeddings.adoc[Embeddings API] * xref:api/chatclient.adoc[Chat Completion API] +* xref:api/vectordbs.adoc[Vector Database API] * xref:api/[Image Generation API](WIP) == Spring AI Generic Model API [[generic-model-api]] diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs.adoc index faffd8914..8ab6c0382 100644 --- a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs.adoc +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs.adoc @@ -6,7 +6,7 @@ A vector databases is a specialized type of database that plays an essential rol In vector databases, queries differ from traditional relational databases. Instead of exact matches, they perform similarity searches. When given a vector as a query, a vector database returns vectors that are "`similar`" to the query vector. -Further details on how this similarity is calculated at a high-level is provided in a <>. +Further details on how this similarity is calculated at a high-level is provided in a xref:api/vectordbs/understand-vectordbs.adoc#vectordbs-similarity[Vector Similarity]. Vector databases are used to integrate your data with AI models. The first step in their usage is to load your data into a vector database. @@ -49,6 +49,7 @@ public class SearchRequest { private Filter.Expression filterExpression; public static SearchRequest query(String query) { return new SearchRequest(query); } + private SearchRequest(String query) { this.query = query; } public SearchRequest withTopK(int topK) {...} @@ -78,24 +79,23 @@ The `similaritySearch` methods in the interface allow for retrieving documents s * `k`: An integer that specifies the maximum number of similar documents to return. This is often referred to as a 'top K' search, or 'K nearest neighbors' (KNN). * `threshold`: A double value ranging from 0 to 1, where values closer to 1 indicate higher similarity. By default, if you set a threshold of 0.75, for instance, only documents with a similarity above this value are returned. * `Filter.Expression`: A class used for passing a fluent DSL (Domain-Specific Language) expression that functions similarly to a 'where' clause in SQL, but it applies exclusively to the metadata key-value pairs of a `Document`. -* `filterExpression`: An external DSL based on ANTLR4 that accepts filter expressions as strings. For example, with metadata keys like country, year, and `isActive`, you could use an expression such as -``` java -country == 'UK' && year >= 2020 && isActive == true. -``` +* `filterExpression`: An external DSL based on ANTLR4 that accepts filter expressions as strings. For example, with metadata keys like country, year, and `isActive`, you could use an expression such as: `country == 'UK' && year >= 2020 && isActive == true.` + +Find more information on the `Filter.Expression` in the <> section. == Available Implementations These are the available implementations of the `VectorStore` interface: -* Azure Vector Search [`AzureVectorStore`]: The https://learn.microsoft.com/en-us/azure/search/vector-search-overview[Azure] vector store -* Chroma [`ChromaVectorStore`]: The https://www.trychroma.com/[Chroma] vector store -* Milvus [`MilvusVectorStore`]: The https://milvus.io/[Milvus] vector store -* Neo4j [`Neo4jVectorStore`]: The https://neo4j.com/[Neo4j] vector store -* PgVector [`PgVectorStore`]: The https://github.com/pgvector/pgvector[PostgreSQL/PGVector] vector store -* Pinecone: https://www.pinecone.io/[PineCone] vector store -* Redis [`RedisVectorStore`]: The https://redis.io/[Redis] vector store -* Simple Vector Store [`SimpleVectorStore`]: A simple implementation of persistent vector storage, good for educational purposes -* Weaviate [`WeaviateVectorStore`] The https://weaviate.io/[Weaviate] vector store +* xref:api/vectordbs/azure.adoc[ Azure Vector Search] - The https://learn.microsoft.com/en-us/azure/search/vector-search-overview[Azure] vector store. +* xref:api/vectordbs/chroma.adoc[ChromaVectorStore] - The https://www.trychroma.com/[Chroma] vector store. +* xref:api/vectordbs/milvus.adoc[MilvusVectorStore] - The https://milvus.io/[Milvus] vector store. +* xref:api/vectordbs/neo4j.adoc[Neo4jVectorStore] - The https://neo4j.com/[Neo4j] vector store. +* xref:api/vectordbs/pgvector.adoc[PgVectorStore] - The https://github.com/pgvector/pgvector[PostgreSQL/PGVector] vector store. +* xref:api/vectordbs/pinecone.adoc[PineconeVectorStore] - https://www.pinecone.io/[PineCone] vector store. +* xref:api/vectordbs/redis.adoc[RedisVectorStore] - The https://redis.io/[Redis] vector store. +* xref:api/vectordbs/weaviate.adoc[WeaviateVectorStore] - The https://weaviate.io/[Weaviate] vector store. +* link:https://github.com/spring-projects/spring-ai/blob/main/spring-ai-core/src/main/java/org/springframework/ai/vectorstore/SimpleVectorStore.java[SimpleVectorStore] - A simple implementation of persistent vector storage, good for educational purposes. More implementations may be supported in future releases. @@ -137,7 +137,7 @@ Later, when a user question is passed into the AI model, a similarity search is Additional options can be passed into the `similaritySearch` method to define how many documents to retrieve and a threshold of the similarity search. -== Metadata Filters +== Metadata Filters [[metadata-filters]] This section describes various filters that you can use against the results of a query. @@ -209,99 +209,6 @@ Expression exp = b.and(b.eq("genre", "drama"), b.gte("year", 2020)).build(); == Understanding Vectors -Vectors have dimensionality and a direction. -For example, the following image depicts a two-dimensional vector stem:[\vec{a}] in the cartesian coordinate system pictured as an arrow. - -image::vector_2d_coordinates.png[] - -The head of the vector stem:[\vec{a}] is at the point stem:[(a_1, a_2)]. -The *x* coordinate value is stem:[a_1] and the *y* coordinate value is stem:[a_2]. The coordinates are also referred to as the components of the vector. - -[[vectordbs-similarity]] -== Similarity - -Several mathematical formulas can be used to determine if two vectors are similar. - -One of the most intuitive to visualize and understand is cosine similarity. - -Consider the following images that show three sets of graphs: - -image::vector_similarity.png[] - -The vectors stem:[\vec{A}] and stem:[\vec{B}] are considered similar, when they are pointing close to each other, as in the first diagram. -The vectors are considered unrelated when pointing perpendicular to each other and opposite when they point away from each other. - -The angle between them, stem:[\theta], is a good measure of their similarity. -How can the angle stem:[\theta] be computed? - -We are all familiar with the https://en.wikipedia.org/wiki/Pythagorean_theorem#History[Pythagorean Theorem]. - -image:pythagorean-triangle.png[] - -What about when the angle between *a* and *b* is not 90 degrees? - -Enter the https://en.wikipedia.org/wiki/Law_of_cosines[Law of cosines]. +xref:api/vectordbs/understand-vectordbs.adoc[Understanding Vectors] -.Law of Cosines -**** -stem:[a^2 + b^2 - 2ab\cos\theta = c^2] -**** - -The following image shows this approach as a vector diagram: - -image:lawofcosines.png[] - - -The magnitude of this vector is defined in terms of its components as: - -.Magnitude -**** -stem:[\vec{A} * \vec{A} = ||\vec{A}||^2 = A_1^2 + A_2^2 ] -**** - -The dot product between two vectors stem:[\vec{A}] and stem:[\vec{B}] is defined in terms of its components as: - - -.Dot Product -**** -stem:[\vec{A} * \vec{B} = A_1B_1 + A_2B_2] -**** - -Rewriting the Law of Cosines with vector magnitudes and dot products gives the following: - -.Law of Cosines in Vector form -**** -stem:[||\vec{A}||^2 + ||\vec{B}||^2 - 2||\vec{A}||||\vec{B}||\cos\theta = ||\vec{C}||^2] -**** - - -Replacing stem:[||\vec{C}||^2] with stem:[||\vec{B} - \vec{A}||^2] gives the following: - -.Law of Cosines in Vector form only in terms of stem:[\vec{A}] and stem:[\vec{B}] - -**** -stem:[||\vec{A}||^2 + ||\vec{B}||^2 - 2||\vec{A}||||\vec{B}||\cos\theta = ||\vec{B} - \vec{A}||^2] -**** - - -https://towardsdatascience.com/cosine-similarity-how-does-it-measure-the-similarity-maths-behind-and-usage-in-python-50ad30aad7db[Expanding this out] gives us the formula for https://en.wikipedia.org/wiki/Cosine_similarity[Cosine Similarity]. - -.Cosine Similarity -**** -stem:[similarity(vec{A},vec{B}) = \cos(\theta) = \frac{\vec{A}\cdot\vec{B}}{||\vec{A}\||\cdot||\vec{B}||] -**** - -This formula works for dimensions higher than 2 or 3, though it is hard to visualize. However, https://projector.tensorflow.org/[it can be visualized to some extent]. -It is common for vectors in AI/ML applications to have hundreds or even thousands of dimensions. - -The similarity function in higher dimensions using the components of the vector is shown below. -It expands the two-dimensional definitions of Magnitude and Dot Product given previously to *N* dimensions by using https://en.wikipedia.org/wiki/Summation[Summation mathematical syntax]. - -.Cosine Similarity with vector components -**** -stem:[similarity(vec{A},vec{B}) = \cos(\theta) = \frac{ \sum_{i=1}^{n} {A_i B_i} }{ \sqrt{\sum_{i=1}^{n}{A_i^2} \cdot \sum_{i=1}^{n}{B_i^2}}] -**** - -This is the key formula used in the simple implementation of a vector store and can be found in the `InMemoryVectorStore` implementation. - diff --git a/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/understand-vectordbs.adoc b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/understand-vectordbs.adoc new file mode 100644 index 000000000..a5fea74be --- /dev/null +++ b/spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/understand-vectordbs.adoc @@ -0,0 +1,95 @@ +[[understand-vector-databases]] += Understanding Vectors + +image::vector_2d_coordinates.png[width=150, role = "right"] + +Vectors have dimensionality and a direction. +For example, the following image depicts a two-dimensional vector stem:[\vec{a}] in the cartesian coordinate system pictured as an arrow. + +The head of the vector stem:[\vec{a}] is at the point stem:[(a_1, a_2)]. +The *x* coordinate value is stem:[a_1] and the *y* coordinate value is stem:[a_2]. The coordinates are also referred to as the components of the vector. + +[[vectordbs-similarity]] +== Similarity + +Several mathematical formulas can be used to determine if two vectors are similar. +One of the most intuitive to visualize and understand is cosine similarity. +Consider the following images that show three sets of graphs: + +image::vector_similarity.png[align="center",width=600] + +The vectors stem:[\vec{A}] and stem:[\vec{B}] are considered similar, when they are pointing close to each other, as in the first diagram. +The vectors are considered unrelated when pointing perpendicular to each other and opposite when they point away from each other. + +The angle between them, stem:[\theta], is a good measure of their similarity. +How can the angle stem:[\theta] be computed? + +image:pythagorean-triangle.png[align="center",width=100, role="left", trim="10 10 10 100"] + +We are all familiar with the https://en.wikipedia.org/wiki/Pythagorean_theorem#History[Pythagorean Theorem]. + +What about when the angle between *a* and *b* is not 90 degrees? + + +Enter the https://en.wikipedia.org/wiki/Law_of_cosines[Law of cosines]. + + +.Law of Cosines +**** +stem:[a^2 + b^2 - 2ab\cos\theta = c^2] +**** + +The following image shows this approach as a vector diagram: +image:lawofcosines.png[align="center",width=200] + +The magnitude of this vector is defined in terms of its components as: + +.Magnitude +**** +stem:[\vec{A} * \vec{A} = ||\vec{A}||^2 = A_1^2 + A_2^2 ] +**** + +The dot product between two vectors stem:[\vec{A}] and stem:[\vec{B}] is defined in terms of its components as: + +.Dot Product +**** +stem:[\vec{A} * \vec{B} = A_1B_1 + A_2B_2] +**** + +Rewriting the Law of Cosines with vector magnitudes and dot products gives the following: + +.Law of Cosines in Vector form +**** +stem:[||\vec{A}||^2 + ||\vec{B}||^2 - 2||\vec{A}||||\vec{B}||\cos\theta = ||\vec{C}||^2] +**** + + +Replacing stem:[||\vec{C}||^2] with stem:[||\vec{B} - \vec{A}||^2] gives the following: + +.Law of Cosines in Vector form only in terms of stem:[\vec{A}] and stem:[\vec{B}] + +**** +stem:[||\vec{A}||^2 + ||\vec{B}||^2 - 2||\vec{A}||||\vec{B}||\cos\theta = ||\vec{B} - \vec{A}||^2] +**** + + +https://towardsdatascience.com/cosine-similarity-how-does-it-measure-the-similarity-maths-behind-and-usage-in-python-50ad30aad7db[Expanding this out] gives us the formula for https://en.wikipedia.org/wiki/Cosine_similarity[Cosine Similarity]. + +.Cosine Similarity +**** +stem:[similarity(vec{A},vec{B}) = \cos(\theta) = \frac{\vec{A}\cdot\vec{B}}{||\vec{A}\||\cdot||\vec{B}||] +**** + +This formula works for dimensions higher than 2 or 3, though it is hard to visualize. However, https://projector.tensorflow.org/[it can be visualized to some extent]. +It is common for vectors in AI/ML applications to have hundreds or even thousands of dimensions. + +The similarity function in higher dimensions using the components of the vector is shown below. +It expands the two-dimensional definitions of Magnitude and Dot Product given previously to *N* dimensions by using https://en.wikipedia.org/wiki/Summation[Summation mathematical syntax]. + +.Cosine Similarity with vector components +**** +stem:[similarity(vec{A},vec{B}) = \cos(\theta) = \frac{ \sum_{i=1}^{n} {A_i B_i} }{ \sqrt{\sum_{i=1}^{n}{A_i^2} \cdot \sum_{i=1}^{n}{B_i^2}}] +**** + +This is the key formula used in the simple implementation of a vector store and can be found in the `InMemoryVectorStore` implementation. +