Commit Graph

174 Commits

Author SHA1 Message Date
Christian Tzolov
ac9ae589f4 Add NOT operator to VectorStore portable filter expressions
- Add NOT expression type to the portable Filter.Expression model.
 - Add NOT to the Antlr grammar and implement the related parser listener method to generate Filter NOT expressions.
 - Add NOT support to the filter programming DSL.
 - Implement FilterHelper.negation for logically transform any boolean expression with NOT statements into
   semantically equivalent one with NOT applied to the leaf expressions.
 - Add tests for paresers, converters and vectorsores ITs.
 - Move the filter IN/NIN expansion logic to the FilterHelper
 - Factor out the filter IN/NIN boolean expression expansion logic out of Weaviate up to the FilterHelper.
 - add in/nin expantion FilterHelper tests
2023-11-28 18:17:56 -05:00
John Blum
e3e5070256 Fix dependencies in spring-ai-test module.
* Changes dependency on spring-ai-openai to spring-ai-core as a required dependency.
* Include dependency on Jakarta Servlet API.
* Include dependency on Spring Web MVC.
* Include dependnecy on OkHttp3 MockWebServer.
* Add common AI test configuration using mock objects.

Rebase OpenAI mock test configuration on the shared components inside spring-ai-test.

* Renames OpenAiMockTestConfiguration to MockOpenAiTestConfiguration.
* Refactor MockOpenAiTestConfiguration removing mock infrastructure beans and import MockAiTestConfiguration class.
* Declare test dependency on spring-ai-test.=
* Integrate complete AI metadata implementation for Microsoft Azure OpenAI.

This new Spring @Configuration class enables AI developers to test against the AI provider's REST API by mocking Web service endpoints and returning canned AI responses.
This allows the AI provider client (Java) library to be exercised in the same manner as the production application, or even Spring AI framework code without modification.

Closes #122
2023-11-28 18:10:59 -05:00
Harry9656
8e8be5da4a Fix link pointing to arxiv 2023-11-25 12:36:14 +01:00
Christian Tzolov
62146ae768 Update README.md
Further clarifications for cloning the repo.
2023-11-25 12:35:05 +01:00
Fabian Krüger
c831a73cd4 Add hint about cloning the repo 2023-11-25 12:23:33 +01:00
Josh Long
c520e77472 support graalvm native compilation for openai 2023-11-24 12:56:56 +01:00
Christian Tzolov
58a6735a1b Clean and fix the openai ITs 2023-11-23 12:30:28 +01:00
Christian Tzolov
c7f22f9a99 Temp disable ClientIT until huggingface endpoint is available 2023-11-23 12:04:31 +01:00
Christian Tzolov
3a0b5dfab4 Disable MilvusVectorStoreAutoConfigurationIT due to outdated milvus grpc dependecies 2023-11-23 11:58:45 +01:00
Craig Walls
ffe44e24e2 Add equals() and hashCode() for Prompt and SearchRequest 2023-11-22 15:52:56 -05:00
Christian Tzolov
ad7af600fd improve vector-store docs 2023-11-22 01:10:14 +01:00
Christian Tzolov
b9d2add707 Fix azure store test and redundant code 2023-11-22 01:03:40 +01:00
Christian Tzolov
faee5fef6f Add support for Weaviate Vector Store
- Add Weaviate Vector Store implementation.
 - Implement a converter of portable Filter.Expressions into native, Weaviate GraphQL Were expressions.
 - Support for Weaviater schema auto-registration of filtarable metadata fields.
 - Add auto-configration, spring properties and tests.
 - WeaviateVectorStore ITs.
 - Add README.md

 Resolves #100
2023-11-21 18:16:34 -05:00
Christian Tzolov
332c92a57a Support portable filter expressions with AzureVectorStore 2023-11-21 18:16:34 -05:00
John Blum
eadcae7702 Define API for AI provider metadata collected during an AI request.
* Define GenerationMetadata property in AiResponse.
* Add OpenAI implementations of AiMetadata, RateLimit and Usage interfaces.
* Add REST Assured JsonPath dependency to spring-ai-openai module.
* Add OkHttp dependency to spring-ai-openai module.
* Add OkHttp Interceptor to parse OpenAI rate limit metadata from HTTP headers.
* Add OkHttp MockWebServer dependency to spring-ai-openai module, test scope
* Add Jakarta Servlet API dependency to spring-ai-openai module, test scope
* Add Spring Web MVC dependency to spring-ai-open-ai module., test scope
* Define OpenAI API response headers in an Enum.
* Add OpenAI test configuration using mock objects.
* Add integration test to assert successful extraction of OpenAI API response metadata.
* Include Spring Boot auto-configuration for (conditional) OpenAI metadata collection.
* Edit documentation and include information on AI metadata collected by Spring AI.
* Provide AI metadata implementation for Microsoft Azure OpenAI Service.
* Capture optional PromptMetadata in AiResponse.
* Define metadata for an AI generation choice.
* Capture AI choice metadata in Generation.
* Integrate ChoiceMetadata into AiResponse returned by OpenAI.

Fixes #98
2023-11-21 18:16:34 -05:00
Mark Pollack
465211f832 revert back to azure open ai client 1.0.0-beta.3 2023-11-21 18:12:12 -05:00
John Blum
142e799c86 Upgrade to Azure OpenAI client library for Java 1.0.0-beta.5. 2023-11-21 17:30:08 -05:00
Janne Valkealahti
55a02f6356 Fix workflows for repo org change 2023-11-20 18:39:55 +00:00
Eddú Meléndez
91374ea1b2 Cleanup tests using testcontainers
* Specify tag images
* Use getHost() instead of hardcoded `localhost`
2023-11-17 13:05:01 -06:00
Christian Tzolov
1ab7ceedff Fix transformers-embedding test’s cross-platform number precision 2023-11-17 12:04:57 +01:00
Tommy Ludwig
6bdea65cd5 Auto-configuration and starter for Ollama
- Adds auto-configuration and a starter for Ollama.
 - Add testconteiner orca-mini integration test.
 - Improve code style and documentation.

Resolves #68, Resolves #69
2023-11-16 22:40:52 +01:00
John Blum
820eb5f0ab Refine delineation of Embedding properties from OpenAI (client) properties.
Though the property names are different, for example 'spring.ai.openai.api-key' and 'spring.ai.openai.embedding-api-key,
by introducing an additional property namespace, 'spring.ai.openai.embedding.*', this enables the configuration of embeddings
to be properly encapsulated and related in Spring Boot fashion.

Closes #102
2023-11-16 15:51:15 -05:00
John Blum
409fa1521d Refine OpenAI and Microsoft Azure AiClient implementations.
* Apply consistent treatment to the organization of the source code.
* Fix Logger statements missing message placeholder formatting.
* Simplify logic using Java 17 syntax where applicable.

Closes #101
2023-11-16 15:25:31 -05:00
meyerg
0aa8eb651e Add Azure AI Search Vector Store
- Use Azure AI Search end point to implement the VectorStore interface.
 - Add ITs and README.
 - Create/update vector index on after properties set.
 - Add boot auto-configuration and ITs.
 - Add boot starter for the vector store.

 Resolves: #82
2023-11-16 20:38:23 +01:00
Christian Tzolov
621aa7d53e Switch to BOM djl dependecies in the transformer-embedding 2023-11-16 16:14:48 +01:00
Christian Tzolov
98ca3e2a8f Add Chroma VectorStore support
- Implement ChromaApi client, based on Chroma REST API.
 - Implement ChromaVectorStore, including support for filter expression conversion.
 - Common VectorStoreUtil class to share to/from Float/Double list/array convertion as well as Json/Map convertions.
 - Add ITs including for Basic Auth and Token autheticatios.
 - Add ChromaApi security support for BasicAuth and Token.
 - Fix an issue with Text filter expression parser, related to double-quoted identifiers.
 - Add Chroma README.md.
 - Add Chroma boot autoconfiguration

Resolves# #86
2023-11-15 22:43:36 -05:00
KathrynBrusewitz
837c4080aa Make filter expression builder Op public 2023-11-15 19:35:49 +01:00
Christian Tzolov
3ea938c4e1 Transformers-embedding client - align tokenizer with onnx's model inputs 2023-11-15 15:18:22 +01:00
Christian Tzolov
ac2a0cd8e9 Fix auto-configurations missing form the import list 2023-11-15 12:54:13 +01:00
Christian Tzolov
1746bf6b46 Add transformers-embedding boot auto-configuraiton and starter 2023-11-15 12:29:22 +01:00
Christian Tzolov
952ff6c319 Simplify VectorStore interface
- Collapses all VectorStore similiaritySearch methdos into one with SearchRequest builder.
 - Fix all affected code and tests.
 - Bump the project version to 0.7.1.
 - Add tests
 - Add autoconfigurations for milvus, pinecone and pgvecor stores.
 - Improve and unify the VectorStore ITs.
 - Make use of TrasformersEmbeddingClient for auto-configurations ITs.
2023-11-14 15:39:01 -05:00
Christian Tzolov
e0a9d4a8fe Update README.md
add java dependency to doc
2023-11-10 19:02:55 +01:00
Christian Tzolov
0a584f0dc2 Transformers embedding Readme clarifications 2023-11-10 08:05:47 +01:00
Christian Tzolov
e423c3f127 Improve transformers embedding readme 2023-11-10 00:22:14 +01:00
Gerrit Meier
e5fe8ad33d Add readme for Neo4jVectorStore. 2023-11-09 15:44:45 -05:00
Mark Pollack
49656a036f Add link to research paper on effective prompts 2023-11-09 15:43:57 -05:00
Jay Bryant
04d9683f49 Editing pass for clarity, grammar, spelling, punctuation, and usage.
Also added several anchors and links.
2023-11-09 15:40:16 -05:00
John Blum
78f6c8c6e8 Fix anchor to 'Workshop' section in project README. (#85)
Improve formatting around video references.

Closes #85
2023-11-09 15:35:57 -05:00
Mark Pollack
1666cf0845 Allow for up to 3 retries when running an integration test 2023-11-09 15:18:47 -05:00
Mark Pollack
73eae56e65 Add test support for AI programming 2023-11-09 12:21:44 +01:00
John Blum
0a236993e3 Fix additional grammatical mistakes and revise wording in vectordbs.adoc.
* Clean up README.md files in Milvus, PGvector, and Pinecone modules.
* Apply consistent treatment of 'model' when used as an AI concept, e.g. AI model or Embedding model.
* Apply consistent treatment of 'vector store' and 'vector database' references.
* Simplify sentence structures.

Closes #79
2023-11-08 12:24:20 -08:00
Mark Pollack
37e53900ac fix failing test 2023-11-07 21:13:13 -05:00
Mark Pollack
57fae37dec ignore failing test... for now! 2023-11-07 21:07:25 -05:00
Mark Pollack
651a998e11 add some additional educational resources 2023-11-07 21:03:41 -05:00
Mark Pollack
d514f28cd4 update vector store docs 2023-11-07 17:05:14 -05:00
Christian Tzolov
3615667f05 Support for search with metdata filter expressions
- Extend the VectorStore with similaritySearch using metadata filters using internal DSL and external DSL using Antlr
 - Metdata support for Pinecone, Milvus, and pgvector vector stores
 - PGVectorStore uses explict ::jsonpath casting for the pgvector filter expression to avoid injections
 - Add unit tests for the filter converters, parser and DSL.
 - Add ITs for the 3 vector stores

Resolves: #75
2023-11-07 15:47:53 -05:00
Mark Pollack
bc12d4364c add reference docs for AiClient 2023-11-04 13:58:27 -04:00
Mark Pollack
6d956a6f37 Add initial support for Hugging Face Inference Endpoints
- Add maven config to attach swagger generate code to vscode java classpath.
 - Add license headers.
 - Streamline the test.
2023-11-03 18:45:34 +01:00
John Blum
a97976e434 Clean up grammatical errors and misspellings in the reference documentation. (#77)
* Fixes misspelling in concepts.adco, 'Prompts' section.
* Uses plural form of AI Models in aiclient.adoc.
* Fixes several grammatical mistakes in vectordbs.adoc.
2023-11-03 09:46:39 -04:00
Christian Tzolov
be10cc410e vectore store readmes improvements 2023-11-02 10:52:34 +01:00