Commit Graph

170 Commits

Author SHA1 Message Date
Fabian Krüger
c831a73cd4 Add hint about cloning the repo 2023-11-25 12:23:33 +01:00
Josh Long
c520e77472 support graalvm native compilation for openai 2023-11-24 12:56:56 +01:00
Christian Tzolov
58a6735a1b Clean and fix the openai ITs 2023-11-23 12:30:28 +01:00
Christian Tzolov
c7f22f9a99 Temp disable ClientIT until huggingface endpoint is available 2023-11-23 12:04:31 +01:00
Christian Tzolov
3a0b5dfab4 Disable MilvusVectorStoreAutoConfigurationIT due to outdated milvus grpc dependecies 2023-11-23 11:58:45 +01:00
Craig Walls
ffe44e24e2 Add equals() and hashCode() for Prompt and SearchRequest 2023-11-22 15:52:56 -05:00
Christian Tzolov
ad7af600fd improve vector-store docs 2023-11-22 01:10:14 +01:00
Christian Tzolov
b9d2add707 Fix azure store test and redundant code 2023-11-22 01:03:40 +01:00
Christian Tzolov
faee5fef6f Add support for Weaviate Vector Store
- Add Weaviate Vector Store implementation.
 - Implement a converter of portable Filter.Expressions into native, Weaviate GraphQL Were expressions.
 - Support for Weaviater schema auto-registration of filtarable metadata fields.
 - Add auto-configration, spring properties and tests.
 - WeaviateVectorStore ITs.
 - Add README.md

 Resolves #100
2023-11-21 18:16:34 -05:00
Christian Tzolov
332c92a57a Support portable filter expressions with AzureVectorStore 2023-11-21 18:16:34 -05:00
John Blum
eadcae7702 Define API for AI provider metadata collected during an AI request.
* Define GenerationMetadata property in AiResponse.
* Add OpenAI implementations of AiMetadata, RateLimit and Usage interfaces.
* Add REST Assured JsonPath dependency to spring-ai-openai module.
* Add OkHttp dependency to spring-ai-openai module.
* Add OkHttp Interceptor to parse OpenAI rate limit metadata from HTTP headers.
* Add OkHttp MockWebServer dependency to spring-ai-openai module, test scope
* Add Jakarta Servlet API dependency to spring-ai-openai module, test scope
* Add Spring Web MVC dependency to spring-ai-open-ai module., test scope
* Define OpenAI API response headers in an Enum.
* Add OpenAI test configuration using mock objects.
* Add integration test to assert successful extraction of OpenAI API response metadata.
* Include Spring Boot auto-configuration for (conditional) OpenAI metadata collection.
* Edit documentation and include information on AI metadata collected by Spring AI.
* Provide AI metadata implementation for Microsoft Azure OpenAI Service.
* Capture optional PromptMetadata in AiResponse.
* Define metadata for an AI generation choice.
* Capture AI choice metadata in Generation.
* Integrate ChoiceMetadata into AiResponse returned by OpenAI.

Fixes #98
2023-11-21 18:16:34 -05:00
Mark Pollack
465211f832 revert back to azure open ai client 1.0.0-beta.3 2023-11-21 18:12:12 -05:00
John Blum
142e799c86 Upgrade to Azure OpenAI client library for Java 1.0.0-beta.5. 2023-11-21 17:30:08 -05:00
Janne Valkealahti
55a02f6356 Fix workflows for repo org change 2023-11-20 18:39:55 +00:00
Eddú Meléndez
91374ea1b2 Cleanup tests using testcontainers
* Specify tag images
* Use getHost() instead of hardcoded `localhost`
2023-11-17 13:05:01 -06:00
Christian Tzolov
1ab7ceedff Fix transformers-embedding test’s cross-platform number precision 2023-11-17 12:04:57 +01:00
Tommy Ludwig
6bdea65cd5 Auto-configuration and starter for Ollama
- Adds auto-configuration and a starter for Ollama.
 - Add testconteiner orca-mini integration test.
 - Improve code style and documentation.

Resolves #68, Resolves #69
2023-11-16 22:40:52 +01:00
John Blum
820eb5f0ab Refine delineation of Embedding properties from OpenAI (client) properties.
Though the property names are different, for example 'spring.ai.openai.api-key' and 'spring.ai.openai.embedding-api-key,
by introducing an additional property namespace, 'spring.ai.openai.embedding.*', this enables the configuration of embeddings
to be properly encapsulated and related in Spring Boot fashion.

Closes #102
2023-11-16 15:51:15 -05:00
John Blum
409fa1521d Refine OpenAI and Microsoft Azure AiClient implementations.
* Apply consistent treatment to the organization of the source code.
* Fix Logger statements missing message placeholder formatting.
* Simplify logic using Java 17 syntax where applicable.

Closes #101
2023-11-16 15:25:31 -05:00
meyerg
0aa8eb651e Add Azure AI Search Vector Store
- Use Azure AI Search end point to implement the VectorStore interface.
 - Add ITs and README.
 - Create/update vector index on after properties set.
 - Add boot auto-configuration and ITs.
 - Add boot starter for the vector store.

 Resolves: #82
2023-11-16 20:38:23 +01:00
Christian Tzolov
621aa7d53e Switch to BOM djl dependecies in the transformer-embedding 2023-11-16 16:14:48 +01:00
Christian Tzolov
98ca3e2a8f Add Chroma VectorStore support
- Implement ChromaApi client, based on Chroma REST API.
 - Implement ChromaVectorStore, including support for filter expression conversion.
 - Common VectorStoreUtil class to share to/from Float/Double list/array convertion as well as Json/Map convertions.
 - Add ITs including for Basic Auth and Token autheticatios.
 - Add ChromaApi security support for BasicAuth and Token.
 - Fix an issue with Text filter expression parser, related to double-quoted identifiers.
 - Add Chroma README.md.
 - Add Chroma boot autoconfiguration

Resolves# #86
2023-11-15 22:43:36 -05:00
KathrynBrusewitz
837c4080aa Make filter expression builder Op public 2023-11-15 19:35:49 +01:00
Christian Tzolov
3ea938c4e1 Transformers-embedding client - align tokenizer with onnx's model inputs 2023-11-15 15:18:22 +01:00
Christian Tzolov
ac2a0cd8e9 Fix auto-configurations missing form the import list 2023-11-15 12:54:13 +01:00
Christian Tzolov
1746bf6b46 Add transformers-embedding boot auto-configuraiton and starter 2023-11-15 12:29:22 +01:00
Christian Tzolov
952ff6c319 Simplify VectorStore interface
- Collapses all VectorStore similiaritySearch methdos into one with SearchRequest builder.
 - Fix all affected code and tests.
 - Bump the project version to 0.7.1.
 - Add tests
 - Add autoconfigurations for milvus, pinecone and pgvecor stores.
 - Improve and unify the VectorStore ITs.
 - Make use of TrasformersEmbeddingClient for auto-configurations ITs.
2023-11-14 15:39:01 -05:00
Christian Tzolov
e0a9d4a8fe Update README.md
add java dependency to doc
2023-11-10 19:02:55 +01:00
Christian Tzolov
0a584f0dc2 Transformers embedding Readme clarifications 2023-11-10 08:05:47 +01:00
Christian Tzolov
e423c3f127 Improve transformers embedding readme 2023-11-10 00:22:14 +01:00
Gerrit Meier
e5fe8ad33d Add readme for Neo4jVectorStore. 2023-11-09 15:44:45 -05:00
Mark Pollack
49656a036f Add link to research paper on effective prompts 2023-11-09 15:43:57 -05:00
Jay Bryant
04d9683f49 Editing pass for clarity, grammar, spelling, punctuation, and usage.
Also added several anchors and links.
2023-11-09 15:40:16 -05:00
John Blum
78f6c8c6e8 Fix anchor to 'Workshop' section in project README. (#85)
Improve formatting around video references.

Closes #85
2023-11-09 15:35:57 -05:00
Mark Pollack
1666cf0845 Allow for up to 3 retries when running an integration test 2023-11-09 15:18:47 -05:00
Mark Pollack
73eae56e65 Add test support for AI programming 2023-11-09 12:21:44 +01:00
John Blum
0a236993e3 Fix additional grammatical mistakes and revise wording in vectordbs.adoc.
* Clean up README.md files in Milvus, PGvector, and Pinecone modules.
* Apply consistent treatment of 'model' when used as an AI concept, e.g. AI model or Embedding model.
* Apply consistent treatment of 'vector store' and 'vector database' references.
* Simplify sentence structures.

Closes #79
2023-11-08 12:24:20 -08:00
Mark Pollack
37e53900ac fix failing test 2023-11-07 21:13:13 -05:00
Mark Pollack
57fae37dec ignore failing test... for now! 2023-11-07 21:07:25 -05:00
Mark Pollack
651a998e11 add some additional educational resources 2023-11-07 21:03:41 -05:00
Mark Pollack
d514f28cd4 update vector store docs 2023-11-07 17:05:14 -05:00
Christian Tzolov
3615667f05 Support for search with metdata filter expressions
- Extend the VectorStore with similaritySearch using metadata filters using internal DSL and external DSL using Antlr
 - Metdata support for Pinecone, Milvus, and pgvector vector stores
 - PGVectorStore uses explict ::jsonpath casting for the pgvector filter expression to avoid injections
 - Add unit tests for the filter converters, parser and DSL.
 - Add ITs for the 3 vector stores

Resolves: #75
2023-11-07 15:47:53 -05:00
Mark Pollack
bc12d4364c add reference docs for AiClient 2023-11-04 13:58:27 -04:00
Mark Pollack
6d956a6f37 Add initial support for Hugging Face Inference Endpoints
- Add maven config to attach swagger generate code to vscode java classpath.
 - Add license headers.
 - Streamline the test.
2023-11-03 18:45:34 +01:00
John Blum
a97976e434 Clean up grammatical errors and misspellings in the reference documentation. (#77)
* Fixes misspelling in concepts.adco, 'Prompts' section.
* Uses plural form of AI Models in aiclient.adoc.
* Fixes several grammatical mistakes in vectordbs.adoc.
2023-11-03 09:46:39 -04:00
Christian Tzolov
be10cc410e vectore store readmes improvements 2023-11-02 10:52:34 +01:00
Christian Tzolov
f4a4de836d Improve pgvector docs 2023-11-02 09:41:28 +01:00
Christian Tzolov
009a24a59b Minor pinecone store readme fixes 2023-11-02 09:01:12 +01:00
Christian Tzolov
81a5eaf801 Add Pinecone VectorStore
- Based on the official pinecone java library.
   Later expects that indices are created externally via Ops.
 - Map Document metadata to and from Pinecone's internal Struct.
   Later converts the metadata into pinecone json format.
 - Add integration tests and README.
2023-11-01 20:33:20 -04:00
Christian Tzolov
001ee990b3 Suppress pdfbox excessive logging 2023-10-31 14:21:50 +01:00