* Clean up README.md files in Milvus, PGvector, and Pinecone modules.
* Apply consistent treatment of 'model' when used as an AI concept, e.g. AI model or Embedding model.
* Apply consistent treatment of 'vector store' and 'vector database' references.
* Simplify sentence structures.
Closes#79
- Extend the VectorStore with similaritySearch using metadata filters using internal DSL and external DSL using Antlr
- Metdata support for Pinecone, Milvus, and pgvector vector stores
- PGVectorStore uses explict ::jsonpath casting for the pgvector filter expression to avoid injections
- Add unit tests for the filter converters, parser and DSL.
- Add ITs for the 3 vector stores
Resolves: #75
* Fixes misspelling in concepts.adco, 'Prompts' section.
* Uses plural form of AI Models in aiclient.adoc.
* Fixes several grammatical mistakes in vectordbs.adoc.
- Based on the official pinecone java library.
Later expects that indices are created externally via Ops.
- Map Document metadata to and from Pinecone's internal Struct.
Later converts the metadata into pinecone json format.
- Add integration tests and README.
- EmbeddingClient implementation that computes, locally, sentence embeddings with SBERT transformers.
- Uses pre-trained transformer models, serialized into Open Neural Network Exchange (ONNX) format.
- Deep Java Library and the Microsoft ONNX Java Runtime are used to run
the ONNX models and compute the embeddings efficiently.
- Add default tokenizer.json and model.onnx for sentence-transformers/all-MiniLM-L6-v2.
- Add, configurable resource caching service to allow caching
remote (http/https) resources to the local FS.
- README.md provides information on how to serialize ONNX models.
- add Git LFS configuration for large onnx model files.
- Provides a rudimentary text extractions for multitude of document formats,
including PDF, Word Doc/Docx PowerPoint ppt/pptx and many more.
- Generates a single Document for the extracted text.
- No pre or post processing and cleansing for the text.
- Move the ExtractedTextFormatter from pdf reader to the core reader to enable reusability. Improve the tika reader
- Add a set of PDF readers fro per-page (PagePdfDocumentReader) and per-paragraph (ParagraphPdfDocumentReader) readers.
- Use a PDFLayoutTextStripper fork and PDFLayoutTextStripperByArea extension to preserve the structure of the extracted document.
- PdfDocumentReaderConfig and PageExtractedTextFormatter in standalone classes.
- Craeate a new document-readers top level model and the pdf-reader under.
As the LifeCycle#start() occures later in time than the InitializingBean#afterPropertySet()
it could cose some initialization issues with the vector clients.
- Implement dimensions method
- Add MetadataMode support. Defaults to EMBED
- Drop the pgml extension between tests.
- Disable the PostgresMlEmbeddingClientIT by default.
Resolves#33
- Add ContentFormatter and DefaultContentFormatter that can filter the metadata
and format the Document metadata and text according to predefined templates.
- Add content formatter tests
- Allow the TextSplitter to copy the document content-formatter to the children.
When the splitter breaks the parent Document into multiple chunks (e.g.
into a list of children Documents) copy the source content formatter to
the chunks by default. Use the copyContentFormatter flag to enable/disable copping.
- Add TextSplitter IT tests
- Add MetadataExtractors as DocumentTransformers.
- Bump spring-ai project version to 0.7.0-SNAPSHOT
- Configurable metadata-mode for EmbeddingClients
- Make the metadata mode configurable for the EmbeddingClient implementations.
- Use the EMBED mode by default.
Resolves#44
