GH-1826 Fix EmbeddingModel's usage on Document#embedding
- Since the Document object's reference to the `embedding` is deprecated and will be removed, the VectorStore implementations require a way to store the embedding of the corresponding Document objects
- One way to fix this is, to have the EmbeddingModel#embed to return the embeddings in the same order as that of the Documents passed to it.
- Since both the Document and embedding collections use the List object, their iteration operation will make sure to keep them in line with the same order.
- A fix is required to preserve the order when batching strategy is applied.
- Updated the Javadoc for BatchingStrategy
- Fixed the Document List order in TokenCountBatchingStrategy
- Refactored the vector store implementations to update this change
Resolves #GH-1826
This commit is contained in:
committed by
Christian Tzolov
parent
6cfe5e79e8
commit
ebd29e0959
@@ -127,12 +127,13 @@ public class QdrantVectorStore extends AbstractObservationVectorStore implements
|
||||
try {
|
||||
|
||||
// Compute and assign an embedding to the document.
|
||||
this.embeddingModel.embed(documents, EmbeddingOptionsBuilder.builder().build(), this.batchingStrategy);
|
||||
List<float[]> embeddings = this.embeddingModel.embed(documents, EmbeddingOptionsBuilder.builder().build(),
|
||||
this.batchingStrategy);
|
||||
|
||||
List<PointStruct> points = documents.stream()
|
||||
.map(document -> PointStruct.newBuilder()
|
||||
.setId(io.qdrant.client.PointIdFactory.id(UUID.fromString(document.getId())))
|
||||
.setVectors(io.qdrant.client.VectorsFactory.vectors(document.getEmbedding()))
|
||||
.setVectors(io.qdrant.client.VectorsFactory.vectors(embeddings.get(documents.indexOf(document))))
|
||||
.putAllPayload(toPayload(document))
|
||||
.build())
|
||||
.toList();
|
||||
|
||||
Reference in New Issue
Block a user