= ETL Pipeline

The Extract, Transform, and Load (ETL) framework serves as the backbone of data processing within the Retrieval Augmented Generation (RAG) use case.

The ETL pipeline orchestrates the flow from raw data sources to a structured vector store, ensuring data is in the optimal format for retrieval by the AI model.

The RAG use case is text to augment the capabilities of generative models by retrieving relevant information from a body of data to enhance the quality and relevance of the generated output.

== API Overview

There are three main components of the ETL pipeline,

* `DocumentReader` that implements `Supplier<List<Document>>`
* `DocumentTransformer` that implements `Function<List<Document>, List<Document>>`
* `DocumentWriter` that implements `Consumer<List<Document>>`

The `Document` class contains text and metadata and is created from PDFs, text files and other document types via the `DocumentReader`.

To construct a simple ETL pipeline, you can chain together an instance of each type.

image::etl-pipeline.jpg[align="center"]

Let's say we have the following instances of those three ETL types

* `PagePdfDocumentReader` an implementation of `DocumentReader`
* `TokenTextSplitter` an implementation of `DocumentTransformer`
* `VectorStore` an implementation of `DocumentWriter`

To perform the basic loading of data into a Vector Database for use with the Retrieval Augmented Generation pattern, use the following code in Java function style syntax.

[source,java]
----
vectorStore.accept(tokenTextSplitter.apply(pdfReader.get()));
----

Alternatively, you can use method names that are more naturally expressive for the domain

[source,java]
----
vectorStore.write(tokenTextSplitter.split(pdfReader.read()));
----



== Getting Started

To begin creating a Spring AI RAG application, follow these steps:

. Download the latest https://github.com/spring-projects/spring-cli/releases[Spring CLI Release]
and follow the https://docs.spring.io/spring-cli/reference/installation.html#_setting_up_your_path_or_alias[installation instructions].
. To create a simple OpenAI-based application, use the command:
+
```shell
spring boot new --from ai-rag --name myrag
```
. Consult the generated `README.md` file for guidance on obtaining an OpenAI API Key and running your first AI RAG application.

== ETL Interfaces and Implementations

The ETL pipeline is composed of the following interfaces and implementations.
Detailed ETL class diagram is shown in the <<etl-class-diagram>> section.

=== DocumentReader

Provides a source of documents from diverse origins.
[source,java]
----
public interface DocumentReader extends Supplier<List<Document>> {

    default List<Document> read() {
		return get();
	}
}
----

==== JsonReader
The `JsonReader` Parses documents in JSON format.

Example:

[source,java]
----
@Component
class MyAiAppComponent {

	private final Resource resource;

    MyAiAppComponent(@Value("classpath:bikes.json") Resource resource) {
        this.resource = resource;
    }

	List<Document> loadJsonAsDocuments() {
		JsonReader jsonReader = new JsonReader(resource, "description");
		return jsonReader.read();
	}
}
----

==== TextReader
The `TextReader` processes plain text documents.

Example:

[source,java]
----
@Component
class MyTextReader {

    private final Resource resource;

    MyTextReader(@Value("classpath:text-source.txt") Resource resource) {
        this.resource = resource;
    }
	List<Document> loadText() {
		TextReader textReader = new TextReader(resource);
		textReader.getCustomMetadata().put("filename", "text-source.txt");

		return textReader.read();
    }
}
----

==== PagePdfDocumentReader
The `PagePdfDocumentReader` uses Apache PdfBox library to parse PDF documents

Example:

[source,java]
----
@Component
public class MyPagePdfDocumentReader {

	List<Document> getDocsFromPdf() {

		PagePdfDocumentReader pdfReader = new PagePdfDocumentReader("classpath:/sample1.pdf",
				PdfDocumentReaderConfig.builder()
					.withPageTopMargin(0)
					.withPageExtractedTextFormatter(ExtractedTextFormatter.builder()
						.withNumberOfTopTextLinesToDelete(0)
						.build())
					.withPagesPerDocument(1)
					.build());

		return pdfReader.read();
    }

}

----


==== ParagraphPdfDocumentReader
The `ParagraphPdfDocumentReader` uses the PDF catalog (e.g. TOC) information to split the input PDF into text paragraphs and output a single `Document` per paragraph.
NOTE: Not all PDF documents contain the PDF catalog.

Example:

[source,java]
----
@Component
public class MyPagePdfDocumentReader {

	List<Document> getDocsFromPdfwithCatalog() {

        new ParagraphPdfDocumentReader("classpath:/sample1.pdf",
                PdfDocumentReaderConfig.builder()
                    .withPageTopMargin(0)
                    .withPageExtractedTextFormatter(ExtractedTextFormatter.builder()
                        .withNumberOfTopTextLinesToDelete(0)
                        .build())
                    .withPagesPerDocument(1)
                    .build());

		return pdfReader.read();
    }
}
----


==== TikaDocumentReader
The `TikaDocumentReader` uses Apache Tika to extract text from a variety of document formats, such as PDF, DOC/DOCX, PPT/PPTX, and HTML. For a comprehensive list of supported formats, refer to the  https://tika.apache.org/2.9.0/formats.html[Tika documentation].

Example:

[source,java]
----
@Component
class MyTikaDocumentReader {

    private final Resource resource;

    MyTikaDocumentReader(@Value("classpath:/word-sample.docx")
                            Resource resource) {
        this.resource = resource;
    }

    List<Document> loadText() {
        TikaDocumentReader tikaDocumentReader = new TikaDocumentReader(resource);
        return tikaDocumentReader.read();
    }
}
----

=== DocumentTransformer

Transforms a batch of documents as part of the processing workflow.

[source,java]
----
public interface DocumentTransformer extends Function<List<Document>, List<Document>> {

    default List<Document> transform(List<Document> transform) {
		return apply(transform);
	}
}
----

==== TextSplitter
The `TextSplitter` an abstract base class that helps divides documents to fit the AI model's context window.


==== TokenTextSplitter
Splits documents while preserving token-level integrity.

==== ContentFormatTransformer
Ensures uniform content formats across all documents.

==== KeywordMetadataEnricher
Augments documents with essential keyword metadata.

==== SummaryMetadataEnricher
Enriches documents with summarization metadata for enhanced retrieval.

=== DocumentWriter

Manages the final stage of the ETL process, preparing documents for storage.

```java
public interface DocumentWriter extends Consumer<List<Document>> {

    default void write(List<Document> documents) {
		accept(documents);
	}
}
```
==== FileDocumentWriter
Persist documents to a file .

==== VectorStore

Provides integration with various vector stores.
See xref:api/vectordbs.adoc[Vector DB Documentation] for a full listing.

[[etl-class-diagram]]
=== ETL Class Diagram

The following class diagram illustrates the ETL interfaces and implementations.

// image::etl-class-diagram.jpg[align="center", width="800px"]
image::etl-class-diagram.jpg[align="center"]
