Add JSON Pointer support to JsonReader

This enhancement allows users to specify a JSON Pointer to extract
specific parts of a JSON document when reading. Key changes include:

- New get(String pointer) method in JsonReader class
- Updated JsonReaderTests with pointer-based extraction tests
- Added documentation for JSON Pointer usage in etl-pipeline.adoc

This feature enables more flexible parsing of complex JSON structures,
allowing users to easily target nested data for extraction.
This commit is contained in:
ktm
2024-10-01 22:15:45 -04:00
committed by Mark Pollack
parent 89487e3565
commit 2babb2a878
4 changed files with 113 additions and 2 deletions

View File

@@ -103,4 +103,37 @@ public class JsonReader implements DocumentReader {
return new Document(content, metadata);
}
protected List<Document> get(JsonNode rootNode) {
if (rootNode.isArray()) {
return StreamSupport.stream(rootNode.spliterator(), true)
.map(jsonNode -> parseJsonNode(jsonNode, objectMapper))
.toList();
}
else {
return Collections.singletonList(parseJsonNode(rootNode, objectMapper));
}
}
/**
* Retrieves documents from the JSON resource using a JSON Pointer.
* @param pointer A JSON Pointer string (RFC 6901) to locate the desired element
* @return A list of Documents parsed from the located JSON element
* @throws RuntimeException if the JSON cannot be parsed or the pointer is invalid
*/
public List<Document> get(String pointer) {
try {
JsonNode rootNode = objectMapper.readTree(this.resource.getInputStream());
JsonNode targetNode = rootNode.at(pointer);
if (targetNode.isMissingNode()) {
throw new IllegalArgumentException("Invalid JSON Pointer: " + pointer);
}
return get(targetNode);
}
catch (IOException e) {
throw new RuntimeException("Error reading JSON resource", e);
}
}
}

View File

@@ -28,11 +28,14 @@ import static org.assertj.core.api.Assertions.assertThat;
@SpringBootTest
public class JsonReaderTests {
@Value("classpath:person.json")
private Resource ObjectResource;
@Value("classpath:bikes.json")
private Resource arrayResource;
@Value("classpath:person.json")
private Resource ObjectResource;
@Value("classpath:events.json")
private Resource eventsResource;
@Test
void loadJsonArray() {
@@ -56,4 +59,26 @@ public class JsonReaderTests {
}
}
@Test
void loadJsonArrayFromPointer() {
assertThat(arrayResource).isNotNull();
JsonReader jsonReader = new JsonReader(eventsResource, "description");
List<Document> documents = jsonReader.get("/0/sessions");
assertThat(documents).isNotEmpty();
for (Document document : documents) {
assertThat(document.getContent()).isNotEmpty();
assertThat(document.getContent()).contains("Session");
}
}
@Test
void loadJsonObjectFromPointer() {
assertThat(ObjectResource).isNotNull();
JsonReader jsonReader = new JsonReader(ObjectResource, "name");
List<Document> documents = jsonReader.get("/store");
assertThat(documents).isNotEmpty();
assertThat(documents.size()).isEqualTo(1);
assertThat(documents.get(0).getContent()).contains("name: Bike Shop");
}
}

View File

@@ -0,0 +1,15 @@
[
{
"sessions": [
{
"description": "Session one"
},
{
"description": "Session two"
},
{
"description": "Session three"
}
]
}
]

View File

@@ -154,6 +154,44 @@ The `JsonReader` processes JSON content as follows:
** It generates metadata using the provided `JsonMetadataGenerator` (or an empty one if not provided).
** It creates a `Document` object with the extracted content and metadata.
==== Using JSON Pointers
The `JsonReader` now supports retrieving specific parts of a JSON document using JSON Pointers. This feature allows you to easily extract nested data from complex JSON structures.
===== The `get(String pointer)` method
[source,java]
----
public List<Document> get(String pointer)
----
This method allows you to use a JSON Pointer to retrieve a specific part of the JSON document.
====== Parameters
* `pointer`: A JSON Pointer string (as defined in RFC 6901) to locate the desired element within the JSON structure.
====== Return Value
* Returns a `List<Document>` containing the documents parsed from the JSON element located by the pointer.
====== Behavior
* The method uses the provided JSON Pointer to navigate to a specific location in the JSON structure.
* If the pointer is valid and points to an existing element:
** For a JSON object: it returns a list with a single Document.
** For a JSON array: it returns a list of Documents, one for each element in the array.
* If the pointer is invalid or points to a non-existent element, it throws an `IllegalArgumentException`.
====== Example
[source,java]
----
JsonReader jsonReader = new JsonReader(resource, "description");
List<Document> documents = jsonReader.get("/store/books/0");
----
==== Example JSON Structure
[source,json]