Add reference documentation for the new (JSON) Cache Data Import/Export feature.
Resolves gh-67.
This commit is contained in:
John Blum
363
spring-geode-docs/src/docs/asciidoc/_includes/data.adoc
Normal file
363
spring-geode-docs/src/docs/asciidoc/_includes/data.adoc
Normal file
@@ -0,0 +1,363 @@
|
||||
[[geode-data-working]]
|
||||
== Working with Data
|
||||
|
||||
One of the most important tasks during development is ensuring your Spring Boot application handles data correctly.
|
||||
In order to verify the integrity, accuracy and availability of your data, your application needs data to work with.
|
||||
|
||||
For those of you already aware of Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[initializing a database],
|
||||
and specifically, a SQL-based `DataSource`, by creating a `schema.sql` file containing SQL DDL statements and subsequently
|
||||
populating it with a `data.sql` file containing SQL DML statements, then the approach presented here for Apache Geode
|
||||
should be familiar to you.
|
||||
|
||||
Of course, Apache Geode already has built-in support that is similar in function to Spring Boot's support for SQL
|
||||
database initialization, namely with:
|
||||
|
||||
* {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] & {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence]
|
||||
* {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service]
|
||||
* And, by using _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export data commands].
|
||||
|
||||
Therefore, you could move, or {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore]
|
||||
persistent `DiskStore` files from 1 cluster to another.
|
||||
|
||||
Alternatively, you could use Apache Geode's _Snapshot Service_ to export and import data stored in `Regions`
|
||||
of the cache. This affords you the ability to filter the data while its being imported and exported.
|
||||
|
||||
TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence]
|
||||
as well as the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service].
|
||||
|
||||
Finally, the GemFire/Geode Shell tool (_Gfsh_) allows users to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data]
|
||||
or {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data] using the appropriate
|
||||
command.
|
||||
|
||||
However, in all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are
|
||||
all in a proprietary, binary format. Plus, none of these approaches are as convenient as Spring Boot's database
|
||||
initialization automation. Therefore, Spring Boot for Apache Geode (SBDG) now offers support to import data from JSON
|
||||
into Apache Geode, stored as PDX bytes.
|
||||
|
||||
Of course, depending on the source of the data, just importing data is not much good unless you can also export data
|
||||
as well. SBDG supports both _data export_ and _import_, in JSON format, out-of-the-box.
|
||||
|
||||
NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures
|
||||
(i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions`
|
||||
using your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes]
|
||||
or indirectly from Spring's and JCache's (JSR-107) {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations]
|
||||
(e.g. such as with Spring's `@Cacheable`).
|
||||
|
||||
TIP: Also see SBDG's <<geode-configuration-declarative-annotations-productivity-regions,documentation>> on the same.
|
||||
|
||||
WARNING: While this feature has utility and many edge cases have been thought through and tested thoroughly, there are
|
||||
still some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82]
|
||||
and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team
|
||||
strongly recommends that this feature only be used for development and testing purposes.
|
||||
|
||||
Let's being with _import_.
|
||||
|
||||
[[geode-data-working-import]]
|
||||
=== Importing Data
|
||||
|
||||
It is easy to import data into a cache `Region` by defining a JSON file containing the JSON object(s) you wish to load.
|
||||
The JSON file should follow the naming convention below and be put in the root of your application classpath :
|
||||
|
||||
`data-<regionName>.json`
|
||||
|
||||
NOTE: This would be the "name" of the `Region` as defined by
|
||||
{apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()].
|
||||
|
||||
For example, if you have a `Region` called "_Orders_", then you would create a JSON file called, `data-orders.json`.
|
||||
You would place this `data-orders.json` file into the root of your application classpath (e.g. in `src/test/resources`).
|
||||
|
||||
You can create JSON files for each `Region` implicitly defined (e.g. using `@EnableEntityDefinedRegions`) or explicitly
|
||||
defined (i.e. defining a `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration.
|
||||
|
||||
The JSON file could contain JSON data similar to the following:
|
||||
|
||||
.`data-orders.json`
|
||||
[source,json]
|
||||
----
|
||||
[{
|
||||
"@type": "example.app.pos.model.PurchaseOrder",
|
||||
"id": 1,
|
||||
"lineItems": [
|
||||
{
|
||||
"@type": "example.app.pos.model.LineItem",
|
||||
"product": {
|
||||
"@type": "example.app.pos.model.Product",
|
||||
"name": "Apple iPad Pro",
|
||||
"price": 1499.00,
|
||||
"category": "SHOPPING"
|
||||
},
|
||||
"quantity": 1
|
||||
},
|
||||
{
|
||||
"@type": "example.app.pos.model.LineItem",
|
||||
"product": {
|
||||
"@type": "example.app.pos.model.Product",
|
||||
"name": "Apple iPhone 11 Pro Max",
|
||||
"price": 1249.00,
|
||||
"category": "SHOPPING"
|
||||
},
|
||||
"quantity": 2
|
||||
}
|
||||
]
|
||||
}, {
|
||||
"@type": "example.app.pos.model.PurchaseOrder",
|
||||
"id": 2,
|
||||
"lineItems": [
|
||||
{
|
||||
"@type": "example.app.pos.model.LineItem",
|
||||
"product": {
|
||||
"@type": "example.app.pos.model.Product",
|
||||
"name": "Starbucks Vente Carmel Macchiato",
|
||||
"price": 5.49,
|
||||
"category": "SHOPPING"
|
||||
},
|
||||
"quantity": 1
|
||||
}
|
||||
]
|
||||
}]
|
||||
----
|
||||
|
||||
The actual application entity classes for this JSON data might look something like:
|
||||
|
||||
.Point-of-Sale (POS) Application Model Classes
|
||||
[source,java]
|
||||
----
|
||||
@Region("Orders")
|
||||
class PurchaseOrder {
|
||||
|
||||
@Id
|
||||
Long id;
|
||||
|
||||
List<LineItem> lineItems;
|
||||
|
||||
}
|
||||
|
||||
class LineItem {
|
||||
|
||||
Product product;
|
||||
Integer quantity;
|
||||
|
||||
}
|
||||
|
||||
@Region("Products")
|
||||
class Product {
|
||||
|
||||
String name;
|
||||
Category category;
|
||||
BigDecimal price;
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
As you can see, the object model and corresponding JSON can be arbitrarily complex, encapsulating a hierarchy of objects
|
||||
with complex types.
|
||||
|
||||
[[geode-data-working-import-metadata]]
|
||||
==== JSON object metadata
|
||||
|
||||
You will notice a few other details contained in the object model and JSON shown above as well.
|
||||
|
||||
[[geode-data-working-import-metadata-attype]]
|
||||
===== `@type` metadata field
|
||||
|
||||
First, is the use of the `@type` JSON object metadata field. This field does not map to any specific field/property on
|
||||
the application domain model class (e.g. `PurchaseOrder`). Rather it serves to tell the framework and/or Apache Geode's
|
||||
JSON subsystem what type of object this JSON data would map to if you were to call (e.g. `PdxInstance.getObject()`).
|
||||
|
||||
For example:
|
||||
|
||||
.Deserializing PDX as an Object
|
||||
[source,java]
|
||||
----
|
||||
@Repository
|
||||
class OrdersRepository {
|
||||
|
||||
@Resource(name = "Orders")
|
||||
Region<Long, PurchaseOrder> orders;
|
||||
|
||||
PurchaseOrder findBy(Long id) {
|
||||
|
||||
Object value = this.orders.get(id);
|
||||
|
||||
return value instanceof PurchaseOrder ? (PurchaseOrder) value
|
||||
: value instanceof PdxInstance ? ((PdxInstance) value).getObject()
|
||||
: null;
|
||||
}
|
||||
}
|
||||
----
|
||||
|
||||
Basically, the `@type` JSON object metadata field informs the `PdxInstance.getObject()` method about the type of object
|
||||
(POJO) the JSON object will map to. Otherwise, `PdxInstance.getObject()` would simply return a `PdxInstance`.
|
||||
|
||||
It is possible that Apache Geode's PDX subsystem might return an actual `PurchaseOrder` from `Region.get(key)`, but that
|
||||
all depends on the configuration of the PDX `read-serialized` cache-level configuration setting among other factors.
|
||||
|
||||
NOTE: When JSON is imported into a GemFire/Geode cache `Region`, then the [PdxInstance.getClassName()] is not actually
|
||||
a valid Java class, it is {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[JSONFormatter.JSON_CLASSNAME].
|
||||
As such, `Region` "read" data access operations (e.g. `Region.get(key)`) result in returning a `PdxInstance` and not
|
||||
a Java object.
|
||||
|
||||
TIP: You may need to additionally proxy the `Region` "read" data access operations (e.g. `Region.get(key)`) by setting
|
||||
the SBDG property `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`, which proxies the `Region`
|
||||
to wrap the Apache Geode `PdxInstance` in a SBDG `PdxInstanceWrapper` to appropriate handle the `PdxInstance.getObject()`
|
||||
call in your application code.
|
||||
|
||||
[[geode-data-working-import-metadata-id]]
|
||||
===== `id` field
|
||||
|
||||
The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the
|
||||
object's (or `PdxInstance`) identity, or "key" when stored in the `Region` (e.g.. `Region.put(key, object)`).
|
||||
|
||||
You will have noticed the the JSON for the orders above declared an "id" field as the identifier:
|
||||
|
||||
.PurchaseOrder identifier ("id")
|
||||
[source,text]
|
||||
----
|
||||
[{
|
||||
"@type": "example.app.pos.model.PurchaseOrder",
|
||||
"id": 1,
|
||||
...
|
||||
----
|
||||
|
||||
This follows the convention used by Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO field
|
||||
or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no POJO
|
||||
field or property is annotated with `@Id`, then the framework falls back to searching for a POJO field or property
|
||||
named "id".
|
||||
|
||||
In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named POJO field/property is used as the identifier,
|
||||
or key for the object when storing it into a cache `Region`.
|
||||
|
||||
However, what happens when an object, or entity does not have a surrogate id or key defined? Perhaps, the application
|
||||
domain model class is appropriately and simply using "natural" identifiers.
|
||||
|
||||
Consider a `Book` class, which might be defined as:
|
||||
|
||||
.Book class
|
||||
[source,java]
|
||||
----
|
||||
@Region("Books")
|
||||
class Book {
|
||||
|
||||
Author author;
|
||||
|
||||
@Id
|
||||
ISBN isbn;
|
||||
|
||||
LocalDate publishedDate;
|
||||
|
||||
Sring title;
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
As hinted at in the `Book` class above, the identifier of a `Book` is the `ISBN` given the `isbn` field was annotated
|
||||
with Spring Data's `@Id` mapping annotation. Except we cannot know this by searching for an `@Id` annotation in
|
||||
the JSON data.
|
||||
|
||||
You might be tempted to argue that if the `@type` metadata field is set, then we would know the class type. We could
|
||||
then load the class and inspect the class definition to learn about the identifier. That is all fine until the class
|
||||
is not actually on the system/application classpath in the first place, hence the reason SBDG's JSON support serializes
|
||||
the JSON data to Apache Geode's PDX format. Therefore, there might not be a class definition, which would lead to a
|
||||
`NoClassDefFoundError` and subsequent `ClassNotFoundException`.
|
||||
|
||||
So, what can we do?
|
||||
|
||||
Well, in good Spring fashion, you can declare another JSON object metadata field called `@identifier` to inform
|
||||
the framework what the identifier is for the JSON object.
|
||||
|
||||
For example:
|
||||
|
||||
.Using "@identifer"
|
||||
[source,json]
|
||||
----
|
||||
{
|
||||
"@type": "example.app.books.model.Book",
|
||||
"@identifier": "isbn",
|
||||
"author": {
|
||||
"id": 1,
|
||||
"name": "Josh Long"
|
||||
},
|
||||
"isbn": "978-1-449-374640-8",
|
||||
"publishedDate": "2017-08-01",
|
||||
"title": "Cloud Native Java"
|
||||
}
|
||||
----
|
||||
|
||||
Here the `@identifier` JSON object metadata field is informing the framework that the "isbn" field is the identifier
|
||||
for a `Book`.
|
||||
|
||||
[[geode-data-working-import-conditional]]
|
||||
==== Conditionally Importing Data
|
||||
|
||||
While the Spring team recommends that most users should only use this feature while developing and testing their Spring
|
||||
Boot applications with Apache Geode, a user might occasionally use this feature in production.
|
||||
|
||||
1 reason for using this feature in production might be to preload a (REPLICATE) Region containing "reference" data.
|
||||
Reference data is largely static, non-transactional by nature and infrequently changing. Preloading reference data
|
||||
is particularly useful in caching, where you want to "warm up" the cache.
|
||||
|
||||
When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files
|
||||
into `src/test/resources`. This ensures they will not be included in your application artifact (e.g. JAR, WAR) when
|
||||
deployed to production.
|
||||
|
||||
However, if you must use this feature to preload data in your production bound, Spring Boot application(s), then you
|
||||
can still "conditionally" load data from JSON. Simply configure the `spring.boot.data.gemfire.cache.data.import.active-profiles`
|
||||
property to the Spring profile(s) that must be active for the import to have any effect.
|
||||
|
||||
For example:
|
||||
|
||||
.Conditional Importing JSON
|
||||
[source,properties]
|
||||
----
|
||||
# Spring Boot application.properties
|
||||
|
||||
spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA, STAGING
|
||||
----
|
||||
|
||||
More often the not, though, you will be using the `spring.boot.data.gemfire.cache.data.import.active-profiles` property
|
||||
to limit data import usage to development (DEV) and QA (test) environments. When delivering to the QA team, you will
|
||||
likely hand them a production candidate artifact, perhaps containing JSON files.
|
||||
|
||||
NOTE: Of course, there are many ways to conditionally build application artifacts. Some users might prefer to handle
|
||||
this concern in their Gradle or Maven builds.
|
||||
|
||||
In order for the import to have an effect in this scenario, you must specifically set the `spring.profiles.active`
|
||||
property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match.
|
||||
|
||||
In the example above, the data import will only occur if `spring.profiles.active` were set to 1 of `[DEV, QA, STAGING]`
|
||||
or a combination of, e.g. `[DEV, QA]`.
|
||||
|
||||
[[geode-data-working-export]]
|
||||
=== Exporting Data
|
||||
|
||||
Exporting data is **disabled** by default. This is necessary since some data stored in your application's `Regions`
|
||||
may be sensitive in nature, and keeping the data secure and confidential is of the utmost concern and priority.
|
||||
|
||||
However, if you are only using this feature for development and testing purposes then you may want to enable the
|
||||
_export_ capability. This can be useful to move data from 1 environment to another.
|
||||
|
||||
For example, if your QA team finds a bug in the application with a particular data set, then the QQ team can _export_
|
||||
the data thereby enabling the development team to _import_ the data into their local, development environment.
|
||||
|
||||
To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`:
|
||||
|
||||
.Enable Export
|
||||
[source,properties]
|
||||
----
|
||||
# Spring Boot application.properties
|
||||
|
||||
spring.boot.data.gemfire.cache.data.export.enabled=true
|
||||
----
|
||||
|
||||
SBDG is careful to _export_ the JSON in a format that Apache Geode expects on _import_ when reimporting the JSON data,
|
||||
including things such as `@type` JSON object metadata fields, and so on.
|
||||
|
||||
WARNING: `@identifier` JSON object metadata fields are not automatically generated. While it is possible for POJOs
|
||||
stored in the `Region` to include an `@identifier` metadata field, mainly because we can inspect the object's class type,
|
||||
it is not possible when the `Region` value is already a `PdxInstance` that did not originate from JSON to begin with. In
|
||||
that case, the user must manually ensure the non-JSON generated `PdxInstance` includes an `@identifier` PDX metadata
|
||||
field before it is exported to JSON, if necessary (e.g. `Book.isbn`). This is only necessary if your entity classes do
|
||||
not specify an explicit identifier field (e.g. using the `@Id` Spring Data mapping annotation), or do not have an "id"
|
||||
field, and are then serialized to PDX. This scenario might also occur when inter-operating with native clients that
|
||||
model the application domain objects differently, and then serialize those objects as PDX that are then stored in
|
||||
Apache Geode on the server, which are then later consumed by your Spring Boot application client.
|
||||
@@ -26,7 +26,7 @@ John Blum
|
||||
:pivotal-gemfire-javadoc: https://gemfire-{pivotal-gemfire-version}-javadocs.docs.pivotal.io/
|
||||
:pivotal-gemfire-website: https://pivotal.io/pivotal-gemfire
|
||||
:spring-boot-docs: https://docs.spring.io/spring-boot/docs/current/reference
|
||||
:spring-boot-docs-html: {spring-boot-docs}/htmlsingle
|
||||
:spring-boot-docs-html: {spring-boot-docs}/html
|
||||
:spring-boot-javadoc: https://docs.spring.io/spring-boot/docs/current/api
|
||||
:spring-boot-website: https://spring.io/projects/spring-boot
|
||||
:spring-boot-data-geode-javadoc: https://docs.spring.io/autorepo/docs/spring-boot-data-geode-build/current/api/
|
||||
@@ -235,6 +235,7 @@ include::{include-dir}/templates.adoc[]
|
||||
include::{include-dir}/repositories.adoc[]
|
||||
include::{include-dir}/functions.adoc[]
|
||||
include::{include-dir}/continuous-query.adoc[]
|
||||
include::{include-dir}/data.adoc[]
|
||||
include::{include-dir}/data-serialization.adoc[]
|
||||
include::{include-dir}/security.adoc[]
|
||||
include::{include-dir}/logging.adoc[]
|
||||
|
||||
Reference in New Issue
Block a user