Spring/spring-boot-data-geode
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Edit the 'Working with Data' chapter.

Browse Source 
Change title to 'Using Data'.

Revise the reference documentation to be more concise.
This commit is contained in:
John Blum
2020-07-24 11:50:19 -07:00
parent 20b8c2d172
commit adbbda54f4
1 changed files with 116 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

249
spring-geode-docs/src/docs/asciidoc/_includes/data.adoc
View File

@@ -1,75 +1,71 @@
[[geode-data-working]]
== Working with Data
[[geode-data-using]]
== Using Data
:geode-name: Apache Geode
One of the most important tasks during development is ensuring your Spring Boot application handles data correctly.
In order to verify the integrity, accuracy and availability of your data, your application needs data to work with.
In order to verify the accuracy, integrity and availability of your data, your application needs data to work with.
For those of you already aware of Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[initializing a database],
and specifically, a SQL-based `DataSource`, by creating a `schema.sql` file containing SQL DDL statements and subsequently
populating it with a `data.sql` file containing SQL DML statements, then the approach presented here for Apache Geode
should be familiar to you.
For those already familiar with Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[SQL database initialization],
the approach when using {geode-name} should be easy to understand.
Of course, Apache Geode already has built-in support that is similar in function to Spring Boot's support for SQL
database initialization, namely with:
{geode-name} provides built-in support, similar in function to Spring Boot's SQL database initialization, by using:
* {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] & {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence]
* _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export] data commands.
* {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service]
* And, by using _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export data commands].
* {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] with {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage]
Therefore, you could move, or {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore]
persistent `DiskStore` files from 1 cluster to another.
For example, by enabling Persistence with Disk Storage, you could {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore]
persistent `DiskStore` files from one cluster to another.
Alternatively, you could use Apache Geode's _Snapshot Service_ to export and import data stored in `Regions`
of the cache. This affords you the ability to filter the data while its being imported and exported.
Alternatively, using {geode-name}'s _Snapshot Service_, you can export data contained in targeted `Regions` from one
cluster during shutdown and import the data into another cluster on startup. The _Snapshot Service_ allows you to filter
data while its being imported and exported.
Finally, GemFire/Geode Shell (_Gfsh_) commands can be used to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data]
and {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data].
TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence]
as well as the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service].
and the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service].
Finally, the GemFire/Geode Shell tool (_Gfsh_) allows users to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data]
or {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data] using the appropriate
command.
In all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are in a
proprietary, binary format.
However, in all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are
all in a proprietary, binary format. Plus, none of these approaches are as convenient as Spring Boot's database
initialization automation. Therefore, Spring Boot for Apache Geode (SBDG) now offers support to import data from JSON
into Apache Geode, stored as PDX bytes.
Furthermore, none of these approaches are as convenient as Spring Boot's database initialization automation. Therefore,
Spring Boot for Apache Geode (SBDG) offers support to import data from JSON into {geode-name} as PDX.
Of course, depending on the source of the data, just importing data is not much good unless you can also export data
as well. SBDG supports both _data export_ and _import_, in JSON format, out-of-the-box.
Unlike Spring Boot, SBDG offers support to export data as well. Data is imported and exported in JSON format, by default.
NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures
(i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions`
using your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes]
or indirectly from Spring's and JCache's (JSR-107) {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations]
(e.g. such as with Spring's `@Cacheable`).
from your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] or indirectly
from Spring and JSR-107, JCache {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations].
TIP: Also see SBDG's <<geode-configuration-declarative-annotations-productivity-regions,documentation>> on the same.
TIP: Refer to SBDG's <<geode-configuration-declarative-annotations-productivity-regions,documentation>> on the same.
WARNING: While this feature has utility and many edge cases have been thought through and tested thoroughly, there are
still some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82]
WARNING: While this feature has utility and many edge cases were thought through and tested thoroughly, there are still
some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82]
and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team
strongly recommends that this feature only be used for development and testing purposes.
Let's being with _import_.
[[geode-data-working-import]]
[[geode-data-using-import]]
=== Importing Data
It is easy to import data into a cache `Region` by defining a JSON file containing the JSON object(s) you wish to load.
The JSON file should follow the naming convention below and be put in the root of your application classpath :
You can import data into a `Region` by defining a JSON file containing the JSON object(s) you wish to load. The JSON
file must follow the naming convention below and be placed in the root of your application classpath:
`data-<regionName>.json`
NOTE: This would be the "name" of the `Region` as defined by
NOTE: `<regionName>` refers to the lowercase "name" of the `Region` as defined by
{apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()].
For example, if you have a `Region` called "_Orders_", then you would create a JSON file called, `data-orders.json`.
You would place this `data-orders.json` file into the root of your application classpath (e.g. in `src/test/resources`).
For example, if you have a `Region` named "_Orders_", then you would create a JSON file called `data-orders.json`
and place it in the root of your application classpath (e.g. in `src/test/resources`).
You can create JSON files for each `Region` implicitly defined (e.g. using `@EnableEntityDefinedRegions`) or explicitly
defined (i.e. defining a `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration.
Create JSON files for each `Region` implicitly defined (e.g. by using `@EnableEntityDefinedRegions`) or explicitly
defined (i.e. with `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration that you
want to load with data.
The JSON file could contain JSON data similar to the following:
The JSON file containing JSON data for _Orders_ might appear as follows:
.`data-orders.json`
[source,json]
@@ -117,7 +113,7 @@ The JSON file could contain JSON data similar to the following:
}]
----
The actual application entity classes for this JSON data might look something like:
The application entity classes matching the JSON data might look something like:
.Point-of-Sale (POS) Application Model Classes
[source,java]
@@ -149,20 +145,21 @@ class Product {
}
----
As you can see, the object model and corresponding JSON can be arbitrarily complex, encapsulating a hierarchy of objects
with complex types.
As seen above, the object model and corresponding JSON can be arbitrarily complex with a hierarchy of objects
having complex types.
[[geode-data-working-import-metadata]]
==== JSON object metadata
[[geode-data-using-import-metadata]]
==== JSON metadata
You will notice a few other details contained in the object model and JSON shown above as well.
You will notice a few other details contained in the object model and JSON shown above.
[[geode-data-working-import-metadata-attype]]
===== `@type` metadata field
[[geode-data-using-import-metadata-attype]]
===== The `@type` metadata field
First, is the use of the `@type` JSON object metadata field. This field does not map to any specific field/property on
the application domain model class (e.g. `PurchaseOrder`). Rather it serves to tell the framework and/or Apache Geode's
JSON subsystem what type of object this JSON data would map to if you were to call (e.g. `PdxInstance.getObject()`).
First, we declared an `@type` JSON metadata field. This field does not map to any specific field or property of
the application domain model class (e.g. `PurchaseOrder`). Rather, it tells the framework and {geode-name}'s JSON/PDX
converter the type of object the JSON data would map to if you were to request an object (i.e. by calling
`PdxInstance.getObject()`).
For example:
@@ -186,29 +183,28 @@ class OrdersRepository {
}
----
Basically, the `@type` JSON object metadata field informs the `PdxInstance.getObject()` method about the type of object
(POJO) the JSON object will map to. Otherwise, `PdxInstance.getObject()` would simply return a `PdxInstance`.
Basically, the `@type` JSON metadata field informs the `PdxInstance.getObject()` method about the type of Java object
the JSON object will map to. Otherwise, the `PdxInstance.getObject()` method would silently return a `PdxInstance`.
It is possible that Apache Geode's PDX subsystem might return an actual `PurchaseOrder` from `Region.get(key)`, but that
all depends on the configuration of the PDX `read-serialized` cache-level configuration setting among other factors.
It is possible for {geode-name}'s PDX serialization framework to return a `PurchaseOrder` from `Region.get(key)` as well,
but it depends on the value of PDX's `read-serialized`, cache-level configuration setting, among other factors.
NOTE: When JSON is imported into a GemFire/Geode cache `Region`, then the [PdxInstance.getClassName()] is not actually
a valid Java class, it is {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[JSONFormatter.JSON_CLASSNAME].
As such, `Region` "read" data access operations (e.g. `Region.get(key)`) result in returning a `PdxInstance` and not
a Java object.
NOTE: When JSON is imported into a `Region` as PDX, the {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[PdxInstance.getClassName()]
does not refer to a valid Java class. It is {apache-geode-javadoc}/org/apache/geode/pdx/JSONFormatter.html#JSON_CLASSNAME[JSONFormatter.JSON_CLASSNAME].
As a result, `Region` data access operations, such as `Region.get(key)`, return a `PdxInstance` and not a Java object.
TIP: You may need to additionally proxy the `Region` "read" data access operations (e.g. `Region.get(key)`) by setting
the SBDG property `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`, which proxies the `Region`
to wrap the Apache Geode `PdxInstance` in a SBDG `PdxInstanceWrapper` to appropriate handle the `PdxInstance.getObject()`
call in your application code.
TIP: You may need to proxy `Region` "read" data access operations (e.g. `Region.get(key)`) by setting the SBDG property
`spring.boot.data.gemfire.cache.region.advice.enabled` to `true`. When this property is set, `Regions` are proxied to
wrap a `PdxInstance` in a `PdxInstanceWrapper` in order to appropriately handle the `PdxInstance.getObject()` call in
your application code.
[[geode-data-working-import-metadata-id]]
===== `id` field
[[geode-data-using-import-metadata-id]]
===== The `id` field & `@identifier` metadata field
The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the
object's (or `PdxInstance`) identity, or "key" when stored in the `Region` (e.g.. `Region.put(key, object)`).
object's (or `PdxInstance`'s) identity and "key" when stored in the `Region` (e.g. `Region.put(key, object)`).
You will have noticed the the JSON for the orders above declared an "id" field as the identifier:
You will have noticed the the JSON for the _Orders_ above declared an "id" field as the identifier:
.PurchaseOrder identifier ("id")
[source,text]
@@ -219,18 +215,17 @@ You will have noticed the the JSON for the orders above declared an "id" field a
...
----
This follows the convention used by Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO field
or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no POJO
field or property is annotated with `@Id`, then the framework falls back to searching for a POJO field or property
named "id".
This follows the same convention used in Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO
field or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no
field or property is annotated with `@Id`, then the framework falls back to searching for a field or property named "id".
In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named POJO field/property is used as the identifier,
or key for the object when storing it into a cache `Region`.
In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named field or property is used as the identifier,
and as the key for the object when storing it into a `Region`.
However, what happens when an object, or entity does not have a surrogate id or key defined? Perhaps, the application
domain model class is appropriately and simply using "natural" identifiers.
However, what happens when an object, or entity does not have a surrogate id defined? Perhaps the application domain
model class is appropriately and simply using "natural" identifiers, which is quite common in practice.
Consider a `Book` class, which might be defined as:
Consider a `Book` class defined as follows:
.Book class
[source,java]
@@ -250,20 +245,18 @@ class Book {
}
----
As hinted at in the `Book` class above, the identifier of a `Book` is the `ISBN` given the `isbn` field was annotated
with Spring Data's `@Id` mapping annotation. Except we cannot know this by searching for an `@Id` annotation in
the JSON data.
As declared in the `Book` class above, the identifier for `Book` is its `ISBN` since the `isbn` field was annotated with
Spring Data's `@Id` mapping annotation. However, we cannot know this by searching for an `@Id` annotation in JSON.
You might be tempted to argue that if the `@type` metadata field is set, then we would know the class type. We could
then load the class and inspect the class definition to learn about the identifier. That is all fine until the class
is not actually on the system/application classpath in the first place, hence the reason SBDG's JSON support serializes
the JSON data to Apache Geode's PDX format. Therefore, there might not be a class definition, which would lead to a
`NoClassDefFoundError` and subsequent `ClassNotFoundException`.
You might be tempted to argue that if the `@type` metadata field is set, we would know the class type and could load
the class definition to learn about the identifier. That is all fine until the class is not actually on the application
classpath in the first place. This is one of the reasons why SBDG's JSON support serializes JSON to {geode-name}'s PDX
format. There might not be a class definition, which would lead to a `NoClassDefFoundError` or `ClassNotFoundException`.
So, what can we do?
So, what then?
Well, in good Spring fashion, you can declare another JSON object metadata field called `@identifier` to inform
the framework what the identifier is for the JSON object.
In this case, SBDG allows you to declare the `@identifier` JSON metadata field to inform the framework
what to use as the identifier for the object.
For example:
@@ -283,26 +276,25 @@ For example:
}
----
Here the `@identifier` JSON object metadata field is informing the framework that the "isbn" field is the identifier
for a `Book`.
Here, the `@identifier` JSON metadata field informs the framework that the "isbn" field is the identifier for a `Book`.
[[geode-data-working-import-conditional]]
[[geode-data-using-import-conditional]]
==== Conditionally Importing Data
While the Spring team recommends that most users should only use this feature while developing and testing their Spring
Boot applications with Apache Geode, a user might occasionally use this feature in production.
While the Spring team recommends that users should only use this feature when developing and testing their Spring Boot
applications with Apache Geode, a user may occasionally use this feature in production.
1 reason for using this feature in production might be to preload a (REPLICATE) Region containing "reference" data.
Reference data is largely static, non-transactional by nature and infrequently changing. Preloading reference data
is particularly useful in caching, where you want to "warm up" the cache.
Users might use this feature in production to preload a (REPLICATE) Region with "reference" data. Reference data is
largely static, infrequently changing and non-transactional. Preloading reference data is particularly useful in caching
use cases, where you want to "warm" the cache.
When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files
into `src/test/resources`. This ensures they will not be included in your application artifact (e.g. JAR, WAR) when
When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files in
`src/test/resources`. This ensures the files will not be included in your application artifact (e.g. JAR, WAR) when
deployed to production.
However, if you must use this feature to preload data in your production bound, Spring Boot application(s), then you
can still "conditionally" load data from JSON. Simply configure the `spring.boot.data.gemfire.cache.data.import.active-profiles`
property to the Spring profile(s) that must be active for the import to have any effect.
However, if you must use this feature to preload data in your production environment, then you can still "conditionally"
load data from JSON. To do so configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` property set to
the Spring profile(s) that must be active for the import to take effect.
For example:
@@ -311,35 +303,27 @@ For example:
----
# Spring Boot application.properties
spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA, STAGING
spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA
----
More often the not, though, you will be using the `spring.boot.data.gemfire.cache.data.import.active-profiles` property
to limit data import usage to development (DEV) and QA (test) environments. When delivering to the QA team, you will
likely hand them a production candidate artifact, perhaps containing JSON files.
NOTE: Of course, there are many ways to conditionally build application artifacts. Some users might prefer to handle
this concern in their Gradle or Maven builds.
In order for the import to have an effect in this scenario, you must specifically set the `spring.profiles.active`
In order for import to have an effect in this example, you must specifically set the `spring.profiles.active`
property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match.
In the example above, the data import will only occur if `spring.profiles.active` were set to 1 of `[DEV, QA, STAGING]`
or a combination of, e.g. `[DEV, QA]`.
NOTE: There are many ways to conditionally build application artifacts. Some users might prefer to handle this concern
in their Gradle or Maven builds.
[[geode-data-working-export]]
[[geode-data-using-export]]
=== Exporting Data
Exporting data is **disabled** by default. This is necessary since some data stored in your application's `Regions`
may be sensitive in nature, and keeping the data secure and confidential is of the utmost concern and priority.
Certain data stored in your application's `Regions` may be sensitive or confidential and keeping the data secure is of
the utmost concern and priority. Therefore, exporting data is **disabled** by default.
However, if you are only using this feature for development and testing purposes then you may want to enable the
_export_ capability. This can be useful to move data from 1 environment to another.
However, if you are using this feature for development and testing purposes then enabling the _export_ capability may be
useful to move data from 1 environment to another. For example, if your QA team finds a bug in the application using a
particular data set, then they can _export_ the data and pass it back to the development team to _import_ in their local
development environment to help debug the issue.
For example, if your QA team finds a bug in the application with a particular data set, then the QQ team can _export_
the data thereby enabling the development team to _import_ the data into their local, development environment.
To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`:
To enable _export_, set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`:
.Enable Export
[source,properties]
@@ -349,15 +333,14 @@ To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.d
spring.boot.data.gemfire.cache.data.export.enabled=true
----
SBDG is careful to _export_ the JSON in a format that Apache Geode expects on _import_ when reimporting the JSON data,
including things such as `@type` JSON object metadata fields, and so on.
SBDG is careful to _export_ data to JSON in a format that {geode-name} expects on _import_ and includes things such as
`@type` metadata fields.
WARNING: `@identifier` JSON object metadata fields are not automatically generated. While it is possible for POJOs
stored in the `Region` to include an `@identifier` metadata field, mainly because we can inspect the object's class type,
it is not possible when the `Region` value is already a `PdxInstance` that did not originate from JSON to begin with. In
that case, the user must manually ensure the non-JSON generated `PdxInstance` includes an `@identifier` PDX metadata
field before it is exported to JSON, if necessary (e.g. `Book.isbn`). This is only necessary if your entity classes do
not specify an explicit identifier field (e.g. using the `@Id` Spring Data mapping annotation), or do not have an "id"
field, and are then serialized to PDX. This scenario might also occur when inter-operating with native clients that
model the application domain objects differently, and then serialize those objects as PDX that are then stored in
Apache Geode on the server, which are then later consumed by your Spring Boot application client.
WARNING: The `@identifier` metadata field is not generated automatically. While it is possible for POJOs stored in a
`Region` to include an `@identifier` metadata field when exported to JSON it is not possible when the `Region` value
is a `PdxInstance` that did not originate from JSON. In this case, you must manually ensure the `PdxInstance` includes
an `@identifier` metadata field before it is exported to JSON if necessary (e.g. `Book.isbn`). This is only necessary
if your entity classes do not declare an explicit identifier field, such as with the `@Id` mapping annotation, or do
not have an "id" field. This scenario can also occur when inter-operating with native clients that model the application
domain objects differently, then serialize the objects using PDX storing them in Regions on the server that are then
later consumed by your Spring Boot application.