Spring/spring-boot-data-geode
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Edit the 'Working with Data' chapter.

Browse Source 
Change title to 'Using Data'.

Revise the reference documentation to be more concise.
This commit is contained in:
John Blum
2020-07-24 11:50:19 -07:00
parent 20b8c2d172
commit adbbda54f4
1 changed files with 116 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

249
spring-geode-docs/src/docs/asciidoc/_includes/data.adoc
View File

@@ -1,75 +1,71 @@
[[geode-data-working]] [[geode-data-using]]
== Working with Data == Using Data
:geode-name: Apache Geode
One of the most important tasks during development is ensuring your Spring Boot application handles data correctly. One of the most important tasks during development is ensuring your Spring Boot application handles data correctly.
In order to verify the integrity, accuracy and availability of your data, your application needs data to work with. In order to verify the accuracy, integrity and availability of your data, your application needs data to work with.
For those of you already aware of Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[initializing a database], For those already familiar with Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[SQL database initialization],
and specifically, a SQL-based `DataSource`, by creating a `schema.sql` file containing SQL DDL statements and subsequently the approach when using {geode-name} should be easy to understand.
populating it with a `data.sql` file containing SQL DML statements, then the approach presented here for Apache Geode
should be familiar to you.
Of course, Apache Geode already has built-in support that is similar in function to Spring Boot's support for SQL {geode-name} provides built-in support, similar in function to Spring Boot's SQL database initialization, by using:
database initialization, namely with:
* {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] & {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] * _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export] data commands.
* {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service] * {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service]
* And, by using _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export data commands]. * {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] with {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage]
Therefore, you could move, or {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore] For example, by enabling Persistence with Disk Storage, you could {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore]
persistent `DiskStore` files from 1 cluster to another. persistent `DiskStore` files from one cluster to another.
Alternatively, you could use Apache Geode's _Snapshot Service_ to export and import data stored in `Regions` Alternatively, using {geode-name}'s _Snapshot Service_, you can export data contained in targeted `Regions` from one
of the cache. This affords you the ability to filter the data while its being imported and exported. cluster during shutdown and import the data into another cluster on startup. The _Snapshot Service_ allows you to filter
data while its being imported and exported.
Finally, GemFire/Geode Shell (_Gfsh_) commands can be used to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data]
and {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data].
TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence] TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence]
as well as the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service]. and the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service].
Finally, the GemFire/Geode Shell tool (_Gfsh_) allows users to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data] In all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are in a
or {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data] using the appropriate proprietary, binary format.
command.
However, in all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are Furthermore, none of these approaches are as convenient as Spring Boot's database initialization automation. Therefore,
all in a proprietary, binary format. Plus, none of these approaches are as convenient as Spring Boot's database Spring Boot for Apache Geode (SBDG) offers support to import data from JSON into {geode-name} as PDX.
initialization automation. Therefore, Spring Boot for Apache Geode (SBDG) now offers support to import data from JSON
into Apache Geode, stored as PDX bytes.
Of course, depending on the source of the data, just importing data is not much good unless you can also export data Unlike Spring Boot, SBDG offers support to export data as well. Data is imported and exported in JSON format, by default.
as well. SBDG supports both _data export_ and _import_, in JSON format, out-of-the-box.
NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures
(i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions` (i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions`
using your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] from your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] or indirectly
or indirectly from Spring's and JCache's (JSR-107) {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations] from Spring and JSR-107, JCache {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations].
(e.g. such as with Spring's `@Cacheable`).
TIP: Also see SBDG's <<geode-configuration-declarative-annotations-productivity-regions,documentation>> on the same. TIP: Refer to SBDG's <<geode-configuration-declarative-annotations-productivity-regions,documentation>> on the same.
WARNING: While this feature has utility and many edge cases have been thought through and tested thoroughly, there are WARNING: While this feature has utility and many edge cases were thought through and tested thoroughly, there are still
still some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82] some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82]
and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team
strongly recommends that this feature only be used for development and testing purposes. strongly recommends that this feature only be used for development and testing purposes.
Let's being with _import_. [[geode-data-using-import]]
[[geode-data-working-import]]
=== Importing Data === Importing Data
It is easy to import data into a cache `Region` by defining a JSON file containing the JSON object(s) you wish to load. You can import data into a `Region` by defining a JSON file containing the JSON object(s) you wish to load. The JSON
The JSON file should follow the naming convention below and be put in the root of your application classpath : file must follow the naming convention below and be placed in the root of your application classpath:
`data-<regionName>.json` `data-<regionName>.json`
NOTE: This would be the "name" of the `Region` as defined by NOTE: `<regionName>` refers to the lowercase "name" of the `Region` as defined by
{apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()]. {apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()].
For example, if you have a `Region` called "_Orders_", then you would create a JSON file called, `data-orders.json`. For example, if you have a `Region` named "_Orders_", then you would create a JSON file called `data-orders.json`
You would place this `data-orders.json` file into the root of your application classpath (e.g. in `src/test/resources`). and place it in the root of your application classpath (e.g. in `src/test/resources`).
You can create JSON files for each `Region` implicitly defined (e.g. using `@EnableEntityDefinedRegions`) or explicitly Create JSON files for each `Region` implicitly defined (e.g. by using `@EnableEntityDefinedRegions`) or explicitly
defined (i.e. defining a `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration. defined (i.e. with `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration that you
want to load with data.
The JSON file could contain JSON data similar to the following: The JSON file containing JSON data for _Orders_ might appear as follows:
.`data-orders.json` .`data-orders.json`
[source,json] [source,json]
@@ -117,7 +113,7 @@ The JSON file could contain JSON data similar to the following:
}] }]
---- ----
The actual application entity classes for this JSON data might look something like: The application entity classes matching the JSON data might look something like:
.Point-of-Sale (POS) Application Model Classes .Point-of-Sale (POS) Application Model Classes
[source,java] [source,java]
@@ -149,20 +145,21 @@ class Product {
} }
---- ----
As you can see, the object model and corresponding JSON can be arbitrarily complex, encapsulating a hierarchy of objects As seen above, the object model and corresponding JSON can be arbitrarily complex with a hierarchy of objects
with complex types. having complex types.
[[geode-data-working-import-metadata]] [[geode-data-using-import-metadata]]
==== JSON object metadata ==== JSON metadata
You will notice a few other details contained in the object model and JSON shown above as well. You will notice a few other details contained in the object model and JSON shown above.
[[geode-data-working-import-metadata-attype]] [[geode-data-using-import-metadata-attype]]
===== `@type` metadata field ===== The `@type` metadata field
First, is the use of the `@type` JSON object metadata field. This field does not map to any specific field/property on First, we declared an `@type` JSON metadata field. This field does not map to any specific field or property of
the application domain model class (e.g. `PurchaseOrder`). Rather it serves to tell the framework and/or Apache Geode's the application domain model class (e.g. `PurchaseOrder`). Rather, it tells the framework and {geode-name}'s JSON/PDX
JSON subsystem what type of object this JSON data would map to if you were to call (e.g. `PdxInstance.getObject()`). converter the type of object the JSON data would map to if you were to request an object (i.e. by calling
`PdxInstance.getObject()`).
For example: For example:
@@ -186,29 +183,28 @@ class OrdersRepository {
} }
---- ----
Basically, the `@type` JSON object metadata field informs the `PdxInstance.getObject()` method about the type of object Basically, the `@type` JSON metadata field informs the `PdxInstance.getObject()` method about the type of Java object
(POJO) the JSON object will map to. Otherwise, `PdxInstance.getObject()` would simply return a `PdxInstance`. the JSON object will map to. Otherwise, the `PdxInstance.getObject()` method would silently return a `PdxInstance`.
It is possible that Apache Geode's PDX subsystem might return an actual `PurchaseOrder` from `Region.get(key)`, but that It is possible for {geode-name}'s PDX serialization framework to return a `PurchaseOrder` from `Region.get(key)` as well,
all depends on the configuration of the PDX `read-serialized` cache-level configuration setting among other factors. but it depends on the value of PDX's `read-serialized`, cache-level configuration setting, among other factors.
NOTE: When JSON is imported into a GemFire/Geode cache `Region`, then the [PdxInstance.getClassName()] is not actually NOTE: When JSON is imported into a `Region` as PDX, the {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[PdxInstance.getClassName()]
a valid Java class, it is {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[JSONFormatter.JSON_CLASSNAME]. does not refer to a valid Java class. It is {apache-geode-javadoc}/org/apache/geode/pdx/JSONFormatter.html#JSON_CLASSNAME[JSONFormatter.JSON_CLASSNAME].
As such, `Region` "read" data access operations (e.g. `Region.get(key)`) result in returning a `PdxInstance` and not As a result, `Region` data access operations, such as `Region.get(key)`, return a `PdxInstance` and not a Java object.
a Java object.
TIP: You may need to additionally proxy the `Region` "read" data access operations (e.g. `Region.get(key)`) by setting TIP: You may need to proxy `Region` "read" data access operations (e.g. `Region.get(key)`) by setting the SBDG property
the SBDG property `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`, which proxies the `Region` `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`. When this property is set, `Regions` are proxied to
to wrap the Apache Geode `PdxInstance` in a SBDG `PdxInstanceWrapper` to appropriate handle the `PdxInstance.getObject()` wrap a `PdxInstance` in a `PdxInstanceWrapper` in order to appropriately handle the `PdxInstance.getObject()` call in
call in your application code. your application code.
[[geode-data-working-import-metadata-id]] [[geode-data-using-import-metadata-id]]
===== `id` field ===== The `id` field & `@identifier` metadata field
The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the
object's (or `PdxInstance`) identity, or "key" when stored in the `Region` (e.g.. `Region.put(key, object)`). object's (or `PdxInstance`'s) identity and "key" when stored in the `Region` (e.g. `Region.put(key, object)`).
You will have noticed the the JSON for the orders above declared an "id" field as the identifier: You will have noticed the the JSON for the _Orders_ above declared an "id" field as the identifier:
.PurchaseOrder identifier ("id") .PurchaseOrder identifier ("id")
[source,text] [source,text]
@@ -219,18 +215,17 @@ You will have noticed the the JSON for the orders above declared an "id" field a
... ...
---- ----
This follows the convention used by Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO field This follows the same convention used in Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO
or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no POJO field or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no
field or property is annotated with `@Id`, then the framework falls back to searching for a POJO field or property field or property is annotated with `@Id`, then the framework falls back to searching for a field or property named "id".
named "id".
In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named POJO field/property is used as the identifier, In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named field or property is used as the identifier,
or key for the object when storing it into a cache `Region`. and as the key for the object when storing it into a `Region`.
However, what happens when an object, or entity does not have a surrogate id or key defined? Perhaps, the application However, what happens when an object, or entity does not have a surrogate id defined? Perhaps the application domain
domain model class is appropriately and simply using "natural" identifiers. model class is appropriately and simply using "natural" identifiers, which is quite common in practice.
Consider a `Book` class, which might be defined as: Consider a `Book` class defined as follows:
.Book class .Book class
[source,java] [source,java]
@@ -250,20 +245,18 @@ class Book {
} }
---- ----
As hinted at in the `Book` class above, the identifier of a `Book` is the `ISBN` given the `isbn` field was annotated As declared in the `Book` class above, the identifier for `Book` is its `ISBN` since the `isbn` field was annotated with
with Spring Data's `@Id` mapping annotation. Except we cannot know this by searching for an `@Id` annotation in Spring Data's `@Id` mapping annotation. However, we cannot know this by searching for an `@Id` annotation in JSON.
the JSON data.
You might be tempted to argue that if the `@type` metadata field is set, then we would know the class type. We could You might be tempted to argue that if the `@type` metadata field is set, we would know the class type and could load
then load the class and inspect the class definition to learn about the identifier. That is all fine until the class the class definition to learn about the identifier. That is all fine until the class is not actually on the application
is not actually on the system/application classpath in the first place, hence the reason SBDG's JSON support serializes classpath in the first place. This is one of the reasons why SBDG's JSON support serializes JSON to {geode-name}'s PDX
the JSON data to Apache Geode's PDX format. Therefore, there might not be a class definition, which would lead to a format. There might not be a class definition, which would lead to a `NoClassDefFoundError` or `ClassNotFoundException`.
`NoClassDefFoundError` and subsequent `ClassNotFoundException`.
So, what can we do? So, what then?
Well, in good Spring fashion, you can declare another JSON object metadata field called `@identifier` to inform In this case, SBDG allows you to declare the `@identifier` JSON metadata field to inform the framework
the framework what the identifier is for the JSON object. what to use as the identifier for the object.
For example: For example:
@@ -283,26 +276,25 @@ For example:
} }
---- ----
Here the `@identifier` JSON object metadata field is informing the framework that the "isbn" field is the identifier Here, the `@identifier` JSON metadata field informs the framework that the "isbn" field is the identifier for a `Book`.
for a `Book`.
[[geode-data-working-import-conditional]] [[geode-data-using-import-conditional]]
==== Conditionally Importing Data ==== Conditionally Importing Data
While the Spring team recommends that most users should only use this feature while developing and testing their Spring While the Spring team recommends that users should only use this feature when developing and testing their Spring Boot
Boot applications with Apache Geode, a user might occasionally use this feature in production. applications with Apache Geode, a user may occasionally use this feature in production.
1 reason for using this feature in production might be to preload a (REPLICATE) Region containing "reference" data. Users might use this feature in production to preload a (REPLICATE) Region with "reference" data. Reference data is
Reference data is largely static, non-transactional by nature and infrequently changing. Preloading reference data largely static, infrequently changing and non-transactional. Preloading reference data is particularly useful in caching
is particularly useful in caching, where you want to "warm up" the cache. use cases, where you want to "warm" the cache.
When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files in
into `src/test/resources`. This ensures they will not be included in your application artifact (e.g. JAR, WAR) when `src/test/resources`. This ensures the files will not be included in your application artifact (e.g. JAR, WAR) when
deployed to production. deployed to production.
However, if you must use this feature to preload data in your production bound, Spring Boot application(s), then you However, if you must use this feature to preload data in your production environment, then you can still "conditionally"
can still "conditionally" load data from JSON. Simply configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` load data from JSON. To do so configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` property set to
property to the Spring profile(s) that must be active for the import to have any effect. the Spring profile(s) that must be active for the import to take effect.
For example: For example:
@@ -311,35 +303,27 @@ For example:
---- ----
# Spring Boot application.properties # Spring Boot application.properties
spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA, STAGING spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA
---- ----
More often the not, though, you will be using the `spring.boot.data.gemfire.cache.data.import.active-profiles` property In order for import to have an effect in this example, you must specifically set the `spring.profiles.active`
to limit data import usage to development (DEV) and QA (test) environments. When delivering to the QA team, you will
likely hand them a production candidate artifact, perhaps containing JSON files.
NOTE: Of course, there are many ways to conditionally build application artifacts. Some users might prefer to handle
this concern in their Gradle or Maven builds.
In order for the import to have an effect in this scenario, you must specifically set the `spring.profiles.active`
property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match. property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match.
In the example above, the data import will only occur if `spring.profiles.active` were set to 1 of `[DEV, QA, STAGING]` NOTE: There are many ways to conditionally build application artifacts. Some users might prefer to handle this concern
or a combination of, e.g. `[DEV, QA]`. in their Gradle or Maven builds.
[[geode-data-working-export]] [[geode-data-using-export]]
=== Exporting Data === Exporting Data
Exporting data is **disabled** by default. This is necessary since some data stored in your application's `Regions` Certain data stored in your application's `Regions` may be sensitive or confidential and keeping the data secure is of
may be sensitive in nature, and keeping the data secure and confidential is of the utmost concern and priority. the utmost concern and priority. Therefore, exporting data is **disabled** by default.
However, if you are only using this feature for development and testing purposes then you may want to enable the However, if you are using this feature for development and testing purposes then enabling the _export_ capability may be
_export_ capability. This can be useful to move data from 1 environment to another. useful to move data from 1 environment to another. For example, if your QA team finds a bug in the application using a
particular data set, then they can _export_ the data and pass it back to the development team to _import_ in their local
development environment to help debug the issue.
For example, if your QA team finds a bug in the application with a particular data set, then the QQ team can _export_ To enable _export_, set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`:
the data thereby enabling the development team to _import_ the data into their local, development environment.
To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`:
.Enable Export .Enable Export
[source,properties] [source,properties]
@@ -349,15 +333,14 @@ To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.d
spring.boot.data.gemfire.cache.data.export.enabled=true spring.boot.data.gemfire.cache.data.export.enabled=true
---- ----
SBDG is careful to _export_ the JSON in a format that Apache Geode expects on _import_ when reimporting the JSON data, SBDG is careful to _export_ data to JSON in a format that {geode-name} expects on _import_ and includes things such as
including things such as `@type` JSON object metadata fields, and so on. `@type` metadata fields.
WARNING: `@identifier` JSON object metadata fields are not automatically generated. While it is possible for POJOs WARNING: The `@identifier` metadata field is not generated automatically. While it is possible for POJOs stored in a
stored in the `Region` to include an `@identifier` metadata field, mainly because we can inspect the object's class type, `Region` to include an `@identifier` metadata field when exported to JSON it is not possible when the `Region` value
it is not possible when the `Region` value is already a `PdxInstance` that did not originate from JSON to begin with. In is a `PdxInstance` that did not originate from JSON. In this case, you must manually ensure the `PdxInstance` includes
that case, the user must manually ensure the non-JSON generated `PdxInstance` includes an `@identifier` PDX metadata an `@identifier` metadata field before it is exported to JSON if necessary (e.g. `Book.isbn`). This is only necessary
field before it is exported to JSON, if necessary (e.g. `Book.isbn`). This is only necessary if your entity classes do if your entity classes do not declare an explicit identifier field, such as with the `@Id` mapping annotation, or do
not specify an explicit identifier field (e.g. using the `@Id` Spring Data mapping annotation), or do not have an "id" not have an "id" field. This scenario can also occur when inter-operating with native clients that model the application
field, and are then serialized to PDX. This scenario might also occur when inter-operating with native clients that domain objects differently, then serialize the objects using PDX storing them in Regions on the server that are then
model the application domain objects differently, and then serialize those objects as PDX that are then stored in later consumed by your Spring Boot application.
Apache Geode on the server, which are then later consumed by your Spring Boot application client.