From adbbda54f4bfb50418f4e991eb3eac39142e9f4d Mon Sep 17 00:00:00 2001 From: John Blum Date: Fri, 24 Jul 2020 11:50:19 -0700 Subject: [PATCH] Edit the 'Working with Data' chapter. Change title to 'Using Data'. Revise the reference documentation to be more concise. --- .../src/docs/asciidoc/_includes/data.adoc | 249 ++++++++---------- 1 file changed, 116 insertions(+), 133 deletions(-) diff --git a/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc b/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc index 1e84f67d..8526cef9 100644 --- a/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc +++ b/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc @@ -1,75 +1,71 @@ -[[geode-data-working]] -== Working with Data +[[geode-data-using]] +== Using Data +:geode-name: Apache Geode One of the most important tasks during development is ensuring your Spring Boot application handles data correctly. -In order to verify the integrity, accuracy and availability of your data, your application needs data to work with. +In order to verify the accuracy, integrity and availability of your data, your application needs data to work with. -For those of you already aware of Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[initializing a database], -and specifically, a SQL-based `DataSource`, by creating a `schema.sql` file containing SQL DDL statements and subsequently -populating it with a `data.sql` file containing SQL DML statements, then the approach presented here for Apache Geode -should be familiar to you. +For those already familiar with Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[SQL database initialization], +the approach when using {geode-name} should be easy to understand. -Of course, Apache Geode already has built-in support that is similar in function to Spring Boot's support for SQL -database initialization, namely with: +{geode-name} provides built-in support, similar in function to Spring Boot's SQL database initialization, by using: -* {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] & {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] +* _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export] data commands. * {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service] -* And, by using _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export data commands]. +* {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] with {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] -Therefore, you could move, or {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore] -persistent `DiskStore` files from 1 cluster to another. +For example, by enabling Persistence with Disk Storage, you could {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore] +persistent `DiskStore` files from one cluster to another. -Alternatively, you could use Apache Geode's _Snapshot Service_ to export and import data stored in `Regions` -of the cache. This affords you the ability to filter the data while its being imported and exported. +Alternatively, using {geode-name}'s _Snapshot Service_, you can export data contained in targeted `Regions` from one +cluster during shutdown and import the data into another cluster on startup. The _Snapshot Service_ allows you to filter +data while its being imported and exported. + +Finally, GemFire/Geode Shell (_Gfsh_) commands can be used to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data] +and {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data]. TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence] -as well as the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service]. +and the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service]. -Finally, the GemFire/Geode Shell tool (_Gfsh_) allows users to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data] -or {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data] using the appropriate -command. +In all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are in a +proprietary, binary format. -However, in all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are -all in a proprietary, binary format. Plus, none of these approaches are as convenient as Spring Boot's database -initialization automation. Therefore, Spring Boot for Apache Geode (SBDG) now offers support to import data from JSON -into Apache Geode, stored as PDX bytes. +Furthermore, none of these approaches are as convenient as Spring Boot's database initialization automation. Therefore, +Spring Boot for Apache Geode (SBDG) offers support to import data from JSON into {geode-name} as PDX. -Of course, depending on the source of the data, just importing data is not much good unless you can also export data -as well. SBDG supports both _data export_ and _import_, in JSON format, out-of-the-box. +Unlike Spring Boot, SBDG offers support to export data as well. Data is imported and exported in JSON format, by default. NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures (i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions` -using your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] -or indirectly from Spring's and JCache's (JSR-107) {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations] -(e.g. such as with Spring's `@Cacheable`). +from your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] or indirectly +from Spring and JSR-107, JCache {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations]. -TIP: Also see SBDG's <> on the same. +TIP: Refer to SBDG's <> on the same. -WARNING: While this feature has utility and many edge cases have been thought through and tested thoroughly, there are -still some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82] +WARNING: While this feature has utility and many edge cases were thought through and tested thoroughly, there are still +some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82] and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team strongly recommends that this feature only be used for development and testing purposes. -Let's being with _import_. - -[[geode-data-working-import]] +[[geode-data-using-import]] === Importing Data -It is easy to import data into a cache `Region` by defining a JSON file containing the JSON object(s) you wish to load. -The JSON file should follow the naming convention below and be put in the root of your application classpath : +You can import data into a `Region` by defining a JSON file containing the JSON object(s) you wish to load. The JSON +file must follow the naming convention below and be placed in the root of your application classpath: `data-.json` -NOTE: This would be the "name" of the `Region` as defined by +NOTE: `` refers to the lowercase "name" of the `Region` as defined by {apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()]. -For example, if you have a `Region` called "_Orders_", then you would create a JSON file called, `data-orders.json`. -You would place this `data-orders.json` file into the root of your application classpath (e.g. in `src/test/resources`). +For example, if you have a `Region` named "_Orders_", then you would create a JSON file called `data-orders.json` +and place it in the root of your application classpath (e.g. in `src/test/resources`). -You can create JSON files for each `Region` implicitly defined (e.g. using `@EnableEntityDefinedRegions`) or explicitly -defined (i.e. defining a `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration. +Create JSON files for each `Region` implicitly defined (e.g. by using `@EnableEntityDefinedRegions`) or explicitly +defined (i.e. with `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration that you +want to load with data. -The JSON file could contain JSON data similar to the following: +The JSON file containing JSON data for _Orders_ might appear as follows: .`data-orders.json` [source,json] @@ -117,7 +113,7 @@ The JSON file could contain JSON data similar to the following: }] ---- -The actual application entity classes for this JSON data might look something like: +The application entity classes matching the JSON data might look something like: .Point-of-Sale (POS) Application Model Classes [source,java] @@ -149,20 +145,21 @@ class Product { } ---- -As you can see, the object model and corresponding JSON can be arbitrarily complex, encapsulating a hierarchy of objects -with complex types. +As seen above, the object model and corresponding JSON can be arbitrarily complex with a hierarchy of objects +having complex types. -[[geode-data-working-import-metadata]] -==== JSON object metadata +[[geode-data-using-import-metadata]] +==== JSON metadata -You will notice a few other details contained in the object model and JSON shown above as well. +You will notice a few other details contained in the object model and JSON shown above. -[[geode-data-working-import-metadata-attype]] -===== `@type` metadata field +[[geode-data-using-import-metadata-attype]] +===== The `@type` metadata field -First, is the use of the `@type` JSON object metadata field. This field does not map to any specific field/property on -the application domain model class (e.g. `PurchaseOrder`). Rather it serves to tell the framework and/or Apache Geode's -JSON subsystem what type of object this JSON data would map to if you were to call (e.g. `PdxInstance.getObject()`). +First, we declared an `@type` JSON metadata field. This field does not map to any specific field or property of +the application domain model class (e.g. `PurchaseOrder`). Rather, it tells the framework and {geode-name}'s JSON/PDX +converter the type of object the JSON data would map to if you were to request an object (i.e. by calling +`PdxInstance.getObject()`). For example: @@ -186,29 +183,28 @@ class OrdersRepository { } ---- -Basically, the `@type` JSON object metadata field informs the `PdxInstance.getObject()` method about the type of object -(POJO) the JSON object will map to. Otherwise, `PdxInstance.getObject()` would simply return a `PdxInstance`. +Basically, the `@type` JSON metadata field informs the `PdxInstance.getObject()` method about the type of Java object +the JSON object will map to. Otherwise, the `PdxInstance.getObject()` method would silently return a `PdxInstance`. -It is possible that Apache Geode's PDX subsystem might return an actual `PurchaseOrder` from `Region.get(key)`, but that -all depends on the configuration of the PDX `read-serialized` cache-level configuration setting among other factors. +It is possible for {geode-name}'s PDX serialization framework to return a `PurchaseOrder` from `Region.get(key)` as well, +but it depends on the value of PDX's `read-serialized`, cache-level configuration setting, among other factors. -NOTE: When JSON is imported into a GemFire/Geode cache `Region`, then the [PdxInstance.getClassName()] is not actually -a valid Java class, it is {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[JSONFormatter.JSON_CLASSNAME]. -As such, `Region` "read" data access operations (e.g. `Region.get(key)`) result in returning a `PdxInstance` and not -a Java object. +NOTE: When JSON is imported into a `Region` as PDX, the {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[PdxInstance.getClassName()] +does not refer to a valid Java class. It is {apache-geode-javadoc}/org/apache/geode/pdx/JSONFormatter.html#JSON_CLASSNAME[JSONFormatter.JSON_CLASSNAME]. +As a result, `Region` data access operations, such as `Region.get(key)`, return a `PdxInstance` and not a Java object. -TIP: You may need to additionally proxy the `Region` "read" data access operations (e.g. `Region.get(key)`) by setting -the SBDG property `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`, which proxies the `Region` -to wrap the Apache Geode `PdxInstance` in a SBDG `PdxInstanceWrapper` to appropriate handle the `PdxInstance.getObject()` -call in your application code. +TIP: You may need to proxy `Region` "read" data access operations (e.g. `Region.get(key)`) by setting the SBDG property +`spring.boot.data.gemfire.cache.region.advice.enabled` to `true`. When this property is set, `Regions` are proxied to +wrap a `PdxInstance` in a `PdxInstanceWrapper` in order to appropriately handle the `PdxInstance.getObject()` call in +your application code. -[[geode-data-working-import-metadata-id]] -===== `id` field +[[geode-data-using-import-metadata-id]] +===== The `id` field & `@identifier` metadata field The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the -object's (or `PdxInstance`) identity, or "key" when stored in the `Region` (e.g.. `Region.put(key, object)`). +object's (or `PdxInstance`'s) identity and "key" when stored in the `Region` (e.g. `Region.put(key, object)`). -You will have noticed the the JSON for the orders above declared an "id" field as the identifier: +You will have noticed the the JSON for the _Orders_ above declared an "id" field as the identifier: .PurchaseOrder identifier ("id") [source,text] @@ -219,18 +215,17 @@ You will have noticed the the JSON for the orders above declared an "id" field a ... ---- -This follows the convention used by Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO field -or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no POJO -field or property is annotated with `@Id`, then the framework falls back to searching for a POJO field or property -named "id". +This follows the same convention used in Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO +field or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no +field or property is annotated with `@Id`, then the framework falls back to searching for a field or property named "id". -In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named POJO field/property is used as the identifier, -or key for the object when storing it into a cache `Region`. +In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named field or property is used as the identifier, +and as the key for the object when storing it into a `Region`. -However, what happens when an object, or entity does not have a surrogate id or key defined? Perhaps, the application -domain model class is appropriately and simply using "natural" identifiers. +However, what happens when an object, or entity does not have a surrogate id defined? Perhaps the application domain +model class is appropriately and simply using "natural" identifiers, which is quite common in practice. -Consider a `Book` class, which might be defined as: +Consider a `Book` class defined as follows: .Book class [source,java] @@ -250,20 +245,18 @@ class Book { } ---- -As hinted at in the `Book` class above, the identifier of a `Book` is the `ISBN` given the `isbn` field was annotated -with Spring Data's `@Id` mapping annotation. Except we cannot know this by searching for an `@Id` annotation in -the JSON data. +As declared in the `Book` class above, the identifier for `Book` is its `ISBN` since the `isbn` field was annotated with +Spring Data's `@Id` mapping annotation. However, we cannot know this by searching for an `@Id` annotation in JSON. -You might be tempted to argue that if the `@type` metadata field is set, then we would know the class type. We could -then load the class and inspect the class definition to learn about the identifier. That is all fine until the class -is not actually on the system/application classpath in the first place, hence the reason SBDG's JSON support serializes -the JSON data to Apache Geode's PDX format. Therefore, there might not be a class definition, which would lead to a -`NoClassDefFoundError` and subsequent `ClassNotFoundException`. +You might be tempted to argue that if the `@type` metadata field is set, we would know the class type and could load +the class definition to learn about the identifier. That is all fine until the class is not actually on the application +classpath in the first place. This is one of the reasons why SBDG's JSON support serializes JSON to {geode-name}'s PDX +format. There might not be a class definition, which would lead to a `NoClassDefFoundError` or `ClassNotFoundException`. -So, what can we do? +So, what then? -Well, in good Spring fashion, you can declare another JSON object metadata field called `@identifier` to inform -the framework what the identifier is for the JSON object. +In this case, SBDG allows you to declare the `@identifier` JSON metadata field to inform the framework +what to use as the identifier for the object. For example: @@ -283,26 +276,25 @@ For example: } ---- -Here the `@identifier` JSON object metadata field is informing the framework that the "isbn" field is the identifier -for a `Book`. +Here, the `@identifier` JSON metadata field informs the framework that the "isbn" field is the identifier for a `Book`. -[[geode-data-working-import-conditional]] +[[geode-data-using-import-conditional]] ==== Conditionally Importing Data -While the Spring team recommends that most users should only use this feature while developing and testing their Spring -Boot applications with Apache Geode, a user might occasionally use this feature in production. +While the Spring team recommends that users should only use this feature when developing and testing their Spring Boot +applications with Apache Geode, a user may occasionally use this feature in production. -1 reason for using this feature in production might be to preload a (REPLICATE) Region containing "reference" data. -Reference data is largely static, non-transactional by nature and infrequently changing. Preloading reference data -is particularly useful in caching, where you want to "warm up" the cache. +Users might use this feature in production to preload a (REPLICATE) Region with "reference" data. Reference data is +largely static, infrequently changing and non-transactional. Preloading reference data is particularly useful in caching +use cases, where you want to "warm" the cache. -When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files -into `src/test/resources`. This ensures they will not be included in your application artifact (e.g. JAR, WAR) when +When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files in +`src/test/resources`. This ensures the files will not be included in your application artifact (e.g. JAR, WAR) when deployed to production. -However, if you must use this feature to preload data in your production bound, Spring Boot application(s), then you -can still "conditionally" load data from JSON. Simply configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` -property to the Spring profile(s) that must be active for the import to have any effect. +However, if you must use this feature to preload data in your production environment, then you can still "conditionally" +load data from JSON. To do so configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` property set to +the Spring profile(s) that must be active for the import to take effect. For example: @@ -311,35 +303,27 @@ For example: ---- # Spring Boot application.properties -spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA, STAGING +spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA ---- -More often the not, though, you will be using the `spring.boot.data.gemfire.cache.data.import.active-profiles` property -to limit data import usage to development (DEV) and QA (test) environments. When delivering to the QA team, you will -likely hand them a production candidate artifact, perhaps containing JSON files. - -NOTE: Of course, there are many ways to conditionally build application artifacts. Some users might prefer to handle -this concern in their Gradle or Maven builds. - -In order for the import to have an effect in this scenario, you must specifically set the `spring.profiles.active` +In order for import to have an effect in this example, you must specifically set the `spring.profiles.active` property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match. -In the example above, the data import will only occur if `spring.profiles.active` were set to 1 of `[DEV, QA, STAGING]` -or a combination of, e.g. `[DEV, QA]`. +NOTE: There are many ways to conditionally build application artifacts. Some users might prefer to handle this concern +in their Gradle or Maven builds. -[[geode-data-working-export]] +[[geode-data-using-export]] === Exporting Data -Exporting data is **disabled** by default. This is necessary since some data stored in your application's `Regions` -may be sensitive in nature, and keeping the data secure and confidential is of the utmost concern and priority. +Certain data stored in your application's `Regions` may be sensitive or confidential and keeping the data secure is of +the utmost concern and priority. Therefore, exporting data is **disabled** by default. -However, if you are only using this feature for development and testing purposes then you may want to enable the -_export_ capability. This can be useful to move data from 1 environment to another. +However, if you are using this feature for development and testing purposes then enabling the _export_ capability may be +useful to move data from 1 environment to another. For example, if your QA team finds a bug in the application using a +particular data set, then they can _export_ the data and pass it back to the development team to _import_ in their local +development environment to help debug the issue. -For example, if your QA team finds a bug in the application with a particular data set, then the QQ team can _export_ -the data thereby enabling the development team to _import_ the data into their local, development environment. - -To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`: +To enable _export_, set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`: .Enable Export [source,properties] @@ -349,15 +333,14 @@ To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.d spring.boot.data.gemfire.cache.data.export.enabled=true ---- -SBDG is careful to _export_ the JSON in a format that Apache Geode expects on _import_ when reimporting the JSON data, -including things such as `@type` JSON object metadata fields, and so on. +SBDG is careful to _export_ data to JSON in a format that {geode-name} expects on _import_ and includes things such as +`@type` metadata fields. -WARNING: `@identifier` JSON object metadata fields are not automatically generated. While it is possible for POJOs -stored in the `Region` to include an `@identifier` metadata field, mainly because we can inspect the object's class type, -it is not possible when the `Region` value is already a `PdxInstance` that did not originate from JSON to begin with. In -that case, the user must manually ensure the non-JSON generated `PdxInstance` includes an `@identifier` PDX metadata -field before it is exported to JSON, if necessary (e.g. `Book.isbn`). This is only necessary if your entity classes do -not specify an explicit identifier field (e.g. using the `@Id` Spring Data mapping annotation), or do not have an "id" -field, and are then serialized to PDX. This scenario might also occur when inter-operating with native clients that -model the application domain objects differently, and then serialize those objects as PDX that are then stored in -Apache Geode on the server, which are then later consumed by your Spring Boot application client. +WARNING: The `@identifier` metadata field is not generated automatically. While it is possible for POJOs stored in a +`Region` to include an `@identifier` metadata field when exported to JSON it is not possible when the `Region` value +is a `PdxInstance` that did not originate from JSON. In this case, you must manually ensure the `PdxInstance` includes +an `@identifier` metadata field before it is exported to JSON if necessary (e.g. `Book.isbn`). This is only necessary +if your entity classes do not declare an explicit identifier field, such as with the `@Id` mapping annotation, or do +not have an "id" field. This scenario can also occur when inter-operating with native clients that model the application +domain objects differently, then serialize the objects using PDX storing them in Regions on the server that are then +later consumed by your Spring Boot application.