diff --git a/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc b/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc new file mode 100644 index 00000000..1e84f67d --- /dev/null +++ b/spring-geode-docs/src/docs/asciidoc/_includes/data.adoc @@ -0,0 +1,363 @@ +[[geode-data-working]] +== Working with Data + +One of the most important tasks during development is ensuring your Spring Boot application handles data correctly. +In order to verify the integrity, accuracy and availability of your data, your application needs data to work with. + +For those of you already aware of Spring Boot's support for {spring-boot-docs-html}/howto.html#howto-initialize-a-database-using-spring-jdbc[initializing a database], +and specifically, a SQL-based `DataSource`, by creating a `schema.sql` file containing SQL DDL statements and subsequently +populating it with a `data.sql` file containing SQL DML statements, then the approach presented here for Apache Geode +should be familiar to you. + +Of course, Apache Geode already has built-in support that is similar in function to Spring Boot's support for SQL +database initialization, namely with: + +* {apache-geode-docs}/managing/disk_storage/chapter_overview.html[Disk Storage] & {apache-geode-docs}/developing/storing_data_on_disk/chapter_overview.html[Persistence] +* {apache-geode-docs}/managing/cache_snapshots/chapter_overview.html[Snapshot Service] +* And, by using _Gfsh's_ {apache-geode-docs}/tools_modules/gfsh/quick_ref_commands_by_area.html#topic_C7DB8A800D6244AE8FF3ADDCF139DCE4[import/export data commands]. + +Therefore, you could move, or {apache-geode-docs}/managing/disk_storage/backup_restore_disk_store.html[backup and restore] +persistent `DiskStore` files from 1 cluster to another. + +Alternatively, you could use Apache Geode's _Snapshot Service_ to export and import data stored in `Regions` +of the cache. This affords you the ability to filter the data while its being imported and exported. + +TIP: Spring Data for Apache Geode (SDG) contains dedicated support for {spring-data-geode-docs-html}/#bootstrap:region:persistence[Persistence] +as well as the {spring-data-geode-docs-html}/#bootstrap:snapshot[Snapshot Service]. + +Finally, the GemFire/Geode Shell tool (_Gfsh_) allows users to {spring-data-geode-docs-html}/tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734[export data] +or {apache-geode-docs}/tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l[import data] using the appropriate +command. + +However, in all cases, the files generated by _persistence_, the _Snapshot Service_ and _Gfsh's_ `export` command are +all in a proprietary, binary format. Plus, none of these approaches are as convenient as Spring Boot's database +initialization automation. Therefore, Spring Boot for Apache Geode (SBDG) now offers support to import data from JSON +into Apache Geode, stored as PDX bytes. + +Of course, depending on the source of the data, just importing data is not much good unless you can also export data +as well. SBDG supports both _data export_ and _import_, in JSON format, out-of-the-box. + +NOTE: SBDG does not provide an equivalent to Spring Boot's `schema.sql` file. The best way to define the data structures +(i.e. `Regions`) managing your data is with SDG's Annotation-based configuration support for defining cache `Regions` +using your application's {spring-data-geode-docs-html}/#bootstrap-annotation-config-regions[entity classes] +or indirectly from Spring's and JCache's (JSR-107) {spring-data-geode-docs-html}/#bootstrap-annotation-config-caching[caching annotations] +(e.g. such as with Spring's `@Cacheable`). + +TIP: Also see SBDG's <> on the same. + +WARNING: While this feature has utility and many edge cases have been thought through and tested thoroughly, there are +still some limitations that need to be ironed out. See https://github.com/spring-projects/spring-boot-data-geode/issues/82[Issue-82] +and https://github.com/spring-projects/spring-boot-data-geode/issues/83[Issue-83] for more details. The Spring team +strongly recommends that this feature only be used for development and testing purposes. + +Let's being with _import_. + +[[geode-data-working-import]] +=== Importing Data + +It is easy to import data into a cache `Region` by defining a JSON file containing the JSON object(s) you wish to load. +The JSON file should follow the naming convention below and be put in the root of your application classpath : + +`data-.json` + +NOTE: This would be the "name" of the `Region` as defined by +{apache-geode-javadoc}/org/apache/geode/cache/Region.html#getName--[Region.getName()]. + +For example, if you have a `Region` called "_Orders_", then you would create a JSON file called, `data-orders.json`. +You would place this `data-orders.json` file into the root of your application classpath (e.g. in `src/test/resources`). + +You can create JSON files for each `Region` implicitly defined (e.g. using `@EnableEntityDefinedRegions`) or explicitly +defined (i.e. defining a `ClientRegionFactoryBean` in _JavaConfig_) in your Spring Boot application configuration. + +The JSON file could contain JSON data similar to the following: + +.`data-orders.json` +[source,json] +---- +[{ + "@type": "example.app.pos.model.PurchaseOrder", + "id": 1, + "lineItems": [ + { + "@type": "example.app.pos.model.LineItem", + "product": { + "@type": "example.app.pos.model.Product", + "name": "Apple iPad Pro", + "price": 1499.00, + "category": "SHOPPING" + }, + "quantity": 1 + }, + { + "@type": "example.app.pos.model.LineItem", + "product": { + "@type": "example.app.pos.model.Product", + "name": "Apple iPhone 11 Pro Max", + "price": 1249.00, + "category": "SHOPPING" + }, + "quantity": 2 + } + ] +}, { + "@type": "example.app.pos.model.PurchaseOrder", + "id": 2, + "lineItems": [ + { + "@type": "example.app.pos.model.LineItem", + "product": { + "@type": "example.app.pos.model.Product", + "name": "Starbucks Vente Carmel Macchiato", + "price": 5.49, + "category": "SHOPPING" + }, + "quantity": 1 + } + ] +}] +---- + +The actual application entity classes for this JSON data might look something like: + +.Point-of-Sale (POS) Application Model Classes +[source,java] +---- +@Region("Orders") +class PurchaseOrder { + + @Id + Long id; + + List lineItems; + +} + +class LineItem { + + Product product; + Integer quantity; + +} + +@Region("Products") +class Product { + + String name; + Category category; + BigDecimal price; + +} +---- + +As you can see, the object model and corresponding JSON can be arbitrarily complex, encapsulating a hierarchy of objects +with complex types. + +[[geode-data-working-import-metadata]] +==== JSON object metadata + +You will notice a few other details contained in the object model and JSON shown above as well. + +[[geode-data-working-import-metadata-attype]] +===== `@type` metadata field + +First, is the use of the `@type` JSON object metadata field. This field does not map to any specific field/property on +the application domain model class (e.g. `PurchaseOrder`). Rather it serves to tell the framework and/or Apache Geode's +JSON subsystem what type of object this JSON data would map to if you were to call (e.g. `PdxInstance.getObject()`). + +For example: + +.Deserializing PDX as an Object +[source,java] +---- +@Repository +class OrdersRepository { + + @Resource(name = "Orders") + Region orders; + + PurchaseOrder findBy(Long id) { + + Object value = this.orders.get(id); + + return value instanceof PurchaseOrder ? (PurchaseOrder) value + : value instanceof PdxInstance ? ((PdxInstance) value).getObject() + : null; + } +} +---- + +Basically, the `@type` JSON object metadata field informs the `PdxInstance.getObject()` method about the type of object +(POJO) the JSON object will map to. Otherwise, `PdxInstance.getObject()` would simply return a `PdxInstance`. + +It is possible that Apache Geode's PDX subsystem might return an actual `PurchaseOrder` from `Region.get(key)`, but that +all depends on the configuration of the PDX `read-serialized` cache-level configuration setting among other factors. + +NOTE: When JSON is imported into a GemFire/Geode cache `Region`, then the [PdxInstance.getClassName()] is not actually +a valid Java class, it is {apache-geode-javadoc}/org/apache/geode/pdx/PdxInstance.html#getClassName--[JSONFormatter.JSON_CLASSNAME]. +As such, `Region` "read" data access operations (e.g. `Region.get(key)`) result in returning a `PdxInstance` and not +a Java object. + +TIP: You may need to additionally proxy the `Region` "read" data access operations (e.g. `Region.get(key)`) by setting +the SBDG property `spring.boot.data.gemfire.cache.region.advice.enabled` to `true`, which proxies the `Region` +to wrap the Apache Geode `PdxInstance` in a SBDG `PdxInstanceWrapper` to appropriate handle the `PdxInstance.getObject()` +call in your application code. + +[[geode-data-working-import-metadata-id]] +===== `id` field + +The top-level objects in your JSON must have an identifier, such as an "id" field. This identifier is used as the +object's (or `PdxInstance`) identity, or "key" when stored in the `Region` (e.g.. `Region.put(key, object)`). + +You will have noticed the the JSON for the orders above declared an "id" field as the identifier: + +.PurchaseOrder identifier ("id") +[source,text] +---- +[{ + "@type": "example.app.pos.model.PurchaseOrder", + "id": 1, + ... +---- + +This follows the convention used by Spring Data. Typically, Spring Data mapping infrastructure looks for a POJO field +or property annotated with {spring-data-commons-javadoc}/org/springframework/data/annotation/Id.html[@Id]. If no POJO +field or property is annotated with `@Id`, then the framework falls back to searching for a POJO field or property +named "id". + +In Spring Data for Apache Geode (SDG), this `@Id` annotated, or "id" named POJO field/property is used as the identifier, +or key for the object when storing it into a cache `Region`. + +However, what happens when an object, or entity does not have a surrogate id or key defined? Perhaps, the application +domain model class is appropriately and simply using "natural" identifiers. + +Consider a `Book` class, which might be defined as: + +.Book class +[source,java] +---- +@Region("Books") +class Book { + + Author author; + + @Id + ISBN isbn; + + LocalDate publishedDate; + + Sring title; + +} +---- + +As hinted at in the `Book` class above, the identifier of a `Book` is the `ISBN` given the `isbn` field was annotated +with Spring Data's `@Id` mapping annotation. Except we cannot know this by searching for an `@Id` annotation in +the JSON data. + +You might be tempted to argue that if the `@type` metadata field is set, then we would know the class type. We could +then load the class and inspect the class definition to learn about the identifier. That is all fine until the class +is not actually on the system/application classpath in the first place, hence the reason SBDG's JSON support serializes +the JSON data to Apache Geode's PDX format. Therefore, there might not be a class definition, which would lead to a +`NoClassDefFoundError` and subsequent `ClassNotFoundException`. + +So, what can we do? + +Well, in good Spring fashion, you can declare another JSON object metadata field called `@identifier` to inform +the framework what the identifier is for the JSON object. + +For example: + +.Using "@identifer" +[source,json] +---- +{ + "@type": "example.app.books.model.Book", + "@identifier": "isbn", + "author": { + "id": 1, + "name": "Josh Long" + }, + "isbn": "978-1-449-374640-8", + "publishedDate": "2017-08-01", + "title": "Cloud Native Java" +} +---- + +Here the `@identifier` JSON object metadata field is informing the framework that the "isbn" field is the identifier +for a `Book`. + +[[geode-data-working-import-conditional]] +==== Conditionally Importing Data + +While the Spring team recommends that most users should only use this feature while developing and testing their Spring +Boot applications with Apache Geode, a user might occasionally use this feature in production. + +1 reason for using this feature in production might be to preload a (REPLICATE) Region containing "reference" data. +Reference data is largely static, non-transactional by nature and infrequently changing. Preloading reference data +is particularly useful in caching, where you want to "warm up" the cache. + +When using this feature for development and testing purposes, you can simply put your `Region` specific JSON files +into `src/test/resources`. This ensures they will not be included in your application artifact (e.g. JAR, WAR) when +deployed to production. + +However, if you must use this feature to preload data in your production bound, Spring Boot application(s), then you +can still "conditionally" load data from JSON. Simply configure the `spring.boot.data.gemfire.cache.data.import.active-profiles` +property to the Spring profile(s) that must be active for the import to have any effect. + +For example: + +.Conditional Importing JSON +[source,properties] +---- +# Spring Boot application.properties + +spring.boot.data.gemfire.cache.data.import.active-profiles=DEV, QA, STAGING +---- + +More often the not, though, you will be using the `spring.boot.data.gemfire.cache.data.import.active-profiles` property +to limit data import usage to development (DEV) and QA (test) environments. When delivering to the QA team, you will +likely hand them a production candidate artifact, perhaps containing JSON files. + +NOTE: Of course, there are many ways to conditionally build application artifacts. Some users might prefer to handle +this concern in their Gradle or Maven builds. + +In order for the import to have an effect in this scenario, you must specifically set the `spring.profiles.active` +property to 1 of the valid, "_active-profiles_" listed in the import property (e.g. `QA`). Only 1 needs to match. + +In the example above, the data import will only occur if `spring.profiles.active` were set to 1 of `[DEV, QA, STAGING]` +or a combination of, e.g. `[DEV, QA]`. + +[[geode-data-working-export]] +=== Exporting Data + +Exporting data is **disabled** by default. This is necessary since some data stored in your application's `Regions` +may be sensitive in nature, and keeping the data secure and confidential is of the utmost concern and priority. + +However, if you are only using this feature for development and testing purposes then you may want to enable the +_export_ capability. This can be useful to move data from 1 environment to another. + +For example, if your QA team finds a bug in the application with a particular data set, then the QQ team can _export_ +the data thereby enabling the development team to _import_ the data into their local, development environment. + +To enable the _export_ feature, simply set the `spring.boot.data.gemfire.cache.data.export.enabled` property to `true`: + +.Enable Export +[source,properties] +---- +# Spring Boot application.properties + +spring.boot.data.gemfire.cache.data.export.enabled=true +---- + +SBDG is careful to _export_ the JSON in a format that Apache Geode expects on _import_ when reimporting the JSON data, +including things such as `@type` JSON object metadata fields, and so on. + +WARNING: `@identifier` JSON object metadata fields are not automatically generated. While it is possible for POJOs +stored in the `Region` to include an `@identifier` metadata field, mainly because we can inspect the object's class type, +it is not possible when the `Region` value is already a `PdxInstance` that did not originate from JSON to begin with. In +that case, the user must manually ensure the non-JSON generated `PdxInstance` includes an `@identifier` PDX metadata +field before it is exported to JSON, if necessary (e.g. `Book.isbn`). This is only necessary if your entity classes do +not specify an explicit identifier field (e.g. using the `@Id` Spring Data mapping annotation), or do not have an "id" +field, and are then serialized to PDX. This scenario might also occur when inter-operating with native clients that +model the application domain objects differently, and then serialize those objects as PDX that are then stored in +Apache Geode on the server, which are then later consumed by your Spring Boot application client. diff --git a/spring-geode-docs/src/docs/asciidoc/index.adoc b/spring-geode-docs/src/docs/asciidoc/index.adoc index b8b7b48f..235d1b59 100644 --- a/spring-geode-docs/src/docs/asciidoc/index.adoc +++ b/spring-geode-docs/src/docs/asciidoc/index.adoc @@ -26,7 +26,7 @@ John Blum :pivotal-gemfire-javadoc: https://gemfire-{pivotal-gemfire-version}-javadocs.docs.pivotal.io/ :pivotal-gemfire-website: https://pivotal.io/pivotal-gemfire :spring-boot-docs: https://docs.spring.io/spring-boot/docs/current/reference -:spring-boot-docs-html: {spring-boot-docs}/htmlsingle +:spring-boot-docs-html: {spring-boot-docs}/html :spring-boot-javadoc: https://docs.spring.io/spring-boot/docs/current/api :spring-boot-website: https://spring.io/projects/spring-boot :spring-boot-data-geode-javadoc: https://docs.spring.io/autorepo/docs/spring-boot-data-geode-build/current/api/ @@ -235,6 +235,7 @@ include::{include-dir}/templates.adoc[] include::{include-dir}/repositories.adoc[] include::{include-dir}/functions.adoc[] include::{include-dir}/continuous-query.adoc[] +include::{include-dir}/data.adoc[] include::{include-dir}/data-serialization.adoc[] include::{include-dir}/security.adoc[] include::{include-dir}/logging.adoc[]