Spring/spring-boot
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Polish Liveness and Readiness support

Browse Source 
This commit moves the core Liveness and Readiness support to its own
`availability` package. We've made this a core concept independent of
Kubernetes.

Spring Boot now produces `LivenessStateChanged` and
`ReadinessStateChanged` events as part of the typical application
lifecycle.

Liveness and Readiness Probes (`HealthIndicator` components and health
groups) are still configured only when deployed on Kubernetes.

This commit also improves the documentation around Probes best practices
and container lifecycle considerations.

See gh-19593
This commit is contained in:
Brian Clozel
2020-03-20 23:49:28 +01:00
parent 3edc1c3a7d
commit ffdf9a422f
30 changed files with 521 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
spring-boot-project/spring-boot-docs/src/docs/asciidoc/deployment.adoc
View File

@@ -184,8 +184,27 @@ TIP: The https://github.com/pivotal-cf/java-cfenv/[Java CFEnv] project is a bett
[[cloud-deployment-kubernetes]]
=== Kubernetes
Spring Boot auto-detects Kubernetes deployment environments by checking the environment for `"*_SERVICE_HOST"` and `"*_SERVICE_PORT"` variables.
You can override this detection with the configprop:spring.main.cloud-platform[] configuration property.
Spring Boot helps you to <<spring-boot-features.adoc#boot-features-kubernetes-application-state,manage the state of your application>> and export it with <<production-ready-features.adoc#production-ready-kubernetes-probes, HTTP Kubernetes Probes using Actuator>>.
Spring Boot helps you to <<spring-boot-features.adoc#boot-features-application-availability-state,manage the state of your application>> and export it with <<production-ready-features.adoc#production-ready-kubernetes-probes, HTTP Kubernetes Probes using Actuator>>.
[[cloud-deployment-kubernetes-container-lifecycle]]
==== Kubernetes Container Lifecycle
When Kubernetes deletes an application instance, the shutdown process involves several subsystems concurrently: shutdown hooks, unregistering the service, removing the instance from the load-balancer...
Because this shutdown processing happens in parallel (and due to the nature of distributed systems), there is a window during which traffic can be routed to a pod that has also begun its shutdown processing.
You can configure a sleep execution in a pre-stop hook to avoid requests being routed to a pod that has already begun shutting down.
This sleep should be long enough for new requests to stop being routed to the pod and its duration will vary from deployment to deployment.
[source,yml,indent=0]
----
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 10"]
----
Once the pre-stop hook has completed, SIGTERM will be sent to the container and <<spring-boot-features#boot-features-graceful-shutdown,graceful shutdown>> will begin, allowing any remaining in-flight requests to complete.
[[cloud-deployment-heroku]]

104
spring-boot-project/spring-boot-docs/src/docs/asciidoc/production-ready-features.adoc
View File

@@ -875,34 +875,110 @@ TIP: You can use `@Qualifier("groupname")` if you need to register custom `Statu
Applications deployed on Kubernetes can provide information about their internal state with https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes[Container Probes].
Depending on https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/[your Kubernetes configuration], the kubelet will call those probes and react to the result.
When deployed in a Kubernetes environment, Spring Boot will manage your <<spring-boot-features.adoc#boot-features-kubernetes-application-state,Kubernetes Application State>>.
Spring Boot Actuator will gather the "Liveness" and "Readiness" information using <<production-ready-health-indicators,Health Indicators>>
Spring Boot manages your <<spring-boot-features.adoc#boot-features-application-availability-state,Application Availability State>> out-of-the-box.
If deployed in a Kubernetes environment, Actuator will gather the "Liveness" and "Readiness" information from the `ApplicationAvailabilityProvider`, create dedicated <<production-ready-health-indicators,Health Indicators>>
and expose them as HTTP Probes using <<production-ready-health-groups, Health Groups>>.
You can then point your Kubernetes `"livenessProbe"` to `"/actuator/health/liveness"` and `"readinessProbe"` to `"/actuator/health/readiness"`.
If your application context startup can be longer than the configured `"livenessProbe"` timeout, configuring a `"startupProbe"` can solve startup issues.
You can then configure your Kubernetes infrastructure with the following endpoint information:
TIP: Application should perform long-running startup tasks with `ApplicationRunners`; in this case, you don't necessarily need a `"startupProbe"` as such tasks are performed after the "Liveness" state is marked as successful.
The application won't receive traffic until tasks are finished - only then the `"readinessProbe"` will be successful.
[source,yml,indent=0]
----
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: liveness-port
failureThreshold: ...
periodSeconds: ...
As explained in the <<spring-boot-features.adoc#boot-features-kubernetes-application-state,Kubernetes Application State section>>, depending on external systems for defining the "Liveness" state might result in cascading failures.
In case your application needs to rely on a specific subset of health checks for "Liveness" or "Readiness", you should consider configuring <<production-ready-health-groups, Health Groups>> manually:
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: liveness-port
failureThreshold: ...
periodSeconds: ...
----
NOTE: If an application takes longer to start than the configured liveness period, Kubernetes mention the `"startupProbe"` as a possible solution.
The `"startupProbe"` is not necessarily needed here as the `"readinessProbe"` fails until all startup tasks are done, see <<production-ready-features.adoc#production-ready-kubernetes-probes-lifecycle,how Probes behave during the application lifecycle>>.
WARNING: If your Actuator endpoints are deployed on a separate management context, be aware that endpoints are then not using the same web infrastructure (port, connection pools, framework components) as the main application.
In this case, a Probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections).
[[production-ready-kubernetes-probes-external-state]]
==== Checking external state with Kubernetes Probes
Actuator configures the "liveness" and "readiness" Probes as Health Groups; this means that all the <<production-ready-health-groups, Health Groups features>> are available for them.
You can, for example, configure additional Health Indicators:
[source,properties,indent=0,configprops]
----
management.endpoint.health.group.readiness.include=readinessProbe,customCheck
----
WARNING: In general, "Liveness" and "Readiness" probes should avoid being based on external checks, such as <<production-ready-features.adoc#production-ready-health, external Health checks>>.
If an external system fails (e.g. a database, a Web API, an external cache), Kubernetes would react by restarting application instances or spinning up many new instances.
You should carefully consider external checks and how the platform should handle such failures.
By default, Spring Boot does not add other Health Indicators to these groups.
WARNING: If your Actuator endpoints are deployed on a separate management context, be aware that endpoints are then not using the same web infrastructure (port, connection pools, framework components) as the main application.
In this case, a Probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections).
The "liveness" Probe should not depend on health checks for external systems.
If the <<spring-boot-features.adoc#boot-features-application-availability-liveness,Liveness State of an application>> is broken, Kubernetes will try to solve that problem by restarting the application instance.
This means that if an external system fails (e.g. a database, a Web API, an external cache), Kubernetes might restart all application instances and create cascading failures.
You'll find more information in the <<deployment.adoc#cloud-deployment-kubernetes, deploying on Kubernetes>> section.
As for the "readiness" Probe, the choice of checking external systems should be made carefully by the application developers.
If the <<spring-boot-features.adoc#boot-features-application-availability-readiness,Readiness State of an application>> is unready, Kubernetes will not route traffic to that application instance.
Some external systems might not be shared by application instances or not essential to the application (the application could have circuit breakers and fallbacks).
Also, Kubernetes will react differently to applications being taken out of the load-balancer, depending on its https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[autoscaling configuration].
[[production-ready-kubernetes-probes-lifecycle]]
==== Application lifecycle and Probes states
An important aspect of the Kubernetes Probes support is its consistency with the application lifecycle.
Spring Boot publishes <<spring-boot-features.adoc#boot-features-application-events-and-listeners,Application Events during startup and shutdown>>.
When a Spring Boot application starts:
[cols="3,2,2,6"]
|===
|Application startup phase |Liveness State |Readiness State |Notes
|Starting
|broken
|unready
|Kubernetes checks the "liveness" Probe and restarts the application if it takes too long.
|Started
|live
|unready
|The application context is refreshed. The application performs startup tasks and does not receive traffic yet.
|Ready
|live
|ready
|Startup tasks are finished. The application is receiving traffic.
|===
When a Spring Boot application shuts down:
[cols="3,2,2,6"]
|===
|Application shutdown phase |Liveness State |Readiness State |Notes
|Running
|live
|ready
|Shutdown has been requested.
|Graceful shutdown
|live
|unready
|If enabled, <<spring-boot-features#boot-features-graceful-shutdown,graceful shutdown processes in-flight requests>>.
|Shutdown complete
|broken
|unready
|The application context is closed and the application cannot serve traffic.
|===
TIP: Check out the <<deployment.adoc#cloud-deployment-kubernetes-container-lifecycle,Kubernetes container lifecycle section>> for more information about Kubernetes deployment.
[[production-ready-application-info]]
=== Application Information

172
spring-boot-project/spring-boot-docs/src/docs/asciidoc/spring-boot-features.adoc
View File

@@ -191,6 +191,90 @@ NOTE: There are some restrictions when creating an `ApplicationContext` hierarch
For example, Web components *must* be contained within the child context, and the same `Environment` is used for both parent and child contexts.
See the {spring-boot-module-api}/builder/SpringApplicationBuilder.html[`SpringApplicationBuilder` Javadoc] for full details.
[[boot-features-application-availability-state]]
== Application Availability State
When deployed on plaftorms, applications can provide information about their availability to the platform using infrastructure like https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/[Kubernetes Probes].
Spring Boot manages this application state with the `ApplicationAvailabilityProvider` and makes it available to application components and the platform itself.
[[boot-features-application-availability-liveness]]
=== Liveness State
The "Liveness" state of an application tells whether its internal state allows it to work correctly, or recover by itself if it's currently failing.
An invalid "Liveness" state means that the application is in a broken state and cannot recover from it; the infrastructure should then restart the application to mitigate that problem.
NOTE: In general, the "Liveness" state should not be based on external checks, such as <<production-ready-features.adoc#production-ready-health, Health checks>>.
If it did, a failing external system (a database, a Web API, an external cache) would trigger massive restarts and cascading failures across the platform.
The internal state of Spring Boot applications is mostly represented by the Spring application context.
If the application context has started successfully, Spring Boot assumes that the application is in a valid state.
An application is considered live as soon as the context has been refreshed, see <<boot-features-application-events-and-listeners, Spring Boot application lifecycle and related Application Events>>.
[[boot-features-application-availability-readiness]]
=== Readiness State
The "Readiness" state of an application tells whether the application is ready to handle traffic.
A failing "Readiness" state tells the platform that it should not route traffic to the application for now.
This typically happens during startup, while `CommandLineRunner` and `ApplicationRunner` components are being processed, or at any time if the application decides that it's too busy for additional traffic.
An application is considered ready as soon as application and command-line runners have been called, see <<boot-features-application-events-and-listeners, Spring Boot application lifecycle and related Application Events>>.
TIP: Tasks expected to run during startup should be executed by `CommandLineRunner` and `ApplicationRunner` components instead of using Spring component lifecycle callbacks such as `@PostConstruct`.
[[boot-features-managing-application-availability]]
=== Managing the Application Availability State
Application components can retrieve the current availability state at any time, by injecting the `ApplicationAvailabilityProvider` and calling methods on it.
More often, applications will want to listen to state updates or update the state of the application.
For example, we can export the "Readiness" state of the application to a file so that a Kubernetes "exec Probe" can look at this file:
[source,java,indent=0]
----
@Component
public class ReadinessStateExporter implements ApplicationListener<LivenessStateChangedEvent> {
@Override
public void onApplicationEvent(LivenessStateChangedEvent event) {
switch (event.getReadinessState().getStatus()) {
case READY:
// create file /tmp/healthy
break;
case UNREADY:
// remove file /tmp/healthy
break;
}
}
}
----
We can also update the state of the application, when the application breaks and cannot recover:
[source,java,indent=0]
----
@Component
public class LocalCacheVerifier {
private final ApplicationEventPublisher eventPublisher;
public LocalCacheVerifier(ApplicationEventPublisher eventPublisher) {
this.eventPublisher = eventPublisher;
}
public void checkLocalCache() {
try {
//...
}
catch (CacheCompletelyBrokenException ex) {
this.eventPublisher.publishEvent(LivenessStateChangedEvent.broken(ex));
}
}
}
----
Spring Boot provides <<production-ready-features.adoc#production-ready-kubernetes-probes,Kubernetes HTTP probes for "Liveness" and "Readiness" with Actuator Health Endpoints>>.
You can get more guidance about <<deployment.adoc#cloud-deployment-kubernetes,deploying Spring Boot applications on Kubernetes in the dedicated section>>.
[[boot-features-application-events-and-listeners]]
@@ -218,8 +302,9 @@ Application events are sent in the following order, as your application runs:
. An `ApplicationContextInitializedEvent` is sent when the `ApplicationContext` is prepared and ApplicationContextInitializers have been called but before any bean definitions are loaded.
. An `ApplicationPreparedEvent` is sent just before the refresh is started but after bean definitions have been loaded.
. An `ApplicationStartedEvent` is sent after the context has been refreshed but before any application and command-line runners have been called.
. An `LivenessStateChangedEvent` is sent right after to indicate that the application is considered as live.
. An `ApplicationReadyEvent` is sent after any application and command-line runners have been called.
It indicates that the application is ready to service requests.
. An `ReadinessStateChangedEvent` is sent right after to indicate that the application is ready to service requests.
. An `ApplicationFailedEvent` is sent if there is an exception on startup.
The above list only includes ``SpringApplicationEvent``s that are tied to a `SpringApplication`.
@@ -6013,91 +6098,6 @@ For instance, it is possible to customize the name of the table for the JDBC sto
For setting the timeout of the session you can use the configprop:spring.session.timeout[] property.
If that property is not set, the auto-configuration falls back to the value of configprop:server.servlet.session.timeout[].
[[boot-features-kubernetes-application-state]]
== Kubernetes Application State
When deployed on Kubernetes, applications can provide information about their state to the platform using https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/[Kubernetes Probes].
Spring Boot manages this application state with the `ApplicationStateProvider` and makes it available to application components and the cloud platform itself.
This feature is only enabled when the application is running on Kubernetes.
TIP: The cloud platform detection relies on platform-specific environment variables, but you can override this detection with the configprop:spring.main.cloud-platform[] configuration property.
[[boot-features-kubernetes-liveness-state]]
=== Liveness State
The "Liveness" state of an application tells whether its internal state allows it to work correctly, or recover by itself if it's currently failing.
An invalid "Liveness" state means that the application is in a broken state and cannot recover from it; the infrastructure should then restart the application to mitigate that problem.
NOTE: In general, the "Liveness" state should not be based external checks, such as <<production-ready-features.adoc#production-ready-health, Health checks>>.
If it did, a failing external system (a database, a Web API, an external cache) would trigger massive restarts and cascading failures across the platform.
The internal state of Spring Boot applications is mostly represented by the Spring application context.
If the application context has started successfully, Spring Boot assumes that the application is in a valid state.
An application is considered live as soon as the `ApplicationStartedEvent` has been published, see <<boot-features-application-events-and-listeners, Spring Boot application lifecycle and related Application Events>>.
[[boot-features-kubernetes-readiness-state]]
=== Readiness State
The "Readiness" state of an application tells whether the application is ready to handle traffic.
A failing "Readiness" state tells Kubernetes that it should not route traffic to the application for now.
This typically happens during startup, while `CommandLineRunner` and `ApplicationRunner` components are being processed, or at any time if the application decides that it's too busy for additional traffic.
An application is considered ready as soon as the `ApplicationReadyEvent` has been published, see <<boot-features-application-events-and-listeners, Spring Boot application lifecycle and related Application Events>>.
TIP: Tasks expected to run during startup should be executed by `CommandLineRunner` and `ApplicationRunner` components instead of using Spring component lifecycle callbacks such as `@PostConstruct`.
[[boot-features-kubernetes-managing-state]]
=== Managing Kubernetes Application State
Application components can retrieve the current application state at any time, by injecting the `ApplicationStateProvider` and calling methods on it.
More often, applications will want to listen to state updates or update the state of the application.
For example, we can export the "Readiness" state of the application to a file so that a Kubernetes "exec Probe" can look at this file:
[source,java,indent=0]
----
@Component
public class LivenessStateExporter implements ApplicationListener<LivenessStateChangedEvent> {
@Override
public void onApplicationEvent(LivenessStateChangedEvent event) {
switch (event.getLivenessState().getStatus()) {
case LIVE:
// create file /tmp/healthy
break;
case BROKEN:
// remove file /tmp/healthy
break;
}
}
}
----
We can also update the state of the application, when the application breaks and cannot recover:
[source,java,indent=0]
----
@Component
public class LocalCacheVerifier {
private final ApplicationEventPublisher eventPublisher;
public LocalCacheVerifier(ApplicationEventPublisher eventPublisher) {
this.eventPublisher = eventPublisher;
}
public void checkLocalCache() {
try {
//...
}
catch (CacheCompletelyBroken ex) {
this.eventPublisher.publishEvent(LivenessStateChangedEvent.broken(ex));
}
}
}
----
Spring Boot provides <<production-ready-features.adoc#production-ready-kubernetes-probes,Kubernetes HTTP probes for "Liveness" and "Readiness" with Actuator Health Endpoints>>.
You can get more guidance about <<deployment.adoc#cloud-deployment-kubernetes,deploying Spring Boot applications on Kubernetes in the dedicated section>>.
[[boot-features-jmx]]
== Monitoring and Management over JMX