diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__introduction.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__introduction.html index 9a0bf898..7c48a0d0 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__introduction.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__introduction.html @@ -10,9 +10,11 @@ step and describe it in details.

scripts are reused by both Concourse and Jenkins pipelines.

In the concourse folder you can find all the necessary scripts and setup to run Concourse demo.

In the docs section you have the whole documentation of the project.

In the jenkins folder you can find all the necessary scripts and setup to run Jenkins demo.

1.2 How to use it?

This repository can be treated as a template for your pipeline. We provide some opinionated implementation that you can alter to suit your needs. The best approach to use it to build your production projects would be to download the Spring Cloud Pipelines repository as ZIP, then -init a Git project there and modify it as you wish.

$ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/v1.0.0.M6.zip
-$ unzip v1.0.0.M6.zip
-$ cd spring-cloud-pipelines-v1.0.0.M6
+init a Git project there and modify it as you wish.

$ # pass the branch (e.g. master) or a particular tag (e.g. v1.0.0.RELEASE)
+$ SC_PIPELINES_RELEASE=...
+$ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/${SC_PIPELINES_RELEASE}.zip
+$ unzip ${SC_PIPELINES_RELEASE}.zip
+$ cd spring-cloud-pipelines-${SC_PIPELINES_RELEASE}
 $ git init
 $ # modify the pipelines to suit your needs
 $ git add .
diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_faq.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_faq.html
index 051f02d9..c951ef98 100644
--- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_faq.html
+++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_faq.html
@@ -22,14 +22,14 @@ scripts without changing the whole pipeline.

git tag -l | xargs -n 1 git push --delete origin

10.8 I want to provide a different JDK version

  • by default we assume that you have jdk with id jdk8 configured
  • if you want a different one just override JDK_VERSION env var and point to the proper one
[Tip]Tip

The docker image comes in with Java installed at /usr/lib/jvm/java-8-openjdk-amd64. You can go to Global Tools and create a JDK with jdk8 id and JAVA_HOME pointing to /usr/lib/jvm/java-8-openjdk-amd64

To change the default one just follow these steps:

  - 

Figure 10.1. Click 'Manage Jenkins'

manage jenkins

  - 

Figure 10.2. Click 'Global Tool'

global tool

  - 

Figure 10.3. Click 'JDK Installations'

jdk installation

  - 

Figure 10.4. Fill out JDK Installation with path to your JDK

jdk

  + 

Figure 10.1. Click 'Manage Jenkins'

manage jenkins

  + 

Figure 10.2. Click 'Global Tool'

global tool

  + 

Figure 10.3. Click 'JDK Installations'

jdk installation

  + 

Figure 10.4. Fill out JDK Installation with path to your JDK

jdk

   

And that’s it!

10.9 Enable Groovy Token Macro Processing

With scripted that but if you needed to this manually then this is how to do it:

  - 

Figure 10.5. Click 'Manage Jenkins'

manage jenkins

  - 

Figure 10.6. Click 'Configure System'

configure system

  - 

Figure 10.7. Click 'Allow token macro processing'

groovy token

10.10 I want deployment to stage and prod be automatic

No problem, just set the property / env var to true

  • AUTO_DEPLOY_TO_STAGE to automatically deploy to stage
  • AUTO_DEPLOY_TO_PROD to automatically deploy to prod

10.11 I don’t want to test API compativility

No problem, just set the API_COMPATIBILITY_STEP_REQUIRED env variable + 

Figure 10.5. Click 'Manage Jenkins'

manage jenkins

  + 

Figure 10.6. Click 'Configure System'

configure system

  + 

Figure 10.7. Click 'Allow token macro processing'

groovy token

10.10 I want deployment to stage and prod be automatic

No problem, just set the property / env var to true

  • AUTO_DEPLOY_TO_STAGE to automatically deploy to stage
  • AUTO_DEPLOY_TO_PROD to automatically deploy to prod

10.11 I don’t want to test API compativility

No problem, just set the API_COMPATIBILITY_STEP_REQUIRED env variable to false and rerun the seed (you can pick it from the seed job’s properties too).

10.12 I can’t tag the repo!

When you get sth like this:

19:01:44 stderr: remote: Invalid username or password.
 19:01:44 fatal: Authentication failed for 'https://github.com/marcingrzejszczak/github-webhook/'
diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_pipeline_common.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_pipeline_common.html
index f003a9bf..0b02713b 100644
--- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_pipeline_common.html
+++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__jenkins_pipeline_common.html
@@ -59,15 +59,15 @@ of Java installation and the others to another one.

< you’d have to go to Configure of the build step and modify the Git name / email. If you want to set it globally you’ll have to remove the section from the build step and follow these steps to set it globally.

You can set Git email / user globally like this:

  - 

Figure 7.1. Click 'Manage Jenkins'

manage jenkins

  - 

Figure 7.2. Click 'Configure System'

configure system

  - 

Figure 7.3. Fill out Git user information

git

  + 

Figure 7.1. Click 'Manage Jenkins'

manage jenkins

  + 

Figure 7.2. Click 'Configure System'

configure system

  + 

Figure 7.3. Fill out Git user information

git

   

Add Jenkins credentials for GitHub

The scripts will need to access the credential in order to tag the repo.

You have to set credentials with id: git.

Below you can find instructions on how to set a credential (e.g. for Cloud Foundry cf-test credential but remember to provide the one with id git).

  - 

Figure 7.4. Click 'Credentials, System'

credentials system

  - 

Figure 7.5. Click 'Global Credentials'

credentials global

  - 

Figure 7.6. Click 'Add credentials'

credentials add

  - 

Figure 7.7. Fill out the user / password and provide the git credential ID (in this example cf-test)

credentials example

  + 

Figure 7.4. Click 'Credentials, System'

credentials system

  + 

Figure 7.5. Click 'Global Credentials'

credentials global

  + 

Figure 7.6. Click 'Add credentials'

credentials add

  + 

Figure 7.7. Fill out the user / password and provide the git credential ID (in this example cf-test)

credentials example

   

7.3 Testing Jenkins scripts

./gradlew clean build

[Warning]Warning

The ran test only checks if your scripts compile.

7.4 How to work with Jenkins Job DSL plugin

Check out the tutorial. Provide the link to this repository in your Jenkins installation.

[Warning]Warning

Remember that views can be overridden that’s why the suggestion is to contain in one script all the logic needed to build a view for a single project (check out that spring_cloud_views.groovy is building all the spring-cloud views).

7.5 Docker Image

If you would like to run the pre-configured Jenkins image somewhere other than your local machine, we diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__opinionated_implementation.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__opinionated_implementation.html index 7d754a5c..2a3e1a5f 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__opinionated_implementation.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__opinionated_implementation.html @@ -1,16 +1,16 @@ 2. Opinionated implementation

2. Opinionated implementation

For the demo purposes we’re providing Docker Compose setup with Artifactory and Concourse / Jenkins tools. -Regardless of the picked CD application for the pipeline to pass one needs either

  • [CLOUD FOUNDRY] a Cloud Foundry instance (for example Pivotal Web Services or PCF Dev)
  • [KUBERNETES] a Kubernetes cluster (for example Minikube)
  • the infrastructure applications deployed to the JAR hosting application (for the demo we’re providing Artifactory).
  • Eureka for Service Discovery
  • Stub Runner Boot for running Spring Cloud Contract stubs.
[Tip]Tip

In the demos we’re showing you how to first build the github-webhook project. That’s because +Regardless of the picked CD application for the pipeline to pass one needs either

  • a Cloud Foundry instance (for example Pivotal Web Services or PCF Dev)
  • a Kubernetes cluster (for example Minikube)
  • the infrastructure applications deployed to the JAR hosting application (for the demo we’re providing Artifactory).
  • Eureka for Service Discovery
  • Stub Runner Boot for running Spring Cloud Contract stubs.
[Tip]Tip

In the demos we’re showing you how to first build the github-webhook project. That’s because the github-analytics needs the stubs of github-webhook to pass the tests. Below you’ll find references to github-analytics project since it contains more interesting pieces as far as testing -is concerned.

2.1 Build

Figure 2.1. Build and upload artifacts

build

In this step we’re generating a version of the pipeline, next we’re - running unit, integration and contract tests. Finally we’re:

  • publishing a fat jar of the application
  • publishing a Spring Cloud Contract jar containing stubs of the application
  • [KUBERNETES] uploading a Docker image of the application

During this phase we’re executing a Maven build using Maven Wrapper or a Gradle build using Gradle Wrapper +is concerned.

2.1 Build

Figure 2.1. Build and upload artifacts

build

In this step we’re generating a version of the pipeline, next we’re + running unit, integration and contract tests. Finally we’re:

  • publishing a fat jar of the application
  • publishing a Spring Cloud Contract jar containing stubs of the application
  • for Kubernetes - uploading a Docker image of the application

During this phase we’re executing a Maven build using Maven Wrapper or a Gradle build using Gradle Wrapper , with unit and integration tests. We’re also tagging the repository with dev/${version} format. That way in each subsequent step of the pipeline we’re able to retrieve the tagged version. Also we know exactly which version of the pipeline corresponds to which Git hash.

Once the artifact got built we’re running API compatibility check.

  • we’re searching for the latest production deployment
  • we’re retrieving the contracts that were used by that deployment
  • from the contracts we’re generating API tests to see if the current implementation is fulfilling the HTTP / messaging contracts that the current production deployment -has defined (we’re checking backward compatibility of the API)

2.2 Test

Figure 2.2. Smoke test and rollback test on test environment

test

Here we’re

  • starting a RabbitMQ service in PaaS
  • deploying Eureka infrastructure application to PaaS
  • downloading the fat jar from Nexus and we’re uploading it to PaaS. We want the application +has defined (we’re checking backward compatibility of the API)

2.2 Test

Figure 2.2. Smoke test and rollback test on test environment

test

Here we’re

  • starting a RabbitMQ service in PaaS
  • deploying Eureka infrastructure application to PaaS
  • downloading the fat jar from Nexus and we’re uploading it to PaaS. We want the application to run in isolation (be surrounded by stubs).
[Tip]Tip

Currently due to port constraints in Cloud Foundry we cannot run multiple stubbed HTTP services in the cloud so to fix this issue we’re running the application with smoke Spring profile on which you can stub out all HTTP calls to return @@ -30,12 +30,12 @@ release (we’re checking out the tag), we’re downloading the appropri or Docker image for Kubernetes) and we’re uploading it to PaaS. IMPORTANT the old artifact is running against the NEW version of the database.

  • we’re running the old smoke tests against the freshly deployed application surrounded by stubs. If those tests pass then we have a high probability that the application is backwards compatible
  • the default behaviour is that after all of those steps the user can manually click to deploy the -application to a stage environment
  • 2.3 Stage

    Figure 2.3. End to end tests on stage environment

    stage

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the artifact (either JAR for Cloud Foundry or Docker image for Kubernetes) +application to a stage environment

    2.3 Stage

    Figure 2.3. End to end tests on stage environment

    stage

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the artifact (either JAR for Cloud Foundry or Docker image for Kubernetes) from and we’re uploading it to PaaS.

    Next we have a manual step in which:

    • from the checked out code we’re running the tests available under the e2e profile. In the case of GitHub Analytics application we’re sending a HTTP message to GitHub Analytic’s endpoint. Then we’re checking if the received message count has increased.

    The step is manual by default due to the fact that stage environment is often shared between teams and some preparations on databases / infrastructure have to take place before running the tests. -Ideally these step should be fully automatic.

    2.4 Prod

    Figure 2.4. Deployment to production

    prod

    The step to deploy to production is manual but ideally it should be automatic.

    [Important]Important

    This step does deployment to production. On production you would assume +Ideally these step should be fully automatic.

    2.4 Prod

    Figure 2.4. Deployment to production

    prod

    The step to deploy to production is manual but ideally it should be automatic.

    [Important]Important

    This step does deployment to production. On production you would assume that you have the infrastructure running. That’s why before you run this step you must execute a script that will provision the services on the production environment. For Cloud Foundry just call tools/cf-helper.sh setup-prod-infra and diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_cloud_foundry.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_cloud_foundry.html index d5dd2400..5bb1a657 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_cloud_foundry.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_cloud_foundry.html @@ -2,14 +2,14 @@ 12. The demo setup (Cloud Foundry)

    12. The demo setup (Cloud Foundry)

    The demo uses 2 applications. Github Webhook and Github analytics code. Below you can -see an image of how these application communicate with each other.

    Figure 12.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

      +see an image of how these application communicate with each other.

    Figure 12.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

       

    For the demo scenario we have two applications. Github Analytics and Github Webhook. Let’s imagine a case where Github is emitting events via HTTP. Github Webhook has an API that could register to such hooks and receive those messages. Once this happens Github Webhook sends a message by RabbitMQ to a channel. Github Analytics is - listening to those messages and stores them in a MySQL database.

    Figure 12.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

      + listening to those messages and stores them in a MySQL database.

    Figure 12.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

       

    Github Analytics has its KPIs (Key Performance Indicators) monitored. In the case -of that application the KPI is number of issues.

    Figure 12.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

      +of that application the KPI is number of issues.

    Figure 12.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

       

    Let’s assume that if we go below the threshold of X issues then an alert should be sent to Slack.

    12.1 Deploying production applications to PCF Dev

    In the real world scenario we wouldn’t want to automatically provision services like RabbitMQ, MySQL or Eureka each time we deploy a new application to production. Typically diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_kubernetes.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_kubernetes.html index a28d5492..73a6665a 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_kubernetes.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi__the_demo_setup_kubernetes.html @@ -2,14 +2,14 @@ 13. The demo setup (Kubernetes)

    13. The demo setup (Kubernetes)

    The demo uses 2 applications. Github Webhook and Github analytics code. Below you can -see an image of how these application communicate with each other.

    Figure 13.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

      +see an image of how these application communicate with each other.

    Figure 13.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

       

    For the demo scenario we have two applications. Github Analytics and Github Webhook. Let’s imagine a case where Github is emitting events via HTTP. Github Webhook has an API that could register to such hooks and receive those messages. Once this happens Github Webhook sends a message by RabbitMQ to a channel. Github Analytics is - listening to those messages and stores them in a MySQL database.

    Figure 13.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

      + listening to those messages and stores them in a MySQL database.

    Figure 13.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

       

    Github Analytics has its KPIs (Key Performance Indicators) monitored. In the case -of that application the KPI is number of issues.

    Figure 13.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

      +of that application the KPI is number of issues.

    Figure 13.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

       

    Let’s assume that if we go below the threshold of X issues then an alert should be sent to Slack.

    13.1 Deploying production applications to Minikube

    In the real world scenario we wouldn’t want to automatically provision services like RabbitMQ, MySQL or Eureka each time we deploy a new application to production. Typically diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-cf.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-cf.html index 0017f6a9..af082d78 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-cf.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-cf.html @@ -3,18 +3,17 @@ 4. Concourse Pipeline (Cloud Foundry)

    4. Concourse Pipeline (Cloud Foundry)

    [Important]Important

    In this chapter we assume that you perform deployment of your application to Cloud Foundry PaaS

    The Spring Cloud Pipelines repository contains opinionated Concourse pipeline definition. Those jobs will form an empty pipeline and a -sample, opinionated one that you can use in your company.

    All in all there are the following projects taking part in the whole microservice setup for this demo.

    • Github Analytics - the app that has a REST endpoint and uses messaging. Our business application.
    • Github Webhook - project that emits messages that are used by Github Analytics. Our business application.
    • Eureka - simple Eureka Server. This is an infrastructure application.
    • Github Analytics Stub Runner Boot - Stub Runner Boot server to be used for tests with Github Analytics. Uses Eureka and Messaging. This is an infrastructure application.

    4.1 Step by step

    If you want to just run the demo as far as possible using PCF Dev and Docker Compose

    Below you can find optional steps needed to be taken when you want to customize the pipeline

    4.1.1 Fork repos

    There are 4 apps that are composing the pipeline

    You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

    4.1.2 Start Concourse and Artifactory

    Concourse + Artifactory can be run locally. To do that just execute the +sample, opinionated one that you can use in your company.

    All in all there are the following projects taking part in the whole microservice setup for this demo.

    • Github Analytics - the app that has a REST endpoint and uses messaging. Our business application.
    • Github Webhook - project that emits messages that are used by Github Analytics. Our business application.
    • Eureka - simple Eureka Server. This is an infrastructure application.
    • Github Analytics Stub Runner Boot - Stub Runner Boot server to be used for tests with Github Analytics. Uses Eureka and Messaging. This is an infrastructure application.

    4.1 Step by step

    If you want to just run the demo as far as possible using PCF Dev and Docker Compose

    4.1.1 Fork repos

    There are 4 apps that are composing the pipeline

    You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

    4.1.2 Start Concourse and Artifactory

    Concourse + Artifactory can be run locally. To do that just execute the start.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
     cd spring-cloud-pipelines/concourse
     ./setup_docker_compose.sh
     ./start.sh 192.168.99.100

    The setup_docker_compose.sh script should be executed once only to allow generation of keys.

    The 192.168.99.100 param is an example of an external URL of Concourse -(equal to Docker-Machine ip in this example).

    Then Concourse will be running on port 8080 and Artifactory 8081.

    Deploy the infra JARs to Artifactory

    When Artifactory is running, just execute the tools/deploy-infra.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
    +(equal to Docker-Machine ip in this example).

    Then Concourse will be running on port 8080 and Artifactory 8081.

    Deploy the infra JARs to Artifactory

    When Artifactory is running, just execute the tools/deploy-infra.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
     cd spring-cloud-pipelines/
     ./tools/deploy-infra.sh

    As a result both eureka and stub runner repos will be cloned, built -and uploaded to Artifactory.

    4.1.3 Start PCF Dev

    TIP: You can skip this step if you have CF installed and don’t want to use PCF Dev -The only thing you have to do is to set up spaces.

    [Warning]Warning

    It’s more than likely that you’ll run out of resources when you reach stage step. +and uploaded to Artifactory.

    4.1.3 Start PCF Dev

    [Tip]Tip

    You can skip this step if you have CF installed and don’t want to use PCF Dev +The only thing you have to do is to set up spaces.

    [Warning]Warning

    It’s more than likely that you’ll run out of resources when you reach stage step. Don’t worry! Keep calm and clear some apps from PCF Dev and continue.

    You have to download and start PCF Dev. A link how to do it is available here.

    The default credentials when using PCF Dev are:

    username: user
     password: pass
     email: user
    @@ -27,19 +26,19 @@ cf set-s
     cf create-space pcfdev-stage
     cf set-space-role user pcfdev-org pcfdev-stage SpaceDeveloper
     cf create-space pcfdev-prod
    -cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

    You can also execute the ./tools/cf-helper.sh setup-spaces to do this.

    4.1.4 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

      +cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

    You can also execute the ./tools/cf-helper.sh setup-spaces to do this.

    4.1.4 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

       

    running concourse

      - 

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    4.1.5 Setup your credentials.yml

    The repo comes with credentials-sample-cf.yml which is set up with sample data (most credentials) are set to be applicable for PCF Dev. Copy this file to a new file credentials.yml (the file is added to .gitignore so don’t worry that you’ll push it with your passwords) and edit it as you wish. For our demo just setup:

    • app-url - url pointing to your forked github-webhook repo
    • github-private-key - your private key to clone / tag GitHub repos
    • repo-with-binaries - the IP is set to the defaults for Docker Machine. You should update it to point to your setup

    If you don’t have a Docker Machine just execute ./whats_my_ip.sh script to + 

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    4.1.5 Setup your credentials.yml

    The repo comes with credentials-sample-cf.yml which is set up with sample data (most credentials) are set to be applicable for PCF Dev. Copy this file to a new file credentials.yml (the file is added to .gitignore so don’t worry that you’ll push it with your passwords) and edit it as you wish. For our demo just setup:

    • app-url - url pointing to your forked github-webhook repo
    • github-private-key - your private key to clone / tag GitHub repos
    • repo-with-binaries - the IP is set to the defaults for Docker Machine. You should update it to point to your setup

    If you don’t have a Docker Machine just execute ./whats_my_ip.sh script to get an external IP that you can pass to your repo-with-binaries instead of the default -Docker Machine IP.

    Below you can see what environment variables are required by the scripts. To the right hand side you can see the default values for PCF Dev that we set in the credentials-sample-cf.yml.

    Property NameProperty DescriptionDefault value
     

    BUILD_OPTIONS

    Additional options you would like to pass to the Maven / Gradle build

    PAAS_TEST_API_URL

    The URL to the CF Api for TEST env

    api.local.pcfdev.io

    PAAS_STAGE_API_URL

    The URL to the CF Api for STAGE env

    api.local.pcfdev.io

    PAAS_PROD_API_URL

    The URL to the CF Api for PROD env

    api.local.pcfdev.io

    PAAS_TEST_ORG

    Name of the org for the test env

    pcfdev-org

    PAAS_TEST_SPACE

    Name of the space for the test env

    pcfdev-space

    PAAS_STAGE_ORG

    Name of the org for the stage env

    pcfdev-org

    PAAS_STAGE_SPACE

    Name of the space for the stage env

    pcfdev-space

    PAAS_PROD_ORG

    Name of the org for the prod env

    pcfdev-org

    PAAS_PROD_SPACE

    Name of the space for the prod env

    pcfdev-space

    REPO_WITH_BINARIES

    URL to repo with the deployed jars

    http://192.168.99.100:8081/artifactory/libs-release-local

    M2_SETTINGS_REPO_ID

    The id of server from Maven settings.xml

    artifactory-local

    PAAS_HOSTNAME_UUID

    Additional suffix for the route. In a shared environment the default routes can be already taken

     

    APP_MEMORY_LIMIT

    How much memory should be used by the infra apps (Eureka, Stub Runner etc.)

    256m

    JAVA_BUILDPACK_URL

    The URL to the Java buildpack to be used by CF

    https://github.com/cloudfoundry/java-buildpack.git#v3.8.1

    4.1.6 Build the pipeline

    Log in (e.g. for Concourse running at 192.168.99.100 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    ./login.sh 192.168.99.100

    Next run the command to create the pipeline.

    ./set_pipeline.sh

    Then you’ll create a github-webhook pipeline under the docker alias, using the provided credentials.yml file. -You can override these values in exactly that order (e.g. ./set-pipeline.sh some-project another-target some-other-credentials.yml)

    4.1.7 Run the github-webhook pipeline

      - 

    Figure 4.1. Click Login

    concourse login

      - 

    Figure 4.2. Pick main team

    concourse team main

      - 

    Figure 4.3. Log in with concourse user and changeme password

    concourse user pass

      - 

    Figure 4.4. Your screen should look more or less like this

    concourse pipeline

      - 

    Figure 4.5. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      - 

    Figure 4.6. Click 'generate-version'

    generate version

      - 

    Figure 4.7. Click + sign to start a new build

    run pipeline

      - 

    Figure 4.8. The job is pending

    concourse pending

      - 

    Figure 4.9. Job is pending in the main screen

    job running

      - 

    Figure 4.10. Job is running in the main screen

    running pipeline

    \ No newline at end of file +Docker Machine IP.

    Below you can see what environment variables are required by the scripts. To the right hand side you can see the default values for PCF Dev that we set in the credentials-sample-cf.yml.

    Property NameProperty DescriptionDefault value
     

    BUILD_OPTIONS

    Additional options you would like to pass to the Maven / Gradle build

    PAAS_TEST_API_URL

    The URL to the CF Api for TEST env

    api.local.pcfdev.io

    PAAS_STAGE_API_URL

    The URL to the CF Api for STAGE env

    api.local.pcfdev.io

    PAAS_PROD_API_URL

    The URL to the CF Api for PROD env

    api.local.pcfdev.io

    PAAS_TEST_ORG

    Name of the org for the test env

    pcfdev-org

    PAAS_TEST_SPACE

    Name of the space for the test env

    pcfdev-space

    PAAS_STAGE_ORG

    Name of the org for the stage env

    pcfdev-org

    PAAS_STAGE_SPACE

    Name of the space for the stage env

    pcfdev-space

    PAAS_PROD_ORG

    Name of the org for the prod env

    pcfdev-org

    PAAS_PROD_SPACE

    Name of the space for the prod env

    pcfdev-space

    REPO_WITH_BINARIES

    URL to repo with the deployed jars

    http://192.168.99.100:8081/artifactory/libs-release-local

    M2_SETTINGS_REPO_ID

    The id of server from Maven settings.xml

    artifactory-local

    PAAS_HOSTNAME_UUID

    Additional suffix for the route. In a shared environment the default routes can be already taken

     

    APP_MEMORY_LIMIT

    How much memory should be used by the infra apps (Eureka, Stub Runner etc.)

    256m

    JAVA_BUILDPACK_URL

    The URL to the Java buildpack to be used by CF

    https://github.com/cloudfoundry/java-buildpack.git#v3.8.1

    4.1.6 Build the pipeline

    Log in (e.g. for Concourse running at 192.168.99.100 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    ./login.sh 192.168.99.100

    Next run the command to create the pipeline.

    ./set_pipeline.sh

    Then you’ll create a github-webhook pipeline under the docker alias, using the provided credentials.yml file. +You can override these values in exactly that order (e.g. ./set-pipeline.sh some-project another-target some-other-credentials.yml)

    4.1.7 Run the github-webhook pipeline

      + 

    Figure 4.1. Click Login

    concourse login

      + 

    Figure 4.2. Pick main team

    concourse team main

      + 

    Figure 4.3. Log in with concourse user and changeme password

    concourse user pass

      + 

    Figure 4.4. Your screen should look more or less like this

    concourse pipeline

      + 

    Figure 4.5. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      + 

    Figure 4.6. Click 'generate-version'

    generate version

      + 

    Figure 4.7. Click + sign to start a new build

    run pipeline

      + 

    Figure 4.8. The job is pending

    concourse pending

      + 

    Figure 4.9. Job is pending in the main screen

    job running

      + 

    Figure 4.10. Job is running in the main screen

    running pipeline

    \ No newline at end of file diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-k8s.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-k8s.html index 1aaf6fc7..ea765dab 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-k8s.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi_concourse-pipeline-k8s.html @@ -34,7 +34,7 @@ Congratulations. You have just deployed JFrog Artifactory Pro! Default credential for Artifactory: user: admin password: password

    Next, we need to set up the repositories.

    First, access the Artifactory URL and log in with -user, admin and password password.

    Figure 5.1. Click on the Quick Setup

    artifactory quick setup

    Then, click on the Maven setup and click Create.

    Figure 5.2. Create the Maven Repository

    artifactory maven repo

    5.2.2 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

      +user, admin and password password.

    Figure 5.1. Click on the Quick Setup

    artifactory quick setup

    Then, click on the Maven setup and click Create.

    Figure 5.2. Create the Maven Repository

    artifactory maven repo

    5.2.2 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

       

    running concourse

       

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    5.2.3 Setup your credentials.yml

    There is a sample credentials file called credentials-sample-k8s.yml prepared for k8s. You can use it as a base for your credentials.yml.

    To allow the Concourse worker’s spawned container to connect to @@ -43,13 +43,13 @@ auth token.

    To get the contents of CA for GCE just execute

    echo "Visit http://127.0.0.1:8080 to use Concourse"
     $ kubectl port-forward --namespace default $POD_NAME 8080:8080
     Visit http://127.0.0.1:8080 to use Concourse

    Log in (e.g. for Concourse running at 127.0.0.1 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    $ fly -t k8s login -c http://localhost:8080 -u concourse -p concourse

    Next run the command to create the pipeline.

    $ ./set_pipeline.sh github-webhook k8s credentials-k8s.yml

    5.2.5 Run the github-webhook pipeline

      - 

    Figure 5.3. Click Login

    concourse login

      - 

    Figure 5.4. Pick main team

    concourse team main

      - 

    Figure 5.5. Log in with concourse user and concourse password

    concourse user pass

      - 

    Figure 5.6. Your screen should look more or less like this

    concourse pipeline

      - 

    Figure 5.7. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      - 

    Figure 5.8. Click 'generate-version'

    generate version

      - 

    Figure 5.9. Click + sign to start a new build

    run pipeline

      - 

    Figure 5.10. The job is pending

    concourse pending

      - 

    Figure 5.11. Job is pending in the main screen

    job running

      - 

    Figure 5.12. Job is running in the main screen

    running pipeline

    \ No newline at end of file + 

    Figure 5.3. Click Login

    concourse login

      + 

    Figure 5.4. Pick main team

    concourse team main

      + 

    Figure 5.5. Log in with concourse user and concourse password

    concourse user pass

      + 

    Figure 5.6. Your screen should look more or less like this

    concourse pipeline

      + 

    Figure 5.7. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      + 

    Figure 5.8. Click 'generate-version'

    generate version

      + 

    Figure 5.9. Click + sign to start a new build

    run pipeline

      + 

    Figure 5.10. The job is pending

    concourse pending

      + 

    Figure 5.11. Job is pending in the main screen

    job running

      + 

    Figure 5.12. Job is running in the main screen

    running pipeline

    \ No newline at end of file diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-cf.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-cf.html index b33371eb..1bd3f55e 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-cf.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-cf.html @@ -32,35 +32,35 @@ has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and `github-analytics'.

      - 

    Figure 8.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      - 

    Figure 8.2. Click the 'Build with parameters'

    seed run

      - 

    Figure 8.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      - 

    Figure 8.4. This is how the results of seed should look like

    seed built

    8.1.5 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do + 

    Figure 8.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      + 

    Figure 8.2. Click the 'Build with parameters'

    seed run

      + 

    Figure 8.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      + 

    Figure 8.4. This is how the results of seed should look like

    seed built

    8.1.5 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do run it you have to provide some properties. By default we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and github-analytics.

      - 

    Figure 8.5. Click the 'github-webhook' view

    seed views

      - 

    Figure 8.6. Run the pipeline

    pipeline run

      + 

    Figure 8.5. Click the 'github-webhook' view

    seed views

      + 

    Figure 8.6. Run the pipeline

    pipeline run

       

    [Important]Important

    If your build fails on the deploy previous version to stage due to missing jar, that means that you’ve forgotten to clear the tags in your repo. Typically that’s due to the fact that you’ve removed the Artifactory volume with deployed JAR whereas a tag in the repo is still pointing there. Check out this section on how to remove the tag.

      - 

    Figure 8.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

      + 

    Figure 8.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

       

    [Important]Important

    Most likely you will run out of memory so when reaching the stage environment it’s good to kill all apps on test. Check out the FAQ section for more details!

      - 

    Figure 8.8. The full pipeline should look like this

    pipeline finished

      + 

    Figure 8.8. The full pipeline should look like this

    pipeline finished

       

    8.2 Declarative pipeline & Blue Ocean

    You can also use the declarative pipeline approach with the Blue Ocean UI. Here is a step by step guide to run a pipeline via this approach.

    The Blue Ocean UI is available under the blue/ URL. E.g. for Docker Machine based setup http://192.168.99.100:8080/blue.

      - 

    Figure 8.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      - 

    Figure 8.10. Your first run will look like this. Click Run button

    blue 2

      - 

    Figure 8.11. Enter parameters required for the build and click run

    blue 3

      - 

    Figure 8.12. A list of pipelines will be shown. Click your first run.

    blue 4

      - 

    Figure 8.13. State if you want to go to production or not and click Proceed

    blue 5

      - 

    Figure 8.14. The build is in progress…​

    blue 6

      - 

    Figure 8.15. The pipeline is done!

    blue 7

      + 

    Figure 8.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      + 

    Figure 8.10. Your first run will look like this. Click Run button

    blue 2

      + 

    Figure 8.11. Enter parameters required for the build and click run

    blue 3

      + 

    Figure 8.12. A list of pipelines will be shown. Click your first run.

    blue 4

      + 

    Figure 8.13. State if you want to go to production or not and click Proceed

    blue 5

      + 

    Figure 8.14. The build is in progress…​

    blue 6

      + 

    Figure 8.15. The pipeline is done!

    blue 7

       

    [Important]Important

    There is no possibility of restarting pipeline from specific stage, after failure. Please check out this issue for more information

    [Warning]Warning

    Currently there is no way to introduce manual steps in a performant way. Jenkins is blocking an executor when manual step is required. That means that you’ll run out of executors diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-k8s.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-k8s.html index 955b157e..3ec2f99d 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-k8s.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi_jenkins-pipeline-k8s.html @@ -24,35 +24,35 @@ has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and `github-analytics'.

      - 

    Figure 9.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      - 

    Figure 9.2. Click the 'Build with parameters'

    seed run

      - 

    Figure 9.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      - 

    Figure 9.4. This is how the results of seed should look like

    seed built

    9.1.4 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do + 

    Figure 9.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      + 

    Figure 9.2. Click the 'Build with parameters'

    seed run

      + 

    Figure 9.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      + 

    Figure 9.4. This is how the results of seed should look like

    seed built

    9.1.4 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do run it you have to provide some properties. By default we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and github-analytics.

      - 

    Figure 9.5. Click the 'github-webhook' view

    seed views

      - 

    Figure 9.6. Run the pipeline

    pipeline run

      + 

    Figure 9.5. Click the 'github-webhook' view

    seed views

      + 

    Figure 9.6. Run the pipeline

    pipeline run

       

    [Important]Important

    If your build fails on the deploy previous version to stage due to missing jar, that means that you’ve forgotten to clear the tags in your repo. Typically that’s due to the fact that you’ve removed the Artifactory volume with deployed JAR whereas a tag in the repo is still pointing there. Check out this section on how to remove the tag.

      - 

    Figure 9.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

      + 

    Figure 9.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

       

    [Important]Important

    Most likely you will run out of memory so when reaching the stage environment it’s good to kill all apps on test. Check out the FAQ section for more details!

      - 

    Figure 9.8. The full pipeline should look like this

    pipeline finished

      + 

    Figure 9.8. The full pipeline should look like this

    pipeline finished

       

    9.2 Declarative pipeline & Blue Ocean

    You can also use the declarative pipeline approach with the Blue Ocean UI. Here is a step by step guide to run a pipeline via this approach.

    The Blue Ocean UI is available under the blue/ URL. E.g. for Docker Machine based setup http://192.168.99.100:8080/blue.

      - 

    Figure 9.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      - 

    Figure 9.10. Your first run will look like this. Click Run button

    blue 2

      - 

    Figure 9.11. Enter parameters required for the build and click run

    blue 3

      - 

    Figure 9.12. A list of pipelines will be shown. Click your first run.

    blue 4

      - 

    Figure 9.13. State if you want to go to production or not and click Proceed

    blue 5

      - 

    Figure 9.14. The build is in progress…​

    blue 6

      - 

    Figure 9.15. The pipeline is done!

    blue 7

      + 

    Figure 9.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      + 

    Figure 9.10. Your first run will look like this. Click Run button

    blue 2

      + 

    Figure 9.11. Enter parameters required for the build and click run

    blue 3

      + 

    Figure 9.12. A list of pipelines will be shown. Click your first run.

    blue 4

      + 

    Figure 9.13. State if you want to go to production or not and click Proceed

    blue 5

      + 

    Figure 9.14. The build is in progress…​

    blue 6

      + 

    Figure 9.15. The pipeline is done!

    blue 7

       

    [Important]Important

    There is no possibility of restarting pipeline from specific stage, after failure. Please check out this issue for more information

    [Warning]Warning

    Currently there is no way to introduce manual steps in a performant way. Jenkins is blocking an executor when manual step is required. That means that you’ll run out of executors diff --git a/spring-cloud-pipelines/1.0.0.M7/multi/multi_spring-cloud-pipelines.html b/spring-cloud-pipelines/1.0.0.M7/multi/multi_spring-cloud-pipelines.html index a9da3cfa..5e12025a 100644 --- a/spring-cloud-pipelines/1.0.0.M7/multi/multi_spring-cloud-pipelines.html +++ b/spring-cloud-pipelines/1.0.0.M7/multi/multi_spring-cloud-pipelines.html @@ -1,3 +1,3 @@ - Spring Cloud Pipelines

    Spring Cloud Pipelines


    Table of Contents

    1. Introduction
    1.1. Project setup
    1.2. How to use it?
    1.2.1. Centralized pipeline creation
    1.2.2. Pipeline per repository
    1.3. The flow
    1.4. Environments
    1.5. Tests
    1.5.1. Testing against stubs
    1.5.2. General view
    1.6. CI Server worker prerequisites
    1.7. Pipeline descriptor
    2. Opinionated implementation
    2.1. Build
    2.2. Test
    2.3. Stage
    2.4. Prod
    3. Project opinions
    3.1. Cloud Foundry project opinions
    3.2. Kubernetes project opinions
    4. Concourse Pipeline (Cloud Foundry)
    4.1. Step by step
    4.1.1. Fork repos
    4.1.2. Start Concourse and Artifactory
    Deploy the infra JARs to Artifactory
    4.1.3. Start PCF Dev
    4.1.4. Setup the fly CLI
    4.1.5. Setup your credentials.yml
    4.1.6. Build the pipeline
    4.1.7. Run the github-webhook pipeline
    5. Concourse Pipeline (Kubernetes)
    5.1. Step by step
    5.1.1. Fork repos
    5.2. Concourse in K8S (Kubernetes)
    5.2.1. Deploying Artifactory to K8S
    5.2.2. Setup the fly CLI
    5.2.3. Setup your credentials.yml
    5.2.4. Build the pipeline
    5.2.5. Run the github-webhook pipeline
    6. Concourse FAQ
    6.1. Can I use the pipeline for some other repos?
    6.2. Will this work for ANY project out of the box?
    6.3. Can I modify this to reuse in my project?
    6.4. I ran out of resources!! (PCF Dev)
    6.5. The rollback step fails due to missing JAR ?!
    6.6. Can I see the output of a job from the terminal?
    6.7. I clicked the job and it’s constantly pending…​
    6.8. The route is already in use (CF)
    6.9. I’m unauthorized to deploy infrastructure jars
    6.10. version resource is broken
    7. Jenkins Pipeline (Common)
    7.1. Project setup
    7.2. Optional customization steps
    7.2.1. Deploying infra jars to a different location
    7.2.2. Setup settings.xml for Maven deployment
    7.2.3. Setup Jenkins env vars
    Seed properties
    Global envs
    7.2.4. Set Git email / user
    Add Jenkins credentials for GitHub
    7.3. Testing Jenkins scripts
    7.4. How to work with Jenkins Job DSL plugin
    7.5. Docker Image
    8. Jenkins Pipeline (Cloud Foundry)
    8.1. Step by step
    8.1.1. Fork repos
    8.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    8.1.3. Start PCF Dev
    8.1.4. Run the seed job
    8.1.5. Run the github-webhook pipeline
    8.2. Declarative pipeline & Blue Ocean
    8.3. Jenkins Cloud Foundry customization
    8.3.1. All env vars
    8.3.2. Jenkins Credentials
    9. Jenkins Pipeline (Kubernetes)
    9.1. Step by step
    9.1.1. Fork repos
    9.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    9.1.3. Run the seed job
    9.1.4. Run the github-webhook pipeline
    9.2. Declarative pipeline & Blue Ocean
    9.3. Jenkins Kubernetes customization
    9.3.1. All env vars
    9.4. Preparing to connect to GCE
    9.5. Connecting to a Kubo or GCE cluster
    10. Jenkins FAQ
    10.1. Pipeline version contains ${PIPELINE_VERSION}
    10.2. Pipeline version is not passed to the build
    10.3. The build times out with pipeline.sh info
    10.4. Can I use the pipeline for some other repos?
    10.5. Will this work for ANY project out of the box?
    10.6. Can I modify this to reuse in my project?
    10.7. The rollback step fails due to missing JAR ?!
    10.8. I want to provide a different JDK version
    10.9. Enable Groovy Token Macro Processing
    10.10. I want deployment to stage and prod be automatic
    10.11. I don’t want to test API compativility
    10.12. I can’t tag the repo!
    10.13. I’m unauthorized to deploy infrastructure jars
    10.14. Signing Artifacts
    10.15. Using SSH keys for git
    10.16. Deploy to stage fails and doesn’t redeploy a service (Kubernetes)
    10.17. I ran out of resources!! (Cloud Foundry)
    10.18. Deploying to test / stage / prod fails - error finding space (Cloud Foundry)
    10.19. The route is already in use (Cloud Foundry)
    10.20. How to execute helper scripts against a real CF instance I’m logged into (Cloud Foundry)
    11. Kubernetes setup
    11.1. Kubernetes CLI Installation
    11.1.1. Script Installation
    11.1.2. Manual Installation
    11.2. Kubernetes Cluster setup
    11.2.1. Script Installation
    11.2.2. Manual Installation
    11.3. Run Minikube
    11.4. Certificates and Workers
    11.4.1. Minikube Certificates and Workers
    11.4.2. Manual Certificates and Workers Setup
    11.5. Generate Minikube namespaces
    12. The demo setup (Cloud Foundry)
    12.1. Deploying production applications to PCF Dev
    12.2. Running Prometheus on CF
    12.3. Running Grafana on CF
    13. The demo setup (Kubernetes)
    13.1. Deploying production applications to Minikube
    13.2. Running Prometheus on Kubernetes
    13.3. Running Grafana on Kubernetes
    14. Building the project
    14.1. Prerequisites
    14.2. Bats submodules
    14.3. Build and test
    14.4. Generate docs
    14.5. Making a release
    15. Customizing the project
    16. Releasing the project
    16.1. Publishing A Docker Image
    \ No newline at end of file + Spring Cloud Pipelines

    Spring Cloud Pipelines


    Table of Contents

    1. Introduction
    1.1. Project setup
    1.2. How to use it?
    1.2.1. Centralized pipeline creation
    1.2.2. Pipeline per repository
    1.3. The flow
    1.4. Environments
    1.5. Tests
    1.5.1. Testing against stubs
    1.5.2. General view
    1.6. CI Server worker prerequisites
    1.7. Pipeline descriptor
    2. Opinionated implementation
    2.1. Build
    2.2. Test
    2.3. Stage
    2.4. Prod
    3. Project opinions
    3.1. Cloud Foundry project opinions
    3.2. Kubernetes project opinions
    4. Concourse Pipeline (Cloud Foundry)
    4.1. Step by step
    4.1.1. Fork repos
    4.1.2. Start Concourse and Artifactory
    Deploy the infra JARs to Artifactory
    4.1.3. Start PCF Dev
    4.1.4. Setup the fly CLI
    4.1.5. Setup your credentials.yml
    4.1.6. Build the pipeline
    4.1.7. Run the github-webhook pipeline
    5. Concourse Pipeline (Kubernetes)
    5.1. Step by step
    5.1.1. Fork repos
    5.2. Concourse in K8S (Kubernetes)
    5.2.1. Deploying Artifactory to K8S
    5.2.2. Setup the fly CLI
    5.2.3. Setup your credentials.yml
    5.2.4. Build the pipeline
    5.2.5. Run the github-webhook pipeline
    6. Concourse FAQ
    6.1. Can I use the pipeline for some other repos?
    6.2. Will this work for ANY project out of the box?
    6.3. Can I modify this to reuse in my project?
    6.4. I ran out of resources!! (PCF Dev)
    6.5. The rollback step fails due to missing JAR ?!
    6.6. Can I see the output of a job from the terminal?
    6.7. I clicked the job and it’s constantly pending…​
    6.8. The route is already in use (CF)
    6.9. I’m unauthorized to deploy infrastructure jars
    6.10. version resource is broken
    7. Jenkins Pipeline (Common)
    7.1. Project setup
    7.2. Optional customization steps
    7.2.1. Deploying infra jars to a different location
    7.2.2. Setup settings.xml for Maven deployment
    7.2.3. Setup Jenkins env vars
    Seed properties
    Global envs
    7.2.4. Set Git email / user
    Add Jenkins credentials for GitHub
    7.3. Testing Jenkins scripts
    7.4. How to work with Jenkins Job DSL plugin
    7.5. Docker Image
    8. Jenkins Pipeline (Cloud Foundry)
    8.1. Step by step
    8.1.1. Fork repos
    8.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    8.1.3. Start PCF Dev
    8.1.4. Run the seed job
    8.1.5. Run the github-webhook pipeline
    8.2. Declarative pipeline & Blue Ocean
    8.3. Jenkins Cloud Foundry customization
    8.3.1. All env vars
    8.3.2. Jenkins Credentials
    9. Jenkins Pipeline (Kubernetes)
    9.1. Step by step
    9.1.1. Fork repos
    9.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    9.1.3. Run the seed job
    9.1.4. Run the github-webhook pipeline
    9.2. Declarative pipeline & Blue Ocean
    9.3. Jenkins Kubernetes customization
    9.3.1. All env vars
    9.4. Preparing to connect to GCE
    9.5. Connecting to a Kubo or GCE cluster
    10. Jenkins FAQ
    10.1. Pipeline version contains ${PIPELINE_VERSION}
    10.2. Pipeline version is not passed to the build
    10.3. The build times out with pipeline.sh info
    10.4. Can I use the pipeline for some other repos?
    10.5. Will this work for ANY project out of the box?
    10.6. Can I modify this to reuse in my project?
    10.7. The rollback step fails due to missing JAR ?!
    10.8. I want to provide a different JDK version
    10.9. Enable Groovy Token Macro Processing
    10.10. I want deployment to stage and prod be automatic
    10.11. I don’t want to test API compativility
    10.12. I can’t tag the repo!
    10.13. I’m unauthorized to deploy infrastructure jars
    10.14. Signing Artifacts
    10.15. Using SSH keys for git
    10.16. Deploy to stage fails and doesn’t redeploy a service (Kubernetes)
    10.17. I ran out of resources!! (Cloud Foundry)
    10.18. Deploying to test / stage / prod fails - error finding space (Cloud Foundry)
    10.19. The route is already in use (Cloud Foundry)
    10.20. How to execute helper scripts against a real CF instance I’m logged into (Cloud Foundry)
    11. Kubernetes setup
    11.1. Kubernetes CLI Installation
    11.1.1. Script Installation
    11.1.2. Manual Installation
    11.2. Kubernetes Cluster setup
    11.2.1. Script Installation
    11.2.2. Manual Installation
    11.3. Run Minikube
    11.4. Certificates and Workers
    11.4.1. Minikube Certificates and Workers
    11.4.2. Manual Certificates and Workers Setup
    11.5. Generate Minikube namespaces
    12. The demo setup (Cloud Foundry)
    12.1. Deploying production applications to PCF Dev
    12.2. Running Prometheus on CF
    12.3. Running Grafana on CF
    13. The demo setup (Kubernetes)
    13.1. Deploying production applications to Minikube
    13.2. Running Prometheus on Kubernetes
    13.3. Running Grafana on Kubernetes
    14. Building the project
    14.1. Prerequisites
    14.2. Bats submodules
    14.3. Build and test
    14.4. Generate docs
    14.5. Making a release
    15. Customizing the project
    16. Releasing the project
    16.1. Publishing A Docker Image
    \ No newline at end of file diff --git a/spring-cloud-pipelines/1.0.0.M7/single/spring-cloud-pipelines.html b/spring-cloud-pipelines/1.0.0.M7/single/spring-cloud-pipelines.html index bb55f40c..6a50a98d 100644 --- a/spring-cloud-pipelines/1.0.0.M7/single/spring-cloud-pipelines.html +++ b/spring-cloud-pipelines/1.0.0.M7/single/spring-cloud-pipelines.html @@ -1,6 +1,6 @@ - Spring Cloud Pipelines

    Spring Cloud Pipelines


    Table of Contents

    1. Introduction
    1.1. Project setup
    1.2. How to use it?
    1.2.1. Centralized pipeline creation
    1.2.2. Pipeline per repository
    1.3. The flow
    1.4. Environments
    1.5. Tests
    1.5.1. Testing against stubs
    1.5.2. General view
    1.6. CI Server worker prerequisites
    1.7. Pipeline descriptor
    2. Opinionated implementation
    2.1. Build
    2.2. Test
    2.3. Stage
    2.4. Prod
    3. Project opinions
    3.1. Cloud Foundry project opinions
    3.2. Kubernetes project opinions
    4. Concourse Pipeline (Cloud Foundry)
    4.1. Step by step
    4.1.1. Fork repos
    4.1.2. Start Concourse and Artifactory
    Deploy the infra JARs to Artifactory
    4.1.3. Start PCF Dev
    4.1.4. Setup the fly CLI
    4.1.5. Setup your credentials.yml
    4.1.6. Build the pipeline
    4.1.7. Run the github-webhook pipeline
    5. Concourse Pipeline (Kubernetes)
    5.1. Step by step
    5.1.1. Fork repos
    5.2. Concourse in K8S (Kubernetes)
    5.2.1. Deploying Artifactory to K8S
    5.2.2. Setup the fly CLI
    5.2.3. Setup your credentials.yml
    5.2.4. Build the pipeline
    5.2.5. Run the github-webhook pipeline
    6. Concourse FAQ
    6.1. Can I use the pipeline for some other repos?
    6.2. Will this work for ANY project out of the box?
    6.3. Can I modify this to reuse in my project?
    6.4. I ran out of resources!! (PCF Dev)
    6.5. The rollback step fails due to missing JAR ?!
    6.6. Can I see the output of a job from the terminal?
    6.7. I clicked the job and it’s constantly pending…​
    6.8. The route is already in use (CF)
    6.9. I’m unauthorized to deploy infrastructure jars
    6.10. version resource is broken
    7. Jenkins Pipeline (Common)
    7.1. Project setup
    7.2. Optional customization steps
    7.2.1. Deploying infra jars to a different location
    7.2.2. Setup settings.xml for Maven deployment
    7.2.3. Setup Jenkins env vars
    Seed properties
    Global envs
    7.2.4. Set Git email / user
    Add Jenkins credentials for GitHub
    7.3. Testing Jenkins scripts
    7.4. How to work with Jenkins Job DSL plugin
    7.5. Docker Image
    8. Jenkins Pipeline (Cloud Foundry)
    8.1. Step by step
    8.1.1. Fork repos
    8.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    8.1.3. Start PCF Dev
    8.1.4. Run the seed job
    8.1.5. Run the github-webhook pipeline
    8.2. Declarative pipeline & Blue Ocean
    8.3. Jenkins Cloud Foundry customization
    8.3.1. All env vars
    8.3.2. Jenkins Credentials
    9. Jenkins Pipeline (Kubernetes)
    9.1. Step by step
    9.1.1. Fork repos
    9.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    9.1.3. Run the seed job
    9.1.4. Run the github-webhook pipeline
    9.2. Declarative pipeline & Blue Ocean
    9.3. Jenkins Kubernetes customization
    9.3.1. All env vars
    9.4. Preparing to connect to GCE
    9.5. Connecting to a Kubo or GCE cluster
    10. Jenkins FAQ
    10.1. Pipeline version contains ${PIPELINE_VERSION}
    10.2. Pipeline version is not passed to the build
    10.3. The build times out with pipeline.sh info
    10.4. Can I use the pipeline for some other repos?
    10.5. Will this work for ANY project out of the box?
    10.6. Can I modify this to reuse in my project?
    10.7. The rollback step fails due to missing JAR ?!
    10.8. I want to provide a different JDK version
    10.9. Enable Groovy Token Macro Processing
    10.10. I want deployment to stage and prod be automatic
    10.11. I don’t want to test API compativility
    10.12. I can’t tag the repo!
    10.13. I’m unauthorized to deploy infrastructure jars
    10.14. Signing Artifacts
    10.15. Using SSH keys for git
    10.16. Deploy to stage fails and doesn’t redeploy a service (Kubernetes)
    10.17. I ran out of resources!! (Cloud Foundry)
    10.18. Deploying to test / stage / prod fails - error finding space (Cloud Foundry)
    10.19. The route is already in use (Cloud Foundry)
    10.20. How to execute helper scripts against a real CF instance I’m logged into (Cloud Foundry)
    11. Kubernetes setup
    11.1. Kubernetes CLI Installation
    11.1.1. Script Installation
    11.1.2. Manual Installation
    11.2. Kubernetes Cluster setup
    11.2.1. Script Installation
    11.2.2. Manual Installation
    11.3. Run Minikube
    11.4. Certificates and Workers
    11.4.1. Minikube Certificates and Workers
    11.4.2. Manual Certificates and Workers Setup
    11.5. Generate Minikube namespaces
    12. The demo setup (Cloud Foundry)
    12.1. Deploying production applications to PCF Dev
    12.2. Running Prometheus on CF
    12.3. Running Grafana on CF
    13. The demo setup (Kubernetes)
    13.1. Deploying production applications to Minikube
    13.2. Running Prometheus on Kubernetes
    13.3. Running Grafana on Kubernetes
    14. Building the project
    14.1. Prerequisites
    14.2. Bats submodules
    14.3. Build and test
    14.4. Generate docs
    14.5. Making a release
    15. Customizing the project
    16. Releasing the project
    16.1. Publishing A Docker Image

    Documentation Authors: Marcin Grzejszczak

    Spring, Spring Boot and Spring Cloud are tools that allow developers speed up the + Spring Cloud Pipelines

    Spring Cloud Pipelines


    Table of Contents

    1. Introduction
    1.1. Project setup
    1.2. How to use it?
    1.2.1. Centralized pipeline creation
    1.2.2. Pipeline per repository
    1.3. The flow
    1.4. Environments
    1.5. Tests
    1.5.1. Testing against stubs
    1.5.2. General view
    1.6. CI Server worker prerequisites
    1.7. Pipeline descriptor
    2. Opinionated implementation
    2.1. Build
    2.2. Test
    2.3. Stage
    2.4. Prod
    3. Project opinions
    3.1. Cloud Foundry project opinions
    3.2. Kubernetes project opinions
    4. Concourse Pipeline (Cloud Foundry)
    4.1. Step by step
    4.1.1. Fork repos
    4.1.2. Start Concourse and Artifactory
    Deploy the infra JARs to Artifactory
    4.1.3. Start PCF Dev
    4.1.4. Setup the fly CLI
    4.1.5. Setup your credentials.yml
    4.1.6. Build the pipeline
    4.1.7. Run the github-webhook pipeline
    5. Concourse Pipeline (Kubernetes)
    5.1. Step by step
    5.1.1. Fork repos
    5.2. Concourse in K8S (Kubernetes)
    5.2.1. Deploying Artifactory to K8S
    5.2.2. Setup the fly CLI
    5.2.3. Setup your credentials.yml
    5.2.4. Build the pipeline
    5.2.5. Run the github-webhook pipeline
    6. Concourse FAQ
    6.1. Can I use the pipeline for some other repos?
    6.2. Will this work for ANY project out of the box?
    6.3. Can I modify this to reuse in my project?
    6.4. I ran out of resources!! (PCF Dev)
    6.5. The rollback step fails due to missing JAR ?!
    6.6. Can I see the output of a job from the terminal?
    6.7. I clicked the job and it’s constantly pending…​
    6.8. The route is already in use (CF)
    6.9. I’m unauthorized to deploy infrastructure jars
    6.10. version resource is broken
    7. Jenkins Pipeline (Common)
    7.1. Project setup
    7.2. Optional customization steps
    7.2.1. Deploying infra jars to a different location
    7.2.2. Setup settings.xml for Maven deployment
    7.2.3. Setup Jenkins env vars
    Seed properties
    Global envs
    7.2.4. Set Git email / user
    Add Jenkins credentials for GitHub
    7.3. Testing Jenkins scripts
    7.4. How to work with Jenkins Job DSL plugin
    7.5. Docker Image
    8. Jenkins Pipeline (Cloud Foundry)
    8.1. Step by step
    8.1.1. Fork repos
    8.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    8.1.3. Start PCF Dev
    8.1.4. Run the seed job
    8.1.5. Run the github-webhook pipeline
    8.2. Declarative pipeline & Blue Ocean
    8.3. Jenkins Cloud Foundry customization
    8.3.1. All env vars
    8.3.2. Jenkins Credentials
    9. Jenkins Pipeline (Kubernetes)
    9.1. Step by step
    9.1.1. Fork repos
    9.1.2. Start Jenkins and Artifactory
    Deploy the infra JARs to Artifactory
    9.1.3. Run the seed job
    9.1.4. Run the github-webhook pipeline
    9.2. Declarative pipeline & Blue Ocean
    9.3. Jenkins Kubernetes customization
    9.3.1. All env vars
    9.4. Preparing to connect to GCE
    9.5. Connecting to a Kubo or GCE cluster
    10. Jenkins FAQ
    10.1. Pipeline version contains ${PIPELINE_VERSION}
    10.2. Pipeline version is not passed to the build
    10.3. The build times out with pipeline.sh info
    10.4. Can I use the pipeline for some other repos?
    10.5. Will this work for ANY project out of the box?
    10.6. Can I modify this to reuse in my project?
    10.7. The rollback step fails due to missing JAR ?!
    10.8. I want to provide a different JDK version
    10.9. Enable Groovy Token Macro Processing
    10.10. I want deployment to stage and prod be automatic
    10.11. I don’t want to test API compativility
    10.12. I can’t tag the repo!
    10.13. I’m unauthorized to deploy infrastructure jars
    10.14. Signing Artifacts
    10.15. Using SSH keys for git
    10.16. Deploy to stage fails and doesn’t redeploy a service (Kubernetes)
    10.17. I ran out of resources!! (Cloud Foundry)
    10.18. Deploying to test / stage / prod fails - error finding space (Cloud Foundry)
    10.19. The route is already in use (Cloud Foundry)
    10.20. How to execute helper scripts against a real CF instance I’m logged into (Cloud Foundry)
    11. Kubernetes setup
    11.1. Kubernetes CLI Installation
    11.1.1. Script Installation
    11.1.2. Manual Installation
    11.2. Kubernetes Cluster setup
    11.2.1. Script Installation
    11.2.2. Manual Installation
    11.3. Run Minikube
    11.4. Certificates and Workers
    11.4.1. Minikube Certificates and Workers
    11.4.2. Manual Certificates and Workers Setup
    11.5. Generate Minikube namespaces
    12. The demo setup (Cloud Foundry)
    12.1. Deploying production applications to PCF Dev
    12.2. Running Prometheus on CF
    12.3. Running Grafana on CF
    13. The demo setup (Kubernetes)
    13.1. Deploying production applications to Minikube
    13.2. Running Prometheus on Kubernetes
    13.3. Running Grafana on Kubernetes
    14. Building the project
    14.1. Prerequisites
    14.2. Bats submodules
    14.3. Build and test
    14.4. Generate docs
    14.5. Making a release
    15. Customizing the project
    16. Releasing the project
    16.1. Publishing A Docker Image

    Documentation Authors: Marcin Grzejszczak

    Spring, Spring Boot and Spring Cloud are tools that allow developers speed up the time of creating new business features. It’s common knowledge however that the feature is only valuable if it’s in production. That’s why companies spend a lot of time and resources on building their own deployment pipelines.

    This project tries to solve the following problems:

    • Creation of a common deployment pipeline
    • Propagation of good testing & deployment practices
    • Speed up the time required to deploy a feature to production

    A common way of running, configuring and deploying applications lowers support costs @@ -14,9 +14,11 @@ step and describe it in details.

    scripts are reused by both Concourse and Jenkins pipelines.

    In the concourse folder you can find all the necessary scripts and setup to run Concourse demo.

    In the docs section you have the whole documentation of the project.

    In the jenkins folder you can find all the necessary scripts and setup to run Jenkins demo.

    1.2 How to use it?

    This repository can be treated as a template for your pipeline. We provide some opinionated implementation that you can alter to suit your needs. The best approach to use it to build your production projects would be to download the Spring Cloud Pipelines repository as ZIP, then -init a Git project there and modify it as you wish.

    $ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/v1.0.0.M6.zip
    -$ unzip v1.0.0.M6.zip
    -$ cd spring-cloud-pipelines-v1.0.0.M6
    +init a Git project there and modify it as you wish.

    $ # pass the branch (e.g. master) or a particular tag (e.g. v1.0.0.RELEASE)
    +$ SC_PIPELINES_RELEASE=...
    +$ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/${SC_PIPELINES_RELEASE}.zip
    +$ unzip ${SC_PIPELINES_RELEASE}.zip
    +$ cd spring-cloud-pipelines-${SC_PIPELINES_RELEASE}
     $ git init
     $ # modify the pipelines to suit your needs
     $ git add .
    @@ -139,16 +141,16 @@ or false.

    Example:

          coordinates: com.example.eureka:github-eureka:0.0.1.M1

    When the deployment to test or deployment to stage occurs, Spring Cloud Pipelines will:

    • for test environment, delete existing services and redeploy the ones from the list
    • for stage environment, if the service is not available it will get deployed. Otherwise nothing will happen

    2. Opinionated implementation

    For the demo purposes we’re providing Docker Compose setup with Artifactory and Concourse / Jenkins tools. -Regardless of the picked CD application for the pipeline to pass one needs either

    • [CLOUD FOUNDRY] a Cloud Foundry instance (for example Pivotal Web Services or PCF Dev)
    • [KUBERNETES] a Kubernetes cluster (for example Minikube)
    • the infrastructure applications deployed to the JAR hosting application (for the demo we’re providing Artifactory).
    • Eureka for Service Discovery
    • Stub Runner Boot for running Spring Cloud Contract stubs.
    [Tip]Tip

    In the demos we’re showing you how to first build the github-webhook project. That’s because +Regardless of the picked CD application for the pipeline to pass one needs either

    • a Cloud Foundry instance (for example Pivotal Web Services or PCF Dev)
    • a Kubernetes cluster (for example Minikube)
    • the infrastructure applications deployed to the JAR hosting application (for the demo we’re providing Artifactory).
    • Eureka for Service Discovery
    • Stub Runner Boot for running Spring Cloud Contract stubs.
    [Tip]Tip

    In the demos we’re showing you how to first build the github-webhook project. That’s because the github-analytics needs the stubs of github-webhook to pass the tests. Below you’ll find references to github-analytics project since it contains more interesting pieces as far as testing -is concerned.

    2.1 Build

    Figure 2.1. Build and upload artifacts

    build

    In this step we’re generating a version of the pipeline, next we’re - running unit, integration and contract tests. Finally we’re:

    • publishing a fat jar of the application
    • publishing a Spring Cloud Contract jar containing stubs of the application
    • [KUBERNETES] uploading a Docker image of the application

    During this phase we’re executing a Maven build using Maven Wrapper or a Gradle build using Gradle Wrapper +is concerned.

    2.1 Build

    Figure 2.1. Build and upload artifacts

    build

    In this step we’re generating a version of the pipeline, next we’re + running unit, integration and contract tests. Finally we’re:

    • publishing a fat jar of the application
    • publishing a Spring Cloud Contract jar containing stubs of the application
    • for Kubernetes - uploading a Docker image of the application

    During this phase we’re executing a Maven build using Maven Wrapper or a Gradle build using Gradle Wrapper , with unit and integration tests. We’re also tagging the repository with dev/${version} format. That way in each subsequent step of the pipeline we’re able to retrieve the tagged version. Also we know exactly which version of the pipeline corresponds to which Git hash.

    Once the artifact got built we’re running API compatibility check.

    • we’re searching for the latest production deployment
    • we’re retrieving the contracts that were used by that deployment
    • from the contracts we’re generating API tests to see if the current implementation is fulfilling the HTTP / messaging contracts that the current production deployment -has defined (we’re checking backward compatibility of the API)

    2.2 Test

    Figure 2.2. Smoke test and rollback test on test environment

    test

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the fat jar from Nexus and we’re uploading it to PaaS. We want the application +has defined (we’re checking backward compatibility of the API)

    2.2 Test

    Figure 2.2. Smoke test and rollback test on test environment

    test

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the fat jar from Nexus and we’re uploading it to PaaS. We want the application to run in isolation (be surrounded by stubs).
    [Tip]Tip

    Currently due to port constraints in Cloud Foundry we cannot run multiple stubbed HTTP services in the cloud so to fix this issue we’re running the application with smoke Spring profile on which you can stub out all HTTP calls to return @@ -168,12 +170,12 @@ release (we’re checking out the tag), we’re downloading the appropri or Docker image for Kubernetes) and we’re uploading it to PaaS. IMPORTANT the old artifact is running against the NEW version of the database.

  • we’re running the old smoke tests against the freshly deployed application surrounded by stubs. If those tests pass then we have a high probability that the application is backwards compatible
  • the default behaviour is that after all of those steps the user can manually click to deploy the -application to a stage environment
  • 2.3 Stage

    Figure 2.3. End to end tests on stage environment

    stage

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the artifact (either JAR for Cloud Foundry or Docker image for Kubernetes) +application to a stage environment

    2.3 Stage

    Figure 2.3. End to end tests on stage environment

    stage

    Here we’re

    • starting a RabbitMQ service in PaaS
    • deploying Eureka infrastructure application to PaaS
    • downloading the artifact (either JAR for Cloud Foundry or Docker image for Kubernetes) from and we’re uploading it to PaaS.

    Next we have a manual step in which:

    • from the checked out code we’re running the tests available under the e2e profile. In the case of GitHub Analytics application we’re sending a HTTP message to GitHub Analytic’s endpoint. Then we’re checking if the received message count has increased.

    The step is manual by default due to the fact that stage environment is often shared between teams and some preparations on databases / infrastructure have to take place before running the tests. -Ideally these step should be fully automatic.

    2.4 Prod

    Figure 2.4. Deployment to production

    prod

    The step to deploy to production is manual but ideally it should be automatic.

    [Important]Important

    This step does deployment to production. On production you would assume +Ideally these step should be fully automatic.

    2.4 Prod

    Figure 2.4. Deployment to production

    prod

    The step to deploy to production is manual but ideally it should be automatic.

    [Important]Important

    This step does deployment to production. On production you would assume that you have the infrastructure running. That’s why before you run this step you must execute a script that will provision the services on the production environment. For Cloud Foundry just call tools/cf-helper.sh setup-prod-infra and @@ -183,18 +185,17 @@ to fulfill the DNS name requirements) e.g. fooService-1-0- structure and project properties.

    3.1 Cloud Foundry project opinions

    We’ve taken the following opinionated decisions for a Cloud Foundry based project:

    • application built using Maven or Gradle wrappers
    • application deployment to Cloud Foundry
    • For Maven (example project):

      • usage of Maven Wrapper
      • settings.xml is parametrized to pass the credentials to push code to Artifactory

        • M2_SETTINGS_REPO_ID - server id for Artifactory / Nexus deployment
        • M2_SETTINGS_REPO_USERNAME - username for Artifactory / Nexus deployment
        • M2_SETTINGS_REPO_PASSWORD - password for Artifactory / Nexus deployment
      • artifacts deployment by ./mvnw clean deploy
      • stubrunner.ids property to retrieve list of collaborators for which stubs should be downloaded
      • repo.with.binaries property - (Injected by the pipeline) will contain the URL to the repo containing binaries (e.g. Artifactory)
      • distribution.management.release.id property - (Injected by the pipeline) ID of the distribution management. Corresponds to server id in settings.xml
      • distribution.management.release.url property - (Injected by the pipeline) Will contain the URL to the repo containing binaries (e.g. Artifactory)
      • running API compatibility tests via the apicompatibility Maven profile
      • latest.production.version property - (Injected by the pipeline) will contain the latest production version for the repo (retrieved from Git tags)
      • running smoke tests on a deployed app via the smoke Maven profile
      • running end to end tests on a deployed app via the e2e Maven profile
    • For Gradle (example project check the gradle/pipeline.gradle file):

      • usage of Gradlew Wrapper
      • deploy task for artifacts deployment
      • REPO_WITH_BINARIES env var - (Injected by the pipeline) will contain the URL to the repo containing binaries (e.g. Artifactory)
      • M2_SETTINGS_REPO_USERNAME env var - Username used to send the binary to the repo containing binaries (e.g. Artifactory)
      • M2_SETTINGS_REPO_PASSWORD env var - Password used to send the binary to the repo containing binaries (e.g. Artifactory)
      • running API compatibility tests via the apiCompatibility task
      • latestProductionVersion property - (Injected by the pipeline) will contain the latest production version for the repo (retrieved from Git tags)
      • running smoke tests on a deployed app via the smoke task
      • running end to end tests on a deployed app via the e2e task
      • groupId task to retrieve group id
      • artifactId task to retrieve artifact id
      • currentVersion task to retrieve the current version
      • stubIds task to retrieve list of collaborators for which stubs should be downloaded

    3.2 Kubernetes project opinions

    We’ve taken the following opinionated decisions for a Cloud Foundry based project:

    • application built using Maven or Gradle wrappers
    • application deployment to Kubernetes
    • The produced Java Docker image needs to allow passing of system properties via SYSTEM_PROPS env variable
    • For Maven (example project):

      • usage of Maven Wrapper
      • settings.xml is parametrized to pass the credentials to push code to Artifactory and Docker repository

        • M2_SETTINGS_REPO_ID - server id for Artifactory / Nexus deployment
        • M2_SETTINGS_REPO_USERNAME - username for Artifactory / Nexus deployment
        • M2_SETTINGS_REPO_PASSWORD - password for Artifactory / Nexus deployment
        • DOCKER_SERVER_ID - server id for Docker image pushing
        • DOCKER_USERNAME - username for Docker image pushing
        • DOCKER_PASSWORD - password for Docker image pushing
        • DOCKER_EMAIL - email for Artifactory / Nexus deployment
      • DOCKER_REGISTRY_URL env var - (Overridable - defaults to DockerHub) URL of the Docker registry
      • DOCKER_REGISTRY_ORGANIZATION - env var containing the organization where your Docker repo lays
      • artifacts and Docker image deployment by ./mvnw clean deploy
      • stubrunner.ids property to retrieve list of collaborators for which stubs should be downloaded
      • repo.with.binaries property - (Injected by the pipeline) will contain the URL to the repo containing binaries (e.g. Artifactory)
      • distribution.management.release.id property - (Injected by the pipeline) ID of the distribution management. Corresponds to server id in settings.xml
      • distribution.management.release.url property - (Injected by the pipeline) Will contain the URL to the repo containing binaries (e.g. Artifactory)
      • deployment.yml contains the Kubernetes deployment descriptor
      • service.yml contains the Kubernetes service descriptor
      • running API compatibility tests via the apicompatibility Maven profile
      • latest.production.version property - (Injected by the pipeline) will contain the latest production version for the repo (retrieved from Git tags)
      • running smoke tests on a deployed app via the smoke Maven profile
      • running end to end tests on a deployed app via the e2e Maven profile
    • For Gradle (example project check the gradle/pipeline.gradle file):

      • usage of Gradlew Wrapper
      • deploy task for artifacts deployment
      • REPO_WITH_BINARIES env var - (Injected by the pipeline) will contain the URL to the repo containing binaries (e.g. Artifactory)
      • M2_SETTINGS_REPO_USERNAME env var - Username used to send the binary to the repo containing binaries (e.g. Artifactory)
      • M2_SETTINGS_REPO_PASSWORD env var - Password used to send the binary to the repo containing binaries (e.g. Artifactory)
      • DOCKER_REGISTRY_URL env var - (Overridable - defaults to DockerHub) URL of the Docker registry
      • DOCKER_USERNAME env var - Username used to send the the Docker image
      • DOCKER_PASSWORD env var - Password used to send the the Docker image
      • DOCKER_EMAIL env var - Email used to send the the Docker image
      • DOCKER_REGISTRY_ORGANIZATION - env var containing the organization where your Docker repo lays
      • deployment.yml contains the Kubernetes deployment descriptor
      • service.yml contains the Kubernetes service descriptor
      • running API compatibility tests via the apiCompatibility task
      • latestProductionVersion property - (Injected by the pipeline) will contain the latest production version for the repo (retrieved from Git tags)
      • running smoke tests on a deployed app via the smoke task
      • running end to end tests on a deployed app via the e2e task
      • groupId task to retrieve group id
      • artifactId task to retrieve artifact id
      • currentVersion task to retrieve the current version
      • stubIds task to retrieve list of collaborators for which stubs should be downloaded

    4. Concourse Pipeline (Cloud Foundry)

    [Important]Important

    In this chapter we assume that you perform deployment of your application to Cloud Foundry PaaS

    The Spring Cloud Pipelines repository contains opinionated Concourse pipeline definition. Those jobs will form an empty pipeline and a -sample, opinionated one that you can use in your company.

    All in all there are the following projects taking part in the whole microservice setup for this demo.

    • Github Analytics - the app that has a REST endpoint and uses messaging. Our business application.
    • Github Webhook - project that emits messages that are used by Github Analytics. Our business application.
    • Eureka - simple Eureka Server. This is an infrastructure application.
    • Github Analytics Stub Runner Boot - Stub Runner Boot server to be used for tests with Github Analytics. Uses Eureka and Messaging. This is an infrastructure application.

    4.1 Step by step

    If you want to just run the demo as far as possible using PCF Dev and Docker Compose

    Below you can find optional steps needed to be taken when you want to customize the pipeline

    4.1.1 Fork repos

    There are 4 apps that are composing the pipeline

    You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

    4.1.2 Start Concourse and Artifactory

    Concourse + Artifactory can be run locally. To do that just execute the +sample, opinionated one that you can use in your company.

    All in all there are the following projects taking part in the whole microservice setup for this demo.

    • Github Analytics - the app that has a REST endpoint and uses messaging. Our business application.
    • Github Webhook - project that emits messages that are used by Github Analytics. Our business application.
    • Eureka - simple Eureka Server. This is an infrastructure application.
    • Github Analytics Stub Runner Boot - Stub Runner Boot server to be used for tests with Github Analytics. Uses Eureka and Messaging. This is an infrastructure application.

    4.1 Step by step

    If you want to just run the demo as far as possible using PCF Dev and Docker Compose

    4.1.1 Fork repos

    There are 4 apps that are composing the pipeline

    You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

    4.1.2 Start Concourse and Artifactory

    Concourse + Artifactory can be run locally. To do that just execute the start.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
     cd spring-cloud-pipelines/concourse
     ./setup_docker_compose.sh
     ./start.sh 192.168.99.100

    The setup_docker_compose.sh script should be executed once only to allow generation of keys.

    The 192.168.99.100 param is an example of an external URL of Concourse -(equal to Docker-Machine ip in this example).

    Then Concourse will be running on port 8080 and Artifactory 8081.

    Deploy the infra JARs to Artifactory

    When Artifactory is running, just execute the tools/deploy-infra.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
    +(equal to Docker-Machine ip in this example).

    Then Concourse will be running on port 8080 and Artifactory 8081.

    Deploy the infra JARs to Artifactory

    When Artifactory is running, just execute the tools/deploy-infra.sh script from this repo.

    git clone https://github.com/spring-cloud/spring-cloud-pipelines
     cd spring-cloud-pipelines/
     ./tools/deploy-infra.sh

    As a result both eureka and stub runner repos will be cloned, built -and uploaded to Artifactory.

    4.1.3 Start PCF Dev

    TIP: You can skip this step if you have CF installed and don’t want to use PCF Dev -The only thing you have to do is to set up spaces.

    [Warning]Warning

    It’s more than likely that you’ll run out of resources when you reach stage step. +and uploaded to Artifactory.

    4.1.3 Start PCF Dev

    [Tip]Tip

    You can skip this step if you have CF installed and don’t want to use PCF Dev +The only thing you have to do is to set up spaces.

    [Warning]Warning

    It’s more than likely that you’ll run out of resources when you reach stage step. Don’t worry! Keep calm and clear some apps from PCF Dev and continue.

    You have to download and start PCF Dev. A link how to do it is available here.

    The default credentials when using PCF Dev are:

    username: user
     password: pass
     email: user
    @@ -207,22 +208,22 @@ cf set-s
     cf create-space pcfdev-stage
     cf set-space-role user pcfdev-org pcfdev-stage SpaceDeveloper
     cf create-space pcfdev-prod
    -cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

    You can also execute the ./tools/cf-helper.sh setup-spaces to do this.

    4.1.4 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

      +cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

    You can also execute the ./tools/cf-helper.sh setup-spaces to do this.

    4.1.4 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

       

    running concourse

      - 

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    4.1.5 Setup your credentials.yml

    The repo comes with credentials-sample-cf.yml which is set up with sample data (most credentials) are set to be applicable for PCF Dev. Copy this file to a new file credentials.yml (the file is added to .gitignore so don’t worry that you’ll push it with your passwords) and edit it as you wish. For our demo just setup:

    • app-url - url pointing to your forked github-webhook repo
    • github-private-key - your private key to clone / tag GitHub repos
    • repo-with-binaries - the IP is set to the defaults for Docker Machine. You should update it to point to your setup

    If you don’t have a Docker Machine just execute ./whats_my_ip.sh script to + 

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    4.1.5 Setup your credentials.yml

    The repo comes with credentials-sample-cf.yml which is set up with sample data (most credentials) are set to be applicable for PCF Dev. Copy this file to a new file credentials.yml (the file is added to .gitignore so don’t worry that you’ll push it with your passwords) and edit it as you wish. For our demo just setup:

    • app-url - url pointing to your forked github-webhook repo
    • github-private-key - your private key to clone / tag GitHub repos
    • repo-with-binaries - the IP is set to the defaults for Docker Machine. You should update it to point to your setup

    If you don’t have a Docker Machine just execute ./whats_my_ip.sh script to get an external IP that you can pass to your repo-with-binaries instead of the default -Docker Machine IP.

    Below you can see what environment variables are required by the scripts. To the right hand side you can see the default values for PCF Dev that we set in the credentials-sample-cf.yml.

    Property NameProperty DescriptionDefault value
     

    BUILD_OPTIONS

    Additional options you would like to pass to the Maven / Gradle build

    PAAS_TEST_API_URL

    The URL to the CF Api for TEST env

    api.local.pcfdev.io

    PAAS_STAGE_API_URL

    The URL to the CF Api for STAGE env

    api.local.pcfdev.io

    PAAS_PROD_API_URL

    The URL to the CF Api for PROD env

    api.local.pcfdev.io

    PAAS_TEST_ORG

    Name of the org for the test env

    pcfdev-org

    PAAS_TEST_SPACE

    Name of the space for the test env

    pcfdev-space

    PAAS_STAGE_ORG

    Name of the org for the stage env

    pcfdev-org

    PAAS_STAGE_SPACE

    Name of the space for the stage env

    pcfdev-space

    PAAS_PROD_ORG

    Name of the org for the prod env

    pcfdev-org

    PAAS_PROD_SPACE

    Name of the space for the prod env

    pcfdev-space

    REPO_WITH_BINARIES

    URL to repo with the deployed jars

    http://192.168.99.100:8081/artifactory/libs-release-local

    M2_SETTINGS_REPO_ID

    The id of server from Maven settings.xml

    artifactory-local

    PAAS_HOSTNAME_UUID

    Additional suffix for the route. In a shared environment the default routes can be already taken

     

    APP_MEMORY_LIMIT

    How much memory should be used by the infra apps (Eureka, Stub Runner etc.)

    256m

    JAVA_BUILDPACK_URL

    The URL to the Java buildpack to be used by CF

    https://github.com/cloudfoundry/java-buildpack.git#v3.8.1

    4.1.6 Build the pipeline

    Log in (e.g. for Concourse running at 192.168.99.100 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    ./login.sh 192.168.99.100

    Next run the command to create the pipeline.

    ./set_pipeline.sh

    Then you’ll create a github-webhook pipeline under the docker alias, using the provided credentials.yml file. -You can override these values in exactly that order (e.g. ./set-pipeline.sh some-project another-target some-other-credentials.yml)

    4.1.7 Run the github-webhook pipeline

      - 

    Figure 4.1. Click Login

    concourse login

      - 

    Figure 4.2. Pick main team

    concourse team main

      - 

    Figure 4.3. Log in with concourse user and changeme password

    concourse user pass

      - 

    Figure 4.4. Your screen should look more or less like this

    concourse pipeline

      - 

    Figure 4.5. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      - 

    Figure 4.6. Click 'generate-version'

    generate version

      - 

    Figure 4.7. Click + sign to start a new build

    run pipeline

      - 

    Figure 4.8. The job is pending

    concourse pending

      - 

    Figure 4.9. Job is pending in the main screen

    job running

      - 

    Figure 4.10. Job is running in the main screen

    running pipeline

    5. Concourse Pipeline (Kubernetes)

    [Important]Important

    In this chapter we assume that you perform deployment of your application +Docker Machine IP.

    Below you can see what environment variables are required by the scripts. To the right hand side you can see the default values for PCF Dev that we set in the credentials-sample-cf.yml.

    Property NameProperty DescriptionDefault value
     

    BUILD_OPTIONS

    Additional options you would like to pass to the Maven / Gradle build

    PAAS_TEST_API_URL

    The URL to the CF Api for TEST env

    api.local.pcfdev.io

    PAAS_STAGE_API_URL

    The URL to the CF Api for STAGE env

    api.local.pcfdev.io

    PAAS_PROD_API_URL

    The URL to the CF Api for PROD env

    api.local.pcfdev.io

    PAAS_TEST_ORG

    Name of the org for the test env

    pcfdev-org

    PAAS_TEST_SPACE

    Name of the space for the test env

    pcfdev-space

    PAAS_STAGE_ORG

    Name of the org for the stage env

    pcfdev-org

    PAAS_STAGE_SPACE

    Name of the space for the stage env

    pcfdev-space

    PAAS_PROD_ORG

    Name of the org for the prod env

    pcfdev-org

    PAAS_PROD_SPACE

    Name of the space for the prod env

    pcfdev-space

    REPO_WITH_BINARIES

    URL to repo with the deployed jars

    http://192.168.99.100:8081/artifactory/libs-release-local

    M2_SETTINGS_REPO_ID

    The id of server from Maven settings.xml

    artifactory-local

    PAAS_HOSTNAME_UUID

    Additional suffix for the route. In a shared environment the default routes can be already taken

     

    APP_MEMORY_LIMIT

    How much memory should be used by the infra apps (Eureka, Stub Runner etc.)

    256m

    JAVA_BUILDPACK_URL

    The URL to the Java buildpack to be used by CF

    https://github.com/cloudfoundry/java-buildpack.git#v3.8.1

    4.1.6 Build the pipeline

    Log in (e.g. for Concourse running at 192.168.99.100 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    ./login.sh 192.168.99.100

    Next run the command to create the pipeline.

    ./set_pipeline.sh

    Then you’ll create a github-webhook pipeline under the docker alias, using the provided credentials.yml file. +You can override these values in exactly that order (e.g. ./set-pipeline.sh some-project another-target some-other-credentials.yml)

    4.1.7 Run the github-webhook pipeline

      + 

    Figure 4.1. Click Login

    concourse login

      + 

    Figure 4.2. Pick main team

    concourse team main

      + 

    Figure 4.3. Log in with concourse user and changeme password

    concourse user pass

      + 

    Figure 4.4. Your screen should look more or less like this

    concourse pipeline

      + 

    Figure 4.5. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      + 

    Figure 4.6. Click 'generate-version'

    generate version

      + 

    Figure 4.7. Click + sign to start a new build

    run pipeline

      + 

    Figure 4.8. The job is pending

    concourse pending

      + 

    Figure 4.9. Job is pending in the main screen

    job running

      + 

    Figure 4.10. Job is running in the main screen

    running pipeline

    5. Concourse Pipeline (Kubernetes)

    [Important]Important

    In this chapter we assume that you perform deployment of your application to Kubernetes PaaS

    The Spring Cloud Pipelines repository contains opinionated Concourse pipeline definition. Those jobs will form an empty pipeline and a sample, opinionated one that you can use in your company.

    All in all there are the following projects taking part in the whole microservice setup for this demo.

    • Github Analytics - the app that has a REST endpoint and uses messaging. Our business application.
    • Github Webhook - project that emits messages that are used by Github Analytics. Our business application.
    • Eureka - simple Eureka Server. This is an infrastructure application.
    • Github Analytics Stub Runner Boot - Stub Runner Boot server to be used for tests with Github Analytics. Uses Eureka and Messaging. This is an infrastructure application.

    5.1 Step by step

    This is a guide for Concourse pipeline.

    If you want to just run the demo as far as possible using PCF Dev and Docker Compose

    5.1.1 Fork repos

    There are 4 apps that are composing the pipeline

    You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

    5.2 Concourse in K8S (Kubernetes)

    The simplest way to deploy Concourse to K8S is to use Helm. @@ -256,7 +257,7 @@ Congratulations. You have just deployed JFrog Artifactory Pro! Default credential for Artifactory: user: admin password: password

    Next, we need to set up the repositories.

    First, access the Artifactory URL and log in with -user, admin and password password.

    Figure 5.1. Click on the Quick Setup

    artifactory quick setup

    Then, click on the Maven setup and click Create.

    Figure 5.2. Create the Maven Repository

    artifactory maven repo

    5.2.2 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

      +user, admin and password password.

    Figure 5.1. Click on the Quick Setup

    artifactory quick setup

    Then, click on the Maven setup and click Create.

    Figure 5.2. Create the Maven Repository

    artifactory maven repo

    5.2.2 Setup the fly CLI

    If you go to Concourse website you should see sth like this:

       

    running concourse

       

    You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

    fly --version

    If fly is properly installed then it should print out the version.

    5.2.3 Setup your credentials.yml

    There is a sample credentials file called credentials-sample-k8s.yml prepared for k8s. You can use it as a base for your credentials.yml.

    To allow the Concourse worker’s spawned container to connect to @@ -265,16 +266,16 @@ auth token.

    To get the contents of CA for GCE just execute

    echo "Visit http://127.0.0.1:8080 to use Concourse"
     $ kubectl port-forward --namespace default $POD_NAME 8080:8080
     Visit http://127.0.0.1:8080 to use Concourse

    Log in (e.g. for Concourse running at 127.0.0.1 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

    $ fly -t k8s login -c http://localhost:8080 -u concourse -p concourse

    Next run the command to create the pipeline.

    $ ./set_pipeline.sh github-webhook k8s credentials-k8s.yml

    5.2.5 Run the github-webhook pipeline

      - 

    Figure 5.3. Click Login

    concourse login

      - 

    Figure 5.4. Pick main team

    concourse team main

      - 

    Figure 5.5. Log in with concourse user and concourse password

    concourse user pass

      - 

    Figure 5.6. Your screen should look more or less like this

    concourse pipeline

      - 

    Figure 5.7. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      - 

    Figure 5.8. Click 'generate-version'

    generate version

      - 

    Figure 5.9. Click + sign to start a new build

    run pipeline

      - 

    Figure 5.10. The job is pending

    concourse pending

      - 

    Figure 5.11. Job is pending in the main screen

    job running

      - 

    Figure 5.12. Job is running in the main screen

    running pipeline

    6. Concourse FAQ

    6.1 Can I use the pipeline for some other repos?

    Sure! Just change the app-url in credentials.yml!

    6.2 Will this work for ANY project out of the box?

    Not really. This is an opinionated pipeline that’s why we took some + 

    Figure 5.3. Click Login

    concourse login

      + 

    Figure 5.4. Pick main team

    concourse team main

      + 

    Figure 5.5. Log in with concourse user and concourse password

    concourse user pass

      + 

    Figure 5.6. Your screen should look more or less like this

    concourse pipeline

      + 

    Figure 5.7. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

    start pipeline

      + 

    Figure 5.8. Click 'generate-version'

    generate version

      + 

    Figure 5.9. Click + sign to start a new build

    run pipeline

      + 

    Figure 5.10. The job is pending

    concourse pending

      + 

    Figure 5.11. Job is pending in the main screen

    job running

      + 

    Figure 5.12. Job is running in the main screen

    running pipeline

    6. Concourse FAQ

    6.1 Can I use the pipeline for some other repos?

    Sure! Just change the app-url in credentials.yml!

    6.2 Will this work for ANY project out of the box?

    Not really. This is an opinionated pipeline that’s why we took some opinionated decisions. Check out the documentation to see what those decisions are.

    6.3 Can I modify this to reuse in my project?

    Sure! It’s open-source! The important thing is that the core part of the logic is written in Bash scripts. That way, in the majority of cases, you could change only the bash scripts without changing the @@ -367,15 +368,15 @@ of Java installation and the others to another one.

    < you’d have to go to Configure of the build step and modify the Git name / email. If you want to set it globally you’ll have to remove the section from the build step and follow these steps to set it globally.

    You can set Git email / user globally like this:

      - 

    Figure 7.1. Click 'Manage Jenkins'

    manage jenkins

      - 

    Figure 7.2. Click 'Configure System'

    configure system

      - 

    Figure 7.3. Fill out Git user information

    git

      + 

    Figure 7.1. Click 'Manage Jenkins'

    manage jenkins

      + 

    Figure 7.2. Click 'Configure System'

    configure system

      + 

    Figure 7.3. Fill out Git user information

    git

       

    Add Jenkins credentials for GitHub

    The scripts will need to access the credential in order to tag the repo.

    You have to set credentials with id: git.

    Below you can find instructions on how to set a credential (e.g. for Cloud Foundry cf-test credential but remember to provide the one with id git).

      - 

    Figure 7.4. Click 'Credentials, System'

    credentials system

      - 

    Figure 7.5. Click 'Global Credentials'

    credentials global

      - 

    Figure 7.6. Click 'Add credentials'

    credentials add

      - 

    Figure 7.7. Fill out the user / password and provide the git credential ID (in this example cf-test)

    credentials example

      + 

    Figure 7.4. Click 'Credentials, System'

    credentials system

      + 

    Figure 7.5. Click 'Global Credentials'

    credentials global

      + 

    Figure 7.6. Click 'Add credentials'

    credentials add

      + 

    Figure 7.7. Fill out the user / password and provide the git credential ID (in this example cf-test)

    credentials example

       

    7.3 Testing Jenkins scripts

    ./gradlew clean build

    [Warning]Warning

    The ran test only checks if your scripts compile.

    7.4 How to work with Jenkins Job DSL plugin

    Check out the tutorial. Provide the link to this repository in your Jenkins installation.

    [Warning]Warning

    Remember that views can be overridden that’s why the suggestion is to contain in one script all the logic needed to build a view for a single project (check out that spring_cloud_views.groovy is building all the spring-cloud views).

    7.5 Docker Image

    If you would like to run the pre-configured Jenkins image somewhere other than your local machine, we @@ -413,35 +414,35 @@ has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and `github-analytics'.

      - 

    Figure 8.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      - 

    Figure 8.2. Click the 'Build with parameters'

    seed run

      - 

    Figure 8.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      - 

    Figure 8.4. This is how the results of seed should look like

    seed built

    8.1.5 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do + 

    Figure 8.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      + 

    Figure 8.2. Click the 'Build with parameters'

    seed run

      + 

    Figure 8.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      + 

    Figure 8.4. This is how the results of seed should look like

    seed built

    8.1.5 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do run it you have to provide some properties. By default we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and github-analytics.

      - 

    Figure 8.5. Click the 'github-webhook' view

    seed views

      - 

    Figure 8.6. Run the pipeline

    pipeline run

      + 

    Figure 8.5. Click the 'github-webhook' view

    seed views

      + 

    Figure 8.6. Run the pipeline

    pipeline run

       

    [Important]Important

    If your build fails on the deploy previous version to stage due to missing jar, that means that you’ve forgotten to clear the tags in your repo. Typically that’s due to the fact that you’ve removed the Artifactory volume with deployed JAR whereas a tag in the repo is still pointing there. Check out this section on how to remove the tag.

      - 

    Figure 8.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

      + 

    Figure 8.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

       

    [Important]Important

    Most likely you will run out of memory so when reaching the stage environment it’s good to kill all apps on test. Check out the FAQ section for more details!

      - 

    Figure 8.8. The full pipeline should look like this

    pipeline finished

      + 

    Figure 8.8. The full pipeline should look like this

    pipeline finished

       

    8.2 Declarative pipeline & Blue Ocean

    You can also use the declarative pipeline approach with the Blue Ocean UI. Here is a step by step guide to run a pipeline via this approach.

    The Blue Ocean UI is available under the blue/ URL. E.g. for Docker Machine based setup http://192.168.99.100:8080/blue.

      - 

    Figure 8.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      - 

    Figure 8.10. Your first run will look like this. Click Run button

    blue 2

      - 

    Figure 8.11. Enter parameters required for the build and click run

    blue 3

      - 

    Figure 8.12. A list of pipelines will be shown. Click your first run.

    blue 4

      - 

    Figure 8.13. State if you want to go to production or not and click Proceed

    blue 5

      - 

    Figure 8.14. The build is in progress…​

    blue 6

      - 

    Figure 8.15. The pipeline is done!

    blue 7

      + 

    Figure 8.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      + 

    Figure 8.10. Your first run will look like this. Click Run button

    blue 2

      + 

    Figure 8.11. Enter parameters required for the build and click run

    blue 3

      + 

    Figure 8.12. A list of pipelines will be shown. Click your first run.

    blue 4

      + 

    Figure 8.13. State if you want to go to production or not and click Proceed

    blue 5

      + 

    Figure 8.14. The build is in progress…​

    blue 6

      + 

    Figure 8.15. The pipeline is done!

    blue 7

       

    [Important]Important

    There is no possibility of restarting pipeline from specific stage, after failure. Please check out this issue for more information

    [Warning]Warning

    Currently there is no way to introduce manual steps in a performant way. Jenkins is blocking an executor when manual step is required. That means that you’ll run out of executors @@ -473,35 +474,35 @@ has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and `github-analytics'.

      - 

    Figure 9.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      - 

    Figure 9.2. Click the 'Build with parameters'

    seed run

      - 

    Figure 9.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      - 

    Figure 9.4. This is how the results of seed should look like

    seed built

    9.1.4 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do + 

    Figure 9.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

    seed click

      + 

    Figure 9.2. Click the 'Build with parameters'

    seed run

      + 

    Figure 9.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

    seed

      + 

    Figure 9.4. This is how the results of seed should look like

    seed built

    9.1.4 Run the github-webhook pipeline

    We already create the seed job for you but you’ll have to run it. When you do run it you have to provide some properties. By default we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global env variables you have to remove them from the seed.

    Anyways, to run the demo just provide in the REPOS var the comma separated list of URLs of the 2 aforementioned forks of github-webhook and github-analytics.

      - 

    Figure 9.5. Click the 'github-webhook' view

    seed views

      - 

    Figure 9.6. Run the pipeline

    pipeline run

      + 

    Figure 9.5. Click the 'github-webhook' view

    seed views

      + 

    Figure 9.6. Run the pipeline

    pipeline run

       

    [Important]Important

    If your build fails on the deploy previous version to stage due to missing jar, that means that you’ve forgotten to clear the tags in your repo. Typically that’s due to the fact that you’ve removed the Artifactory volume with deployed JAR whereas a tag in the repo is still pointing there. Check out this section on how to remove the tag.

      - 

    Figure 9.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

      + 

    Figure 9.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

    pipeline manual

       

    [Important]Important

    Most likely you will run out of memory so when reaching the stage environment it’s good to kill all apps on test. Check out the FAQ section for more details!

      - 

    Figure 9.8. The full pipeline should look like this

    pipeline finished

      + 

    Figure 9.8. The full pipeline should look like this

    pipeline finished

       

    9.2 Declarative pipeline & Blue Ocean

    You can also use the declarative pipeline approach with the Blue Ocean UI. Here is a step by step guide to run a pipeline via this approach.

    The Blue Ocean UI is available under the blue/ URL. E.g. for Docker Machine based setup http://192.168.99.100:8080/blue.

      - 

    Figure 9.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      - 

    Figure 9.10. Your first run will look like this. Click Run button

    blue 2

      - 

    Figure 9.11. Enter parameters required for the build and click run

    blue 3

      - 

    Figure 9.12. A list of pipelines will be shown. Click your first run.

    blue 4

      - 

    Figure 9.13. State if you want to go to production or not and click Proceed

    blue 5

      - 

    Figure 9.14. The build is in progress…​

    blue 6

      - 

    Figure 9.15. The pipeline is done!

    blue 7

      + 

    Figure 9.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

    blue 1

      + 

    Figure 9.10. Your first run will look like this. Click Run button

    blue 2

      + 

    Figure 9.11. Enter parameters required for the build and click run

    blue 3

      + 

    Figure 9.12. A list of pipelines will be shown. Click your first run.

    blue 4

      + 

    Figure 9.13. State if you want to go to production or not and click Proceed

    blue 5

      + 

    Figure 9.14. The build is in progress…​

    blue 6

      + 

    Figure 9.15. The pipeline is done!

    blue 7

       

    [Important]Important

    There is no possibility of restarting pipeline from specific stage, after failure. Please check out this issue for more information

    [Warning]Warning

    Currently there is no way to introduce manual steps in a performant way. Jenkins is blocking an executor when manual step is required. That means that you’ll run out of executors @@ -545,14 +546,14 @@ scripts without changing the whole pipeline.

    git tag -l | xargs -n 1 git push --delete origin

    10.8 I want to provide a different JDK version

    • by default we assume that you have jdk with id jdk8 configured
    • if you want a different one just override JDK_VERSION env var and point to the proper one
    [Tip]Tip

    The docker image comes in with Java installed at /usr/lib/jvm/java-8-openjdk-amd64. You can go to Global Tools and create a JDK with jdk8 id and JAVA_HOME pointing to /usr/lib/jvm/java-8-openjdk-amd64

    To change the default one just follow these steps:

      - 

    Figure 10.1. Click 'Manage Jenkins'

    manage jenkins

      - 

    Figure 10.2. Click 'Global Tool'

    global tool

      - 

    Figure 10.3. Click 'JDK Installations'

    jdk installation

      - 

    Figure 10.4. Fill out JDK Installation with path to your JDK

    jdk

      + 

    Figure 10.1. Click 'Manage Jenkins'

    manage jenkins

      + 

    Figure 10.2. Click 'Global Tool'

    global tool

      + 

    Figure 10.3. Click 'JDK Installations'

    jdk installation

      + 

    Figure 10.4. Fill out JDK Installation with path to your JDK

    jdk

       

    And that’s it!

    10.9 Enable Groovy Token Macro Processing

    With scripted that but if you needed to this manually then this is how to do it:

      - 

    Figure 10.5. Click 'Manage Jenkins'

    manage jenkins

      - 

    Figure 10.6. Click 'Configure System'

    configure system

      - 

    Figure 10.7. Click 'Allow token macro processing'

    groovy token

    10.10 I want deployment to stage and prod be automatic

    No problem, just set the property / env var to true

    • AUTO_DEPLOY_TO_STAGE to automatically deploy to stage
    • AUTO_DEPLOY_TO_PROD to automatically deploy to prod

    10.11 I don’t want to test API compativility

    No problem, just set the API_COMPATIBILITY_STEP_REQUIRED env variable + 

    Figure 10.5. Click 'Manage Jenkins'

    manage jenkins

      + 

    Figure 10.6. Click 'Configure System'

    configure system

      + 

    Figure 10.7. Click 'Allow token macro processing'

    groovy token

    10.10 I want deployment to stage and prod be automatic

    No problem, just set the property / env var to true

    • AUTO_DEPLOY_TO_STAGE to automatically deploy to stage
    • AUTO_DEPLOY_TO_PROD to automatically deploy to prod

    10.11 I don’t want to test API compativility

    No problem, just set the API_COMPATIBILITY_STEP_REQUIRED env variable to false and rerun the seed (you can pick it from the seed job’s properties too).

    10.12 I can’t tag the repo!

    When you get sth like this:

    19:01:44 stderr: remote: Invalid username or password.
     19:01:44 fatal: Authentication failed for 'https://github.com/marcingrzejszczak/github-webhook/'
    @@ -627,14 +628,14 @@ $ kubectl config set-context default-system --cluster=default-cluster --user=def
     $ kubectl config use-context default-system

    11.5 Generate Minikube namespaces

    With the running Minikube cluster we need to generate namespaces. Just execute the ./tools/k8s-helper.sh setup-namespaces to do this.

    12. The demo setup (Cloud Foundry)

    The demo uses 2 applications. Github Webhook and Github analytics code. Below you can -see an image of how these application communicate with each other.

    Figure 12.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

      +see an image of how these application communicate with each other.

    Figure 12.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

       

    For the demo scenario we have two applications. Github Analytics and Github Webhook. Let’s imagine a case where Github is emitting events via HTTP. Github Webhook has an API that could register to such hooks and receive those messages. Once this happens Github Webhook sends a message by RabbitMQ to a channel. Github Analytics is - listening to those messages and stores them in a MySQL database.

    Figure 12.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

      + listening to those messages and stores them in a MySQL database.

    Figure 12.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

       

    Github Analytics has its KPIs (Key Performance Indicators) monitored. In the case -of that application the KPI is number of issues.

    Figure 12.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

      +of that application the KPI is number of issues.

    Figure 12.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

       

    Let’s assume that if we go below the threshold of X issues then an alert should be sent to Slack.

    12.1 Deploying production applications to PCF Dev

    In the real world scenario we wouldn’t want to automatically provision services like RabbitMQ, MySQL or Eureka each time we deploy a new application to production. Typically @@ -683,14 +684,14 @@ scrape_configs: - targets: ['github-analytics-sc-pipelines.cfapps.io']

    A deployed version for the Spring Cloud Pipelines demo is available here

    12.3 Running Grafana on CF

    You can check out Toshiaki Maki’s code on how to automate Prometheus installation on CF.

    Download tarball from https://grafana.com/grafana/download?platform=linux Next set http_port = 8080 in conf/default.ini. Then call

    cf push sc-pipelines-grafana -b binary_buildpack -c './bin/grafana-server web' -m 64m

    The demo is using Grafana Dashboard with ID 2471.

    A deployed version for the Spring Cloud Pipelines demo is available here

    13. The demo setup (Kubernetes)

    The demo uses 2 applications. Github Webhook and Github analytics code. Below you can -see an image of how these application communicate with each other.

    Figure 13.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

      +see an image of how these application communicate with each other.

    Figure 13.1. Github Webhook listens to HTTP calls and sends a message to Github Analytics

    demo

       

    For the demo scenario we have two applications. Github Analytics and Github Webhook. Let’s imagine a case where Github is emitting events via HTTP. Github Webhook has an API that could register to such hooks and receive those messages. Once this happens Github Webhook sends a message by RabbitMQ to a channel. Github Analytics is - listening to those messages and stores them in a MySQL database.

    Figure 13.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

      + listening to those messages and stores them in a MySQL database.

    Figure 13.2. Github Analytics exposes metrics that are polled by Prometheus

    demo metrics

       

    Github Analytics has its KPIs (Key Performance Indicators) monitored. In the case -of that application the KPI is number of issues.

    Figure 13.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

      +of that application the KPI is number of issues.

    Figure 13.3. Grafana alerts Slack over Prometheus metrics

    demo alerting

       

    Let’s assume that if we go below the threshold of X issues then an alert should be sent to Slack.

    13.1 Deploying production applications to Minikube

    In the real world scenario we wouldn’t want to automatically provision services like RabbitMQ, MySQL or Eureka each time we deploy a new application to production. Typically