Adding release pipeline

Integrating existing pipeline with a new release pipeline

It is composed by three different groups:

- The first group is a basic build so that we can always be sure which build we will be getting when releasing, followed by the acceptance tests.

- The second group are the releases.

- The last group is the CI image used by the different tasks.
This commit is contained in:
Alberto Rios
2019-08-20 13:56:07 +02:00
committed by Alberto Ríos
parent 60d60b7b71
commit 7b8c877cf0
19 changed files with 599 additions and 61 deletions

View File

@@ -1,12 +1,84 @@
#App Broker CI Pipeline
### Running CI pipeline locally
## Running CI pipeline locally
CircleCI allows pipelines to be run locally, this is very useful development purposes
* Install circle command line tool to run local builds:
```$bash
curl -o /usr/local/bin/circleci https://circle-downloads.s3.amazonaws.com/releases/build_agent_wrapper/circleci && \
chmod +x /usr/local/bin/circleci
```
* Run script to execute pipeline
`ci/scripts/run-local-ci-job.sh`
- Install circle command line tool to run local builds:
```$bash
curl -o /usr/local/bin/circleci https://circle-downloads.s3.amazonaws.com/releases/build_agent_wrapper/circleci && \
chmod +x /usr/local/bin/circleci
```
- Run script to execute pipeline
```$bash
ci/scripts/run-local-ci-job.sh
```
## Running the acceptance tests on Cloud Foundry locally
With the following variables set:
```
API_HOST
API_PORT
USERNAME
PASSWORD
CLIENT_ID
CLIENT_SECRET
DEFAULT_ORG
DEFAULT_SPACE
SKIP_SSL_VALIDATION
```
Run script to execute the acceptance tests suite:
```$bash
ci/scripts/acceptance-tests.sh
```
## Releasing
A release pipeline inspired by the one created by the Spring boot team https://github.com/spring-projects/spring-boot/blob/master/ci/
It is composed by three different groups:
- The first group is a basic build so that we can always be sure which build we will be getting when releasing, followed by the acceptance tests.
- The second group are the releases.
- The last group is the CI image used by the different tasks and the source can be found here: `ci/images/release-ci-image/Dockerfile`.
### Releases
The original pipeline was decomposed into different jobs so that we could recover from each of them manually
### Fly
The pipeline can be run using the following script:
```$bash
$ ./ci/scripts/set-pipeline.sh
```
### Release commands
If you don't want to click, you can trigger each job using the CLI:
To release a milestone:
```$bash
$ fly -t scs trigger-job -j release-test/stage-milestone
$ fly -t scs trigger-job -j release-test/promote-milestone
```
To release an RC:
```$bash
$ fly -t scs trigger-job -j release-test/stage-rc
$ fly -t scs trigger-job -j release-test/promote-rc
```
To release a GA:
```$bash
$ fly -t scs trigger-job -j release-test/stage-release
$ fly -t scs trigger-job -j release-test/promote-release
$ fly -t scs trigger-job -j release-test/distribute-release
$ fly -t scs trigger-job -j release-test/sync-to-maven-central
```

View File

@@ -1,3 +1,8 @@
app-broker-github-url: https://github.com/spring-cloud/spring-cloud-app-broker
app-broker-git-branch-current: master
scs-slack-failure-channel: "#sc-app-broker"
scs-slack-failure-channel: "#sc-app-broker"
bintray-distribution-repo: spring-cloud-app-broker
bintray-package: spring-cloud-app-broker
bintray-repo: jars
bintray-subject: spring
build-name: spring-cloud-app-broker-build
branch: master

View File

@@ -0,0 +1,13 @@
FROM ubuntu:bionic-20181018
RUN apt-get update
RUN apt-get install --no-install-recommends -y ca-certificates net-tools git curl jq
RUN rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /opt/openjdk
ENV PATH $JAVA_HOME/bin:$PATH
RUN mkdir -p /opt/openjdk && \
cd /opt/openjdk && \
curl -L https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u192-b12/OpenJDK8U-jdk_x64_linux_hotspot_8u192b12.tar.gz | tar xz --strip-components=2
ADD https://raw.githubusercontent.com/spring-io/concourse-java-scripts/v0.0.2/concourse-java.sh /opt/

View File

@@ -1,32 +1,57 @@
---
aliases:
- &slack-failure-notification
put: alert
params:
icon_emoji: ":animal:"
username: concourse
channel: {{scs-slack-failure-channel}}
text: <!here> Build <https://scs.ci.springapps.io/builds/$BUILD_ID|$BUILD_NAME> of job $BUILD_JOB_NAME in the $BUILD_PIPELINE_NAME pipeline has failed!
- &slack-failure-notification
put: alert
params:
icon_emoji: ":animal:"
username: concourse
channel: {{scs-slack-failure-channel}}
text: <!here> Build <https://scs.ci.springapps.io/builds/$BUILD_ID|$BUILD_NAME> of job $BUILD_JOB_NAME in the $BUILD_PIPELINE_NAME pipeline has failed!
jobs:
- name: build
- name: build-release-ci-images
plan:
- get: app-broker
- get: ci-images-git-repo
trigger: true
- task: build
file: app-broker/ci/tasks/build.yml
on_failure:
*slack-failure-notification
- put: release-ci-image
params:
build: ci-images-git-repo/ci/images/release-ci-image
- name: build
serial: true
public: true
plan:
- get: release-ci-image
- get: git-repo
trigger: true
- do:
- task: build-project
privileged: true
timeout: 1h30m
image: release-ci-image
file: git-repo/ci/tasks/build-project.yml
- put: artifactory-repo
params: &artifactory-params
repo: libs-snapshot-local
folder: distribution-repository
build_uri: "https://ci.spring.io/teams/${BUILD_TEAM_NAME}/pipelines/${BUILD_PIPELINE_NAME}/jobs/${BUILD_JOB_NAME}/builds/${BUILD_NAME}"
build_number: "${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-${BUILD_NAME}"
disable_checksum_uploads: true
artifact_set:
- include:
- "/**/releasetest-*-docs.zip"
properties:
"zip.type": "docs"
"zip.deployed": "false"
- name: run-acceptance-tests
serial: true
disable_manual_trigger: true
plan:
- get: app-broker
- get: git-repo
passed: [build]
trigger: true
- task: acceptance-tests
file: app-broker/ci/tasks/acceptance-tests.yml
file: git-repo/ci/tasks/acceptance-tests.yml
params:
API_HOST: {{API_HOST}}
API_PORT: {{API_PORT}}
@@ -40,7 +65,176 @@ jobs:
on_failure:
*slack-failure-notification
- name: stage-milestone
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- task: stage
image: release-ci-image
file: git-repo/ci/tasks/stage.yml
params:
RELEASE_TYPE: M
- put: artifactory-repo
params:
<<: *artifactory-params
repo: libs-staging-local
- put: git-repo
params:
repository: stage-git-repo
- name: stage-rc
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- task: stage
image: release-ci-image
file: git-repo/ci/tasks/stage.yml
params:
RELEASE_TYPE: RC
- put: artifactory-repo
params:
<<: *artifactory-params
repo: libs-staging-local
- put: git-repo
params:
repository: stage-git-repo
- name: stage-release
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- task: stage
image: release-ci-image
file: git-repo/ci/tasks/stage.yml
params:
RELEASE_TYPE: RELEASE
- put: artifactory-repo
params:
<<: *artifactory-params
repo: libs-staging-local
- put: git-repo
params:
repository: stage-git-repo
- name: promote-milestone
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- get: artifactory-repo
trigger: false
passed: [stage-milestone]
params:
save_build_info: true
- task: promote
image: release-ci-image
file: git-repo/ci/tasks/promote.yml
params:
RELEASE_TYPE: M
ARTIFACTORY_SERVER: ((artifactory-server))
ARTIFACTORY_USERNAME: ((artifactory-username))
ARTIFACTORY_PASSWORD: ((artifactory-password))
- name: promote-rc
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- get: artifactory-repo
trigger: false
passed: [stage-rc]
params:
save_build_info: true
- task: promote
image: release-ci-image
file: git-repo/ci/tasks/promote.yml
params:
RELEASE_TYPE: RC
ARTIFACTORY_SERVER: ((artifactory-server))
ARTIFACTORY_USERNAME: ((artifactory-username))
ARTIFACTORY_PASSWORD: ((artifactory-password))
- name: promote-release
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- get: artifactory-repo
trigger: false
passed: [stage-release]
params:
save_build_info: true
- task: promote
image: release-ci-image
file: git-repo/ci/tasks/promote.yml
params:
RELEASE_TYPE: RELEASE
ARTIFACTORY_SERVER: ((artifactory-server))
ARTIFACTORY_USERNAME: ((artifactory-username))
ARTIFACTORY_PASSWORD: ((artifactory-password))
BINTRAY_SUBJECT: ((bintray-subject))
BINTRAY_REPO: ((bintray-repo))
BINTRAY_USERNAME: ((bintray-username))
BINTRAY_API_KEY: ((bintray-api-key))
BINTRAY_PACKAGE: ((bintray-package))
BINTRAY_DISTRIBUTION_REPO: ((bintray-distribution-repo))
- name: distribute-release
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- get: artifactory-repo
trigger: false
passed: [promote-release]
params:
save_build_info: true
- task: distribute
image: release-ci-image
file: git-repo/ci/tasks/distribute.yml
params:
ARTIFACTORY_SERVER: ((artifactory-server))
ARTIFACTORY_USERNAME: ((artifactory-username))
ARTIFACTORY_PASSWORD: ((artifactory-password))
BINTRAY_SUBJECT: ((bintray-subject))
BINTRAY_REPO: ((bintray-repo))
BINTRAY_USERNAME: ((bintray-username))
BINTRAY_API_KEY: ((bintray-api-key))
BINTRAY_PACKAGE: ((bintray-package))
BINTRAY_DISTRIBUTION_REPO: ((bintray-distribution-repo))
- name: sync-to-maven-central
serial: true
plan:
- get: release-ci-image
- get: git-repo
trigger: false
- get: artifactory-repo
trigger: false
passed: [distribute-release]
params:
save_build_info: true
- task: sync-to-maven-central
image: release-ci-image
file: git-repo/ci/tasks/sync-to-maven-central.yml
params:
BINTRAY_USERNAME: ((bintray-username))
BINTRAY_API_KEY: ((bintray-api-key))
SONATYPE_USER_TOKEN: ((sonatype-user-token))
SONATYPE_PASSWORD_TOKEN: ((sonatype-user-token-password))
BINTRAY_SUBJECT: ((bintray-subject))
BINTRAY_REPO: ((bintray-repo))
BINTRAY_PACKAGE: ((bintray-package))
resource_types:
- name: artifactory-resource
type: docker-image
source:
repository: springio/artifactory-resource
tag: 0.0.7
- name: slack-notification
type: docker-image
source:
@@ -48,13 +242,42 @@ resource_types:
tag: latest
resources:
- name: app-broker
- name: git-repo
type: git
source:
uri: {{app-broker-github-url}}
branch: {{app-broker-git-branch-current}}
uri: ((github-repo))
username: ((github-username))
password: ((github-password))
branch: ((branch))
ignore_paths: ["ci/images/*"]
- name: ci-images-git-repo
type: git
source:
uri: ((github-repo))
branch: ((branch))
paths: ["ci/images/*"]
- name: release-ci-image
type: docker-image
source:
repository: ((docker-hub-organization))/release-ci-image
username: ((docker-hub-username))
password: ((docker-hub-password))
tag: ((branch))
- name: artifactory-repo
type: artifactory-resource
source:
uri: ((artifactory-server))
username: ((artifactory-username))
password: ((artifactory-password))
build_name: ((build-name))
- name: alert
type: slack-notification
source:
url: {{scs-slack-webhook}}
groups:
- name: "Build"
jobs: ["build", "run-acceptance-tests"]
- name: "Release"
jobs: ["stage-milestone", "stage-rc", "stage-release", "promote-milestone", "promote-rc", "promote-release", "distribute-release", "sync-to-maven-central"]
- name: "CI Images"
jobs: ["build-release-ci-images"]

10
ci/scripts/build-project.sh Executable file
View File

@@ -0,0 +1,10 @@
#!/bin/bash
set -e
# shellcheck source=scripts/common.sh
source "$(dirname "$0")/common.sh"
repository=$(pwd)/distribution-repository
pushd git-repo >/dev/null
./gradlew --no-daemon clean build install -Dmaven.repo.local="${repository}"
popd >/dev/null

3
ci/scripts/common.sh Executable file
View File

@@ -0,0 +1,3 @@
source /opt/concourse-java.sh
export TERM=dumb
setup_symlinks

52
ci/scripts/distribute.sh Executable file
View File

@@ -0,0 +1,52 @@
#!/bin/bash
set -e
# shellcheck source=scripts/common.sh
source "$(dirname "$0")/common.sh"
buildName=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.name')
buildNumber=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.number')
version=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.modules[0].id' | sed 's/.*:.*:\(.*\)/\1/')
echo "Distributing ${buildName}/${buildNumber} to ${BINTRAY_DISTRIBUTION_REPO}"
curl \
-s \
--connect-timeout 240 \
--max-time 2700 \
-u "${ARTIFACTORY_USERNAME}":"${ARTIFACTORY_PASSWORD}" \
-H "Content-type:application/json" \
-d "{\"sourceRepos\": [\"libs-release-local\"], \"targetRepo\" : \"${BINTRAY_DISTRIBUTION_REPO}\", \"async\":\"true\"}" \
-f \
-X \
POST "${ARTIFACTORY_SERVER}/api/build/distribute/${buildName}/${buildNumber}" >/dev/null || {
echo "Failed to promote" >&2
exit 1
}
echo "Waiting for artifacts to be distributed"
WAIT_TIME=20
WAIT_ATTEMPTS=120
artifacts_published=false
retry_counter=0
while [ $artifacts_published == "false" ] && [ $retry_counter -lt $WAIT_ATTEMPTS ]; do
result=$(curl -s -f -u "${BINTRAY_USERNAME}":"${BINTRAY_API_KEY}" https://api.bintray.com/packages/"${BINTRAY_SUBJECT}"/"${BINTRAY_REPO}"/"${BINTRAY_PACKAGE}")
if [ $? -eq 0 ]; then
versions=$(echo "$result" | jq -r '.versions')
exists=$(echo "$versions" | grep "$version" -o || true)
if [ "$exists" = "$version" ]; then
artifacts_published=true
fi
fi
retry_counter=$((retry_counter + 1))
sleep $WAIT_TIME
done
if [[ $artifacts_published == "false" ]]; then
echo "Failed to ditribute"
exit 1
fi
echo "Distribution complete"
echo $version >version/version

39
ci/scripts/promote.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/bin/bash
set -e
# shellcheck source=scripts/common.sh
source "$(dirname "$0")/common.sh"
buildName=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.name')
buildNumber=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.number')
version=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.modules[0].id' | sed 's/.*:.*:\(.*\)/\1/')
if [[ $RELEASE_TYPE == "M" ]]; then
targetRepo="libs-milestone-local"
elif [[ $RELEASE_TYPE == "RC" ]]; then
targetRepo="libs-milestone-local"
elif [[ $RELEASE_TYPE == "RELEASE" ]]; then
targetRepo="libs-release-local"
else
echo "Unknown release type $RELEASE_TYPE" >&2
exit 1
fi
echo "Promoting ${buildName}/${buildNumber} to ${targetRepo}"
curl \
-s \
--connect-timeout 240 \
--max-time 900 \
-u "${ARTIFACTORY_USERNAME}":"${ARTIFACTORY_PASSWORD}" \
-H "Content-type:application/json" \
-d "{\"status\": \"staged\", \"sourceRepo\": \"libs-staging-local\", \"targetRepo\": \"${targetRepo}\"}" \
-f \
-X \
POST "${ARTIFACTORY_SERVER}/api/build/promote/${buildName}/${buildNumber}" >/dev/null || {
echo "Failed to promote" >&2
exit 1
}
echo "Promotion complete"
echo $version >version/version

52
ci/scripts/stage.sh Executable file
View File

@@ -0,0 +1,52 @@
#!/bin/bash
set -e
# shellcheck source=scripts/common.sh
source "$(dirname "$0")/common.sh"
repository=$(pwd)/distribution-repository
pushd git-repo >/dev/null
git fetch --tags --all >/dev/null
popd >/dev/null
git clone git-repo stage-git-repo >/dev/null
pushd stage-git-repo >/dev/null
snapshotVersion=$(awk -F '=' '$1 == "version" { print $2 }' gradle.properties)
if [[ $RELEASE_TYPE == "M" ]]; then
stageVersion=$(get_next_milestone_release $snapshotVersion)
nextVersion=$snapshotVersion
elif [[ $RELEASE_TYPE == "RC" ]]; then
stageVersion=$(get_next_rc_release $snapshotVersion)
nextVersion=$snapshotVersion
elif [[ $RELEASE_TYPE == "RELEASE" ]]; then
stageVersion=$(get_next_release $snapshotVersion)
nextVersion=$(bump_version_number $snapshotVersion)
else
echo "Unknown release type $RELEASE_TYPE" >&2
exit 1
fi
echo "Staging $stageVersion (next version will be $nextVersion)"
sed -i "s/version=$snapshotVersion/version=$stageVersion/" gradle.properties
git config user.name "Spring Buildmaster" >/dev/null
git config user.email "buildmaster@springframework.org" >/dev/null
git add gradle.properties >/dev/null
git commit -m"Release v$stageVersion" >/dev/null
git tag -a "v$stageVersion" -m"Release v$stageVersion" >/dev/null
./gradlew --no-daemon clean build install -Dmaven.repo.local="${repository}"
git reset --hard HEAD^ >/dev/null
if [[ $nextVersion != $snapshotVersion ]]; then
echo "Setting next development version (v$nextVersion)"
sed -i "s/version=$snapshotVersion/version=$nextVersion/" gradle.properties
git add gradle.properties >/dev/null
git commit -m"Next development version (v$nextVersion)" >/dev/null
fi
echo "DONE"
popd >/dev/null

View File

@@ -0,0 +1,21 @@
#!/bin/bash
set -e
buildName=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.name')
buildNumber=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.number')
version=$(cat artifactory-repo/build-info.json | jq -r '.buildInfo.modules[0].id' | sed 's/.*:.*:\(.*\)/\1/')
echo "Syncing ${buildName}/${buildNumber} to Maven Central"
curl \
-s \
--connect-timeout 240 \
--max-time 2700 \
-u "${BINTRAY_USERNAME}":"${BINTRAY_API_KEY}" \
-H "Content-Type: application/json" -d "{\"username\": \"${SONATYPE_USER_TOKEN}\", \"password\": \"${SONATYPE_PASSWORD_TOKEN}\"}" \
-f \
-X \
POST "https://api.bintray.com/maven_central_sync/${BINTRAY_SUBJECT}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/versions/${version}" >/dev/null || {
echo "Failed to sync" >&2
exit 1
}
echo "Sync complete"

View File

@@ -8,10 +8,10 @@ image_resource:
tag: 8
inputs:
- name: app-broker
- name: git-repo
run:
path: app-broker/ci/tasks/acceptance-tests.sh
path: git-repo/ci/scripts/acceptance-tests.sh
params:
API_HOST:

View File

@@ -0,0 +1,15 @@
---
platform: linux
inputs:
- name: git-repo
outputs:
- name: distribution-repository
caches:
- path: maven
- path: gradle
run:
path: bash
args:
- -ec
- |
${PWD}/git-repo/ci/scripts/build-project.sh

View File

@@ -1,15 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
build() {
./gradlew --no-daemon --no-parallel check
}
main() {
pushd "app-broker" > /dev/null
build
popd > /dev/null
}
main

View File

@@ -1,14 +0,0 @@
---
platform: linux
image_resource:
type: docker-image
source:
repository: openjdk
tag: 8
inputs:
- name: app-broker
run:
path: app-broker/ci/tasks/build.sh

19
ci/tasks/distribute.yml Normal file
View File

@@ -0,0 +1,19 @@
---
platform: linux
inputs:
- name: git-repo
- name: artifactory-repo
outputs:
- name: version
params:
ARTIFACTORY_SERVER:
ARTIFACTORY_USERNAME:
ARTIFACTORY_PASSWORD:
BINTRAY_SUBJECT:
BINTRAY_REPO:
BINTRAY_USERNAME:
BINTRAY_API_KEY:
BINTRAY_PACKAGE:
BINTRAY_DISTRIBUTION_REPO:
run:
path: git-repo/ci/scripts/distribute.sh

14
ci/tasks/promote.yml Normal file
View File

@@ -0,0 +1,14 @@
---
platform: linux
inputs:
- name: git-repo
- name: artifactory-repo
outputs:
- name: version
params:
RELEASE_TYPE:
ARTIFACTORY_SERVER:
ARTIFACTORY_USERNAME:
ARTIFACTORY_PASSWORD:
run:
path: git-repo/ci/scripts/promote.sh

14
ci/tasks/stage.yml Normal file
View File

@@ -0,0 +1,14 @@
---
platform: linux
inputs:
- name: git-repo
outputs:
- name: stage-git-repo
- name: distribution-repository
params:
RELEASE_TYPE:
caches:
- path: maven
- path: gradle
run:
path: git-repo/ci/scripts/stage.sh

View File

@@ -0,0 +1,15 @@
---
platform: linux
inputs:
- name: git-repo
- name: artifactory-repo
params:
BINTRAY_REPO:
BINTRAY_SUBJECT:
BINTRAY_USERNAME:
BINTRAY_API_KEY:
BINTRAY_PACKAGE:
SONATYPE_USER_TOKEN:
SONATYPE_PASSWORD_TOKEN:
run:
path: git-repo/ci/scripts/sync-to-maven-central.sh