Commit a6d0ea34 authored by Robert Knight's avatar Robert Knight

Attempt to fix issue where wrong code was sometimes deployed

This commit fixes an issue with the client's Jenkins pipeline that could
cause code for a recent but not-the-latest commit to master to be deployed to QA,
but with the version number corresponding to the latest commit.

---

The client's Jenkins CI/CD build is split into three phases:

 1. Checkout source, build, lint and test
 2. Publish QA release
 3. Publish prod release

Each of these phases was executed in a separate `node` block. Each node
block defines a task and Jenkins allocates a workspace directory for
that task. Workspace directories are named after the project, branch and
concurrent build number (eg. `/data/jenkins/workspace/client_master@2`).

Each workspace directory is "locked" for use by one task at a
time, but workspace directories are re-used when not locked. New
workspace directories are allocated as needed when concurrent builds
happen and all existing directories for a given (project, branch) are
locked.

If there was only one active build of the client's "master" branch,
phases (2) and (3) would re-use the same workspace directory and
everything would work as expected. If however there was a concurrent build,
phase (2) could end up using a different workspace directory than phase (1):

 1. Build A starts phase 1 and uses workspace dir #1
 2. Build B starts phase 1 and uses workspace dir #2
 3. Build A finishes phase 1, freeing workspace #1
 4. Build A starts phase 2, using workspace #1
 5. Build A completes phase 2, freeing workspace #1
 6. Build B completes phase 1 and starts phase 2, using workspace #1

In step (6), build B runs phase 2 (the "Publish QA" step) using the
workspace created by build A and publishes the code checked out by build
A with the version number associated with build B.

The fact that this kind of screw-up is even possible shows a deeper
issue with our current CI setup. Nevertheless, this commit addresses the
immediate issue by:

 1. Putting phases (1) and (2) in the same `node` block so that they are
    guaranteed to use the same workspace directory. There was no benefit
    to making them separate tasks.

 2. Adding a `checkout scm` command at the start of phase (3) so that it
    will definitely deploy the same code that was tested and deployed to
    QA in steps (1) and (2)

[1] https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#understanding-syntax
parent 53d4d897
......@@ -82,17 +82,16 @@ node {
}
}
}
}
}
if (env.BRANCH_NAME != releaseFromBranch) {
echo "Skipping deployment because ${env.BRANCH_NAME} is not the ${releaseFromBranch} branch"
return
}
// // TESTING
// if (env.BRANCH_NAME != releaseFromBranch) {
// echo "Skipping deployment because ${env.BRANCH_NAME} is not the ${releaseFromBranch} branch"
// return
// }
milestone()
stage('Publish to QA') {
node {
milestone()
stage('Publish to QA') {
qaVersion = pkgVersion + "-${lastCommitHash}"
nodeEnv.inside("-e HOME=${workspace}") {
withCredentials([
......@@ -128,6 +127,7 @@ stage('Publish to QA') {
}
}
}
}
milestone()
stage('Publish') {
......@@ -135,6 +135,8 @@ stage('Publish') {
milestone()
node {
checkout scm
echo "Publishing ${pkgName} v${newPkgVersion} from ${releaseFromBranch} branch."
nodeEnv.inside("-e HOME=${workspace} -e BRANCH_NAME=${env.BRANCH_NAME}") {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment