• Robert Knight's avatar
    Attempt to fix issue where wrong code was sometimes deployed · a6d0ea34
    Robert Knight authored
    This commit fixes an issue with the client's Jenkins pipeline that could
    cause code for a recent but not-the-latest commit to master to be deployed to QA,
    but with the version number corresponding to the latest commit.
    
    ---
    
    The client's Jenkins CI/CD build is split into three phases:
    
     1. Checkout source, build, lint and test
     2. Publish QA release
     3. Publish prod release
    
    Each of these phases was executed in a separate `node` block. Each node
    block defines a task and Jenkins allocates a workspace directory for
    that task. Workspace directories are named after the project, branch and
    concurrent build number (eg. `/data/jenkins/workspace/client_master@2`).
    
    Each workspace directory is "locked" for use by one task at a
    time, but workspace directories are re-used when not locked. New
    workspace directories are allocated as needed when concurrent builds
    happen and all existing directories for a given (project, branch) are
    locked.
    
    If there was only one active build of the client's "master" branch,
    phases (2) and (3) would re-use the same workspace directory and
    everything would work as expected. If however there was a concurrent build,
    phase (2) could end up using a different workspace directory than phase (1):
    
     1. Build A starts phase 1 and uses workspace dir #1
     2. Build B starts phase 1 and uses workspace dir #2
     3. Build A finishes phase 1, freeing workspace #1
     4. Build A starts phase 2, using workspace #1
     5. Build A completes phase 2, freeing workspace #1
     6. Build B completes phase 1 and starts phase 2, using workspace #1
    
    In step (6), build B runs phase 2 (the "Publish QA" step) using the
    workspace created by build A and publishes the code checked out by build
    A with the version number associated with build B.
    
    The fact that this kind of screw-up is even possible shows a deeper
    issue with our current CI setup. Nevertheless, this commit addresses the
    immediate issue by:
    
     1. Putting phases (1) and (2) in the same `node` block so that they are
        guaranteed to use the same workspace directory. There was no benefit
        to making them separate tasks.
    
     2. Adding a `checkout scm` command at the start of phase (3) so that it
        will definitely deploy the same code that was tested and deployed to
        QA in steps (1) and (2)
    
    [1] https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#understanding-syntax
    a6d0ea34
Jenkinsfile 7.57 KB