Speeding up Jenkins build

Some good ideas - here's what I would also consider:

First, try to measure how long each step (cloning, npm install, run tests, ...) takes, and then try to work on improving each step. You can either try to focus on the one that's taking up most of the time, or you can try to get some quick wins before trying to crack the harder nuts.

Some ideas:

• Do you need to do a git clone every time? Maybe consider doing a git pull instead. At the end of the build, do a git reset --hard ; git clean -f -d as the last step to get rid of any uncommitted or locally changed files. I found that doing this is a lot faster than doing a full git clone every time.
• npm install: In line with the above step, if you keep the node_modules folder from the previous build, this step should be a lot quicker as well. There is of the course the risk of updates or removed dependencies, which will still linger around. Many code changes don't require you to reinstall all of the node modules. You might even be able to put in a bit of logic that detects whether the package.json file has changed since the last build, and only run a full npm install in that case.
• I've solved the above by running a nightly build that does a full, clean build: A fresh git clone and a full npm install once per day. All other builds during the day use the previous build state and are a lot quicker as a result of this. If you find that this leads to wrong results, you could schedule the clean build to run several times per day.
• Tune the s3cmd/aws command - I think these can work in a similar way as rsync, which can be really fast and efficient if used the right way. Only try to the git diff route as a last resort. Doing a git pull instead of a git clone might already solve this issue.
• In general, try to understand whether all of the steps need to be run on every build, or whether you can move some of them to other build plans or to other time slots (at night?). If your main purpose of the build plan is to run the unit tests to give quick feedback to developers, then maybe you don't need to zip/copy/deploy the app every time. Or split it up into two build plans, one that runs the tests and returns quickly, and a second one that does all of the deployment. Having this in separate build plans will allow you to be flexible with the scheduling.

These are some initial ideas that I would try as a first step.

Performing a shallow clone speeds up the cloning part. As it does not download all commits.

git clone --depth 1 clone_url...

You should be able to cache node_modules with this plugin: https://plugins.jenkins.io/jobcacher

I created such script to check md5sum of package.json

stage('NPM Build') {  steps {    sh '''    node -v && npm -v    '''    // rm -rf node_modules    sh '''    CACHE_FOLDER=${HOME}/.cache/md5    echo "EXECUTOR_NUMBER: ${EXECUTOR_NUMBER}"    MD5_FILE_NAME=package-json_${EXECUTOR_NUMBER}.md5sum    [ -d ${CACHE_FOLDER} ] || mkdir -p ${CACHE_FOLDER}    ls ${CACHE_FOLDER}    if [ -f ${CACHE_FOLDER}/${MD5_FILE_NAME} ];then      cp ${CACHE_FOLDER}/${MD5_FILE_NAME} ${MD5_FILE_NAME}      md5sum package.json      cat ${MD5_FILE_NAME}      md5sum -c ${MD5_FILE_NAME} || npm ci    else      echo "No md5sum backup"      npm ci    fi    echo "create new md5sum backup"    md5sum package.json    md5sum package.json > ${MD5_FILE_NAME}    cp ${MD5_FILE_NAME} ${CACHE_FOLDER}    '''    sh '''    npm run ngcc    '''    sh '''    npm run build    '''  }}