Single jenkins job for all repositories in a Github organization

Answering my own question:

As several people had pointed out, Jenkins assumes one job per repository. The Github Organization plugin didn't work well because it is clumsy and requires you commit and maintain a Jenkinsfile to each of your repos, which is specifically what I wanted to avoid.

The critical piece of information that I unaware of is that Jenkins has an excellent CLI and REST api to control jobs, and a single job configuration can easily be exported as a simple xml file.

So what I did is setup a the Jenkins job for one of the repos via the Jenkins GUI. Then I wrote a simple REST client that downloads the config.xml for this jobs, and creates or updates the Jenkins jobs for each of the repositories in our Github organization.

The builds automatically get triggered by Github (organization-wide) webhook if the URL matches that of any of the repositories. No special Github Organization plugin is needed.

I am not sure how much of this answer will help you, but I will be happy even if it provides some insight into Jenkins pipelines.

I am elaborating the procedure to follow using Jenkins pipelines, if not now at some point of time you need to move your build and deploy to pipelines for Infrastructure as code.

Starting with Jenkins plugins, the following are mandatory plugins for the procedure that I will be explaining here:

• Github organization - for scanning the organization with multiple repos
• Multi-branch pipeline - for creating pipelines automatically for all the branches/PRs in a repo. This helps to validate feature branches and PR changes.

Jenkins Configuration

1. Create Github organization from the options below:

2. Configure the newly created organization, from the above step. Owner should be your Organization where hundred of repos are available.

also, configure what file and what branches to look into a repo to trigger a build. script path is the file that does the steps (probably build and deploy) for the repos. So all the repos will be detected or shown in Jenkins only if a file with this name is available in the repos.

Jenkins scans the configured organization as per the interval mentioned here. It detects any additions/deletions of repos and also commits. Good to configure numbers of builds to store, as needed.

Git repo/organization configuration

Configure webhooks in github

Configure the events that require notifications to Jenkins.

Branch protection and status checks for PRs

Protecting the branch by enabling proper checks will help to restrict commits from a few sets of people after status checks are passed. This helps to maintain good code quality.

Here is the snapshot that shows the status checks status when a PR is raised. Based on this reviewers will be able to decide for approving the PR.

This link explains in detail about the procedure that I have described here.

https://github.com/gitbucket/gitbucket/wiki/Setup-Jenkins-Multibranch-Pipeline-and-Organization

The standard approach would be to create a new multibranch pipeline which scans your organization for new repositories. Every repository should have a jenkinsfile with the instructions to build. But in general it is also possible to achieve what you are trying on a programmatical way.

What my approach would be:

1. Create a Job Template as config.xml (shell script to run docker to check certain things)
2. Scan GitHub to find new Repositories
3. Create a new jenkins job based on the temnplate (ideally just replace the SCM link to the new location) How-to-create-a-job-using-the-REST-API-and-cURL
4. Run that job

I would use the Folders Plugin to create a folder for this type of jobs.

If that is what you are really trying to do I could elaborate further.