What Is Jenkins and Why Every DevOps Engineer Needs to Understand CI/CD Pipelines

The Problem Jenkins Solves

Let us start with a story every developer knows.

A developer writes code, pushes it to GitHub, and then someone has to manually run mvn compile, mvn test, mvn package one by one to build the application. After that, someone has to manually check the code quality, manually upload the artifact to a storage location, and manually deploy it to a server.

Now imagine ten developers doing this every single day. You can see how quickly things become chaotic.

Jenkins exists precisely to eliminate this manual chaos. It is a tool that sits in the middle of all your other tools and automates the entire sequence from code commit to deployment.

What Is a Jenkins Job?

A Jenkins job is simply a task. Think of it as a set of instructions Jenkins follows whenever you tell it to run.

For a typical Java application, a Jenkins job might do this sequence:

1. Pull the latest code from GitHub

2. Run mvn compile to compile the source

3. Run mvn test to execute the tests

4. Run mvn package to generate a deployable artifact (a WAR file)

Previously, developers and DevOps engineers were running these Maven commands one by one, manually. Jenkins lets you define all of these steps once inside a job, and from that point forward you just hit a button (or Jenkins triggers it automatically) and everything happens on its own.

You can even replace those four separate Maven commands with a single command inside the job:

mvn clean package

Jenkins executes it, and the WAR file gets generated automatically in the Jenkins workspace.

Where Does Jenkins Store Its Files?

Jenkins has a default working directory. Whenever a job runs, all files including cloned repositories and generated artifacts live here:

/var/lib/jenkins/workspace

Inside this workspace directory, Jenkins creates a folder named after each job. So if your job is named my-app-build, the WAR file generated by Maven will land in:

/var/lib/jenkins/workspace/my-app-build/target/my-app.war

The target directory is created by Maven as part of the build process, and it always contains the final artifact.

The Full CI/CD Pipeline Architecture

Now that you understand what a Jenkins job is, let us zoom out and look at the bigger picture.

A complete CI/CD pipeline in a typical DevOps setup looks like this:

Code -> Build -> Test -> Artifact Storage -> Deploy

Here is how each stage maps to a real tool:

And sitting at the center of all of this, coordinating every stage? That is Jenkins.

Jenkins integrates with GitHub. Jenkins integrates with Maven. Jenkins integrates with SonarQube. Jenkins integrates with Nexus. Jenkins integrates with Tomcat on EC2. Jenkins can also integrate with Docker and Kubernetes.

Because Jenkins is constantly integrating all these tools and connecting them into a seamless automated workflow, it earns the name CI server - short for Continuous Integration server.

Understanding CI/CD: What Do Those Letters Actually Mean?

People throw these terms around constantly, so let us be precise.

Continuous Integration (CI)

This is the "build and test" part. Whenever a developer commits code to the repository, Jenkins automatically pulls that code, builds it, and tests it. No human needs to kick off the process manually.

Continuous Integration = Continuously building + continuously testing

Continuous Delivery (CD - Delivery)

When the build passes, the artifact is ready to be deployed. With continuous delivery, a human still makes the final decision to push the deployment. The pipeline delivers the artifact to a staging area, and a person clicks "deploy."

Continuous Deployment (CD - Deployment)

This is the fully automated version. When the build passes and all checks are green, Jenkins automatically deploys the application to the target environment without any human approval.

So the key distinction:

• Continuous Delivery = manual deployment trigger

• Continuous Deployment = automatic deployment, zero human involvement

Together, CI/CD is the automated highway that takes code from a developer's laptop to a running production server.

The Jenkins Pipeline Walkthrough

Let us trace exactly what happens when a developer pushes version 1 of their code to GitHub:

1. The developer pushes version 1 code to GitHub.

2. Jenkins detects the change (via a webhook or polling) and pulls the code.

3. Jenkins runs Maven (mvn clean package) to compile the code and generate a WAR file.

4. Jenkins sends the code to SonarQube for a quality and security scan.

5. Jenkins stores the WAR file in Nexus or S3.

6. Jenkins deploys the WAR file to a Tomcat server running on an EC2 instance.

Now the developer pushes version 2. The entire sequence above runs again, automatically. This is the continuous cycle. This is exactly what DevOps engineers get paid to set up, maintain, and improve.

Why Jenkins Over Everything Else?

Jenkins is open source and completely free. Most companies do not want to pay for proprietary CI tools when Jenkins handles the job perfectly well at no cost.

There are alternatives worth knowing about:

• Bamboo (Atlassian, paid)

• CircleCI (cloud-based)

• GitHub Actions (built into GitHub)

• GitLab CI/CD (built into GitLab)

• Semaphore

• Harness

• Buddy

For Kubernetes deployments specifically, there is also ArgoCD, which is a dedicated GitOps deployment tool. ArgoCD is not a replacement for Jenkins - it is specialized for Kubernetes deployments and is often used alongside Jenkins in modern pipelines.

In AWS specifically, Amazon offers AWS CodePipeline as a managed CI/CD service. This is also worth learning, especially for cloud-native projects.

But in the vast majority of real-world companies that use self-hosted CI/CD, Jenkins is the standard choice.

Key Facts to Remember

Before moving on, lock these facts into your memory because they will come up constantly:

• Jenkins default port: 8080

• Jenkins default workspace path: /var/lib/jenkins/workspace

• Jenkins requires Java to run - without Java installed, Jenkins will not start

• Jenkins is open source - free to use

• Jenkins was previously called Hudson - it was later forked and renamed

• Plugins are central to Jenkins - Jenkins does not natively understand GitHub, Maven, Docker, or SonarQube. Every integration requires a plugin to be installed first.

What Gets Built on Top of This

Over the coming sections, you will learn to install Jenkins, create your first job, write pipelines using Groovy scripting, set up and integrate each of these tools (Tomcat, SonarQube, Nexus), and connect everything into a working end-to-end CI/CD pipeline.

The architecture diagram in your head should now look something like this:

Developer pushes code
         |
       GitHub
         |
       Jenkins  < The main orchestrator
      /  |  |  \
  Maven Sonar Nexus Tomcat/EC2

Jenkins is the brain. Every other tool in this pipeline is a muscle that Jenkins coordinates.

Summary

• A Jenkins job is a task or a set of automated steps.

• Jenkins sits at the center of a CI/CD pipeline and integrates with every tool in the software delivery lifecycle.

• CI (Continuous Integration) means automatically building and testing code on every commit.

• CD (Continuous Deployment) means automatically deploying successful builds to the target environment.

• Jenkins is open source, runs on port 8080, requires Java, and extends its capabilities through plugins.

• The typical pipeline flow is: GitHub -> Maven -> SonarQube -> Nexus/S3 -> Tomcat/EC2.

In the next post, we will install Jenkins on an Amazon Linux 2 EC2 instance.