1. Introducing Jenkins X – Jenkins X

Chapter 1. Introducing Jenkins X

In this chapter, I’ll introduce you to Jenkins X, a tool for creating automated cloud-ready, CI/CD pipelines based on DevOps best practices and leveraging Kubernetes.  I’ll explain some of the key features and characteristics associated with Jenkins X, and how it can simplify creating and delivering software products via automation, targeted development environments, a governed promotion model, and more.

You’ll also become acquainted with the constructs and conventions that Jenkins X provides to streamline software development, centering around an opinionated approach to setup and workflow management.  

Wrapping up the chapter, I’ll tie all of the concepts and workflows together with a high-level view of the overall Jenkins X workflow - from installation to doing development to cleaning up when done. You’ll also learn about some of the Jenkins X command line commands that are used in each part of the workflow.

With this set of knowledge, you’ll have a complete high-level picture of what Jenkins X is and what it provides, and be in a solid position to dive in deeper in the following chapters.

What is Jenkins X?

At a high level, Jenkins X is a framework.  It is a framework with a set of tools and processes that simplifies and automates creation, management, and use of a CI/CD pipeline. And it can be applied in any external or on-prem cloud environment. It allows developers and others to focus on the code being developed, instead of how it is transformed into deliverables and deployments.  It creates a pipeline that becomes the primary tool for getting the code delivered, instead of the pipeline being a barrier to delivery. And it does this by automatically implementing industry best practices and workflows.

The next few sections will drill in on these claims and explore the key functionalities that allow Jenkins X to make good on them.

Cloud-ready Development Environment

What are the best ways to produce cloud-ready software?  An old tenet of software development holds that the development environment should resemble, as close as possible, the same kinds of environments that customers will run the software in. Following through on this idea for developing cloud-ready software, your development environment should be cloud-ready as well.

How does that translate in reality?  You want a development environment that can let you develop in any cloud, spinning up or tearing down environments as you need them.  It needs to provide an easy pipeline for software development, automatically implementing CI/CD and DevOps best practices. And it needs to do this while leveraging best-of-breed applications, such as Jenkins and Kubernetes while keeping the usage simple. Jenkins X provides this kind of development environment. And keeping the usage simple starts from the very beginning - with automation during setup. 

Automation of the hard stuff for setup

Rather than needing to spend lots of time and resources creating CI/CD pipelines and supporting processes, running a single command in Jenkins X can do that for you.  The result is an end-to-end  CI/CD pipeline, complete with git repositories, promotion environments, webhooks to trigger actions, and more – all deployed into a Kubernetes cluster. And you don’t even have to create the cluster – Jenkins X can do that for you as well.  In most cases, all that is really required to bootstrap the process is to download the Jenkins X application – known as “jx” - on a system with access to Kubernetes.

Jenkins X also includes the ability to create projects tailored for any number of common programming environments.  “Quickstart” project templates are included for Go, Python, and many others.  Once setup, the projects can be easily managed from prototype to production via Jenkins X’s managed promotion model.

Managed promotion model with automatic environments

Jenkins X provides a built-in model for code promotion.  It does this by creating several default environments for you – including ones for development (dev), staging, and production out of the box.  These environments are backed with Git repositories and their own pipelines.  Jenkins X automatically manages the environments for you.  In addition, it helps automate the processes around promotion as you evolve the application you are developing.  Jenkins X can also create “Preview environments” as needed, for any changes you make and target for merging. Preview environments allows you to try changed versions of your work in a separate area of your Kubernetes cluster before promoting them via standard pull request mechanisms.  Environments are discussed in detail in chapter #.

Automatic Feedback

Speaking of pull requests, Jenkins X will update pull requests and tracking tickets with comments based on what is occurring in the pipeline and success/failure of events like test builds.  This includes when code is ready to be previewed, promoted, or when pull requests are generated.

In fact, comments you add in pull requests can be used to trigger certain actions and events in the pipeline.

Support for Multiple SCMs

While the pull request construct is generally associated with GitHub, Jenkins X can be used with other source control management systems such as BitBucket for this kind of workflow.

You may have sensed a theme here – automation.  Jenkins X was designed from the ground up to automate all of the stuff that usually bogs down people trying to work with containers, orchestration, etc.  This includes managing Kubernetes, setting up pipelines, getting code through the pipeline, and even installing and updating tools.

Jenkins X achieves this automation, in part, by incorporating some key design characteristics around DevOps into it’s implementation.  These are worth being aware of as you start your journey learning about Jenkins X. I’ll talk about those in the next section.

Key Design Characteristics of Jenkins X

In this section I’ll cover some key design characteristics of Jenkins X.  These don’t describe specific functionality, but factor into how the application provides the functionality.

Jenkins X is opinionated

Jenkins X utilizes a narrow set of tools and workflows to simplify the process of creating and using a CI/CD environment.  Then it connects these tools together in a predefined way.  This saves the user/team from having to choose, get, and install all the tools needed.  It also saves the user/team from having to learn how to use Kubernetes and the other tooling in the pipeline.  In most cases, the overall workflow and automation decisions implemented by Jenkins X remove the need for interacting with the orchestration (and most other) applications.  In practice, the only tooling that most users will need to interact with (other than Jenkins X itself) is the tooling that they would normally use in a development process (such as Git).

It is possible (and even easy in many cases) to override some of the default choices that Jenkins X makes.  For example, while Jenkins X will default to using GitHub for its Git repositories, it is possible to also select and use GitLab or Gitea or BitBucket instances as the Git application.  

Overriding default choices

While you can override certain choices, in some instances, not all functionality provided through Jenkins X is guaranteed to work.  For example, as of the time of this writing, using Gitea as the Git provider does not allow for the full automation with Pull Requests.

Jenkins X uses industry best practices

The processes in Jenkins X follow many of the best practices outlined in the book Accelerate  from the DevOps institute. That book documents extensive research into what works, and what doesn’t, for high-performing DevOps organizations. These include such simple ideas as having everything managed in source control and employing GitOps.  

 

The GitOps functionality in Jenkins X initiates significant workflow events when code is changed in Git.  An example is the promotion of versioned artifacts between the environments we discussed earlier (dev, staging, production).  Throughout the process, versioning of artifacts is done via semantic versioning – another best practice.

Jenkins X runs closer to the cluster

While you might logically think that Jenkins X would leverage the traditional Jenkins application, it doesn’t, at least not any longer.  That approach was tried early on, and, while it worked, it was not optimal and was even problematic in some respects because it required interfacing with an application which wasn’t designed for running in these kind of environments.

Jenkins X now runs closer to the cluster and in a more native Kubernetes way.  It does this by leveraging two tools called Prow and Tekton to produce the pipeline code and run the CI/CD process.  Basically, these tools allow for creating pipelines-as-code that run natively within a Kubernetes cluster - instead of needing a separate tool (like the Jenkins application) to handle that. More about these tools is covered in chapter 2 and at other key points throughout the book.

Static and Serverless Jenkins X

If you are looking at older documentation or examples of using Jenkins X, you may come across the terms “Static Jenkins X” and “Serverless Jenkins X”. “Static” here refers to the use of an actual Jenkins instance in Jenkins X prior to removal of that functionality.  “Serverless” was used to mean the version of Jenkins X utilizing Prow, Tekton, etc. (not using a static Jenkins instance).   Today, since the static option has been removed, we simply refer to the product as Jenkins X and understand that it is always implementing the serverless version.

Fit to Purpose

The functions and approaches that Jenkins X incorporates and implements allow it to be easily tailored for a variety of specific use cases.  For example, most users may use it to do classic CI (continuous integration) workloads such as building artifacts or orchestrating tests.  But it can also be used for highly specialized use cases such as Machine Learning (ML) operations.  (See https://jenkins-x.io/docs/using-jx/creating/create-mlquickstart/ for more information on ML application usage.)

Beyond the basic ideas and goals of Jenkins X, let’s discuss the overall workflow.  Chapter 3 will cover this topic in detail. But before we leave this chapter, I’ll provide a high level overview for you.

Overall Workflow

As mentioned earlier, creation of a full multi-stage development environment can be done with a single command.  Most other key operations for the developer workflow (outside of the actual development) can also be done with single commands.   These all use the Jenkins X command line executable – “jx”.   Figure 1-2 below shows the main parts of the Jenkins X workflow with general references to the jx commands involved. 

 

Figure 1-2. High-level overview of Jenkins X overall workflow

As shown in the figure, the overall flow can be divided into two parts - initial setup (on the left) and the development cycle (on the right).  Though this may look complicated, it’s actually very straightforward once you start using it.   Next, I’ll discuss more details on each of the two main sections.

Setup

As a prerequisite to installing Jenkins X, you need to have some kind of infrastructure that can run Kubernetes (and has Kubernetes installed on it).  This can be any Kubernetes environment – one that is on-prem, or one that is running in an external cloud.  Even minikube is usable (although you may have to jump through some hoops to do that). 

Jenkins X and Minikube

At the time of this writing, minikube is still supported under current versions of Jenkins X. This may change over time as both Jenkins X and minikube evolve.  You should also be aware that it can be problematic and require workarounds for default functionality that Jenkins X relies on, such as the ability to receive webhook payloads from external sources. 

The first step is to install Jenkins X on the system where you will be running your Kubernetes cluster (item 1 in Figure 1-2).   This is typically the only real “bootstrapping” part of the process that you need to do (meaning the only part you can’t use Jenkins X itself for).  You can find instructions for installing Jenkins X at https://jenkins-x.io/getting-started/install/.

Caveats on automatic installation/configuration

The recommended environments for running Jenkins X are kubes (Kubernetes clusters) in the cloud providers.  You can get free instances to try out Jenkins X if you want.  See chapter # on how to get up and running with these.

However, if you are not using a cloud system, there are two caveats regarding the environment that may cause you to have to do some manual installation/configuration. 

  • In order to leverage the GitHub webhook notifications that signal events to proceed in a Jenkins X installation, your system needs to be reachable over the internet.  However, even if you are running on a system that is not normally reachable, there are workarounds for this if you are ok with using them. See chapter # for more info.
  • If needed, Jenkins X will install a VM driver.   This is usually hyperv for Windows or xhyve for Mac OS.   If you wanted to use a different VM driver (such as VMWare or Virtualbox) those would need to be installed manually.

With the jx application installed on the system, you can move on to getting a cluster for your CI/CD pipeline and workflow.  You can either have Jenkins X create a new cluster for you or you can leverage an existing cluster.  Creating a new cluster is done via the “jx create cluster” command (item 2 in Figure 1-2).   If you want to use an existing cluster, it is necessary that it meet the requirements for running Jenkins X.  To assess suitability, Jenkins X provides a “jx compliance” command to run against the cluster.   More info on that can be found in chapter #.

Although you have installed the jx application on the kubernetes system, to get the CI/CD pipeline and related pieces running within the actual cluster, you need to run through the “install in the cluster” process for the related Jenkins X pieces.  Chapter # outlines what happens during this part of the process, but you can essentially think of it as installing a set of client binaries, configuring associated Git repositories, and installing the Jenkins X platform.  By default, if you use the “jx create” command to create a cluster, the install is done on the cluster for you. On the other hand, if you use an existing cluster, you can use the “jx boot” command (item 3 in Figure 1-2) to do the install actions.  There is also an option to the create cluster command (--skip-installation) to not do the installation if you want to do that separately.

jx boot

jx boot is the newest way (as of this writing) to get the needed pieces of Jenkins X put into and running in the cluster.  It essentially sets up the development area (dev environment).  Previously this would be done by the “jx install” command or as part of the “jx create cluster” command.  The jx install command has been deprecated altogether at this point.  

The difference/advantage of “jx boot” is that it is a GitOps-based approach. (See the note explaining GitOps earlier in the chapter if you need more information on that concept.)  Like the former jx install functionality, jx boot will prompt you for needed parameters to get the jx pieces setup in the cluster.  But it will also record them in a file that can be used in a GitOps process to apply the changes automatically to the cluster.  And, because it is stored in a Git repository, it can be used again later to restore things back to that same state if needed.  jx boot is covered in more detail in chapter X.

With the pipeline pieces setup in the cluster, you’re ready to begin working on whatever project you’re interested in.  Similar to the approach with clusters, you can create a new project or import an existing one – via the “jx create <project type>” or “jx import” commands, respectively (item 4 in Figure 1-2).  <project type> can be a limited set of specific types such as “spring”, or it can be of type “quickstart”.  Creating a quickstart project creates a simple project template for many other types such as golang, python, etc., along with the supporting files to create containers around them and run them in a Kubernetes cluster. This process will also establish the connections to the various Git repositories needed to manage the project, as well as creating any needed pipeline jobs or tasks to build and run the project.  Finally, it will also create staging and production environments (shown in the right part of Figure 1-2) – setting up any plumbing required for the overall development workflow.

At this point, you’re finished with setup and ready to start developing.

Development

The development cycle with Jenkins X is a standard Git workflow as far as creating and editing content on the user’s local system.  Once a change is ready to be pushed, the process in Jenkins X (based on best practices we mentioned earlier) will typically be started by creating a pull request.  When a new pull request is created, Jenkins X will automatically create a “preview environment”.  As opposed to the default staging and production environments, preview environments are designed to be temporary.  They are there to be used for testing a development change in a Git branch before it is merged via the pull request.  An existing preview environment, , is updated on each commit to the corresponding branch.  This ensures there is a place to continue testing out commits until the change is merged.  The “jx preview” command (item 7 in Figure 1-2) can be used to manually create or update a preview environment.

So how do Jenkins X environments fit in with the overall workflow?  Think of the environments as an instance of the overall CI/CD pipeline  (item 6 in Figure 1-2) for a particular version of the code (dev, staging, prod, or preview).  That is, when code moves into an environment, there is a Git repository associated with that environment and CI/CD pipeline processes that run based on the code changes.  These provide the verification processes, such as builds, that prove the code is good and ready - to be promoted into the next environment in the chain (if not at production).

Promotion from one environment to another is gated through pull requests (or similar mechanisms depending on the Git provider). Jenkins X keys off of pull requests and updates them with status information as the pipelines run.  The “jx promote” command can also be used to manually promote a version of the code to an environment (item 8 in Figure 1-2).   You can read more about environments and promotion between them in chapter #.

This kind of development workflow cycle can continue via the pull requests and environment promotions until the code is ready to be released.

Once coding is complete, you can clean up your Jenkins X pieces to save on infrastructure use and costs.  I didn’t represent this in the chart, but I’ll briefly talk about the Jenkins X-specific ways to assist with it.

Cleaning up

When you are done with the development work for a particular Jenkins X instance, it’s a good idea to clean up whatever resources you were using - both to reduce costs and to make the infrastructure available for other uses.

Aside from being able to remove the usual Kubernetes pieces via Kubernetes itself, Jenkins X provides the “jx delete” command to delete applications being worked on in the environment.   It also provides the “jx uninstall” command to uninstall the Jenkins X platform from a Kubernetes cluster.  Jenkins X provides an option to keep environments around if you want.

One other note on cleaning up is that Jenkins X runs a job every three hours to remove preview environments for merge requests that have been resolved (merged or closed).  A separate “jx gc” command will run garbage collection right away for resolved preview environments.

That’s a quick explanation from start to finish of the overall workflow typically used in Jenkins X.  As you can see, while there is a lot going on, the amount of setup and interaction required to utilize the extensive CI/CD platform is minimal, requiring only a few separate commands - assuming the default options are accepted.  In future chapters, I’ll delve into the different areas represented by the different commands and processes.

Key Takeaways

Here are the main points to remember from this chapter:

  • Jenkins X is a tool for creating automated cloud-ready, CI/CD pipelines based on DevOps best practices and leveraging Kubernetes
  • Jenkins X heavily leverages automation so only a few basic constructs and steps are generally needed to accomplish your goals
  • Jenkins X provides separate development environments with a promotion model to move content between them
  • Jenkins X is opinionated - using a default set of technologies and implementation decisions to simplify setup and workflow management

At this point, you should have a basic idea of what Jenkins X is and what it can do for you. As noted, Jenkins X depends on Kubernetes and associated cloud-native technologies for its functionality and usage.  A basic understanding of those technologies is helpful (but not required) to better understand the models and environments that Jenkins X leverages and provides.  In the next chapter, I’ll run through the basics of containers, Kubernetes, and the other technologies that Jenkins X is built upon. Feel free to skip those parts if you’re already familiar and comfortable with those.

Finally, there’s one other aspect of Jenkins X though that can be confusing initially - how does it differ from the previous releases of Jenkins? Since it factors into the background for understanding Jenkins X, I’ll also clarify that in the next chapter. Again, feel free to skip over that if you already understand the differences and dive into chapter 3 which will show you how to get “up and running” with Jenkins X.