1. Jenkins : Open source, self contained , automation server to automate project building, testing and Delivery.
Jenkins can be installed on any machine which is having Java RunTime Environment (JRE) installed.
Jenkins is a tool for implementing CI/CD (Continuous Integration - Continuous Delivery)
2. CI/CD : stages are
CI (Continuous Integration ) | (i) Continuous Download (ii) Continuous Build (iii) Continuous Deployment (iv) Continuous Testing |
CD (Continuous Delivery) | (v) Continuous Delivery |
3. Basic Infrastructure for Jenkins : To work on with jenkins we need to follow at least this infrastructure
4. Creating Dev-Instance Environment : It is mainly at Developer side, it contains Continuous Download (From GitHub), For continuous build (Maven) , Continuous Deploy(Jenkins with JRE)
1 | Create and Launch Linux Server (Here choosing , AWS with Ubuntu 20.04) | AWS → EC2 → AMI : Ubuntu 18.04 → Instance Type : t2.micro → Storage : 8GB gp2 SSD → Security Group : SSH(Port 22), http(Port 80) , Custom TCP (Port 8081) → Review and Launch |
2 | Connect to that instance using ssh client | ssh -i <key> <user>@<server-public-ip> |
3 | After connecting, update repository | sudo apt update |
4 | Install JRE - Java RunTime Environment | sudo apt install openjdk-11-jre |
5 | Check the JRE installed - Check version | java -version |
6 | Install git | sudo apt install git -y |
7 | Check the version of git | git -–version |
8 | Install Maven | sudo apt-get install maven -y |
9 | Check the version of maven | mvn -v |
10 | Download and install jenkins | wget https://get.jenkins.io/war-stable/2.361.1/jenkins.war |
11 | Start the jenkins.war file, after this password will shown copy it | java -jar jenkins.war --httpPort=8081 |
12 | Connect to jenkins : Public IP + port | http://ip(public):8081/ |
13 | Once After jenkins opens in browser, paste password , which is copied from terminal, and continue 073ddf902fb54a1bb048954acc421acf → continue |
14 | Select Install suggested plugins |
15 | Create First Admin user | Add username , password , email and continue |
16 | Instance Configuration | Here you can change the port if you wish → save and finish |
17 | Creating a simple Job | (i) Select → Create a job (ii) Name : sample-job (iii) style : Free style Project (iv) Go to Build Environment , add execute shell, add script echo “Hello Jenkins” Save it ! |
18 | Running the job : Go to dashboard, there you can see job name, click the name and in the side select Build now | Build now : click on Build Now , It starts building, Build History : You can see the history & status at Build History , if it’s green build success. Click on the History version to show the console response. |
19 | Logout jenkins and stop/terminate aws ec2 instance |
20 | Reconnect and use, if you terminate the instance then follow from 1st step again | Connect aws ec2 instance, Start jenkins : java -jar jenkins.war --httpPort=8081,after this Open jenkins in Browser : Public IP of server : 8081 |
4. QA-Dev-Instance Environment :
1 | Create and Launch Linux Server (Here choosing , AWS with Ubuntu 18.04) | AWS → EC2 → AMI : Ubuntu 18.04 → Instance Type : t2.micro → Storage : 8GB gp2 SSD → Security Group : SSH(Port 22), http(Port 80) , Custom TCP (Port 8080) → Review and Launch |
2 | Connect to that instance using ssh client | ssh -i <key> <user>@<server-public-ip> |
3 | After connecting, update repository | sudo apt update |
3 | Install tomcat | sudo apt-get install -y tomcat8 |
4 | Install tomcat-admin | Sudo apt-get install -y tomcat-admin |
5 | Checkout tomcat at browser : https:Public ip : 8080 |
6 | Add the user to tomcat Add the text in <tomcat-users> segment : | vim /etc/tomcat9/tomcat-users.xml
<role rolename="manager-gui"/> <user username="tomcat" password="s3cret" roles="manager-gui,manager-script,manager-status"/>
|
7` | Restart tomcat | Sudo systemctl restart tomcat |
5. Infrastructure for production server :
1 | Create and Launch Linux Server (Here choosing , AWS with Ubuntu 20.04) | AWS → EC2 → AMI : Ubuntu 18.04 → Instance Type : t2.micro → Storage : 8GB gp2 SSD → Security Group : SSH(Port 22), http(Port 80) , Custom TCP (Port 8080) → Review and Launch |
2 | Connect to that instance using ssh client | ssh -i <key> <user>@<server-public-ip> |
3 | After connecting, update repository | sudo apt update |
3 | Install tomcat | sudo apt-get install -y tomcat8 |
4 | Install tomcat-admin | Sudo apt-get install -y tomcat-admin |
5 | Checkout tomcat at browser : https:Public ip : 8080 |
6 | Add the user to tomcat Add the text in <tomcat-users> segment : | vim /etc/tomcat9/tomcat-users.xml
<role rolename="manager-gui"/> <user username="tomcat" password="s3cret" roles="manager-gui,manager-script,manager-status"/>
|
7` | Restart tomcat | Sudo systemctl restart tomcat |
6. Jenkins Free style project :
1 | Infrastructure to develop |
|
2 | Dev instance : → Open Jenkins
→ New Job/Item : job-main → Free Style project → Name it save it , open it |
3 | Continuous Download code - github | → Configuration → source code management → Select : Git → Enter Repository URL : <github-repo-url> → Save it |
4 | Continuous build | → go to configure → Build steps : add top level command : package |
5 | Continuous Deploy
Deploy build to QA instance , so
We need to plugin.
The QA instance must have tomcat installed. | → plugin install : Deploy to container manage jenkins → manage plugins → Plugin manager → available → search for it and install
→ Deploy to QA Server using it Private IP with Port : → Goto post build actions → select Deploy war/ear to a container WAR/EAR files : **/*.war Context path : deploy Containers : Username and Password : enter here/save here Container url with port |
6 | Continuous test | → create separate new task/job/item : job-test → run the testing scripts you can download them by using source code management , build, run → for we don't have any automated script, so as a substitube add bash script as echo “Testing Success !”
|
7 | After deploy we need to run test job-test | Testing is done after Deploy right , so after build we have to add next post build add “Build other steps”
Add post build step at job-main : Build other steps Select job-test
|
8 | If the job-test success , the deliver code to production server | artifact/code moved to production from dev-server → prod-server
→ add plugin : copy artifact → at job-main : add post build archive the artifact name **/*.war |
9 | Continuous delivery | → at job-test : add post-build :deploy to container → add tomcat → username and password → prod server private ip : port and save it |
Jenkins Users
7. Users Creation : Jenkins Dashboard → Manage Jenkins → Security → Manage User → Create user,
Then enter username, password, email , it’s done ! (i) test-user , (ii) dev-user
8. Login With other users : logout or just use the jenkins url to login with the new account, by default all new users admin users , so every user admin access it is an issue, ex: test-admin has full then it is issue.
9. Role based authorization strategy plugin : Jenkins Dashboard → Manage Jenkins → Manage plugins → add plugin Role based authorization strategy
(i) Setting up Role based authorization strategy plugin : Manage jenkins → Security → configure Global Security -> Authorization : by default selected logged in use can do anything - deselect it and select the role based strategy , This will add menu Manage and Assign Roles
(ii) Creating Roles and Giving Permissions : Jenkins Dashboard → Manage Jenkins → Security → Manage and Assign Roles → Manage Roles , here we have two sections :
(i) Global Roles , (ii) Item Roles(project roles)
(iii) Creating Global Role (Role to add) : Employee
Add Permissions : (i) Overall : Read , (ii) View : Configure, Create, Delete, Read
By default, the admin role has All permissions.
(iv) Item Role : Add Names(Role to add) with pattern : (i) Developer - dev.* , (ii) Tester - test.*
Add Permissions : (i) Give all permissions at Credentials, job, Run, SCM, Lockable Resources
(v) Assign Roles : Jenkins Dashboard → Manage Jenkins → Security → Manage and Assign Roles
(i) Global roles : add user/group to add : add test-user , dev-user and tick as employee
(i) Item roles : for test-user : tester , for dev-user : developer
(vi) Restart : the Jenkins using http://ip:port/restart
(vii) Verify : if we login with dev-user, we can see only basic things , and items start with name dev, for test-user he can able to see only jobs start with text/name as test
Here, it is all about the Users Creation with role based strategy setup using the plugin.
Jenkins Master-Slave Configuration
10. Jenkins-Master-Slave : Jenkins Server intially limited hardware (ec2-t2.micro), so if we have load more then jenkins server is goes down, everything is collapsed. So here we create a slave jenkins machiene and add it to Jenkins master server.
11. Master-Slave Configuration :
1 | Add New EC2-Machine :
| → Connect to EC2-Machine , → Update the repository : sudo apt update -y → Install JRE with the same version, which is installed on the Master server. : Command : sudo apt install openjdk-11-jre |
2 | Add password for slave machine user | → Check the username : whoami → Change the password : sudo passwd <username> |
3 | Enable password less connection between Master Server to Slave Server | → In slave Machiene go to ssh directory : cd /etc/ssh , → Edit file sshd_config : sudo vim sshd_config → Password authentication change yes : #PasswordAuthentication yes , save wq! → Restart ssh service : sudo service ssh restart |
4 | Connect to slave machiene | ssh <username>@<private-ip> , ask you for password enter it, Then you connected to slave machine. |
5 | Generate ssh key @ Master Machine | Ssh-keygen (don't add any name and passwords, just enter) |
6 | Copy SSH key to slave machine | ssh-copy-id <username>@<private-ip-slave-server> |
7 | Login password less from master to slave | ssh <username>@<private-ip-slave> |
8 | Download slave.jar file in slave machine | sudo wget http://<private-ip-master>:<port>/jnlpJars/slave.jar |
9 | Add Permission to slave.jar as rwxrwxr-- for user to execute slave file | sudo chmod 774 slave.jar or sudo u+x slave.jar |
10 | Create a directory in slave as workspace and move in to directory | Mkdir workspace cd /workspace |
11 | Create Node in Jenkins(Master) | Go to Manage Jenkins → Manage Nodes & Clouds → New Node → Add name : slave1 , add permanent agent - OK → Name remote root directory : /home/username/workspace → Label : slave_one → Launch agent : Launch agent via executing command on the controller → Launch command : ssh <username>@<private-ip-slave> java -jar slave.jar → save it |
12 | Adding jobs to slave machine
Note : sample-slave-job runs on slave machine. Run the job , check log to see process | Go to Dashboard → New Item → new job → → Name : sample-slave-job → Type : Free-style → Build step → shell : echo “Hello Slave ! “ → General : Tick - Restrict where this project can be run → Label Expression : Enter slave label : slave_one |
Jenkins Pipeline
11. Pipeline Job : Implementing the CI-CD from the level of code , The code is created using groovy script,
And this file is also called a jenkins file.
12. Advantages of Pipeline Jobs : if implemented as code, it gives the developers the ability to upload into version controlling systems from where they can edit and review the script.
→ Pipelines can accept interactive human input before continuing with specific stage in CI-CD,
Ex: Before deployment into production into production environment, pipeline script can accept approval from the delivery head and then continue.
→ Pipeline script support complex real time scenario where we can implement conditional statements, loops..etc
Ex: If testing passes, we want to go to delivery.
If testing fails, we want to send automated emails.
13. Script format/syntax for pipeline job:
node(‘master/slave’)
{
Stage (‘Stage in CI-CI;)
{
Groovy code for implementing the stage
}
}
14. Build Pipeline : Build pipeline style has four steps of ci-cd only (i) General, (ii) Build Trigger, (iii) Advanced project options , (iv) pipeline
1 | Install build pipeline plugin | Go to Jenkins Dashboard → Manage Jenkins → Manage plugins → add build pipeline plugin |
2 | Create a new job | → Add Item : pipeline-project → Type : pipeline |
3 | Pipeline steps: Continuous download You can run/build to do single step | node(‘master/slave’) { Stage (‘Continuous download;) { Git 'https://github.com/sunildevops77/maven.git' } } |
3 | Enabling passwordless connection between dev→qa dev→prod | → login to qa and prod → add password for users : sudo passwd <username> → sudo vim /etc/ssh/sshd_config → change #PasswordAuthentication yes , save wq! → sudo service ssh restart or sudo systemctl restart sshd.services → login to dev server , gen ssh key : sudo ssh-keygen → send to qa : ssh-copy-id <qa-username>@<privateip-qa> → send to prod : qa : ssh-copy-id <prod-username>@<privateip-prod> → ask for password entry, done ! you can login without password from dev. |
4 | Adding permission to devserver(others) to write into tomcat(qa) and prod server | Qa-server : cd /var/lib/ Sudo chmod -R 777 tomcat8/ Prod-server : cd /var/lib/ Sudo chmod -R 777 tomcat8/ |
5 | Create build step:
Continuous Download + Continuous Build + Continuous Deploy +
Continuous Testing + Continuous Delivery | node(‘master/slave’) { stage (‘Continuous Download’) { Git '<github-repo/remote-repo-maven-project-repo-url' } stage (‘Continuous Build’) { sh label: '', script: 'mvn package' } stage (‘Continuous Deployment’) { sh label: '', script:'scp /home/ubuntu/.jenkins/workspace/ScriptedPipeline/webapp/target/webapp.war <qa-username>@<qa-private-server-ip>:/var/lib/tomcat8/webapps/qaenv.war' } stage (‘Continuous Testing’) { sh label: '', script: 'echo "Testing Passed"' } stage (‘Continuous Delivery’) { sh label: '', script: 'scp /home/ubuntu/.jenkins/workspace/ScriptedPipeline/webapp/target/webapp.war <prod-server-username>@<prod-server-private-ip>:/var/lib/tomcat8/webapps/prodenv.war' } } |
Jenkins Multibranch pipeline
Developer creates multiple branches for different features , so we have create jenkins file for each branch containing it’s CI/CD stages, along with branches , the jenkins file uploaded to repository(ex: github)
15. Multibranch pipeline : Multi branch pipeline has 7 Configuration steps : (i) General , (ii) Branch Sources , (iii) Build Configuration, (iv) Scan Multibranch Pipeline Triggers , (v) Orphaned Item Strategy , (vi) Appearance , (vii) Health metrics , (viii) Properties
1. Create repository with two branches with jenkins file : https://github.com/sagar-gith/jn-mbp
2. In that repository add Jenkins file with pipeline groovy script with stages
node('master') { stage('Continuous Download') { git 'https://github.com/sunildevops77/maven.git' } stage('Continuous Build') { sh label: '', script: 'mvn package' } stage('Continuous Deployment') { sh label: '', script: 'scp /home/ubuntu/.jenkins/workspace/pipe-line/webapp/target/webapp.war ubuntu@172.31.19.55:/var/lib/tomcat8/webapps/qaenv.war' } stage('Continuous Testing') { sh label: '', script: 'echo "Testing Passed"' } stage('Continuous Delivery') { sh label: '', script: 'scp /home/ubuntu/.jenkins/workspace/pipe-line/webapp/target/webapp.war ubuntu@172.31.17.59:/var/lib/tomcat8/webapps/prodenv.war' } } |
3. Go to jenkins , add Item : jenk-mbp , select Type : multi branch pipeline and click on ,
Next steps (i). Add source : Git , add repository url
(ii). Scan Multibranch Pipeline Triggers: Tick - Periodically if not otherwise run → 1 Min
(iii). Save it, jenkins automatically adds the branches to jobs
(iv). Click on jenkins item : jenk-mbp , there you can see the both branch jobs, click on Each branch, there you can see the jobs running with pipeline using jenkins file.
Jenkins Multibranch pipeline
16. Email Integration : If a job fails. We need to send notification, that mail be Email Notifications
(i) Go to Dashboard > Manage Jenkins > Configure System > Email Notification
(ii) SMTP server :
(iii) SMTP Port :
(iv) Credentials : Email and password
(v) Tick ssl / tls , based on url email client
(vi) Send Test Configuration mail , done !
When the job fails, you will receive the email notification.
17. Build Jobs Periodically : To Build Job Daily or Time basic you can user cron job
Cron Jobs has 5 values :
Minutes | Hour | Day of Month | Month | Dow |
0-59 | 0-23 | 1-31 | 1-12 | 0-6 |
18 . Task - 1: If you want to run a job every day at 10:00PM, then write cron job
Answer : 0 22 * * *
Minutes | Hour | Day of Month | Month | Dow |
0 | 22 | * | * | * |
19. Task - 2 : If you want to run a job every day at 11:00 PM From Monday to Friday only,
then write cron job
Answer : 0 23 * * 1-5
Minutes | Hour | Day of Month | Month | Dow |
0 | 23 | * | * | 1-5 |
20. CronJob Integration with Jenkins :
(i) Select any job, Configure , Go to Build Trigger and select Tick
(ii) If job needs to build/run for every hour : enter at schedule : 0 * * * *, save it.
for every hour the job runs.
21. Wait for approval from Delivery Head before moving to Production :
This is only works with pipeline job
(i) Add Item/New Job : Pipeline-with-delivery-approval , Type : Pipe line
(ii) Go to Pipeline : Add the Groovy Code
(iii) Add stage : wait for interactive input : input 'Waiting for approval'
On before the production Delivery.
node('master') { stage('Continuous Download') { git 'https://github.com/sunildevops77/maven.git' } stage('Continuous Build') { sh label: '', script: 'mvn package' } stage('Continuous Deployment') { sh label: '', script: 'scp /home/ubuntu/.jenkins/workspace/pipe-line/webapp/target/webapp.war ubuntu@172.31.19.55:/var/lib/tomcat8/webapps/qaenv.war' } stage('Continuous Testing') { sh label: '', script: 'echo "Testing Passed"' } stage('Continuous Delivery') { input 'Waiting for approval' sh label: '', script: 'scp /home/ubuntu/.jenkins/workspace/pipe-line/webapp/target/webapp.war ubuntu@172.31.17.59:/var/lib/tomcat8/webapps/prodenv.war' } } |
(iv). Here you can see after Testing phase , it asking for permission, showing Waiting for Approval
(v). You can also specify a particular person(jenkins-user) to approve :
Add this step on before the any step for approve from any specific user
input 'Waiting for approval', submitter:’dev-admin’