DEV Community

Cover image for Jenkins nodes are offline
ChienPang Lee
ChienPang Lee

Posted on

Jenkins nodes are offline

Lab had a re-arrangement. We had decided to add a few new machines, sort the inventory and consolidate resources such as NIC cards, hard drives, memory dimms, etc. This also triggered a conscious decision of moving Jenkins service (the Jenkins master) to a new server which was new on pretty much every aspect, including OS. When all power was resumed and CI jobs kicked off, they hung.

I have a stage “hex” that was supposed to run some tests, estimated to be completed within 15 min. After the lab updates, a job was kicked off; it took around 7 hours without ending. As a result of that I blindly aborted it and gave it another shot, hoping it was only a glitch of CI flows. Another hour passed with the latest job while there’s still no good news. It’s a trouble-shooting task then.

Scrolling down to the bottom of job log, I saw:

Still waiting to schedule task
All nodes of label ‘bldsrv_private_200.13_centos9’ are offline
Enter fullscreen mode Exit fullscreen mode

As I looked into the status of the mentioned node of that label, I ran into numerous unsuccessful executions:


Clicking one of the failed executions gave me further information:

The error of executing agent indicated something wrong with the docker container.

Docker Agent [localhost:5000/centos9-jail:latest on tcp://10.32.200.13:4243 ID 7076e0254ff86c26e7d9c6a8b3762fff5d6ae11401d96349e358570234942068]
Enter fullscreen mode Exit fullscreen mode

Indeed we do use Docker Clouds. The configurations were double-checked as mentioned in previous post Jenkins CI the unseen building bricks of parallelization.

[root@bldsrv-200-13 jenkins]# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
     Active: active (running) since Mon 2026-05-04 19:50:42 CST; 2h 2min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 2170586 (dockerd)
      Tasks: 179
     Memory: 837.3M
     CGroup: /system.slice/docker.service
             ├─2170586 /usr/bin/dockerd -H tcp://10.32.200.13:4243 -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
             ├─2171095 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.17.0.2 -container-port 5000 -use-listen-fd
             ├─2171103 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5000 -container-ip 172.17.0.2 -container-port 5000 -use-listen-fd
             ├─2207093 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 3128 -container-ip 172.17.0.3 -container-port 3128 -use-listen-fd
             ├─2207101 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 3128 -container-ip 172.17.0.3 -container-port 3128 -use-listen-fd
             ├─2207395 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.4 -container-port 22 -use-listen-fd
             ├─2207403 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32768 -container-ip 172.17.0.4 -container-port 22 -use-listen-fd
             ├─2207411 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32769 -container-ip 172.17.0.4 -container-port 443 -use-listen-fd
             ├─2207420 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32769 -container-ip 172.17.0.4 -container-port 443 -use-listen-fd
             ├─2207428 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32770 -container-ip 172.17.0.4 -container-port 6900 -use-listen-fd
             └─2207437 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32770 -container-ip 172.17.0.4 -container-port 6900 -use-listen-fd

May 04 21:52:56 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:56.996976118+08:00" level=info msg="Container failed to exit within 10s of signal 37 - using the force" container=c3dfb867eb11eebeecc524654357fbdaa9e7ffef1192c603a9e5ee2e45da887e
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.010293297+08:00" level=info msg="ignoring event" container=deba996363cb8b8e3fb6fa29d6a5fb6546bd4cd69f38ffdca4ea25db8cf14467 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.020578385+08:00" level=info msg="Container failed to exit within 10s of signal 37 - using the force" container=efe16ccd188d15c25840ec693db96fdd09e519fb3ee1795b0f96e364f5df29c6
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.029078650+08:00" level=info msg="ignoring event" container=a569e6c24e8d3080a8205583587ea84137a1988126181029e659e4cb263b48c2 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.062594392+08:00" level=info msg="ignoring event" container=c3dfb867eb11eebeecc524654357fbdaa9e7ffef1192c603a9e5ee2e45da887e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.071636945+08:00" level=info msg="ignoring event" container=efe16ccd188d15c25840ec693db96fdd09e519fb3ee1795b0f96e364f5df29c6 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:59 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:59.609183494+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:52:59 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:59.609542011+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:53:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:53:09.611275126+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:53:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:53:09.611553781+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
Enter fullscreen mode Exit fullscreen mode

Docker daemon was running, listening on correct port which could be connected by Jenkins Clouds config. Note there were errors in the status:

May 04 21:56:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:56:09.617075720+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:56:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:56:09.617843533+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
Enter fullscreen mode Exit fullscreen mode

For Jenkins Clouds to work, it relies on a docker image that has to be pulled and run. In our case, we organized our local image and push it into a local registry.

[root@bldsrv-200-13 jenkins]# docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
localhost:5000/centos9-jail           latest    3b5e86b7aa54   9 hours ago    5.84GB
registry                              2         26b2eb03618e   2 years ago    25.4MB

[root@bldsrv-200-13 jenkins]# docker ps | grep registry
8fccbf02c57e   registry:2                           "/entrypoint.sh /etc…"   10 days ago          Up 2 hours          0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp
Enter fullscreen mode Exit fullscreen mode

We did have our jail image, tagged ‘localhost:5000/centos9-jail’ Meanwhile, the “registry” service was running. Let’s see if our jail image was actually pushed into ‘registry’.

[root@bldsrv-200-13 jenkins]# curl http://localhost:5000/v2/_catalog
{"repositories":["app-jail","centos9-jail"]}
Enter fullscreen mode Exit fullscreen mode

Right on! It’s there, sitting the the belly of registry service. Next question. Can the image be used?

[root@bldsrv-200-13 ~]# docker run --rm -ti localhost:5000/centos9-jail cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
Enter fullscreen mode Exit fullscreen mode

Looked all good. As a matter of fact, Jenkins could also pull and run the image and it’s been attempting to do so, as shown below

root@bldsrv-200-13 jenkins]# docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED              STATUS              PORTS                                                                                                                                   NAMES
f771b48b1fb6   localhost:5000/centos9-jail:latest   "supervisord"            6 seconds ago        Up 5 seconds        22/tcp, 443/tcp, 6900/tcp                                                                                                               inspiring_noether
ed50f88e0e1e   localhost:5000/centos9-jail:latest   "supervisord"            16 seconds ago       Up 15 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               jovial_hawking
3c6cce9c417e   localhost:5000/centos9-jail:latest   "supervisord"            26 seconds ago       Up 25 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               boring_liskov
21ceedb00793   localhost:5000/centos9-jail:latest   "supervisord"            36 seconds ago       Up 35 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_rubin
0d60e410ad0c   localhost:5000/centos9-jail:latest   "supervisord"            46 seconds ago       Up 45 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               vigilant_ptolemy
083a0201dc5b   localhost:5000/centos9-jail:latest   "supervisord"            56 seconds ago       Up 55 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               flamboyant_benz
3271582c18d4   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               heuristic_elion
72372f1a4869   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               optimistic_chebyshev
0f87f9209861   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               fervent_mayer
2542392eacd1   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               admiring_satoshi
0eef81092693   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               nice_hellman
bed3885c4c43   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               vigorous_haibt
bf755d1e458d   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stupefied_hawking
99df541b4c70   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_jackson
0a0737b47f4c   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eager_wescoff
19a58b56547a   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stoic_mayer
7ac55893caaa   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               great_jang
1a507b140b72   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eloquent_bardeen
51276557fda2   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               busy_perlman
44847b167803   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               recursing_saha
979b9a80d68a   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               mystifying_hodgkin
5707acd16c39   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               intelligent_shtern
0ee9f3464321   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               exciting_mcclintock
7121a36795c5   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               optimistic_diffie
b5a6bd81a5fe   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               upbeat_wilson
718a423f9a48   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               unruffled_boyd
bbddb26e1f10   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               angry_cohen
85ce490a3c81   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               adoring_chandrasekhar
961a1450dd79   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sleepy_wing
dfc296d708f6   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               distracted_pascal
ce7ed1b4ae53   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               kind_mclaren
241bb3ffa64f   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               gallant_ramanujan
d62e31687600   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               adoring_greider
1da6532c8f20   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sleepy_black
cbf98eb3fcf4   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               goofy_sanderson
4e65eebe1e50   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               serene_turing
36d67da7a5f2   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               zealous_bose
9a286435dc64   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               inspiring_franklin
2e91567b689e   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               clever_swanson
f42d26c35459   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               jolly_wozniak
aae69071cf05   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sweet_swirles
66d94d544711   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stoic_mclaren
2b84df260fd8   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               confident_agnesi
d93c712cc0db   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               flamboyant_goodall
3c82531ce7c1   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               mystifying_newton
ba44f12b9a35   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eager_gauss
8b8518ed60da   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               nervous_pike
4e99a80bb0f0   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               cool_hypatia
d76c8e79f357   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               kind_curie
569d813a33e3   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               relaxed_darwin
f8e51f13c4ce   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               lucid_visvesvaraya4
33aee7bf5b58   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               nice_lovelace
8d93a012dbf5   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               magical_curie
90c14aebb7ee   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               cool_mirzakhani
f083bbad20f3   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               thirsty_faraday
087195cae4ad   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               suspicious_boyd
f7bdc9eb5218   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               lucid_germain
24a2e44056e0   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               gracious_vaughan
65f0a0f24e30   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               wizardly_proskuriakova
a23734fa562c   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_cannon
a249f9cd2e1c   localhost:5000/centos9-jail:latest   "supervisord"            10 minutes ago       Up 10 minutes       22/tcp, 443/tcp, 6900/tcp                                                                                                               xenodochial_greider
81ab4f2c90f7   localhost:5000/centos9-jail:latest   "supervisord"            10 minutes ago       Up 10 minutes       22/tcp, 443/tcp, 6900/tcp                                                                                                               agitated_poitras
a100b4aef1bf   localhost:5000/centos9-jail          "/sbin/init"             2 hours ago          Up 2 hours          0.0.0.0:32768->22/tcp, [::]:32768->22/tcp, 0.0.0.0:32769->443/tcp, [::]:32769->443/tcp, 0.0.0.0:32770->6900/tcp, [::]:32770->6900/tcp   centos9_cubecos_chienpang_200.13
f9683335cda9   ubuntu/squid:latest                  "entrypoint.sh -f /e…"   2 hours ago          Up 2 hours          0.0.0.0:3128->3128/tcp, [::]:3128->3128/tcp                                                                                             squid
8fccbf02c57e   registry:2                           "/entrypoint.sh /etc…"   10 days ago          Up 2 hours          0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp                                                                                             registry
[root@bldsrv-200-13 jenkins]#
Enter fullscreen mode Exit fullscreen mode

The numerous attempts of container processes matched what we have seen in the above snapshot of Jenkins agents. So far we had a direction that Jenkins Clouds did not work. The backend services, images, registry and Jenkins configurations all seemed to work. Moreover, Jenkins started the containers (many of them) but none could make him happy.

Without further clues, I turned my attention to Jenkins logs.


At this point, we finally knew what Jenkins did not like - the Java versions

Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0
Enter fullscreen mode Exit fullscreen mode

We had a new server, with newly-installed OS (Ubuntu 24.04.4 LTS). Jenkins server itself is also a container service. I have organized it as one of our Makefile target that it’d pull latest jenkins image from Docker hub upon execution.

root@bldsrv-200-10:~# docker exec -ti jenkins ls -lt /usr/share/jenkins/ref/init.groovy.d/
total 0
root@bldsrv-200-10:~# docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED        STATUS        PORTS                                                                                          NAMES
d3bfdf795397   danieldent/nginx-ssl-proxy   "/init"                  11 hours ago   Up 11 hours   80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp                                                nginx-proxy
33b482e7e628   jenkins/jenkins:lts          "/usr/bin/tini -- /u…"   11 hours ago   Up 11 hours   0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp, 0.0.0.0:50000->50000/tcp, [::]:50000->50000/tcp   jenkins
dd02cbd4bf72   aheimsbakk/munin-alpine      "/usr/bin/dumb-init …"   5 weeks ago    Up 11 hours   0.0.0.0:80->80/tcp, [::]:80->80/tcp                                                            munin-server
d9734f81f8d5   registry:2                   "/entrypoint.sh /etc…"   5 weeks ago    Up 3 days     0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp                                                    registry
root@bldsrv-200-10:~# docker inspect --format ''  jenkins
2.555.1

root@bldsrv-200-10:~# docker exec -ti jenkins java --version
openjdk 21.0.10 2026-01-20 LTS
OpenJDK Runtime Environment Temurin-21.0.10+7 (build 21.0.10+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.10+7 (build 21.0.10+7-LTS, mixed mode)
Enter fullscreen mode Exit fullscreen mode

And here is the culprit, the Java version of latest Jenkins image on https://hub.docker.com/r/jenkins/jenkins is 2.555.1, exactly what we have currently. However, the Java version 21 of Jenkins master does not match that of our agent (jail image Java version 17). See below in our worker nodes.

[root@bldsrv-200-13 jenkins]# docker run --rm -ti localhost:5000/centos9-jail:latest java --version
openjdk 17.0.18 2026-01-20 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.18.0.8-2) (build 17.0.18+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.18.0.8-2) (build 17.0.18+8-LTS, mixed mode, sharing)
Enter fullscreen mode Exit fullscreen mode

As the cause becomes clear so is the fix. Either keeping lowering the Java version in jail or bumping it up in the Docker image launched by Jenkins Clouds. For the develop wheel to move forward, the later is preferred.

RUN dnf install -y java-21-openjdk java-21-openjdk-devel
RUN alternatives --set java java-21-openjdk.x86_64
Enter fullscreen mode Exit fullscreen mode

The final dockerfile was updated with an additional line to adhere it to version 21 because other packages, such as maven, in my relatively old Centos8 base image requres Java 17. We’d end up having multiple versions of Java SDK, not obvious which would be the default one.


Four days later, we had a green pipeline again. The impairment was cleared such that dev work could move forward. Agile enough? Not in my eyes. Tech enough? I’d say so.

Read the full technical article here: Jenkins nodes are offline

Top comments (0)