Unable to start bootstrap cluster on bare metal Ce...

aravindranath · 10-11-2022 11:34 PM

Im trying to install Anthos on baremetal, I followed the QuickStart guide.
When I try to run bmctl create cluster command, I get the following error:

[2022-10-12 06:29:55+0000] Error creating cluster: create kind cluster failed: error creating bootstrap cluster: docker run error: command "docker run --hostname bmctl-control-plane --name bmctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=bmctl --net kind --restart=on-failure:1 --init=false --volume /dev/mapper:/dev/mapper --volume=/root/baremetal/bmctl-workspace/config.toml:/etc/containerd/config.toml --publish=127.0.0.1:42041:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf gcr.io/anthos-baremetal-release/kindest/node:v0.12.0-gke.26-v1.23.10-gke.1000" failed with error: exit status 127

I try the docker command manually and get the following error:

# docker run --hostname bmctl-control-plane --name bmctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=bmctl --net kind --restart=on-failure:1 --init=false --volume /dev/mapper:/dev/mapper --volume=/root/baremetal/bmctl-workspace/config.toml:/etc/containerd/config.toml --publish=127.0.0.1:42041:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf gcr.io/anthos-baremetal-release/kindest/node:v0.12.0-gke.26-v1.23.10-gke.1000

9f2b1e923c9ffbcec97707ee4e9dd45e53cde5f7bb6e168c8104ae2ac90bcf6b
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/entrypoint": stat /usr/local/bin/entrypoint: no such file or directory: unknown.

I have tried on multiple setups and end up with the same error.

OS Version: CentOS Linux release 8.2.2004 (Core)

thebsv

facing the same issue, on a different image tag of the same anthos-baremetal-release image:

I1012 13:27:37.817217 37781 kind.go:303] Start: create a KIND cluster bmctl.
I1012 13:27:38.044617 37781 kind.go:342] Using image registry: "gcr.io/anthos-baremetal-release"
I1012 13:27:38.045745 37781 kind.go:961] Launch containerd restarter.
I1012 13:27:38.098229 37781 kind.go:677] Docker network already exists. Trying to recreate.
I1012 13:27:38.356947 37781 kind.go:745] Detected MTU 1500.
I1012 13:27:38.495194 37781 kind.go:723] Docker network created.
I1012 13:27:38.708865 37781 kind.go:195] Creating cluster "bmctl" ...
I1012 13:27:38.708916 37781 kind.go:195] • Ensuring node image (gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900) ‌🖼‌ ...
I1012 13:27:38.765857 37781 kind.go:195] Image: gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900 present locally
I1012 13:27:38.821137 37781 kind.go:195] ✓ Ensuring node image (gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900) ‌🖼‌
I1012 13:27:38.821152 37781 kind.go:195] • Preparing nodes ‌📦‌ ...
I1012 13:27:40.325813 37781 kind.go:195] ✗ Preparing nodes ‌📦‌
E1012 13:27:40.494641 37781 logs.go:94] "msg"="Failed to bootstrap." "error"="create kind cluster failed: error creating bootstrap cluster: command \"docker run --name bmctl-control-plane --hostname bmctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=bmctl --net kind --restart=on-failure:1 --init=false --volume /dev/mapper:/dev/mapper --volume=/root/baremetal/bmctl-workspace/config.toml:/etc/containerd/config.toml --publish=127.0.0.1:36579:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900\" failed with error: exit status 127"
I1012 13:27:40.494774 37781 kind.go:607] Canceling containerd restarter...
E1012 13:27:40.560559 37781 console.go:110] exit with error ^M
E1012 13:27:40.560635 37781 console.go:110] Error creating cluster: create kind cluster failed: error creating bootstrap cluster: command "docker run --name bmctl-control-plane --hostname bmctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=bmctl --net kind --restart=on-failure:1 --init=false --volume /dev/mapper:/dev/mapper --volume=/root/baremetal/bmctl-workspace/config.toml:/etc/containerd/config.toml --publish=127.0.0.1:36579:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900" failed with error: exit status 127

keab

Does your bootstrap machine have enough disk/memory?

aravindranath

I do not think resources are the problem here:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       449Mi        29Gi       0.0Ki       1.4Gi        30Gi
Swap:            0B          0B          0B
# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs              16G     0   16G   0% /dev
tmpfs                 16G     0   16G   0% /dev/shm
tmpfs                 16G  708K   16G   1% /run
tmpfs                 16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/cl-root   50G  6.1G   44G  13% /
/dev/mapper/cl-home   42G  326M   41G   1% /home
/dev/sda1            976M  130M  780M  15% /boot
tmpfs                3.2G     0  3.2G   0% /run/user/0

keab

Did you prepare your machine per https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/configure-os/centos (and make sure we're using docker 19.03+)?

thebsv

Yes, we have followed the docs and are using docker 19.03+ ; we tried the same using different versions of docker as well however we are still facing the same issue. The issue seems to stem from the fact that the image is missing an entrypoint file, wondering how we can debug the image, tried to exec into it using bash, tried to change the entrypoint to '/bin/sh', however we are still seeing the same error:

exec: "/usr/local/bin/entrypoint": stat /usr/local/bin/entrypoint: no such file or directory

baremetal]# docker run -it gcr.io/anthos-baremetal-release/kindest/node:v0.14.0-gke.11-v1.24.2-gke.1900 bash
ERRO[0000] error waiting for container: context canceled
docker: Error response from daemon: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/entrypoint": stat /usr/local/bin/entrypoint: no such file or directory: unknown.

cozynest

Is there sufficient disk space and memory n your bootstrap machine?

Unable to start bootstrap cluster on bare metal Centos 8.2