[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cannot create a new container with docker inside container #9978

Closed
sctb512 opened this issue Mar 20, 2024 · 5 comments · Fixed by #10021
Closed

cannot create a new container with docker inside container #9978

sctb512 opened this issue Mar 20, 2024 · 5 comments · Fixed by #10021
Labels
kind/bug

Comments

@sctb512
Copy link
Member
sctb512 commented Mar 20, 2024 •
edited
Loading

Description

We use two different versions of Golang to build containerd binaries with the same tag.

containerd tag: v1.6.16

  • build with golang 1.18.x
    Create a new container with docker inside the container created by containerd. This works fine.

  • build with golang 1.20.x
    Create a new container with docker inside the container created by containerd. Gets errors:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/10), flags: 0xe: operation not permitted: unknown.

Steps to reproduce the issue

  1. install lxcfs in node
apt-get update
apt-get install lxcfs -y
  1. replace containerd binaries to the binaries built with Golang 1.20.x and restart containerd service
  2. create pod
apiVersion: v1
kind: Pod
metadata:
  name: docker-mount-sort-test-pod
  namespace: default
spec:
  containers:
  - image: ghcr.io/sctb512/docker-test:latest
    imagePullPolicy: Always
    name: docker
    securityContext:
      privileged: true
      runAsNonRoot: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/containerd
      mountPropagation: HostToContainer
      name: containerd-image
    - mountPath: /proc/meminfo
      name: lxcfs-proc-meminfo
      readOnly: true
    - mountPath: /proc/cpuinfo
      name: lxcfs-proc-cpuinfo
      readOnly: true
    - mountPath: /run/containerd
      name: containerd-dir
  volumes:
  - hostPath:
      path: /var/lib/lxc/lxcfs/proc/meminfo
      type: File
    name: lxcfs-proc-meminfo
  - hostPath:
      path: /var/lib/lxc/lxcfs/proc/cpuinfo
      type: File
    name: lxcfs-proc-cpuinfo
  - hostPath:
      path: /run/containerd
      type: Directory
    name: containerd-dir
  - hostPath:
      path: /var/lib/containerd
      type: Directory
    name: containerd-image
  1. Create a new container with docker in docker container.
docker run --rm -it nginx:latest

Describe the results you received and expected

Result:

Create container failed and I get the following errors:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/10), flags: 0xe: operation not permitted: unknown.

Expect: container created successfully.

What version of containerd are you using?

1.6.16

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response
@sctb512
Copy link
Member Author
sctb512 commented Mar 20, 2024

After diving into this issue for many days. We found that this problem is caused by a Golang commit: golang/go@72e77a7

In this commit, Golang changes the sort.Sort() from stable to unstable, which causes the final mounts sorted by containerd to be different.

containerd/pkg/cri/opts/spec_linux.go

Line 120 in 91a68ed

sort.Sort(orderedMounts(mounts))

@sctb512
Copy link
Member Author
sctb512 commented Mar 20, 2024 •
edited
Loading

If we change the sort.Sort() in the containerd source code to sort.Stable(). It works.

containerd/pkg/cri/opts/spec_linux.go

Line 120 in 91a68ed

sort.Sort(orderedMounts(mounts))

And the problem becomes why the /proc/meminfo and /proc/cpuinfo are mounted before /run/containerd causes this problem.

The containerd adds a rbind option for all mounts, which will mount all submountpoints of source.

containerd/pkg/cri/opts/spec_linux.go

Line 178 in 91a68ed

options := []string{"rbind"}

  • If /proc/meminfo and /proc/cpuinfo are mounted before /run/containerd, runc will see that /run/containerd/xxx/rootfs/proc has two submountpoints. This will cause mount -t ptoc proc /proc to fail because the kernel will refuse the mount operation. The corresponding kernel function is mnt_already_visible.
  • If /proc/meminfo and /proc/cpuinfo are mounted after /run/containererd, runc cannot see any submountpoints in /run/containerd/xxx/rootfs/proc. So it can mount procfs successful.

@sctb512
Copy link
Member Author
sctb512 commented Mar 20, 2024 •
edited
Loading

To fix this problem, we can change sort.Sort() to sort.Stable() to keep the same with the binaries built by Golang 1.18. But I am not sure if this will cause any potential problems. And there are also some places that also use sort.Sort(), I'm not sure if they are affected as well.

If we can do this, users as this case should also keep the /run/containerd is mounted before /proc/xxx. The affected mounts are: https://github.com/opencontainers/runc/blob/0abab45c9b97c113ff2cdc16f3a7388444c3fbec/libcontainer/rootfs_linux.go#L583

@sctb512 sctb512 changed the title cannot create a new container with docker inside a container cannot create a new container with docker inside container Mar 20, 2024
@kzys
Copy link
Member
kzys commented Mar 22, 2024

Hmm, this is odd. The pdqsort change was merged in 2022 and our CI's Go version is relatively up-to-date.

golang/go#50154

@sctb512
Copy link
Member Author
sctb512 commented Mar 24, 2024 •
edited
Loading

I found the UT for the order of the mounts, but the test uses sort.Stable() instead of sort.Sort().
https://github.com/containerd/containerd/blob/v1.6.16/pkg/cri/opts/spec_test.go#L44

The E2E test may also not include this scenario where /run/containerd and /proc/xxx are mounted in the container at the same time.

@sctb512 sctb512 mentioned this issue Mar 30, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(cri): fix unexpected order of mounts since go 1.19 sctb512/containerd
2 participants
@kzys @sctb512