[go: nahoru, domu]
Skip to content
This repository has been archived by the owner on Jan 1, 2021. It is now read-only.

error creating overlay mount #1388

Open
kianmeng opened this issue Apr 20, 2019 · 8 comments
Open

error creating overlay mount #1388

kianmeng opened this issue Apr 20, 2019 · 8 comments

Comments

@kianmeng
Copy link

Dockerfile.txt

Result for most recent released versions.

  • v18.09.5 failed
  • v18.09.4 failed
  • v18.09.3 failed
  • v18.09.2 failed
  • v18.09.1 failed
  • v18.09.0 pass
  • v18.06.1-ce pass

When build Docker images with more than 50-plus layers, the build will fail exactly at step 62 as shown below.

Step 62/100 : RUN touch foobar
error creating overlay mount to /mnt/sda1/var/lib/docker/overlay2/83fc9f0b117631bc8a412c13c44db45801ecb056068df501e6c093c7bc893a55-init/merged: no such file or directory
make: *** [Makefile:2: all] Error 1

To reproduce the bug.

docker build --no-cache -t "test-overlay-mount:alpine" -f Dockerfile.txt .
@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Oh this is bizarre -- my host daemon is 18.09.4 and it works fine there, but I can indeed reproduce in boot2docker (and they're both overlay2). I wonder if there's something in boot2docker's kernel that we need to tweak?

Here's the last few lines of docker.log, in case there's something useful in there:

time="2019-04-23T21:42:04.742912991Z" level=info msg="shim reaped" id=73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd 
time="2019-04-23T21:42:04.753079768Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:04.771233513Z" level=warning msg="73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:04.877586177Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e/shim.sock" debug=false pid=8353 
time="2019-04-23T21:42:05.024312033Z" level=info msg="shim reaped" id=d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e 
time="2019-04-23T21:42:05.034812646Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.056345567Z" level=warning msg="d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.171429505Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7/shim.sock" debug=false pid=8462 
time="2019-04-23T21:42:05.312189027Z" level=info msg="shim reaped" id=ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7 
time="2019-04-23T21:42:05.322298730Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.339279397Z" level=warning msg="ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.445777897Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0/shim.sock" debug=false pid=8571 
time="2019-04-23T21:42:05.588702464Z" level=info msg="shim reaped" id=cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0 
time="2019-04-23T21:42:05.599140762Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.619888477Z" level=warning msg="cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.728919978Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8/shim.sock" debug=false pid=8681 
time="2019-04-23T21:42:05.869297886Z" level=info msg="shim reaped" id=fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8 
time="2019-04-23T21:42:05.879415859Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.895837537Z" level=warning msg="fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.963812737Z" level=error msg="error unmounting /mnt/sda1/var/lib/docker/overlay2/74c47a47159d093611f4f37a0a3b2df4d7e7b7a75ed02f3199ef18a517ea65d7-init/merged: invalid argument" storage-driver=overlay2

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

https://github.com/docker/docker.github.io/blob/b7b1e03f0423c6cbc50ac3e242407be14988ebc6/storage/storagedriver/overlayfs-driver.md#how-the-overlay2-driver-works implies we still have a layer limit, but it's much higher than 100 (and moby/moby#26380 talks about getting up to 122 reliably), so I'm not 100% sure what's happening here.

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Nothing in v18.09.0...v18.09.1 jumps out as particularly interesting, so maybe there's something in the kernel between 4.14.79 and 4.14.92 that can help explain what's changed?

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Crap, definitely not the kernel -- just tried doing a build of master on 4.14.79 and it still stops at Step 62. Trying dropping the Docker version back down now.

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Welp, that works fine, so it's narrowed down to being caused by something in docker-archive/docker-ce@v18.09.0...v18.09.1 🤔

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Looks like docker-archive/engine#84 / moby/moby#37993 is probably the most likely candidate (especially since the error message in moby/moby#37970 appears to match the one we're getting exactly).

What I'm still fuzzy on is why this would be causing the error to show up when the PR is literally supposed to be the fix for it. 😕

cc @thaJeztah

(Also not sure why boot2docker is hitting it but my host isn't -- the value of /sys/module/overlay/parameters/index is N on both and they're similar Docker versions.)

@tianon
Copy link
Contributor
tianon commented Apr 23, 2019

Ran https://github.com/moby/moby/blob/2bfa24b39393f3e6e1ec38e1d76004e5a50d109f/contrib/check-config.sh against our current kernel config (for giggles, just in case) and it shows appropriately that we're only missing CONFIG_AUFS_FS and ZFS-related bits, as expected.

@borjator
Copy link
borjator commented May 9, 2019

We are kind of blocked by this. Any leads I can follow to diagnose and try to fix this issue?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kianmeng @tianon @borjator