I'd have to guess something in the kernel change from 4.14.134 to 4.14.141, but if you're doing IO on the shared folders from VirtualBox, it could very easily be something in those VirtualBox changes (which would be a shame, because we have to drop support for VirtualBox 5 entirely to go with 6.x guest additions, which is why that revert is in there).

Regarding building locally, I just do this:

$ docker build -t boot2docker/boot2docker --pull --force-rm .
$ docker run --rm boot2docker/boot2docker > boot2docker.iso

@tianon i doubt it, im using the same 6.0.x version of vbox on all machines and b2d versions. Thanks for the info, i’ll do a couple of builds and see if i can pinpoint the exact change that replicates the issue. and report back ofc.

the newer versions are unfortunately unusable. the pageload performance of symfony is on par with docker for mac if not worse. (which means real real bad. like 40 sec bad)

docker run --rm boot2docker/boot2docker > boot2docker.iso

Hmm, this does not seem to work, a clean checkout of 19.03.1 does not build properly:

+ for mirror in $TCL_MIRRORS
+ wget -O usr/local/tce.installed/optional/lzo.tcz http://repo.tinycorelinux.net/10.x/x86_64/tcz/lzo.tcz
--2019-11-27 09:33:09--  http://repo.tinycorelinux.net/10.x/x86_64/tcz/lzo.tcz
Resolving repo.tinycorelinux.net (repo.tinycorelinux.net)... 89.22.99.37
Connecting to repo.tinycorelinux.net (repo.tinycorelinux.net)|89.22.99.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57344 (56K) [text/plain]
Saving to: 'usr/local/tce.installed/optional/lzo.tcz'

     0K                                    100% 18.2M=0.003s

2019-11-27 09:33:09 (18.2 MB/s) - 'usr/local/tce.installed/optional/lzo.tcz' saved [57344/57344]

+ cd usr/local/tce.installed/optional
+ md5sum -c lzo.tcz.md5.txt
lzo.tcz: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
+ cd usr/local/tce.installed/optional
+ md5sum -c lzo.tcz.md5.txt
lzo.tcz: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
Removing intermediate container 8d357f2f1737
The command '/bin/bash -Eeuo pipefail -xc for package in $TCL_PACKAGES; do 		eval 'md5="$TCL_PACKAGE_MD5__'"$(echo "$package" | sed -r 's/[^a-zA-Z0-9]+/_/g')"'"'; 		echo "$md5 *$package" > "usr/local/tce.installed/optional/$package.md5.txt"; 		for mirror in $TCL_MIRRORS; do 			if 			wget -O "usr/local/tce.installed/optional/$package" "$mirror/$TCL_MAJOR/x86_64/tcz/$package" 				&& ( cd usr/local/tce.installed/optional && md5sum -c "$package.md5.txt" ) 			; then 			break; 			fi; 		done; 		( cd usr/local/tce.installed/optional && md5sum -c "$package.md5.txt" ); 		unsquashfs -dest . -force "usr/local/tce.installed/optional/$package"; 		touch "usr/local/tce.installed/${package%.tcz}"; 		touch "usr/local/tce.installed/optional/$package.dep"; 	done; 		tcl-chroot ldconfig; 	for script in usr/local/tce.installed/*; do 		[ -f "$script" ] || continue; 		[ -x "$script" ] || continue; 		tcl-chroot "$script"; 	done; 		{ 		echo '#!/bin/bash -Eeux'; 		echo 'tcl-chroot su -c "tce-load -wicl \"\$@\"" docker -- - "$@"'; 	} > /usr/local/bin/tcl-tce-load; 	chmod +x /usr/local/bin/tcl-tce-load' returned a non-zero code: 1

Correct, that older tag will not build as-is because the TCL artifacts have been updated. I'd recommend instead tweaking the knobs (LINUX_ VERSION, etc) on the tip of the 19.03.x branch instead (or playing with ./update.sh which is what I use to update those checksums).

Same problem here, sadly found this issue only after spending hours reinstalling everything.

UPD: Built v19.03.5 with the only change being LINUX_VERSION=4.14.134, got the same performance problem.

UPD: Built v19.03.5 with LINUX_VERSION=4.14.154 and DOCKER_VERSION 19.03.1, the performance problem is still there. Then I tried the v19.03.1 release of boot2docker without any changes to it and there is no performance problem.

UPD: Tried the v19.03.2 release of boot2docker without any changes to it and got the performance problem again. All above tests were performed using VirtualBox 6.0.2 on the host system.
Next thing I tried was v19.03.2 release of boot2docker again, but this time using VirtualBox 5.2.34 on the host - the problem is still there.

UPD: Some other things I tried:
v19.03.5 with VBOX_VERSION=6.0.2 and Vbox 6 on host - mounting does not work, but this issue is already known
v19.03.5 with VBOX_VERSION=6.0.2 and Vbox 5 on host - mounting does not work
v19.03.5 with VBOX_VERSION=5.2.40 - bad performance
v19.03.5 with VBOX_VERSION=5.2.30 - bad performance

Seems like I'm out of options currently. The only working way is to use the v19.03.1 release of boot2docker. No matter what LINUX_VERSION, DOCKER_VERSION or VBOX_VERSION I use to build boot2docker on the v19.03.5 branch and no matter what the host VirtualBox version is - I get unusable performance.

Thank you for the reply @sspat I am very sory that i havent done what you did, but i simply had no time in the previous months. What i did do with my testing is very similar to results you've got. All of my 30 developers still use 19.03.1 and we are stuck with it for good it seems, especially since Docker itself would be discontinuing the docker-machine project.

We will literally have to spin up a CND cluster or something similar. Which is a complete overkill for this kind of development.

I would hope @tianon has some idea of what we may have missed?