Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error creating hard drive during make build-all #77

Open
dprosper opened this issue Oct 21, 2020 · 6 comments
Open

Error creating hard drive during make build-all #77

dprosper opened this issue Oct 21, 2020 · 6 comments
Assignees
@syyang-in-cloud
Labels
bug Something isn't working not reproducible wontfix This will not be worked on
Milestone
v0.3.0

Comments

@dprosper
Copy link

dprosper commented Oct 21, 2020 •
edited
Loading

What's the issue?

I am runing on Mac OS, downloaded the latest docker image and I get the following error while running the make build-all

ibmcloud-image-builder % docker pull syibm/ibmcloud-image-builder
Using default tag: latest
latest: Pulling from syibm/ibmcloud-image-builder
df20fa9351a1: Pull complete 
5ec5dc15ff53: Pull complete 
a7a1cf7e653d: Pull complete 
Digest: sha256:a07f57d0e849293f6923a7dc33d6f701861cfe8c6024dd297f4ae94c2d8e2240
Status: Downloaded newer image for syibm/ibmcloud-image-builder:latest
docker.io/syibm/ibmcloud-image-builder:latest

ibmcloud-image-builder % docker tag  syibm/ibmcloud-image-builder ibmcloud-image-builder

ibmcloud-image-builder % make build-all
/Applications/Xcode.app/Contents/Developer/usr/bin/make build-image DISTRO_NAME=ubuntu DISTRO_VERSION=bionic TYPE=base
docker run --privileged --rm -v `pwd`:/ibmcloud-image-builder ibmcloud-image-builder:latest /bin/bash -c "./packer-build.sh packer/ubuntu/bionic/base"
++ echo packer/ubuntu/bionic/base
++ sed s#packer/##g
+ IMAGE_DIR=ubuntu/bionic/base
++ echo packer/ubuntu/bionic/base
++ sed s#packer/##g
++ tr / -
+ DIR_TO_FILENAME=ubuntu-bionic-base
+ NEW_IMAGE=output-qemu/ibmcloud-ubuntu-bionic-base-amd64-100G.qcow2
+ ENCRYPTED_IMAGE=output-qemu/ibmcloud-encrypted-ubuntu-bionic-base-amd64-100G.qcow2
+ mkdir -p img/ubuntu-bionic-base
+ cp -r packer/ubuntu/bionic/base/ansible packer/ubuntu/bionic/base/http packer/ubuntu/bionic/base/packer.json packer/ubuntu/bionic/base/shell img/ubuntu-bionic-base
+ cd img/ubuntu-bionic-base
+ '[' -n '' ']'
+ SECRET=JustMySimpleSecret
+ [[ -f ~/.ssh/id_rsa.pub ]]
+ mkdir -p ./ssh
+ chmod 0700 ./ssh
+ ssh-keygen -q -t rsa -b 4096 -C [email protected] -N '' -f ./ssh/id_rsa
+ export PACKER_PUBLIC_KEY=./ssh/id_rsa.pub
+ PACKER_PUBLIC_KEY=./ssh/id_rsa.pub
+ export PACKER_PRIVATE_KEY=./ssh/id_rsa
+ PACKER_PRIVATE_KEY=./ssh/id_rsa
++ cat ./ssh/id_rsa.pub
+ public_key='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDTsJTNJeKl63tKdXUzwRxvksqdpsaFk0O178bNUBBiiI7odeH304ViffauxlG7RpOTGudUK/tJGSK5in5vErENfrdyVs9yEIGWpdEsZLyt3wGX3qhCjpyZoBueGN0a5nkykpqfazaNfjhJ7T1p1U3G0a2ofhfh/M0QiqQLBvEwMQenxmwaeHhfB+wpfPBhND/4nFGcBn9shpSkBN50kiGh8Bl/+zh8qtB0vlMOxsxhCNhZ3U3lhs5vrGz3jvYCGVeF+pUIQmvUfI9IGxC9zPPaRClM+fnvK8a+qAX7txGY2l5f2zQzLFI9RU7Ic+4vTBJpO9sVpZh07/qZp/jnvK1hoNc6xu4I3B+HW3H/M2587XaaHK2ZvL5G8oOUVIbzmbXvP0gm+nQCQNWcoovIcLDL5Yzycbmn82IM2bSs3X8CeO3CKV9L1Q2LTZdtlRRi5ybCFyRtQ3EyBz4A09V+LjNjqv/QovQBl8Fe4oVr2YHj1Y1pMqNPh67uBedjMoxsWh+KxxUzEDObjc9Ax9zzkp6h/+Gsh4YhwUmFOzDr2R1gJFWDDWnxkeLT6pz1dccOxlXreg1NR4+1vnk/zAxiFKN7yA3it35b4lFCD2Mn+hTq0Dz75LaWuTk/qhTswLMKTidDIBdU53oG42hW1aF5lW6/kvvfxx+9PpcpDGAsUq54Kw== [email protected]'
+ cat
+ cloud-localds disk-ssh-pub.img user-data
+ ansible-galaxy install geerlingguy.docker
- downloading role 'docker', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-docker/archive/2.9.0.tar.gz
- extracting geerlingguy.docker to /root/.ansible/roles/geerlingguy.docker
- geerlingguy.docker (2.9.0) was installed successfully
+ rm -rf output-qemu
+ PACKER_LOG=0
+ packer build packer.json
qemu: output will be in this color.

==> qemu: Retrieving ISO
==> qemu: Trying https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
==> qemu: Trying https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img?checksum=sha256%3Ac4517e054c398235aa7a09ddcc1db31cd168077049febcc4292ff77fe1e5eab3
bionic-server-cloudimg-amd64.img 342.81 MiB / 342.81 MiB  100.00% 34s01s
==> qemu: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img?checksum=sha256%3Ac4517e054c398235aa7a09ddcc1db31cd168077049febcc4292ff77fe1e5eab3 => /ibmcloud-image-builder/img/ubuntu-bionic-base/packer_cache/b6a683c460d3fffb5de20d59dfc847d8bc255295.iso
==> qemu: Copying hard drive...
==> qemu: Error creating hard drive: QemuImg error: qemu-img: error while writing at byte 0: Invalid argument
==> qemu: Deleting output directory...
Build 'qemu' errored: Error creating hard drive: QemuImg error: qemu-img: error while writing at byte 0: Invalid argument

==> Some builds didn't complete successfully and had errors:
--> qemu: Error creating hard drive: QemuImg error: qemu-img: error while writing at byte 0: Invalid argument

==> Builds finished but no artifacts were created.
make[1]: *** [build-image] Error 1
make: *** [ubuntu-bionic-base] Error 2

The same error is seen if I build the docker image locally first, i.e. make pre-build
@syyang-in-cloud syyang-in-cloud self-assigned this Oct 30, 2020
@syyang-in-cloud
Copy link
Member

Let me try on MacOS, and I will get back to you if I can't reproduce.

@syyang-in-cloud
Copy link
Member

Hmm... it's been almost half year that I'm running from Mac ... here's make pre-build

Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$ make pre-build
docker build . -f Dockerfile -t ibmcloud-image-builder:latest
Sending build context to Docker daemon  18.55GB
Step 1/3 : FROM alpine:latest
latest: Pulling from library/alpine
188c0c94c7c5: Pull complete
Digest: sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Status: Downloaded newer image for alpine:latest
 ---> d6e46aa2470d
Step 2/3 : RUN set -ex         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories         && apk update         && apk add --no-cache --virtual .build-python         && apk add bash ansible libffi-dev openssl-dev openssh qemu qemu-system-x86_64 nettle qemu-img ifupdown-ng iproute2-minimal cloud-init cloud-utils         && wget https://releases.hashicorp.com/packer/1.6.0/packer_1.6.0_linux_amd64.zip         && unzip packer_1.6.0_linux_amd64.zip         && chmod +x packer         && rm packer_1.6.0_linux_amd64.zip         && mv packer /usr/local/bin
 ---> Running in fb738f91c7bf
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/testing
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/main
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/community
+ apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.12.1-10-ge724957f3e [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.1-11-g882c869d81 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
v20200917-2923-g58bb90da52 [http://dl-cdn.alpinelinux.org/alpine/edge/testing]
v20200917-3000-g39ea64e095 [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v20200917-3000-g39ea64e095 [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 29681 distinct packages available
+ apk add --no-cache --virtual .build-python
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/1) Installing .build-python (20201031.032527)
OK: 6 MiB in 15 packages
+ apk add bash ansible libffi-dev openssl-dev openssh qemu qemu-system-x86_64 nettle qemu-img ifupdown-ng iproute2-minimal cloud-init cloud-utils
(1/181) Installing libbz2 (1.0.8-r1)
(2/181) Upgrading libcrypto1.1 (1.1.1g-r0 -> 1.1.1h-r0)
(3/181) Installing expat (2.2.10-r0)
(4/181) Installing libffi (3.3-r2)
(5/181) Installing gdbm (1.18.1-r0)
(6/181) Installing xz-libs (5.2.5-r0)
(7/181) Installing ncurses-terminfo-base (6.2_p20201024-r0)
(8/181) Installing ncurses-libs (6.2_p20201024-r0)
(9/181) Installing readline (8.0.4-r0)
(10/181) Installing sqlite-libs (3.33.0-r0)
(11/181) Upgrading libssl1.1 (1.1.1g-r0 -> 1.1.1h-r0)
(12/181) Installing python3 (3.8.6-r0)
(13/181) Installing py3-markupsafe (1.1.1-r3)
(14/181) Installing py3-jinja2 (2.11.2-r0)
(15/181) Installing py3-parsing (2.4.7-r1)
(16/181) Installing py3-six (1.15.0-r0)
(17/181) Installing py3-packaging (20.4-r0)
(18/181) Installing yaml (0.2.5-r0)
(19/181) Installing py3-yaml (5.3.1-r1)
(20/181) Installing py3-cparser (2.20-r0)
(21/181) Installing py3-cffi (1.14.3-r0)
(22/181) Installing py3-idna (2.10-r0)
(23/181) Installing py3-asn1crypto (1.4.0-r0)
(24/181) Installing py3-cryptography (2.9.2-r0)
(25/181) Installing py3-asn1 (0.4.8-r0)
(26/181) Installing py3-bcrypt (3.2.0-r0)
(27/181) Installing py3-pynacl (1.4.0-r0)
(28/181) Installing py3-paramiko (2.7.2-r0)
(29/181) Installing ansible-base (2.10.2-r0)
(30/181) Installing ansible (2.10.1-r0)
(31/181) Installing bash (5.0.18-r0)
Executing bash-5.0.18-r0.post-install
(32/181) Installing libblkid (2.36-r0)
(33/181) Installing blkid (2.36-r0)
(34/181) Installing libacl (2.2.53-r0)
(35/181) Installing libattr (2.4.48-r0)
(36/181) Installing skalibs (2.9.3.0-r1)
(37/181) Installing s6-ipcserver (2.9.2.0-r1)
(38/181) Installing utmps (0.0.3.2-r1)
Executing utmps-0.0.3.2-r1.pre-install
(39/181) Installing coreutils (8.32-r1)
(40/181) Installing udev-init-scripts (34-r0)
Executing udev-init-scripts-34-r0.post-install
(41/181) Installing eudev-libs (3.2.9-r3)
(42/181) Installing kmod-libs (27-r0)
(43/181) Installing eudev (3.2.9-r3)
(44/181) Installing libuuid (2.36-r0)
(45/181) Installing libfdisk (2.36-r0)
(46/181) Installing libsmartcols (2.36-r0)
(47/181) Installing sfdisk (2.36-r0)
(48/181) Installing libcap-ng (0.8-r0)
(49/181) Installing setpriv (2.36-r0)
(50/181) Installing libmount (2.36-r0)
(51/181) Installing findmnt (2.36-r0)
(52/181) Installing mcookie (2.36-r0)
(53/181) Installing hexdump (2.36-r0)
(54/181) Installing lsblk (2.36-r0)
(55/181) Installing cfdisk (2.36-r0)
(56/181) Installing libeconf (0.3.8-r0)
(57/181) Installing util-linux (2.36-r0)
(58/181) Installing cloud-utils-growpart (0.32-r0)
(59/181) Installing libcom_err (1.45.6-r0)
(60/181) Installing e2fsprogs-libs (1.45.6-r0)
(61/181) Installing e2fsprogs (1.45.6-r0)
(62/181) Installing e2fsprogs-extra (1.45.6-r0)
(63/181) Installing ifupdown-ng (0.10.1-r0)
(64/181) Installing fts (1.2.7-r1)
(65/181) Installing libelf (0.181-r0)
(66/181) Installing iproute2-minimal (5.9.0-r0)
(67/181) Installing ifupdown-ng-iproute2 (0.10.1-r0)
(68/181) Installing py3-configobj (5.0.6-r7)
(69/181) Installing py3-jsonpointer (2.0-r4)
(70/181) Installing py3-jsonpatch (1.26-r1)
(71/181) Installing py3-pyrsistent (0.17.3-r0)
(72/181) Installing py3-attrs (20.2.0-r0)
(73/181) Installing py3-ordered-set (4.0.2-r0)
(74/181) Installing py3-appdirs (1.4.4-r1)
(75/181) Installing py3-setuptools (50.3.2-r0)
(76/181) Installing py3-more-itertools (8.5.0-r0)
(77/181) Installing py3-zipp (1.0.0-r0)
(78/181) Installing py3-importlib-metadata (1.7.0-r0)
(79/181) Installing py3-jsonschema (3.2.0-r0)
(80/181) Installing py3-oauthlib (2.1.0-r4)
(81/181) Installing py3-chardet (3.0.4-r4)
(82/181) Installing py3-urllib3 (1.25.11-r0)
(83/181) Installing py3-requests (2.24.0-r2)
(84/181) Installing linux-pam (1.4.0-r1)
(85/181) Installing shadow (4.8.1-r0)
(86/181) Installing sudo (1.9.3_p1-r0)
(87/181) Installing cloud-init (20.3-r3)
(88/181) Installing libmagic (5.39-r0)
(89/181) Installing file (5.39-r0)
(90/181) Installing bzip2 (1.0.8-r1)
(91/181) Installing libcap (2.44-r0)
(92/181) Installing cdrkit (1.1.11-r3)
(93/181) Installing mtools (4.0.25-r0)
(94/181) Installing libaio (0.3.112-r1)
(95/181) Installing libgcc (10.2.0-r6)
(96/181) Installing libintl (0.20.2-r0)
(97/181) Installing pcre (8.44-r0)
(98/181) Installing glib (2.66.2-r0)
(99/181) Installing gmp (6.2.0-r0)
(100/181) Installing nettle (3.6-r0)
(101/181) Installing p11-kit (0.23.21-r0)
(102/181) Installing libtasn1 (4.16.0-r1)
(103/181) Installing libunistring (0.9.10-r0)
(104/181) Installing gnutls (3.6.15-r0)
(105/181) Installing qemu-img (5.1.0-r1)
(106/181) Installing tar (1.32-r1)
(107/181) Installing cloud-utils (0.32-r0)
(108/181) Installing linux-headers (5.7.8-r0)
(109/181) Installing pkgconf (1.7.3-r0)
(110/181) Installing libffi-dev (3.3-r2)
(111/181) Installing openssh-keygen (8.4_p1-r1)
(112/181) Installing libedit (20191231.3.1-r1)
(113/181) Installing openssh-client (8.4_p1-r1)
(114/181) Installing openssh-sftp-server (8.4_p1-r1)
(115/181) Installing openssh-server-common (8.4_p1-r1)
(116/181) Installing openssh-server (8.4_p1-r1)
(117/181) Installing openssh (8.4_p1-r1)
(118/181) Installing openssl-dev (1.1.1h-r0)
(119/181) Installing ca-certificates (20191127-r5)
(120/181) Installing brotli-libs (1.0.9-r2)
(121/181) Installing nghttp2-libs (1.41.0-r0)
(122/181) Installing libcurl (7.73.0-r0)
(123/181) Installing mesa (20.2.1-r0)
(124/181) Installing libpciaccess (0.16-r0)
(125/181) Installing libdrm (2.4.102-r0)
(126/181) Installing wayland-libs-server (1.18.0-r4)
(127/181) Installing mesa-gbm (20.2.1-r0)
(128/181) Installing pixman (0.40.0-r2)
(129/181) Installing libseccomp (2.5.0-r1)
(130/181) Installing libxau (1.0.9-r0)
(131/181) Installing libbsd (0.10.0-r0)
(132/181) Installing libxdmcp (1.1.3-r0)
(133/181) Installing libxcb (1.14-r1)
(134/181) Installing libx11 (1.6.12-r0)
(135/181) Installing libepoxy (1.5.4-r0)
(136/181) Installing virglrenderer (0.8.2-r1)
(137/181) Installing xkeyboard-config (2.31-r0)
(138/181) Installing libxml2 (2.9.10-r5)
(139/181) Installing libxkbcommon (1.0.1-r0)
(140/181) Installing qemu (5.1.0-r1)
Executing qemu-5.1.0-r1.pre-install
Executing qemu-5.1.0-r1.post-install
*
* If you want to run VM as unprivileged user and let Qemu create tunX devices,
* then you must add that user to the group "qemu".
* If you use KVM for hardware-assisted virtualization, then you may also need
* to add that user to the group "kvm".
*
(141/181) Installing libjpeg-turbo (2.0.5-r0)
(142/181) Installing lzo (2.10-r2)
(143/181) Installing libpng (1.6.37-r1)
(144/181) Installing libstdc++ (10.2.0-r6)
(145/181) Installing snappy (1.1.8-r2)
(146/181) Installing mesa-glapi (20.2.1-r0)
(147/181) Installing wayland-libs-client (1.18.0-r4)
(148/181) Installing libxshmfence (1.3-r1)
(149/181) Installing mesa-egl (20.2.1-r0)
(150/181) Installing libxdamage (1.1.5-r1)
(151/181) Installing libxext (1.3.4-r0)
(152/181) Installing libxfixes (5.0.3-r2)
(153/181) Installing libxxf86vm (1.1.4-r2)
(154/181) Installing mesa-gl (20.2.1-r0)
(155/181) Installing libxv (1.0.11-r2)
(156/181) Installing alsa-lib (1.2.4-r1)
(157/181) Installing libxrender (0.9.10-r3)
(158/181) Installing freetype (2.10.4-r0)
(159/181) Installing fontconfig (2.13.1-r3)
(160/181) Installing cairo (1.16.0-r2)
(161/181) Installing cdparanoia-libs (10.2-r9)
(162/181) Installing gstreamer (1.16.2-r2)
(163/181) Installing libogg (1.3.4-r0)
(164/181) Installing opus (1.3.1-r1)
(165/181) Installing orc (0.4.32-r0)
(166/181) Installing libxft (2.3.3-r0)
(167/181) Installing fribidi (1.0.10-r0)
(168/181) Installing graphite2 (1.3.14-r0)
(169/181) Installing harfbuzz (2.7.2-r0)
(170/181) Installing pango (1.47.0-r0)
(171/181) Installing libtheora (1.1.1-r14)
(172/181) Installing libvorbis (1.3.7-r0)
(173/181) Installing wayland-libs-egl (1.18.0-r4)
(174/181) Installing gst-plugins-base (1.16.2-r3)
(175/181) Installing lz4-libs (1.9.2-r0)
(176/181) Installing libsasl (2.1.27-r10)
(177/181) Installing spice-server (0.14.3-r0)
(178/181) Installing libusb (1.0.23-r0)
(179/181) Installing openssl (1.1.1h-r0)
(180/181) Installing vde2-libs (2.3.2-r12)
(181/181) Installing qemu-system-x86_64 (5.1.0-r1)
Executing busybox-1.31.1-r19.trigger
Executing eudev-3.2.9-r3.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 761 MiB in 194 packages
+ wget https://releases.hashicorp.com/packer/1.6.0/packer_1.6.0_linux_amd64.zip
Connecting to releases.hashicorp.com (151.101.193.183:443)
saving to 'packer_1.6.0_linux_amd64.zip'
packer_1.6.0_linux_a  10% |***                             | 3133k  0:00:08 ETA
packer_1.6.0_linux_a  55% |*****************               | 15.6M  0:00:01 ETA
packer_1.6.0_linux_a 100% |********************************| 28.2M  0:00:00 ETA
'packer_1.6.0_linux_amd64.zip' saved
+ unzip packer_1.6.0_linux_amd64.zip
Archive:  packer_1.6.0_linux_amd64.zip
  inflating: packer
+ chmod +x packer
+ rm packer_1.6.0_linux_amd64.zip
+ mv packer /usr/local/bin
Removing intermediate container fb738f91c7bf
 ---> 8d7a3644c32e
Step 3/3 : WORKDIR /ibmcloud-image-builder
 ---> Running in 186b32ccc3f0
Removing intermediate container 186b32ccc3f0
 ---> 11157797f8f3
Successfully built 11157797f8f3
Successfully tagged ibmcloud-image-builder:latest
Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$

@syyang-in-cloud
Copy link
Member

syyang-in-cloud commented Oct 31, 2020 •
edited
Loading

I've reproduced this issue. ... It seems that packer can't build image inside docker for macOS.
Hmm... probably, I haven't run from docker when I was working with macOS in the beginning, or I have worked form VirtualBox. I'll update README.md that the image cannot be built from docker for macOS for now.

If I have to work from macOS, I would run until cloud-localds from Docker, and the rest of image build using packer from native macOS. Hmm... I need to refactor this repo anyway, I'll come up with something for the longer term. Maybe at the milestone 0.3.0

Thanks for reporting this bug, @dprosper!

@syyang-in-cloud syyang-in-cloud added the bug Something isn't working label Oct 31, 2020
@syyang-in-cloud syyang-in-cloud added this to the v0.3.0 milestone Oct 31, 2020
@syyang-in-cloud
Copy link
Member

syyang-in-cloud commented Jan 14, 2021 •
edited
Loading

I don't know what's been changed ... I just retried on my macOS Big Sur, and it worked fine.

Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$ git pull
Warning: Permanently added 'github.com,140.82.113.3' (RSA) to the list of known hosts.
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 17 (delta 6), reused 15 (delta 6), pack-reused 0
Unpacking objects: 100% (17/17), done.
From github.com:IBM-Cloud/ibmcloud-image-builder
   a8c3e86..d27ff79  main       -> origin/main
Updating a8c3e86..d27ff79
Fast-forward
 README.md                                        |  13 +++----------
 docker-compose.yml                               |   6 +++---
 packer/ubuntu/bionic/base/ansible/playbook.yml   |  37 +++++++++++++++++++++++++------------
 packer/ubuntu/bionic/docker/ansible/playbook.yml | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------
 4 files changed, 92 insertions(+), 73 deletions(-)
Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$

Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$ make pre-build
docker build . -f Dockerfile -t ibmcloud-image-builder:latest
Sending build context to Docker daemon  17.35GB
Step 1/3 : FROM alpine:latest
latest: Pulling from library/alpine
801bfaa63ef2: Pull complete 
Digest: sha256:3c7497bf0c7af93428242d6176e8f7905f2201d8fc5861f45be7a346b5f23436
Status: Downloaded newer image for alpine:latest
 ---> 389fef711851
Step 2/3 : RUN set -ex         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories         && echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories         && apk update         && apk add --no-cache --virtual .build-python         && apk add bash ansible libffi-dev openssl-dev openssh qemu qemu-system-x86_64 nettle qemu-img ifupdown-ng iproute2-minimal cloud-init cloud-utils         && wget https://releases.hashicorp.com/packer/1.6.0/packer_1.6.0_linux_amd64.zip         && unzip packer_1.6.0_linux_amd64.zip         && chmod +x packer         && rm packer_1.6.0_linux_amd64.zip         && mv packer /usr/local/bin
 ---> Running in be81c9fc6da0
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/testing
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/main
+ echo http://dl-cdn.alpinelinux.org/alpine/edge/community
+ apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.12.3-54-gc2a303485f [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.3-54-gc2a303485f [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
v3.13.0_rc5-9-g835bc7eb84 [http://dl-cdn.alpinelinux.org/alpine/edge/testing]
v3.13.0_rc5-11-g4d6171d68a [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.13.0_rc5-8-g37e272e5bd [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 30345 distinct packages available
+ apk add --no-cache --virtual .build-python
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/1) Installing .build-python (20210114.051856)
OK: 6 MiB in 15 packages
+ apk add bash ansible libffi-dev openssl-dev openssh qemu qemu-system-x86_64 nettle qemu-img ifupdown-ng iproute2-minimal cloud-init cloud-utils
(1/148) Installing libbz2 (1.0.8-r1)
(2/148) Installing expat (2.2.10-r1)
(3/148) Installing libffi (3.3-r2)
(4/148) Installing gdbm (1.19-r0)
(5/148) Installing xz-libs (5.2.5-r0)
(6/148) Installing ncurses-terminfo-base (6.2_p20210109-r0)
(7/148) Installing ncurses-libs (6.2_p20210109-r0)
(8/148) Installing readline (8.1.0-r0)
(9/148) Installing sqlite-libs (3.34.0-r1)
(10/148) Installing python3 (3.8.7-r0)
(11/148) Installing py3-markupsafe (1.1.1-r3)
(12/148) Installing py3-jinja2 (2.11.2-r0)
(13/148) Installing py3-parsing (2.4.7-r1)
(14/148) Installing py3-six (1.15.0-r0)
(15/148) Installing py3-packaging (20.8-r1)
(16/148) Installing yaml (0.2.5-r0)
(17/148) Installing py3-yaml (5.3.1-r1)
(18/148) Installing py3-cparser (2.20-r0)
(19/148) Installing py3-cffi (1.14.4-r0)
(20/148) Installing py3-idna (3.1-r0)
(21/148) Installing py3-asn1crypto (1.4.0-r0)
(22/148) Installing py3-cryptography (3.3.1-r0)
(23/148) Installing py3-asn1 (0.4.8-r0)
(24/148) Installing py3-bcrypt (3.2.0-r2)
(25/148) Installing py3-pynacl (1.4.0-r0)
(26/148) Installing py3-paramiko (2.7.2-r0)
(27/148) Installing ansible-base (2.10.4-r0)
(28/148) Installing ansible (2.10.5-r0)
(29/148) Installing bash (5.1.0-r0)
Executing bash-5.1.0-r0.post-install
(30/148) Installing libblkid (2.36.1-r0)
(31/148) Installing blkid (2.36.1-r0)
(32/148) Installing libsmartcols (2.36.1-r0)
(33/148) Installing partx (2.36.1-r0)
(34/148) Installing libuuid (2.36.1-r0)
(35/148) Installing libfdisk (2.36.1-r0)
(36/148) Installing sfdisk (2.36.1-r0)
(37/148) Installing cloud-utils-growpart (0.32-r2)
(38/148) Installing libcom_err (1.45.6-r1)
(39/148) Installing e2fsprogs-libs (1.45.6-r1)
(40/148) Installing e2fsprogs (1.45.6-r1)
(41/148) Installing e2fsprogs-extra (1.45.6-r1)
(42/148) Installing udev-init-scripts (34-r0)
Executing udev-init-scripts-34-r0.post-install
(43/148) Installing eudev-libs (3.2.9-r3)
(44/148) Installing zstd-libs (1.4.5-r3)
(45/148) Installing kmod-libs (28-r0)
(46/148) Installing eudev (3.2.9-r3)
(47/148) Installing ifupdown-ng (0.10.2-r2)
(48/148) Installing fts (1.2.7-r1)
(49/148) Installing libelf (0.182-r0)
(50/148) Installing iproute2-minimal (5.10.0-r1)
(51/148) Installing ifupdown-ng-iproute2 (0.10.2-r2)
(52/148) Installing libmount (2.36.1-r0)
(53/148) Installing lsblk (2.36.1-r0)
(54/148) Installing py3-configobj (5.0.6-r7)
(55/148) Installing py3-jsonpointer (2.0-r4)
(56/148) Installing py3-jsonpatch (1.28-r0)
(57/148) Installing py3-pyrsistent (0.17.3-r0)
(58/148) Installing py3-attrs (20.3.0-r0)
(59/148) Installing py3-ordered-set (4.0.2-r0)
(60/148) Installing py3-appdirs (1.4.4-r1)
(61/148) Installing py3-setuptools (51.0.0-r0)
(62/148) Installing py3-more-itertools (8.6.0-r0)
(63/148) Installing py3-zipp (1.0.0-r0)
(64/148) Installing py3-importlib-metadata (1.7.0-r0)
(65/148) Installing py3-jsonschema (3.2.0-r0)
(66/148) Installing py3-chardet (4.0.0-r0)
(67/148) Installing py3-urllib3 (1.26.2-r1)
(68/148) Installing py3-requests (2.25.1-r1)
(69/148) Installing linux-pam (1.5.1-r0)
(70/148) Installing shadow (4.8.1-r0)
(71/148) Installing sudo (1.9.5p1-r0)
(72/148) Installing cloud-init (20.4-r0)
Executing cloud-init-20.4-r0.post-install
Please run setup-cloud-init to enable required init.d services.

You may also want to read file /usr/share/doc/cloud-init/README.Alpine
in the cloud-init-docs package.
(73/148) Installing libmagic (5.39-r0)
(74/148) Installing file (5.39-r0)
(75/148) Installing bzip2 (1.0.8-r1)
(76/148) Installing libcap (2.46-r0)
(77/148) Installing cdrkit (1.1.11-r3)
(78/148) Installing libacl (2.2.53-r0)
(79/148) Installing libattr (2.4.48-r0)
(80/148) Installing skalibs (2.10.0.0-r0)
(81/148) Installing s6-ipcserver (2.10.0.0-r0)
(82/148) Installing utmps (0.1.0.0-r0)
Executing utmps-0.1.0.0-r0.pre-install
(83/148) Installing coreutils (8.32-r2)
(84/148) Installing mtools (4.0.26-r0)
(85/148) Installing libaio (0.3.112-r1)
(86/148) Installing libintl (0.20.2-r2)
(87/148) Installing pcre (8.44-r0)
(88/148) Installing glib (2.66.4-r0)
(89/148) Installing gmp (6.2.1-r0)
(90/148) Installing nettle (3.7-r0)
(91/148) Installing p11-kit (0.23.22-r0)
(92/148) Installing libtasn1 (4.16.0-r1)
(93/148) Installing libunistring (0.9.10-r0)
(94/148) Installing gnutls (3.7.0-r0)
(95/148) Installing qemu-img (5.2.0-r2)
(96/148) Installing tar (1.33-r1)
(97/148) Installing libcap-ng (0.8.2-r0)
(98/148) Installing setpriv (2.36.1-r0)
(99/148) Installing findmnt (2.36.1-r0)
(100/148) Installing mcookie (2.36.1-r0)
(101/148) Installing hexdump (2.36.1-r0)
(102/148) Installing cfdisk (2.36.1-r0)
(103/148) Installing libeconf (0.3.8-r0)
(104/148) Installing util-linux (2.36.1-r0)
(105/148) Installing cloud-utils (0.32-r2)
(106/148) Installing linux-headers (5.7.8-r0)
(107/148) Installing pkgconf (1.7.3-r0)
(108/148) Installing libffi-dev (3.3-r2)
(109/148) Installing openssh-keygen (8.4_p1-r2)
(110/148) Installing libedit (20191231.3.1-r1)
(111/148) Installing openssh-client (8.4_p1-r2)
(112/148) Installing openssh-sftp-server (8.4_p1-r2)
(113/148) Installing openssh-server-common (8.4_p1-r2)
(114/148) Installing openssh-server (8.4_p1-r2)
(115/148) Installing openssh (8.4_p1-r2)
(116/148) Installing openssl-dev (1.1.1i-r0)
(117/148) Installing ca-certificates (20191127-r5)
(118/148) Installing brotli-libs (1.0.9-r3)
(119/148) Installing nghttp2-libs (1.42.0-r1)
(120/148) Installing libcurl (7.74.0-r0)
(121/148) Installing mesa (20.3.2-r0)
(122/148) Installing libpciaccess (0.16-r0)
(123/148) Installing libdrm (2.4.104-r0)
(124/148) Installing wayland-libs-server (1.18.0-r5)
(125/148) Installing mesa-gbm (20.3.2-r0)
(126/148) Installing pixman (0.40.0-r2)
(127/148) Installing libseccomp (2.5.1-r0)
(128/148) Installing libxau (1.0.9-r0)
(129/148) Installing libbsd (0.10.0-r0)
(130/148) Installing libxdmcp (1.1.3-r0)
(131/148) Installing libxcb (1.14-r1)
(132/148) Installing libx11 (1.7.0-r0)
(133/148) Installing libepoxy (1.5.5-r0)
(134/148) Installing virglrenderer (0.8.2-r1)
(135/148) Installing xkeyboard-config (2.31-r0)
(136/148) Installing libxml2 (2.9.10-r6)
(137/148) Installing libxkbcommon (1.0.3-r0)
(138/148) Installing qemu (5.2.0-r2)
Executing qemu-5.2.0-r2.pre-install
Executing qemu-5.2.0-r2.post-install
*
* If you want to run VM as unprivileged user and let Qemu create tunX devices,
* then you must add that user to the group "qemu".
* If you use KVM for hardware-assisted virtualization, then you may also need
* to add that user to the group "kvm".
*
(139/148) Installing libgcc (10.2.1_pre1-r3)
(140/148) Installing libjpeg-turbo (2.0.6-r0)
(141/148) Installing lzo (2.10-r2)
(142/148) Installing libpng (1.6.37-r1)
(143/148) Installing libstdc++ (10.2.1_pre1-r3)
(144/148) Installing snappy (1.1.8-r2)
(145/148) Installing libusb (1.0.24-r0)
(146/148) Installing openssl (1.1.1i-r0)
(147/148) Installing vde2-libs (2.3.2-r12)
(148/148) Installing qemu-system-x86_64 (5.2.0-r2)
Executing busybox-1.31.1-r19.trigger
Executing eudev-3.2.9-r3.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 765 MiB in 163 packages
+ wget https://releases.hashicorp.com/packer/1.6.0/packer_1.6.0_linux_amd64.zip
Connecting to releases.hashicorp.com (151.101.65.183:443)
saving to 'packer_1.6.0_linux_amd64.zip'
packer_1.6.0_linux_a  11% |***                             | 3405k  0:00:07 ETA
packer_1.6.0_linux_a  50% |****************                | 14.3M  0:00:01 ETA
packer_1.6.0_linux_a 100% |********************************| 28.2M  0:00:00 ETA
'packer_1.6.0_linux_amd64.zip' saved
+ unzip packer_1.6.0_linux_amd64.zip
Archive:  packer_1.6.0_linux_amd64.zip
  inflating: packer
+ chmod +x packer
+ rm packer_1.6.0_linux_amd64.zip
+ mv packer /usr/local/bin
Removing intermediate container be81c9fc6da0
 ---> 2d1d8f464bcb
Step 3/3 : WORKDIR /ibmcloud-image-builder
 ---> Running in 2edf1a0aca93
Removing intermediate container 2edf1a0aca93
 ---> 64eeab8a5178
Successfully built 64eeab8a5178
Successfully tagged ibmcloud-image-builder:latest
Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$


Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$ make clean
docker run --privileged --rm -v `pwd`:/ibmcloud-image-builder ibmcloud-image-builder:latest /bin/bash -c "./packer-delete.sh"
+ rm -rf img


Seungs-MacBook-Pro:ibmcloud-image-builder seungyeop$ make build-all
/Library/Developer/CommandLineTools/usr/bin/make build-image DISTRO_NAME=ubuntu DISTRO_VERSION=bionic TYPE=base
docker run --privileged --rm -v `pwd`:/ibmcloud-image-builder ibmcloud-image-builder:latest /bin/bash -c "./packer-build.sh packer/ubuntu/bionic/base"
++ echo packer/ubuntu/bionic/base
++ sed s#packer/##g
+ IMAGE_DIR=ubuntu/bionic/base
++ echo packer/ubuntu/bionic/base
++ sed s#packer/##g
++ tr / -
+ DIR_TO_FILENAME=ubuntu-bionic-base
+ NEW_IMAGE=output-qemu/ibmcloud-ubuntu-bionic-base-amd64-100G.qcow2
+ ENCRYPTED_IMAGE=output-qemu/ibmcloud-encrypted-ubuntu-bionic-base-amd64-100G.qcow2
+ mkdir -p img/ubuntu-bionic-base
+ cp -r packer/ubuntu/bionic/base/ansible packer/ubuntu/bionic/base/disk-ssh-pub.img packer/ubuntu/bionic/base/http packer/ubuntu/bionic/base/packer.json packer/ubuntu/bionic/base/packer_cache packer/ubuntu/bionic/base/shell packer/ubuntu/bionic/base/ssh packer/ubuntu/bionic/base/user-data img/ubuntu-bionic-base
+ cd img/ubuntu-bionic-base
+ '[' -n '' ']'
+ SECRET=JustMySimpleSecret
+ [[ -f ~/.ssh/id_rsa.pub ]]
+ mkdir -p ./ssh
+ chmod 0700 ./ssh
+ ssh-keygen -q -t rsa -b 4096 -C [email protected] -N '' -f ./ssh/id_rsa
./ssh/id_rsa already exists.
Overwrite (y/n)? + export PACKER_PUBLIC_KEY=./ssh/id_rsa.pub
+ PACKER_PUBLIC_KEY=./ssh/id_rsa.pub
+ export PACKER_PRIVATE_KEY=./ssh/id_rsa
+ PACKER_PRIVATE_KEY=./ssh/id_rsa
++ cat ./ssh/id_rsa.pub
+ public_key='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwTduhJJRNAlLeEPUset1UG5AVdTSp6PNhyEGc4Wnede3WTriZOsKKuSbXvMwPa8qPi1ELo1AlA5JfEaizCHRU+oakqMjshpGkDtMcvedcegGOm/RqSIChR/wtg0Q5D2GECw7aJFjxsjfMa+chsYkDsGETIfQcOnk5WgP7REwrxD6KGcSUX1HJcviURo68nBuKVCl7t0R27/ZcJSU8GjJAX5XUgZ/IiDXpOG0ppH2AYLS4e08zTccZ+t0TWJeh1IPSSFC+Pkf2aPXssLSEZ2xAFwMgOSX8bjfuhMX2zLKjWIh9X7tIW8Uzoo7LKwOHbhPb6aJIE8G4mjHv+I0s7Mz1XqcKxCLHH+ImaeXDQ7ir0L2P+SNcuVHV0HzWvbTlGbh8gkvytzo7tWDyXJRVo3DiP9+eq5sSt1qrumgws+zcHLJ+KM0ItxyNs2ZANc9rzv/OYXjIqc0oSUiAcqorxQd4vbCDA1o96+8J+/nMUeDv9JhGydCCfhobylxD8i+aqbxaEwjHnXO/oY8iq4DrEOoy3K4ezWA9lWUzovxQ7OQzd/l1jPI0J4IVwYUI1Y0NEm3YbXRiYDIAJpbbOoqjpthyXt4vAq/xuLoO7AAUd/uxfHchlaEQZCNFBFu80HvUM+DcwVRHUIcwEX9WtHD4U1V6cAowIcnwYQMMTTKQw1L/kQ== [email protected]'
+ cat
+ cloud-localds disk-ssh-pub.img user-data
+ ansible-galaxy install geerlingguy.docker
Starting galaxy role install process
- downloading role 'docker', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-docker/archive/3.0.0.tar.gz
- extracting geerlingguy.docker to /root/.ansible/roles/geerlingguy.docker
- geerlingguy.docker (3.0.0) was installed successfully
+ rm -rf output-qemu
+ PACKER_LOG=0
+ packer build packer.json
qemu: output will be in this color.

==> qemu: Retrieving ISO
==> qemu: Trying https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
==> qemu: Trying https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img?checksum=sha256%3A190b2d7121f7f39100c7ed20be4355764ac0c2fea9b77025c039233d93957762
bionic-server-cloudimg-amd64.img 344.81 MiB / 344.81 MiB  100.00% 28s01s
==> qemu: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img?checksum=sha256%3A190b2d7121f7f39100c7ed20be4355764ac0c2fea9b77025c039233d93957762 => /ibmcloud-image-builder/img/ubuntu-bionic-base/packer_cache/35035c80eb73b82ceea1d8c6ff4c093533915979.iso
==> qemu: Copying hard drive...
==> qemu: Resizing hard drive...
==> qemu: Starting HTTP server on port 8228
==> qemu: Found port for communicator (SSH, WinRM, etc): 4158.
==> qemu: Looking for available port between 5900 and 6000 on 127.0.0.1
==> qemu: Starting VM, booting disk image
==> qemu: Overriding defaults Qemu arguments with QemuArgs...
==> qemu: Waiting 10s for boot...
==> qemu: Connecting to VM via VNC (127.0.0.1:5907)
==> qemu: Typing the boot command over VNC...
==> qemu: Using ssh communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available...

@syyang-in-cloud
Copy link
Member

syyang-in-cloud commented Jan 14, 2021 •
edited
Loading

Hi @dprosper , I don't know what's changed ... Nothing much changed in the repo. I've upgrade my macOS to Big Sur and my docker version is as below.
image

image

I've updated minor change in Ubuntu images, and I tried again on my macOS, just in case, and it seems working fine. I have no idea what's the difference ... only thing changed for me was a couple of upgrades of macOS (Big Sur) and Docker Desktop since last time when I tried (October 2020).

Could you try again? Thanks a lot!

@syyang-in-cloud syyang-in-cloud added wontfix This will not be worked on not reproducible labels Jan 14, 2021
@dprosper
Copy link
Author

thank you, I will try again over the next couple of days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@syyang-in-cloud syyang-in-cloud

Labels
bug Something isn't working not reproducible wontfix This will not be worked on
Projects
None yet
Milestone
v0.3.0
Development

No branches or pull requests
2 participants
@dprosper @syyang-in-cloud