pkgdev:setup

Contents

1. Introduction
2. OS setup
   1. SmartOS / illumos
   2. macOS
   3. Ubuntu Linux
   4. RedHat Linux
3. Fetch Repositories
4. Create Sandbox

Introduction

In order to get started with package development it is recommended that you use the pkgbuild infrastructure. This provides scripts and configuration files that we use to produce the official binary packages, and helps to ensure your environment is similar.

OS setup

Follow the instructions below for your target OS.

SmartOS / illumos

For SmartOS users we provide a pkgbuild image. This contains everything you need to get going quickly.

Use the image version which matches the package set you want to build. For example, to build packages for 19.4.x/2019Q4 images use the latest 19.4.x pkgbuild image.

$ imgadm update

$ imgadm avail | awk '/pkgbuild.*19.4/ {print}'
d001d2b4-3157-11ea-832d-df421d070030 pkgbuild 19.4.0 smartos zone-dataset 2020-01-07

$ imgadm import d001d2b4-3157-11ea-832d-df421d070030

Alternatively if you wish to develop against trunk, it's recommended to use the latest trunk image:

$ imgadm avail | awk '/pkgbuild-trunk/ {print}'
d7d6c04c-40e5-11ea-8c6c-67ed0cdd2dfe pkgbuild-trunk 20200124 smartos zone-dataset 2020-01-27

$ imgadm import d7d6c04c-40e5-11ea-8c6c-67ed0cdd2dfe

Create a zone using the chosen image uuid, giving it plenty of RAM and quota if you are planning to build large packages. For example:

{
  "brand": "joyent",
  "image_uuid": "d7d6c04c-40e5-11ea-8c6c-67ed0cdd2dfe",
  "alias": "pkgbuild-trunk",
  "max_physical_memory": 8192,
  "quota": 16,
  "resolvers": [
    "8.8.8.8"
  ],
  "dns_domain": "local",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp"
    }
  ]
}

$ vmadm create -f pkgbuild.json

macOS

Create the pbulk user, used for unprivileged builds. The easiest way to do this is via the System Preferences dialog, but if you wish to use the command line then something like might work (adjust UID for your system):

user="pbulk"
uid=500
gid=20
comment="pbulk user"

dscl . -create /users/${user} RecordName ${user}
dscl . -create /users/${user} RecordType dsRecTypeNative:users
dscl . -create /users/${user} UniqueID $uid
dscl . -create /users/${user} PrimaryGroupID $gid
dscl . -create /users/${user} NFSHomeDirectory "/Users/${user}"
dscl . -create /users/${user} UserShell "/bin/bash"
dscl . -create /users/${user} Comment "$comment"
dscl . -delete /users/${user} AuthenticationAuthority
dscl . -create /users/${user} Password '*'

mkdir /Users/${user}
chown ${user}:staff /Users/${user}

Then run passwd pbulk and set some random generated password. In order to build certain SDL packages and any others that require a running Window Server during the build, it is necessary to enable automatic login for the pbulk user.

Create an APFS volume called "data", ensuring that it is case-sensitive. We try to avoid case-insensitive issues in pkgsrc but there's always a chance that something is missed, so this avoids potential problems.

#
# This is only an example, check your local disklist configuration for the
# appropriate container to use, etc.
#
diskutil apfs addVolume disk1 "Case-sensitive APFS" data

In order to perform sandbox builds, a number of directories are loopback mounted over NFS. This requires /etc/exports to be configured with the following contents:

/			-alldirs -maproot=root 127.0.0.1
/Volumes/data		-alldirs -maproot=root 127.0.0.1
/System/Volumes/Data	-alldirs -maproot=root 127.0.0.1

If you are performing bulk builds across multiple hosts then you'll need to set up additional IP addresses accordingly.

Check that the directories are exported. The NFS daemon detects changes to /etc/exports and will update mounts automatically.

$ showmount -e
Exports list on localhost:
/                                   127.0.0.1
/System/Volumes/Data                127.0.0.1
/Volumes/data                       127.0.0.1

If you are running macOS Sierra 10.12.4 or newer you will unfortunately need to disable System Integrity Protection as Apple has restricted the ability to update the mDNS service which is required for DNS resolution inside build chroots. If this isn't possible for your situation, you have no choice but to hardcode every site you need to access in /etc/hosts. You will also see the following warning when trying to create and remove sandboxes:

/var/run/com.apple.mDNSResponder.plist: service already loaded

Create required directories, and fetch bootstrap tarballs and packages:

# Directories that store various pkgsrc artefacts
mkdir -p /Volumes/data/{bulklog,distfiles,packages/Darwin,shadows}

# Creating this directory avoids run-sandbox warnings, so is worth running
# even if you will not be signing packages.
mkdir -p /var/root/.gnupg

# Fetch bootstrap tarballs
rsync -av \
    rsync://pkgsrc.smartos.org/pkgsrc/packages/Darwin/bootstrap-pbulk \
    /Volumes/data/packages/Darwin/

#
# Fetch the appropriate tools packages for your system, currently either
# 10.14 (Mojave) or 10.15 (Catalina).  Use the correct version, as there are
# differences in the setups, as Catalina has forced us to make some changes.
#
mac_version=10.14	# Mojave
mac_version=10.15	# Catalina

rsync -av \
    rsync://pkgsrc.smartos.org/pkgsrc/packages/Darwin/${mac_version} \
    /Volumes/data/packages/Darwin/

Ubuntu Linux

Perform a full upgrade.

$ apt-get update
$ apt-get dist-upgrade
$ apt-get autoremove

Install required packages. git is part of build-essential on 16.04 or later.

$ apt-get install build-essential git mailutils postfix

If using the Joyent Ubuntu image rename the /mnt mount to provide /data.

: Switch /mnt to /data if required
$ sed -ie 's,/mnt,/data,' /etc/fstab

Create the pbulk user, used for unprivileged builds.

$ useradd -U -m -s /bin/bash -c "pbulk user" pbulk

Set hostname if required

$ vi /etc/hostname
pkgsrc-pbulk.local

$ reboot

RedHat Linux

Perform a full upgrade.

$ yum update

Install required packages.

$ yum install bzip2 ed gcc gcc-c++ git psmisc screen

Create the pbulk user, used for unprivileged builds.

$ useradd -U -m -s /bin/bash -c "pbulk user" pbulk

Set hostname if required.

: EL 6.x
$ vi /etc/sysconfig/network
HOSTNAME=pkgsrc-pbulk.local

: EL 7.x
$ hostnamectl set-hostname pkgsrc-pbulk.local

$ reboot

Fetch repositories

Fetch both the pkgsrc and pkgbuild repositories. If you are using the SmartOS pkgbuild image these repositories will have already been cloned, so you only need to perform a git pull instead.

The /data and /Volumes/data prefixes are somewhat hardcoded at this point, though with some work you can use a different prefix if necessary. This is however currently undocumented).

On macOS and Linux you'll also want to add the scripts directory to $PATH, and on macOS you'll probably want to track our release branch.

On Linux:

mkdir -p /data
cd /data
git clone https://github.com/TritonDataCenter/pkgsrc.git
git clone https://github.com/TritonDataCenter/pkgbuild.git

PATH=$PATH:/data/pkgbuild/scripts

On macOS:

cd /Volumes/data
git clone https://github.com/TritonDataCenter/pkgsrc.git
git clone https://github.com/TritonDataCenter/pkgbuild.git

: Track the macOS release branch (recommended)
cd /data/pkgsrc
git checkout joyent/release/macos
git submodule init
git submodule update

PATH=$PATH:/Volumes/data/pkgbuild/scripts

On SmartOS pkgbuild:

(cd /data/pkgbuild; git pull)
(cd /data/pkgsrc; git pull)

Fetch bootstrap and packages

Again if you are using the SmartOS pkgbuild image you do not need to perform this step. If setting things up manually however, you will need to fetch the bootstrap tarballs corresponding to the builds you are targetting.

Each build requires two bootstraps, the target build as well as the tools build that is used to create it. For example:

$ mkdir -p /data/packages/SmartOS/bootstrap-pbulk
$ cd /data/packages/SmartOS/bootstrap-pbulk

: 2018Q4
$ for bs in tools x86_64; do
>   curl -O https://pkgsrc.smartos.org/packages/SmartOS/bootstrap-pbulk/bootstrap-2018Q4-${bs}.tar.gz
> done

: trunk
$ for bs in tools x86_64; do
>   curl -O https://pkgsrc.smartos.org/packages/SmartOS/bootstrap-pbulk/bootstrap-trunk-${bs}.tar.gz
> done

For Darwin trunk:

$ mkdir -p /data/packages/Darwin/bootstrap-pbulk
$ cd /data/packages/Darwin/bootstrap-pbulk
$ for bs in i386 pbulk32 pbulk64 x86_64; do
>   curl -O https://pkgsrc.smartos.org/packages/Darwin/bootstrap-pbulk/bootstrap-trunk-${bs}.tar.gz
> done

If for whatever reason you want to create a tools sandbox, they are built using the x86_64 bootstrap from the previous release. Thus for a 2018Q4-tools sandbox you would require the 2018Q4-tools bootstrap as well as the 2018Q3-x86_64 bootstrap. The trunk tools bootstrap is bumped every so often to the most recent quarterly release.

Create Sandbox

The pkgbuild repository aims to recreate the same environment that produces our official binary package sets. The run-sandbox script prepares a chroot environment with everything set up ready to build packages. It takes a single argument which is the pkgbuild configuration to use, based on one of the directories under pkgbuild/conf.

Some examples:

#
# SmartOS/illumos
#
run-sandbox trunk-x86_64            # Create a trunk 64-bit sandbox
run-sandbox 2019Q4-x86_64           # Create a 19.4.x sandbox
run-sandbox 2019Q4-tools            # Create a 19.4.x GZ "tools" sandbox

#
# macOS
#
run-sandbox macos14-trunk-x86_64    # Create a 10.14 Mojave sandbox
run-sandbox macos15-trunk-x86_64    # Create a 10.15 Catalina sandbox

#
# Linux (CentOS / RedHat Enterprise Linux / Oracle Linux / etc.)
#
run-sandbox el6-trunk-x86_64        # Create an Enterprise Linux 6.x sandbox
run-sandbox el7-trunk-x86_64        # Create an Enterprise Linux 7.x sandbox

Once you have successfully created a sandbox you can move onto building packages, or choose one of the other links in the "Package Development" section of the sidebar at the top of this page.