Administrator’s Manual

The administrator’s manual covers the package installation, setup and maintenance of a mini-buildd instance.

Installation

Debian package ‘mini-buildd’ is available natively in and for Debian.

  • Debian Backports may offer bugfix updates.

  • mini-buildd’s homepage may offer other alternate (Debian or Ubuntu) distributions to install on, or bleeding-edge updates.

To install a mini-buildd instance, just install the Debian package:

apt install mini-buildd

Tip

How much disk space do I need to run mini-buildd?

For building, as a rule of thumb, you will need about:

(<CODENAMES> * <ARCHS> * 0.3G) + (<CORES> * 4G)

disk space free for building.

F.e., if you support codename bullseye and buster builds for archs amd64 and i386 on a machine with 4 cores:

(2 * 2 * 0.3G) + (4 * 4G) = 17.2G

Builds would still not run oo space if max default number of parallel builds is running (assuming base chroot size of 0.3G and a (sort of worst case) Debian Source Tree size of 4G).

No rule of thumb when you host repositories. Just make it big.

Configuration

Package configuration (via debconf) include the home path of the mini-buildd user, the administrator’s password and extra options for the daemon run.

Usually, you set the admin password on the initial install, and just leave the rest on default values.

Of course, you can change your (debconf) settings anytime (including (re-)setting the admin password and changing mini-buildd’s home path) using:

dpkg-reconfigure mini-buildd

Running encrypted

To run HTTPS, you need to adapt the --http-endpoint command line option (see command ‘mini-buildd’), for example by:

dpkg-reconfigure mini-buildd  # Syntax example: 'ssl:port=8066:privateKey=KEY_PATH:certKey=CERT_PATH'

New in version 1.1.21.

Important

The SSL private key must be readable by user or group mini-buildd.

Tip

How to enable encrypted uploads (FTPS)?

If you happen to bootstrap via setup from HTTPS, the created Daemon instance per default already uses FTPS.

To enable it later, adapt the FTP setup in the Daemon instance via mini-buildd’s ‘admin’ page.

Self-signed certificates

If you don’t have/want certificates signed by official authorities, self signed certificates will also work – but be sure…

  • certifcate’s CN is the same as your fqdn hostname (or what you set via the --hostname arg).

  • self signed public get added to the system’s certificate store for all machines you want to access the installation (see Debian package ca-certificates).

  • [dev only] each instance and cert have a different hostname – in case you run multiple instances on the same host (only makes sense for developer testing).

You may use this tool to maintain such a certificate:

mini-buildd-self-signed-certificate

Setup

A setup is the configuration (Daemon, Sources, Repositories, etc.) as well as (potentially) their status (f.e., actual chroots or repositories on the system).

You may use the API call ‘setup’ to simplify this task. setup can be used to bootstrap a setup as well as maintenance (f.e., add new distributions, or update/check for mini-buildd’s defaults).

setup provides a one stop for this task, including some presets for bootstrapping.

Attention

TODO: More testing and clarification needed in this section.

Bootstrap

After initial install, i.e., with no setup, just use the API call ‘setup’, either with default values or using one of the presets in mini-buildd’s ‘setup’ page.

The most likely ‘setup’ options to customize here from the getgo would be identity and repositories.

When the call has successfully finished, you should end up with – assuming the defaults – a fully functional Repository test. Optionally, build the internal test packages via API call ‘test_packages’.

In case you went with the defaults for the repositories (only have a “test” repo), just create a new Repository instance with the actual ID you want to use for production.

Think about how you want to do upload authorization for that production repository; see Auth via SSH wrapper.

Calling API call ‘setup’ is mostly harmless: idempotent, and it will never change existing model instances unless explicitly using --update.

Keyring Packages

Once you are satisfied with your setup, you should finish up by building keyring packages for your archive via the API call ‘keyring_packages’.

This is usually a one-timer, unless of course

  • You changed/renewed your archive key.

  • You added new distributions.

  • You know there is an update of some sort to the internal (mini-buildd-internal) keyring package generator you are inclined to use.

In this case, just repeat the API call…

Maintenance

TBD

Models

When mini-buildd runs, it’s basically acts as a web server, with a django web application running on it.

mini-buildd’s configuration consists of related django model instances, and their configuration is done via Django’s ‘admin’ application. On the mini-buildd home page, just hit on admin (left top) to enter.

You will need to log in as Django user admin, with the password you configured when installing the package (if you chose an insecure password on package install time, now is the time to set a proper one via Django’s user management).

All changes you do here finally wind up in the SQL database at ~/config.sqlite; this config not only represents mere configuration, but also state (of ~mini-buildd/, and in case of chroots, even artifacts on the system, see Model statuses below), so this file cannot be simply interchanged or copied.

Some of the models have a status attached to it.

This usually refers to a model’s associated data on the system (which can be managed via actions in the configuration interface):

Model

Associated prepared system data

File location (~ denoting mini-buildd’s home path)

Daemon

GnuPG Key

~/.gnupg/

Repository

Reprepro repository

~/repositories/REPO_ID/

Chroot

Chroot data and schroot configuration

  • ~/var/chroots/CODENAME/ARCH/

  • /etc/schroot/chroot.d/mini-buildd-CODENAME-ARCH.conf

  • Some backends (like LVM) may taint other system data

Some other models also use the same status infrastructure, but the associated data is prepared internally in the model’s data (SQL database) only:

Model

Associated prepared data

AptKey, Uploader, Remote

Public GnuPG Key

Source

List of matching archives, selected info from Release file

Status semantics:

Status

Check status

Semantic

Removed

No associated data.

Prepared

Associated data exists. With no flags, data is checked and in-sync. Special conditions to the data may apply:

Prepared

Unchecked (-)

Needs a manual check run to set things straight.

Prepared

Changed (*)

Model was changed, but the date is not yet updated. Needs a manual prepare run to set things straight.

Prepared

Failed (x)

Check failed.

Prepared

Failed_Reactivate (A)

Check failed, will be automatically activated again as soon as check succeeds again.

Active

Prepared on the system, checked and activated.

Status actions:

Status actions can be called from a model’s list view in Django’s admin configuration.

Action

Semantic

Prepare

Create associated data on the system, or synchronize it with item changes.

Check

Check item and/or associated data.

Activate

Activate the item, or set the auto-activate flag.

Deactivate

Deactivate the item, or remove the auto-activate flag.

Remove

Remove associated data from system.

Tip

In the web configurator, you may use the PCA shortcut (prepare, check and activate).

Daemon

The Daemon model represents a configured mini-buildd instance. It is limited to have exactly one instance; when activated, it means the internal FTP server is started acting on *.changes.

Don’t confuse this with the mini-buildd Unix daemon, which is always running when the mini-buildd Debian package is installed, and always provides the HTTP server and web application.

The Daemon instance inside of mini-buildd provides the packager and builder engine (triggered by incoming via the FTP server), and can be enabled/disabled inside mini-buildd.

  1. Edit the one Daemon instance. Get the identity and the gnupg template right, changing these later will call for trouble.
    • identity will hereafter be referred to as <ARCHIVE>. Keyring packages will be named <ARCHIVE>-archive-keyring.

    • prepare will generate your instance ID (read: GnuPG key).

      You may need to generate some entropy (install Debian package ‘haveged’ maybe) on the system if this stalls.

  2. PCA the Daemon instance.

Source

This groups all models that determine what APT sources are available, and where to get them.

You will later interface with Source and PrioritySource when dealing with chroots and distributions.

A Source is usually identified sufficiently by Origin and Codename.

When a Source is prepared, it will implicitly update Architectures and Components from the the Release file.

Tip

What APT keys is my Release file signed with?

Run s.th. like:

gpg --verify /var/lib/apt/lists/PATH_Release.gpg /var/lib/apt/lists/PATH_Release

for the Release in question to get a list of key ids the source is actually signed with.

A source must must have at least one valid key the Release file is signed with.

On the System

The actual repositories are managed via reprepro, and live in ~/repositories – each repository in its own subdir.

You normally don’t need to, but it’s technically perfectly fine to manually do package manipulations (on the shell, as user mini-buildd) using reprepro commands. I this case, of course, it’s in your power to meet or loosen restrictions that otherwise mini-buildd inflicts on the repository.

You must not manually change any repository’s configuration (i.e., on local disk) though, as these are handled/overwritten by mini-buildd.

To be able cope with multiple versions (reprepro does only allow one package version per dist) each distribution also has several additional *-rollbackN distributions configured.

Layout

It’s highly recommended to just stick with one of the default Layouts, as produced by the Defaults wizard.

In case you really need a custom layout, it’s recommended not to change the Default Layouts, but to create a new Layout profile with an appropriate name.

The Default Layout’s semantics are outlined in Default Layout.

The Debian Developers Layout is meant for mimicking a layout like in Debian unstable (no version restriction, upload to meta a distribution names like unstable) to test build packages meant for Debian.

You will interface with Layouts in Repositories, determining what suites (and rollback distributions) are available, which suites are uploadable, and which suites migrate, etc…

Stay In Sync With Default Layout Changes

In general, all the wizards never touch existing objects. This means, on existing systems, there is currently (1.0.x) no way (unfortunately) to stay on or easily upgrade to the defaults as provided by mini-buildd wizards.

For those who deliberately want to upgrade to these (recommended) defaults, here are instructions how to do this manually:

1.0.17: New Hotfix Suite

New in version 1.0.17.

  1. Enter the web application’s configuration section admin and login as superuser admin.

  2. For Layouts Default and Default (no rollbacks)
    1. Enter the editor for that layout.

    2. Add a new Suite Option (note: last entry shown in the list is for that purpose).
      • Extra Options: Rollback: 4

      • Suite: hotfix (note: you may need to add this; look for the green “+” sign below suite name.)

      • Uploadable: yes

      • Experimental: no

      • Migrates to: stable

      • Not Automatic: yes

      • But Automatic Upgrades: yes

  3. Re-index (PCA) all affected repositories.

Meta-Distributions

Meta-Distributions can be set in a Layout’s “Extra Options”.

Meta-Distributions may be seen as workaround to be able to upload (i.e., via debian/changelog) to other distributions than to the generic <codename>-<repoid>-<suite> format.

For example, the built-in “Debian Developers” Layout has mappings for unstable and experimental by default.

Note that these mappings are per Layout (and then, eventually, per Repository), but the final overall mapping must still be unique for the whole mini-buildd instance (as we only have one incoming, and the incoming change’s distribution must be unambiguous).

So, when using this feature, this usually means:

  • Make sure only one repository uses a Layout with Meta-Distributions configured (recommended).

  • Make any meta mapping key appear only once in each used Layout.

Changed in version 1.0.25: Ambiguity of the global meta distribution map is now checked for (on repository checks and implicitly on package builds).

Distribution

Distributions determines how and for what architectures a base distribution is to be build:

  • What base distribution? (sid, wheezy, lenny, …)

  • With what extra sources? (Debian Backports, …)

  • What components to support? (main, contrib, non-free, …)

  • With what generic build options? (resolver, lintian, ccache, eatmydata, …)

  • For what architectures? (i386, amd64, armel, …)

Error

eatmydata: Builds fail when linked with openvc!

Only a problem in current (Jan 2014) sid. See Debian Bug #733281.

Repository

A repository represents one apt repository managed via reprepro:

  • What repository identity? (“codename-identity-suite”)

  • What mini-buildd Layout? (“codename-identity-suite”, supported suites and their semantics)

  • What mini-buildd Distributions? (”codename-identity-suite”)

  • What misc configuration to use? (reprepro, static GPG auth, notify, …)

Uploader

Uploader instances are created automatically to each user profile. The administrator may activate GPG keys a user has uploaded, and decide what repositories he is allowed to upload.

Important

Don’t add or delete Uploader instances manually!

These are bound to users, and come automatically when new users are created. The administrator only changes these instances to grant rights.

Chroot

Adding (active) chroots to your mini-buildd instance implicitly makes it a builder.

Preparing a chroots will both bootstrap it, and create configuration on the system so it can be used via schroot.

You can chose amongst a number of schroot backends; to be able to be supported by mini-buildd, the backend must support snapshots (compare man 5 schroot.conf).

At the time (Oct 2016) of this writing, mini-buildd supports these backends:

Type

Options

Build size limit

Speed

Extra fs

Extra dependencies

Dir

aufs, overlayfs, unionfs, overlay

No

Medium

No

Kernel support (aufs <= jessie, overlay >= stretch)

Recommended

File

compression

No

Low

No

No

LVM

loop, given LVM setup

Yes

Fast

Yes

LVM tools, Kernel support (dm, in Debian standard kernel)

BTRFS

none

No

???

Yes

btrfs host file system, btrfs-progs

In short, we recommend directory based chroots via aufs using 3.2.35 =< Debian Linux Kernel < 3.18 (jessie-) and recommend directory based chroots via overlay with kernels > 3.18 (stretch+) as best compromise. It offers acceptable speed, and no limits.

File chroots are also fine, they will just always work; you may think about configuring schroot to use a tmpfs for its snapshots (if you have enough RAM), and use no compression to speed it up.

If you are in for speed, or just already have a LVM setup on your system, LVM chroots are good alternative, too.

Preparing chroots may take a while; if you cancel the HTTP request in your browser, preparation will continue anyway.

Tip

What’s eatmydata and ccache?

You may configure Distributions with generic build options that may also affect the backend (like pre-installing eatmydata) or build (like configuring ccache to be used) speed. See Distributions and Repositories.

Error

debootstrap fails for <=lenny chroots on >=jessie host kernel (uname)!

See Debian Bug #642031. This should ideally be worked around in debootstrap itself eventually.

mini-buildd comes with a workaround wrapper command ‘mini-buildd-debootstrap-uname-2.6’. Just add:

Debootstrap-Command: /usr/sbin/mini-buildd-debootstrap-uname-2.6

to the chroot’s extra options to work around it (the default chroots created with the chroot wizard already include this workaround for lenny and etch chroots, btw).

Fwiw, this is due to older libc6 packaging’s preinst, which will meekly fail if uname -r starts with a two-digit version; i.e.:

FINE : 3.2.0-4-amd64      Standard wheezy kernel
FAILS: 3.10-2-amd64       Standard jessie/sid kernel
FAILS: 3.9-0.bpo.1-amd64  Wheezy backport of the jessie/sid kernel

Error

Fails to build “all” packages with “build archall” flag set to arch “x” in case DSP has >= 1 arch “all” and >=1 arch “y” binary package!

This is due to sbuild and in in more detail explained here Debian Bug #706086.

A bad one-package workaround would be to change the “build archall” flag to arch “y”.

Error

LVM chroots fail running lvcreate with ‘not found: device not cleared’!

Unclear (?). See Debian Bug #705238 or http://lists.debian.org/debian-user/2012/12/msg00407.html .

“–noudevsync” workaround makes lvcreate work again, but the chroot will not work later anyway later.

Tip

My chroot creation fails due to missing arch in archive (partial mirror)?

This might occur, for example, if you use a (local) partial mirror (with debmirror or the like) as mini-buildd archive that does not mirror the arch in question.

At the moment, all archives you add must provide all architectures you are going to support to avoid problems.

Tip

sudo fails with “sudo: no tty present and no askpass program specified”?

Make sure /etc/sudoers has this line:

#includedir /etc/sudoers.d

(This is sudo’s Debian package’s default, but the administrator might have changed it at some point.)

Remote

Remotes can interconnect a mini-buildd instance with another in a peer-to-peer fashion, i.e., you need to add a respective remote instance on both two peers. When interconnected, these two instances automatically share their build chroots.

To interconnect two mini-buildd instances

  1. Add remote on instance0 for instance1; prepare, check, and activate it. Activation will initially fail, but it will be put on auto-reactivate (A).

  2. Add remote on instance1 for instance0; prepare, check, and activate it. Activation will work as instance0 already knows us, and is on auto-reactivate.

  3. Run Activate remote on instance0.

Important

Be sure to use the exact same host names as given in the resp. instance’s Daemon configuration!

Advanced Topics

Custom Event Hooks

Optional custom event hooks may be defined in ~/etc/event-hooks.

Any executable files matching [!_.]*[!~] glob pattern will be considered.

They will always be called with exactly one argument, the file name of the saved (json) event file.

All scripts found will run in alphabetical sort order, in the context of mini-buildd (i.e., as user mini-buildd).

mini-buildd is agnostic to whether these hooks do succeed, rather merely logs results to daemon.log.

Layout example:

00-do-this-first
50-do-that
99-do-that-last

Internal Crontab

New in version 2.0.0.

Periodic task are now handled internally, see mini-buildd’s ‘crontab’ page to monitor runs or potentially debug problems.

I.e., there are no more ‘system-cronned’ service restart workaround that could cause problems in the past.

The mini-buildd user

mini-buildd relies on having a user called mini-buildd; the Unix home of this user is the home of your mini-buildd instance, and the Unix daemon runs with its user id.

In mini-buildd’s home, you will find this top level layout; i.e., these are handled by mini-buildd itself, and should not be touched manually (unless you really know what you are doing, of course):

config.sqlite       mini-buildd's configuration.
incoming/           Directory served by the ftpd.
var/                Variable data: chroots, logs, temp directories, build directories spool.
repositories/       Your valuable repositories.
.gnupg/             The instance's GnuPG key ring.
.mini-buildd.pid    Unix daemon PID file.
.django_secret_key  Some django shit we need.

When you remove the mini-buildd package without purging, it will remove system artifacts (see --remove-system-artifacts option, this currently affects chroots only) that can only be properly removed with mini-buildd installed. Otherwise, mini-buildd’s home (and, of course, the repositories) stay intact.

When you purge the mini-buildd package, all traces will be removed from the system, including your repositories.

Logging and Debugging

Tip

Can I manually instantiate a build chroot to repeat/debug a failed build?

Yes, see command ‘mini-buildd-debug-build’ – you need to be administrator, though.

Per default, mini-buildd logs

  • to mini-buildd’s log file ~/var/log/daemon.log.

  • via syslog, facility USER (which usually ends up in /var/log/user.log).

The former is handled by mini-buildd itself, including rotating and access to it via API calls.

The latter is the same place where sbuild and friends put their logs by default.

You may control the log level via the --verbose, and extra debug options via the --debug command line flag. The latter allows you to keep temporary files, enable python exception traces or to enable debug options for specific software components used by mini-buildd. See command ‘mini-buildd’ for full documentation.

New in version 1.0.19: Debug option sbuild.

Just use dpkg-reconfigure mini-buildd or edit /etc/default/mini-buildd to set any of these options permanently.

Debug run in the console

This example gives you a full treat of logs to the console (you may vary with the arguments to suit your needs):

systemctl stop mini-buildd
PYTHONWARNINGS="default" /usr/sbin/mini-buildd --log-level DEBUG --debug=exception,http,webapp

Profiling run on the console

systemctl stop mini-buildd
python3 -m cProfile -o mini-buildd.profile /usr/sbin/mini-buildd

Auth via SSH wrapper

See command ‘mini-buildd-ssh-uploader’ command ‘mini-buildd-ssh-client’ – i.e., yes, look at the instructions in these script.

HTTP access log

Mini-buildd also keeps a standard HTTP access log in ~/var/log/access.log.

Keyring and test packages

On mini-buildd’s home, you will find action buttons to create+build keyring packages, as well as running test packages.

linux >= 4.8.5 (amd64): Re-enable vsyscall for <= wheezy builds

In Debian kernel packages since 4.8.4-1~exp1:

[ Ben Hutchings ]
* [amd64] Enable LEGACY_VSYSCALL_NONE instead of LEGACY_VSYSCALL_EMULATE.
        This breaks (e)glibc 2.13 and earlier, and can be reverted using the kernel
        parameter: vsyscall=emulate

I.e.: When running the Debian standard kernel and your mini-buildd instance needs to support wheezy or earlier, you need to re-enable this (in /etc/default/grub).

You may use local-vsyscall-emulate.cfg in examples (a symlink should do) for this purpose.

On any running kernel, this is a poor man’s check to see if vsyscall is still enabled on your system:

grep "\[vsyscall\]" /proc/self/maps

See also

Debian Bug #847154, linux package’s NEWS file.

django: Avoid downgrades (does not start after downgrade)

mini-buildd usually is compatible with several django main versions (see control file). This, package-wise, allows for downgrading django (maybe you want to go back from backports to stable for some reason).

This, however, will mostly always cause problems as the SQL database scheme of your app has already been updated.

In case this already has happened, you can only upgrade django again (or somehow try to manually downgrade mini-buildd’s SQL (~/config.sqlite) if you dare).

Cruft in ~/var/log

With mini-buildd <= 1.0.17, rejected packages where logged here too, which may have lead to cruft that is never cleaned up. You may run:

mini-buildd-reject-cleanup

Import a foreign archive key to an existing mini-buildd instance

  1. Stop the mini-buildd service.

  2. Become the mini-buildd user.

  3. Manipulate the user’s GPG keyring
    • Be sure it contains exactly one key (pub+sec) when done.

  4. (Re)start the mini-buildd service.
    • Check that the Daemon key has actually changed (f.e., on the web home, right bottom).

  5. Make a pseudo change to all repository instances.
    • Just enter the repo editor, don’t actually change anything, but do “save”.

    • This fixes the status to “Prepared (Changed)” (matching the external manipulation).

  6. “PCA” ((re)prepare, check, create) all repositories.
    • This should bring the new key to the reprepro indices.

  7. Re-create keyring packages.

Important

The Daemon instance does not touch the GPG setup once it’s created – unless you do an explicit remove on the instance.

Possible problems fetching keys from key servers (gpg 2.1, 2.2)

Since gpg 2.1.22, ‘use-tor’ option is default. Afaiu, there is some magic in dirmgr now trying to autodetect if tor is available, and then uses this (safer) option.

In practice, we have seen that receiving from keyserver has become unreliable, sometimes failing with:

gpg: keyserver receive failed: Connection closed in DNS

and sometimes with:

gpg: WARNING: Tor is not properly configured
gpg: keyserver receive failed: Permission denied

and occasionally working fine.

Changed in version 1.0.34,1.1.9: mini-buildd’s internal importer now first tries to utilize keys from installed Debian or Ubuntu archive key packages (and add ‘Suggests:’ for them) before reverting to ‘recv’ from the configured keyserver. Also, the keyserver import is now being retried.

Ulimately, the GPG’s defaults should be used, and dirmngr should be more reliable. If this bugs you however, you might try the following options to mitigate the problem:

  • Update tor; on a stretch system, updating from 0.2.9.14 -> 0.3.2.9 seems to improve the success rate.

  • Remove tor from the system.

  • Use no-use-tor in dirmngr.conf. This might eventually be an option when there is a systemwide default config for user-context dirmngr instances. Currently, that’s not the case, and also no command line option to tunnel that through.

Migrate packages from 0.8.x

Tip

Can I skip migrating from 0.8.x?

Yes – a much simpler solution might be to just copy the old repository directory (~/rep) somewhere and serve it via some standard web server, and just continue to use it along with your new repo as long as needed.

This roughly explains the steps needed to upgrade a mini-buildd 0.8.x installation to 1.0.x with transferring the packages from the old 0.8.x repositories over, so you can continue with the new 1.0.x repos only:

  1. Upgrade mini-buildd from 0.8.x to 1.0.

    Chances are this might have already implicitly happened, with some update.

    You will then have 1.0 up and running, and ye olde 0.8.x repositories still available as read-only apt repositories.

    Just be sure you don’t purge the old package, and then install 1.0, as this will remove the whole old repository.

  2. Configure mini-buildd 1.0.

    This means you should, in the end, have a 1.0 repository with the same identity as the old 0.8.x repository, and with all distributions you want to migrate.

  3. Import packages.

    Become mini-buildd user, and got to the new 1.0 reprepro repository you want to import to, and use the importer script to migrate packages:

    cd repositories/REPOID
    mini-buildd-import-08x ~/rep/squeeze-REPOID-experimental squeeze-REPOID-experimental
    mini-buildd-import-08x ~/rep/squeeze-REPOID squeeze-REPOID-unstable
    

    This example is for squeeze; repeat the imports for all base distributions you want to migrate.

    Thusly, ye olde *-experimental distribution will be migrated to the distribution with the same name in 1.0. Ye olde squeeze-REPOID goes to squeeze-REPOID-unstable. For the latter, multiple package version will be automatically installed to the new rollback distributions (which are needed with reprepro to support multiple package versions).

  4. (Optional) Fix up package status.

    All the migrated packages are now in 1.0 “unstable” distribution; you may think of bulk-migrating them all to “stable”, if that were your semantics for the 0.8.x non-experimental distributions.

Eventually, when everything is updated, you may of course (re)move the old 0.8.x directory ~/rep/.