Yocto build on a cluster with IceCC

Yocto build on a cluster with IceCC

This section is copied from https://www.openembedded.org/wiki/Using_IceCC

IceCC and OE

It is possible to compile on a cluster of machines with OE, and quite easily so. To do so add the following to your local.conf:

ICECC_PARALLEL_MAKE = "-j 24"
INHERIT += "icecc"

General consensus is that ICECC_PARALLEL_MAKE should be between 2x to 4x the number of cores your CPU has. Too many and your PC will be unusable, too few and you won't keep the cluster busy.

Icecc and sstate interaction

Inheriting icecc will change the hashes for all tasks, so you cannot share sstate between builds that have icecc inherited and those that don't. The recommended solution is add support for icecream to all builds as follows:

INHERIT += "icecc"
ICECC_DISABLED ??= "1"

Now, users that want to use icecream can enable it by setting ICECC_DISABLED = "0", but everyone can still share sstate.

Recipe build failures

Many recipes may fail to build under icecream for a variety of reasons. If you find a recipe that does so, you can add it to ICECC_USER_PACKAGE_BL in local.conf.

We try to keep the system blacklist (ICECC_SYSTEM_PACKAGE_BL) up to date, but packages sometimes get missed (or aren't part of oe-core and don't get tested at all).

Note that any recipe that disables PARALLEL_MAKE by setting it to "" will automatically have icecream disabled.

Dedicating a scheduler

By default, icecream will pseudo-randomly assign one of the compile nodes to be the scheduler (the node that decides where all the compiles get distributed). In some cases, it may be desirable to dedicate a specific computer to be the scheduler. You may want to do this if:

• The cluster is large. The CPU time required to do scheduling can be noticable
• The cluster has a lot of mobile devices (e.g. laptops). If one of these happens to be assigned to be the scheduler and then goes offline, it can take an while to recover.

Network Bandwidth Considerations

Icecream performance is very dependent on the speed of the network between the cluster nodes. As a rule of thumb, 1 Gpbs or better is ideal, and with anything less than 100 Mbps, it's probably not worth using at all. Wifi is dubiously useful at any speed.

Credits

Icecream support was originally implement by zecke, improved by Ifaistos, and most recently by Joshua Watt (JPEW)

Utilities

There are several useful utilities when dealing with icecream:

• icemon - A gui tool for monitoring the cluster
• icecream-sundae - A command line tool for monitoring the cluster

More Information

• Sweeten Your Yocto Build Times with Icecream - Joshua Watt, Embedded Linux Conference North America 2019 (slides)