Gentoo tips and tricks

This page deals mainly with a home network of more than one computer running Gentoo Linux. Some of it applies only to machines of same architecture, some also to mixed networks.

Shared /usr/portage via NFS

This is pretty straightforward after you know the basics of NFS. Set up a server, preferably with a dedicated /usr/portage partition.

Do emerge --sync only on the server. (It's possible, but slower, on any client machine too.)
Set the NFS directory read/write for root, as in my current /etc/exports:
```
  # v4
  /nfs            192.168.4.0/24(rw,fsid=0,insecure,no_subtree_check)
  /nfs/portage    192.168.4.0/24(rw,insecure,no_root_squash,no_subtree_check)
```
This way, you can do full emerges on the client machines, including downloads into distfiles, and building shared binaries. Thus, the shared /usr/portage is not just for ebuilds.

A shared portage tree also works for multiple architectures. The only caveat that I know of is the packages directory -- it's /usr/portage/packages by default. If you use shared binaries, you should set this differently for each architecture. For example, I have this in /etc/make.conf on my iMac:

PKGDIR="/usr/portage/packages/ppc"

Shared binaries for many machines

With Gentoo, you can optimize everything for each machine if you like. On the other hand, for most software it makes sense to use the same binaries. FEATURES=buildpkg in make.conf will build these packages after a succesful emerge. Afterwards, on the other machines, use emerge -k to install the binary if available.

Of course, CFLAGS should be limited to run on all target CPUs. Currently (2012-03-10) I do this between a Core 2 Duo T7200 and an Atom D510, which have almost the same instructions, but not quite. I use "-march=core2 -mtune=atom" as a working compromise.

Compiling in ramdisk

To speed up compilation somewhat, and save the hard drive, I follow this tip. I've modified the script slightly to my tastes.

In some rare cases the ramdisk compilation fails. The obvious case is a lack of space. Note, however, that it's OK to have a ramdisk larger than your actual RAM, provided you have enough swap. Be sure to leave some free memory for the actual compilation processes and whatever else you have running ;)

Distcc and Ccache

You've saved quite some time by reusing the binaries. Save some more by distributing and reusing the compilation process.

Ccache is really straightforward, just emerge it, and add "ccache" in the FEATURES of make.conf. Gentoo Wiki tip has some fine tuning tips, but I have been doing fine with the defaults.

Distcc isn't too complicated either. Emerge it, put "distcc" into FEATURES, and remember two things:

On the server machines (preferably, that means all of your machines), edit /etc/conf.d/distccd to allow the clients in your network.
On the clients, let the compiler know what servers to use. For example, I have the following line in /etc/distcc/hosts:
```
localhost/2 hoo/4
```
This means distcc will run at most two compilers on localhost, and four on the server "hoo". You can leave out the /number and distcc will use some sensible defaults.

DISTCC_HOSTS also denotes the order in which servers are called. For example, put the fastest machines first. Or if you have some machines not always on, put them last.

There are mixed opinions on the suitable number of distcc processes per machine or CPU. Some people suggest N+1 and others 2*N, for a machine with N CPUs. In any case, the idea is to have more processes than CPUs. When a compiler has just finished and is waiting for new work, the CPU can go on doing something useful.

Now that there are 6 compilers available, you need MAKEOPTS="-j6" in make.conf. Don't forget to /etc/init.d/distccd start on the servers ;)

OTOH, managing too many instances of distcc can be heavy on the client machine. This becomes a real problem in those cases that cannot use distcc, for example some stages of compiling GCC. (See below for per-package settings.)

You can also use distcc in a mixed environment with cross compilers. For example to help a lonely PPC machine compile faster in a room full of x86s.

Distcc and Ccache for non-portage uses

Now that you have a caching compile cluster, feel free to use it for whatever compiling, not just Portage. Generally you only need to define the C (and possibly C++) compilers, and use the above MAKEOPTS. The usual ./configure; make invocation then becomes

$ CC="ccache distcc" CXX="ccache distcc g++" ./configure
$ make -j6

Another possibility is to set CC directly as an option for make. This works for the Linux kernel, for example:

# make CC="ccache distcc" -j6 bzImage modules
# make CC="ccache distcc" modules_install install

The -j6 sometimes messes up the installation stage, so it requires another line. Of course you can put this all into a nice script.

Here's another introduction to distcc, ccache and make -j at IBM DeveloperWorks.

Per-package settings for portage

My /etc/portage/bashrc from an ARM server:

case ${CATEGORY}/${PN} in
    dev-db/sqlite|dev-lang/python|app-office/openoffice|dev-python/numpy)
        export CFLAGS="${CFLAGS//-ffast-math/}"
        ;;
    sys-devel/gcc|sys-libs/glibc|dev-libs/ppl)
        # does not use distcc, so this would be heavy on a light machine
        export MAKEOPTS="-j1"
        ;;
    dev-vcs/git|sys-fs/udev)
        # adds its own -march= which conflicts with -mcpu=
        shopt -s extglob
        export CFLAGS="${CFLAGS//-mcpu=+([a-z0-9-])/}"
        ;;
esac

Changing the CHOST

There is again a definitive guide which you might want to follow. However, with distcc and ccache things can get more tricky. The obvious solution is to do this without using them in the emerge. In this case, the job count in MAKEOPTS is reduced, as only local compilers are available.

FEATURES="-distcc -ccache" MAKEOPTS="-j2" emerge -1 binutils gcc glibc

In my experience, binutils and gcc can be emerged with the old distcc and ccache. But once the compiler is changed, emerging glibc must be done by other means, i. e. without distcc and ccache.

Afterwards, distcc and ccache should be recompiled. If you use my crossdistcc script, it should be re-run as well. At this point you may still have the old MACHTYPE in the shell, so you should force the new one for the script.

With distcc, you probably need to emerge correct cross compilers for the distcc servers as well.

Example

This is what worked for me when changing from armv5tejl to armv5tel.

# Change the CHOST
jmacs /etc/make.conf

emerge -1 binutils gcc

gcc-config -l
gcc-config 1 # or whichever gcc you use

FEATURES="-distcc -ccache" MAKEOPTS="-j2" emerge -1 glibc

grep -sr armv5tejl /etc/env.d # then delete old references

# Also remove other old, unused bits
rm -r /usr/armv5tejl-softfloat-linux-gnueabi
rm /usr/bin/armv5tejl-softfloat-linux-gnueabi-*

env-update
. /etc/profile

# As advised in the Gentoo chost manual
FEATURES="-distcc -ccache" MAKEOPTS="-j2" emerge -1 libtool
/usr/share/gcc-data/armv5tel-softfloat-linux-gnueabi/4.4.4/fix_libtool_files.sh 4.4.4 --oldarch armv5tejl-softfloat-linux-gnueabi

# bash is not really essential at this point, but it contains the CHOST
# in some builtin variables, so eventually it needs re-emerging
FEATURES="-distcc -ccache" MAKEOPTS="-j2" emerge -1 bash ccache

# distcc compile needs a correct python
FEATURES="-distcc" MAKEOPTS="-j2" emerge -1 python

# python 3.1.2 is the latest, but this is still the default in Gentoo
FEATURES="-distcc" MAKEOPTS="-j2" emerge -1 python:2.6

FEATURES="-distcc" MAKEOPTS="-j2" emerge -1 distcc

# Force the new machtype, if you still have the old variable in shell
# Alternatively, exit and restart shell to get the new bash variables
MACHTYPE=armv5tel-softfloat-linux-gnueabi crossdistcc.sh

# Perl packages, as advised in the manual
emerge -av1 `qfile /usr/lib/perl* -Cq | sort -u`

The Perl package emerge seemed to have conflicts, which I resolved by removing all of the problematic virtuals, and proceeding with the above command.

Risto A. Paju