[updated: the version of the python script was an obsolete one, I updated it, and changed the title of the post for more clarity]

I have 40 old computers (donation from a corporation) that are to be dispatched among small social work non-profit organizations and needy people in several French cities and probably also in Senegal. How to install a customized and usable version of linux on all of them despite the hardware heterogeneity of that collection of PCs and our lack of time? How to allow them to be reinstalled remotely without requiring any computer person to be present on site? I want the linux distribution to be Ubuntu, with a specific list of packages and configuration parameters. Some PCs have 1 hard drive of 9 GB or more, some others have up to 3 hard drives of sometimes 4GB, etc. The solution I found is to use FAI (Fully Automatic Installation) with a couple of custom enhancements such as a Python script that calculates the optimal partition tables for every PC.

Here are some notes about how I proceeded. If you want to contribute to similar projects (Information Technology and innovation for small non-profit organizations working in the field of social work in France or Africa), please drop me a comment here or by email at sig at akasig dot org.

Requirements and architecture

The way FAI works is as follows. The computer to install boots locally either from a CD-ROM, from a floppy disk or via a local networking protocol such as PXE or BOOTP. It then connects to a central installation server. It is served with instructions about how to install itself. It then downloads and installs packages from official repositories (e.g. Ubuntu repositories) or from the installation server if ever this server contains a mirror of the distribution repository. It is a package-based installer and differs from file-based installers such as System Imager (that relies on rsync).

Therefore, the main requirement is to have a server for centralizing the installation process. For testing purposes, I used my home PC with its DSL line and its Ubuntu Dapper distribution. But the production server is hosted in a data center and runs a debian.

For booting, the usual FAI way is to use a local DHCP server for retrieving information such as the address of the installation server. But in my case, I want to allow computers to (re-)install themselves from the premises and local area networks of non-profit organizations or even at home of individuals. I obviously can’t control the DHCP servers that are usually serving this critical installation information. Therefore, I had to work around this by using some special FAI options when creating the bootable CD-ROMs (see below).

Another issue I had to tackle is that FAI supports a limited amount of hardware heterogeneoity. For instance, if your computers don’t have the exact same amount of hard drive space, that’s usually not a problem for FAI. It comes with configuration mechanisms that handle that quite smoothly. But in my case, I have unknown computers to install, with various numbers and sizes of hard drives for instance. Therefore, I had to let computers calculate by themselves the optimal partitioning scheme for any hard drive setup. I did that with the help of a constraint programming library for Python. I also had to make sure that Python would be available on the computer at that stage of the installation process.

Eventually, I had to work around some access control constraints of FAI so that I could write the calculated optimal partitioning scheme to the computer to install. Indeed, when the computer to install first connects to the installation server, it mounts its root partition via NFS in read-only mode. And it doesn’t have access to the hard drive(s) yet. The solution I adopted is to write the optimal partitioning configuration to the FAI RAMdisk (on the computer to install) and to pre-define a symlink from the NFS-mounted root (on the installation server) to that configuration file so that FAI knows where to find it once it has been calculated (details below).

Other modifications I had to do include correcting some shebang lines in scripts that used sh whereas they should have been using bash in the case of an ubuntu server environment. I also had to correct the path a grub post-installation script to adapt it to Ubuntu. Eventually, I had to find the proper collection of FAI « classes » to define in order for Ubuntu to work properly.

I did not invent all of these tweaks and hacks (except the partitioning one). All of them were suggested by the extremely supportive FAI community via their #fai IRC channel on irc.oftc.net (special thanks to MrFai, sanso and h01ger). And I could not get into FAI without the great (but partly outdated!) FAI guide.

Now come the more detailed notes about how to (hopefully) reproduce the steps and tweaks described above.

A bit of FAI magic

On my Ubuntu dapper, the FAI package was a rather old one (v.2.10). Therefore I retrieved the more recent 3.1ubuntu1 package from the edgy repository and installed it manually. The first thing to do was then to go to /etc/fai and check every configuration file for possible updates to make. In /etc/fai/NFSROOT, for instance, I added python as a package to install in the virtual root partition that will be mounted via NFS by the computers to install. I also made sure that my NFS service would allow the target computers to connect and that the iptables firewall would not block these connections either. Then, I was ready for a sudo faisetup that created this virtual root partition under /srv/fai.

Once the NFS root hierarchy has been created, I manually added to it the python constraint programming library required by my partitioning hack. I downloaded the source tarball, pre-compiled it on my installation server with python setup.py build (probably useless). And I manually copied the .py and .pyc file to the proper site-packages directory of the NFS root (to /srv/fai/nfsroot/usr/lib/python2.4/site-packages/).

In order to create the bootable CD-ROMs that would allow computers to start using their local DHCP server but still to know how to connect to the central installation server, I had to use the following command line and options:

sudo make-fai-bootfloppy -B -Ieth0 -l -f /tmp/fai_floppy.img -i /tmp/fai_cdrom.iso -d f -F -v nfsroot=192.168.0.100:/srv/fai/nfsroot ip=:192.168.0.100:::::dhcp FAI_ACTION=install

It creates an ISO image of that bootable CDROM (/tmp/fai_cdrom.iso) that is then to be burnt. It also tells the path to the installation server. I had first tried without the -l option that asks for LILO to be used instead of GRUB but I could not figure out how to let GRUB not ignore the required nfsroot option. That option always disappeared from the kernel options GRUB specifies for booting. Therefore, I decided to use LILO instead. I also had troubles mixing the use of DHCP and the use of my nfsroot option and had to use the -d f option that is supposed to tell the computer to boot with a fixed IP address whereas it will actually refer to my ip= option that tells it to boot with DHCP but to notice that the installation server is at a given IP address. A bit tricky, isn’t it… Anyway, it worked and you just have to replace 192.168.0.100 by the IP address of your installation server and everything should be fine (let’s be optimistic…). As an alternative, you should refer to the man page of fai-cd which is another FAI command for creating a bootable CDROM. Maybe fai-cd is even more recommended than make-fai-bootfloppy indeed but I did not try because it has not yet been properly documented in the FAI guide.

Then, I added my partitioning script as a FAI hook that gets called just before the « partition » FAI task and only for computers that are assigned to my custom FAI class. In order to do so, I saved my script under /srv/fai/config/hooks with the filename partition.MYCLASS (where MYCLASS is the name you choose for describing the class of computers that will be using this partitioning script). Note that you should remove the .txt extension from the filename once you download it from this site.
When called, that script would create a new file name called MYCLASS that would contain the FAI syntax for specifying how partitions are to be created on disks (it’s called a FAI disk_config file indeed). But since this script is called at install time from a computer that mounted its root partition via NFS in read-only, I had to let the script save this MYCLASS file under /tmp/ which is then a writeable RAM disk. But for FAI to be aware of the existence of that file during its partitioning task, I first had to create a symlink from /srv/fai/config/disk_config/MYCLASS to /tmp/MYCLASS. (ln -s /tmp/MYCLASS /srv/fai/config/disk_config/MYCLASS). After some discussion with FAI folks on IRC, I understood this is not the optimal solution. Ideally, I should use the FAI mkrw script instead: it would create an appropriate writeable path on the RAM disk and the script would be stored there. Anyway, the symlink option also works though it’s less elegant.

Beyond creating this customized disk_config file for MYCLASS computers, I also modified and re-used the simple example files that are provided under the FAI examples directory. I created a FRENCH class by copying and modifying the GERMAN class that is provided there so that it tells that KEYMAP=fr-latin9. I used the FAIBASE class file and just modified it a bit: TIMEZONE=Europe/Paris. In the /srv/fai/class/50-host-classes script that defines default classes, I added to the last case the following classes: FRENCH, FAI_BOOTPART (in desperate hope it would add to the GRUB menu an option for booting the computer using FAI from the hard-drive in case of re-installation without CD), NTP and NETWORK (being unsure these were required for the NTP service to be installed by default and to receive proper configuration parameters. In /srv/fai/config/debconf/ I created a FRENCH debconf file by re-using the GERMAN one given as example. In /srv/fai/config/package_config/, I also copied and modified the GERMAN file into one called FRENCH and containing the identifiers of Ubuntu packages related to French setups.

As explained above, I also had to modify several FAI scripts in order to fully adapt them to the Ubuntu environment. These modifications were suggested to the FAI mailing list and forwarded to me by MrFai. They consisted in changing the #! /bin/sh shebang lines into #! /bin/bash lines for the following scripts: class/10-base-class, class/50-host-class, hooks/instsoft.FAIBASE, scripts/FAIBASE/10-misc, scripts/FAIBASE/30-interface, scripts/LAST/50-misc and also the mount2dir script. Last but not least, these modifications included modifying the config/files/boot/grub/menu.lst/postinst file so that it made reference to /sbin/update-grub instead of /usr/sbin/update-grub. I suppose that these changes are soon to be available right from the downloadable FAI packages once they get propagated there.

That’s it. With all of these operations, you should be able to install a fresh (and French) Ubuntu on any computer using the CD you have burnt. Or at least I could install a couple of them.

More things to do

But even then, we are not done with what’s required for our distributed infrastructure to be remotely maintainable. Here is my to-do list:

• many of the hacks and tricks indicated above should probably not be done directly under the /srv hierarchy but under something like /usr/lib/fai/ or you might face the risk that they some of them get lost next time you recreate your nfsroot using the fai-setup script; there is probably some cleaning to be made here
• check that the FAI_BOOTPART class was really taken into account because, at the moment, I could not see any FAI option in the GRUB menu of the installed computers
• add bcfg2 with custom parameters to the classes to install so that the configuration can be properly managed remotely
• check once again that the way NFS is offered to these remote computers will not create any security issue
• create a new script that will select the flavor of the distribution to install depending on the amount of RAM on the PC. For instance, with less than 256 MB of RAM, it would be preferable to install a basic Ubuntu (without its Gnome desktop) and use another windows manager
• setup the default user because the one provided by the DEMO class does not suit my need
• add some more intelligence to the partitioning script so that it checks if ever there already is a suitable home partition and then ask FAI to preserve it instead of recreating it (and loosing the data it contains)
• setup a proper SSH account on the server so that FAI can save its log files on it once the installation is done
• activate the option that will let FAI save on the installation server the detailed hardware information it could read from the PC
• create a unique and permanent identifier to be stored on the machine and on the server so that we can track PCs; in a first step, the MAC address may be usable but in some future, assigning a permanent UID to the whole list of hardware characteristics could be better if done smartly
• check that the default Xorg options do not put old screens at risk (resolution and speed)
• bring a bit of graphical customization in order to brand the desktops
• add openvpn to the PCs so that we can connect to them remotely even when they are behing NAT routers
• configure the authentication so that it is made against the central database (MySQL) that would also be used for the identity management of our Plone sites, with an nss_update mechanism that will allow authentication to succeed even when the central server is not reachable (caches the credentials on the PC)
• for facilitating the initial installation, I should probably stop using bootable CDs and get back to the orthodox FAI way of booting from the network and using DHCPd for delivering instructions about the location of the installation server; however I first have to figure out how to let the computers’ GRUB menus offer a boot option that will not require the DHCPd to deliver those instructions and that will let them use the central installation server somewhere over the Internet

If you are interested in helping some social non-profit ventures with the maintenance and configuration of their PCs and/or have some clues about how to take action on some items of this todo list, please don’t hesitate to get in touch with me and to leave some comment here. Your help would be very much welcome!