1. What is e-smith?

The e-smith server and gateway system is a Linux distribution designed for small to medium enterprises. It's intended to greatly simplify the process of setting up Internet and file sharing services, and it can be administered by a non-technical user with no prior experience of Linux.

We chose Perl as the main development language for the e-smith server and gateway both because of its widespread popularity (making it easier to recruit developers) and because it's well suited to e-smith's blend of system administration, templating and web applications development.

Of course, the system isn't just Perl. Other parts of the system include the base operating system (based on Red Hat 7.0), a customised installer using Red Hat's Anaconda (which is written in Python), and a range of applications including mail and web servers, file sharing, and web-based email using IMP (which is written in PHP). However, despite the modifications and quick hacks we've made in other languages, the bulk of development performed by the e-smith team is in Perl.

2. The e-smith manager: a Perl CGI application

Administration of an e-smith server and gateway system is performed primarily via a web interface, called the "e-smith manager". This is essentially a collection of CGI programs which display system information and allow the administrator to modify it as necessary.

This allows system owners with no previous knowledge of Linux to administer their systems easily without needing to understand the arcana of the command line, text configuration files, and so on.

The manager interface is based on the CGI module which comes standard with the Perl distribution. However, a module esmith::cgi has been written to provide further abstractions of common tasks such as:

• generating page headers and footers

• generating commonly used widgets

• generating status report pages

It is likely that this module will be further extended in the next version to provide more abstracted ways of building "wizard" style interfaces, so that developers don't have to copy-and-paste huge swathes of code calling the CGI module directly.

3. Global configuration files

The e-smith server and gateway is a collection of many parts, all of which can be configured in different ways depending on the user's needs. All the global configuration data is kept in a central repository, and this information is used as the base for specific configurations for the various software on the system.

Much of the system configuration data is kept in a simple text file, /home/e-smith/configuration. The basic format is name=value, as shown below:

AccessType=dedicated
ExternalDHCP=off
ExternalNetmask=255.255.255.0

Obviously this can be simply parsed with Perl, by using something equivalent to:

my %conf;
while (<>) {
    chomp;
    my ($name, $value) = split(/=/);
    $conf{$name} = $value;
}

print "$conf{DomainName}\n";

However, it is also possible to store more information in a single line by adding zero or more pairs of property names and values delimited by pipe symbols (|), like this:

fetchmail=service|status|enabled
flexbackup=backupservice|erase_rewind_only|true
ftp=service|access|private|status|enabled

Parsing this gets slightly more tricky, requiring an additional split and putting the property names and values into a hash.

As it happens, e-smith have a module with common utilities such as this built in, so in fact a developer would only write something like this:

use esmith::db;

my %conf;
tie %conf, 'esmith::config', '/home/e-smith/configuration';

# in scalar context, for simple configuration entries
my $domain = db_get(\%conf, 'DomainName');

# or get a scalar and a hash when looking at a more complex entry
my ($type, %properties) = db_get(\%conf, 'ftp');

Similarly to the main configuration file, user and group information is kept in /home/e-smith/accounts, in a format which is likewise simple to parse.

This simplicity is intentional; although the data could have been stored in a more complex database, the developers decided to keep the core of e-smith as simple as possible so that the learning curve for new developers would not be steep.

4. Templated configuration files using Text::Template

Things get more complex when we examine the configuration files stored in /etc. Every piece of software has its own configuration format, and writing parsers for each one can be a complex, time-consuming and error-prone process. The e-smith software avoids the whole issue by using a template-driven system instead, using Text::Template.

Templates are stored in a directory hierarchy rooted at /etc/e-smith/templates. Each configuration file is either a Text::Template file, or can be given a subdirectory in which template fragments are stored. These templates are then parsed (and in the case of a subdirectory of fragments, concatenated together) to generate the config files for each service on the system. The fragmented approach is part of e-smith's modular and extensible architecture; it allows third-party modules to add fragments to the configuration if necessary.

For example, let's take a look at the ntpd service (which keeps the system's clock right by querying a time server using the Network Time Protocol). It usually has a config file /etc/ntp.conf. On an e-smith server, this is built out of the template fragments found in the /etc/e-smith/templates/etc/ntp.conf/ directory.

This is a simple template, and only requires basic variable substitutions. (Since Text::Template evaluates anything in braces and replaces it with the return value of the code, some templates have more complex code embedded in them.) Here is what's in the template fragment file /etc/e-smith/templates/etc/ntp.conf/template-begin:

Server { $NTPServer }
driftfile /etc/ntp/drift
authenticate no

In this example, $NTPServer would be replaced with the value of that variable.

Instead of calling Text::Template directly, e-smith developers use the e-smith::util module to automate the process. Here's how we'd generate the ntp.conf file:

use esmith::util;

processTemplate({
    CONFREF => \%conf,  # this is the %conf from the last section
    TEMPLATE_PATH => '/etc/ntp.conf',
});

The above example takes advantage of a number of default values set by the processTemplate(). If we want more control, we can specify such things as the user ID (UID), group ID (GID) and permissions to use when writing out the configuration file.

use esmith::util;

processTemplate({
    CONFREF => \%conf,  # this is the %conf from the last section
    TEMPLATE_PATH => '/etc/ntp.conf',
    UID => $username,
    GID => $group,
    PERMS => 0644,
});

Incidentally, the way in which the processTemplate() routine lets the programmer override the default values is a good example of Perlish idiom:

# the parameter hash the programmer passed to the routine is
# %params_hash, and the defaults are in %defaults.  The two are merged
# together by simply concatenating them as lists:

my %p = (%defaults, %params_hash);

# Because of the way hashes work, any key in %params_hash which is the
# same as one in %defaults_hash will overwrite it.

5. Events and actions

When the user hits "submit" on a web form, a number of things can occur:

• master configuration files are updated

• templated configuration files in /etc are updated

• network services restarted

• new user account created

• backup performed

• ...or any of a number of other events

The model used to make these things happen is one of actions and events. An event is something that happens on the system (such as the user submitting a web form, an installation completing, a reboot, etc). An action is the atomic unit of things-that-need-doing, several of which may be called when an event occurs. For instance, the post-install event calls actions to configure and start up various services, initialise the password file, and so on.

Actions are written as Perl scripts, and stored in /etc/e-smith/events/actions. Some of them just use system() calls to do what's needed, as in this example (/etc/e-smith/events/actions/reboot):

package esmith;

use strict;
use Errno;

exec ("/sbin/shutdown", qw(-r now)) or die "Can't exec shutdown: $!";
exit (2);

Others are more complex, and can contain up to a few hundred lines of Perl code.

An event is defined by creating a subdirectory under /etc/e-smith/events and filling it with symlinks to the actions to be performed.

[root@e-smith events]# ls -l user-create/
total 0
lrwxrwxrwx    1 root     root           27 Jan 24 00:07 S15user-create-unix -> ../actions/user-create-unix
lrwxrwxrwx    1 root     root           27 Jan 24 00:07 S20conf-httpd-admin -> ../actions/conf-httpd-admin
lrwxrwxrwx    1 root     root           28 Jan 24 00:07 S20email-update-user -> ../actions/email-update-user
lrwxrwxrwx    1 root     root           30 Jan 24 00:07 S25email-update-shared -> ../actions/email-update-shared
lrwxrwxrwx    1 root     root           22 Jan 24 00:07 S25ldap-update -> ../actions/ldap-update
lrwxrwxrwx    1 root     root           29 Jan 24 00:07 S25reload-httpd-admin -> ../actions/reload-httpd-admin
lrwxrwxrwx    1 root     root           23 Jan 24 00:07 S50email-assign -> ../actions/email-assign
lrwxrwxrwx    1 root     root           21 Jan 24 00:07 S80pptpd-conf -> ../actions/pptpd-conf

Events are called via a script called /sbin/e-smith/signal-event, itself written in Perl. It's included here nearly-in-full, as a detailed example of e-smith code.

#!/usr/bin/perl -w

package esmith;
use strict;

# Events are signalled by calling this signal-event program and passing
# the event name and any other relevant parameters as arguments. For
# example:
# 
# signal-event ipchange 192.168.1.1 192.168.8.100

my $event = $ARGV [0];
my $handlerDir = "/etc/e-smith/events/$event";

#------------------------------------------------------------
# get event handler filenames
#------------------------------------------------------------

opendir (DIR, $handlerDir)
    || die "Can't open directory /etc/e-smith/events/$event\n";

# drop the "." and ".." directories
my @handlers = sort (grep (!/^\.\.?$/, readdir (DIR)));

closedir (DIR);

#------------------------------------------------------------
# Execute all handlers, sending any output to the system log.
# 
# Event handlers are not supposed to generate error messages
# under normal conditions, so we do not provide a mechanism
# for event handlers to signal errors to the user. Errors can
# only be written to the log file.
#------------------------------------------------------------

open (LOG, "|/usr/bin/logger -i -t e-smith");

#------------------------------------------------------------
# Ensure output is autoflushed.
#------------------------------------------------------------

my $ofh = select (LOG);
$| = 1;
select ($ofh);

print LOG "Processing event: @ARGV\n";

#------------------------------------------------------------
# Run handlers, logging all output.
#------------------------------------------------------------

my $exitcode = 0;
my $handler;
foreach $handler (@handlers)
{
    my $filename = "$handlerDir/$handler";
    if (-f $filename)
    {
	print LOG "Running event handler: $filename\n";
	print LOG `$filename @ARGV 2>&1`;
	if ($? != 0)
	{   
	    $exitcode = 1;
	}
    } 
}

close LOG;
exit ($exitcode);

6. Future development

There is currently a locking issue with the global configuration files. The techniques used to manipulate these files do not allow multiple processes to modify them concurrently. If two programs try to manipulate the files at the same time, one of them will overwrite the other one's changes. This is obviously a serious issue, albeit one that seldom causes problems in normal use, as most e-smith servers do not have multiple administrators working on the system at the same time.

A seemingly obvious solution is to use DBM instead of the current flat text file system. However, the flat text files are important because they make the system config readable and modifiable using a standard text editor from the Linux shell prompt. A simple command can then regenerate other configs or stop or start services based on the changes, without requiring the web interface to be used. This is both useful in the situation where the web interface might have been broken (a rare situation) or where a configuration option is hidden from less technical users.

A solution combining the benefits of text files and DBM has been suggested, in which the routine that reads the config database would check to see whether the text file had changed recently. If it had changed, it would convert it to DBM, otherwise it would just use the DBM directly. When a configuration option is changed, it would be written to both the DBM and the text file.

Another problem is the way that multiple instances of the Perl interpreter are invoked to run events and actions, causing some performance problems. A number of alternatives are being considered, including mod_perl and POE. The goal is to reduce the wait time experienced by the user when they click "submit" via the web interface; ideally, response would be near-instantaneous.

Other forthcoming improvements include a simpler way to create "wizard" interfaces for the e-smith manager (possibly using the FormMagick Perl module currently under development), and internationalisation (probably using the Locale::Maketext module).

7. In conclusion...

The e-smith server and gateway is a great example of a large project using Perl both as a system administration scripting tool and a Serious Programming Language. Although it has around twenty thousand lines of Perl code, the system is easy to understand, and the Perl code itself is maintainable and readable even by relatively inexperienced Perl programmers.

If you're interested in taking a closer look at the e-smith code, or maybe contributing to it, more information is available from the e-smith developer website.