Real admins probably use stuff like cfengine, lcfg or puppet to manage system configurations in massive networks. But what about people's personal configuration files? Like, your zsh setup?

Personally, I like keeping my setup files in ~/etc. That way, I only need to bag that directory and transfer it to another host to have everything behave the way I want to.

The simplest form is where a configuration resides in one file. Like with my ‘git’ setup. That would be ‘~/etc/git/gitconfig’. And it would be great if that file could end up as ‘~/.gitconfig’. It could end up there in different ways; as a copy of the original, as a hardlink or as a symbolic link to it. Which of those probably depends on taste and needs of the user.

Things can - obviously - become way more complicated than that. Think of people using emacs or zsh. They tend to have huge setups, often scattered across many files. Like my zsh setup - which I'm in the process of cleaning up. So let's discuss that and see what a deployment system would have to do in order to help me.

Basically, my zsh setup starts in one file and that sources a bunch of others in the process. That file would be ‘~/etc/zsh/zshrc’. It needs to end up as ‘~/.zshrc’. So far, so easy. The files it sources are in a directory ‘~/etc/zsh/zshrc.d’. The files files from there need to be put into ‘~/.zshrc.d’ so that the .zshrc can find them. There are quite a number of files in there and I really don't feel like adding them all by hand, so a proper system has to help me with that.

Now, there's also a directory ‘~/etc/zsh/functions’ which contains files that define functions (usually - but not always - one at a time). That directory's contents need to be deployed into ‘~/.zfunctions’.

And then, zsh doesn't just load a .zshrc file. No, it can protentially load a whole lot of files like that. I don't use all of them, but some; like .zlogout. They need to be put into place, too.

So we need something that roughly does this:

~/etc/zsh/zshrc                       ->   ~/.zshrc
~/etc/zsh/zlogout                     ->   ~/.zlogout
~/etc/zsh/zshrc.d/*.z                 ->   ~/.zshrc.d/*
~/etc/zsh/functions/[_a-zA-Z]*[a-z]   ->   ~/.zfunctions/*

Before we go on with this, let's see how a system that can help us with all that could look like.

Deploying all sorts of files is one thing. And it should be extremely simple to trigger (a simple "make deploy" should be enough). But another thing is to remove a setup for whatever reason. And it should be just as easy as deploying. "make withdraw" should do the trick. Because of DEploy and WIthdraw, the system is called ‘dewi’.

Why ‘make’? Well. a) it's available virtually everywhere (you just need to try to avoid non-portable extensions). b) it does a great job of hiding complexity from the user. and c) it allows for a great, "talking" interface which a lot of shells can even assist you in, using (like in my zsh setup, I can do "make \" and see a menu of available make target and do "make d\" to complete to "make deploy").

Make has its limitations, though. Writing a system that can fulfill all the requirements in pure portable make isn't possible without the author and probably the users going insane.

The real work needs to be done by something entirely more powerful. The obvious choice for me, was Perl. (And no, I won't even start defending that choice against narrow-minded pissy little bitches who think every Perl program is ugly as shit without knowing squat about the language in the first place, just because some snake disciples said so on the internet.)

Okay, now that we've lost the moronic part of the audience, why Perl? a) Like ‘make’ it's available everywhere. b) it's interpreted, so bringing the actual code into place is very easy. c) it's interpreted. And it's very easy to make it load other files containing Perl code, to make it behave like we want to. d) with a reasonable and well-documented API, such configuration files Perl are pretty simple to grok (actually, with some examples, I'm fairly sure you don't need to know any of the actual language).

So basically, the system would consist of a trivial Makefile, which defines two (or more) simple targets (like ‘deploy’ and ‘withdraw’) which call a Perl-script, that does the actual work. The Perl-script reads Perl-files for customisation. These Perl-files obviously need a name. I chose ‘Dewifile’ in the tradition of the ‘Makefile’ used by make.

Back to our examples. How would I tell our ‘dewi’ system to deploy that simple ‘gitconfig’ file?

register('gitconfig');

...that could do it. It could default to making a dotfile out of the original filename and create a thusly named copy in the user's home directory. That would make handling such easy cases easy enough. So, let's do that - and why copying by default? Well, you can copy on every OS and file-system.

But if stuff gets more complicated? Do we really want to go this way?

register('foo.rc', 'opt0', 'param1', 'anotheropt', 'and-so-on', ...);

How do you remember which option goes where? Do you really want to repeat default values all the time? I think you don't. What you sometimes do in Perl, if you'd otherwise create rather unclear APIs, is using a hash reference as the sole argument to a subroutine.

A what to a what? A hash in Perl lingo is an associative array. In other words something where you can link a key to a value:

{
    key_one => value_one,
    key_two => value_two,
    ...
}

The curly braces are important. The order of the keys does not matter. And if done right, you'd only have to specify the mandatory keys and the ones where you want to change default values.

"Reference" here more or less means that just an address to the hash is handed to the subroutine. For the user, that doesn't make a difference.

Let's look at some complex register() calls to see how those hashref arguments work:

register( { glob        => 'zshrc',
            method      => 'copy',
            destination => '~/' } );

If called with a complex argument, register behaves very generic. It will not touch the given input file name. Therefore with the above call you would get ‘zshrc’ deployed as ‘~/zshrc’ (not a dotfile). To fix that, you may provide a ‘transform’ parameter.

But that ‘transform’ parameter would not be a string. Because we don't know in what way you would want to transform a file name, you may write your own transformation code and provide a reference to it in this parameter (Perl programmers will know this as a coderef).

Don't worry, you don't need to write transformation Perl code to make a dotfile out of a source-filename. ‘dewi’ comes with predefined transformations. What you need to know is how to create a coderef:

\&subroutinename

The dotfile transformation subroutine is called ‘makedotfile’, so a correct register call for the problem would look like this:

register( { glob        => 'zshrc',
            method      => 'copy',
            transform   => \&makedotfile,
            destination => '~/' } );

Dewi uses the ‘coderef’ idea in more places if it wants to make hard things possible. So get used to the idea.

What about the ‘zlogout’ file? Copy that register call? Well, no. The parameter is called ‘glob’. And the value is used by Perl's bsd_glob() function from it's standard File::Glob module to generate file names. So, we can do this (see File::Glob's manual for details):

register( { glob        => '{zshrc,zlogout}',
            method      => 'copy',
            transform   => \&makedotfile,
            destination => '~/' } );

We could have left the ‘method’ and ‘destination’ parameters out, because they do not change any default values. But I left them in for demonstration.

Now you already know everything we need to, to deploy the files from the sub-directories, too:

register( { glob        => 'zshrc.d/*.z',
            destination => '~/.zshrc.d' } );
register( { glob        => 'functions/[_a-zA-Z]*[a-z]',
            destination => '~/.zfunctions' } );

Why not just symlink to the sub-directories? That was done on purpose. For multiple reasons. a) I do not want to litter my ‘~/etc/*/’ directories. Like, when I'd zcompile my setup, I'd end up with a truckload of ‘*.zwc’ files. b) I want to be able to inject code into my ‘~/.zfunctions’ and ‘~/.zshrc.d’ directories without putting it in ‘~/etc/zsh’. Why? Because for me, the latter directory is a git repository and when I want to use my zsh-lookup system, I do not want to put it into my configuration's git repository. Those two pieces of code do not have anything to do with each other. I could work around that by putting more patterns into my ‘.gitignore’ file. But it's unnecessary, unclean and stupid. Oh and this is not a zsh-only thing. With emacs, it's pretty much the same, and it's worse with other applications.

To cut a long story short: dewi deploys files. Not links directories. It makes things cleaner and easier. Just get over it.

While writing or changing Dewifiles, you may not want dewi to always actually do what you just configured. It should just print what it would do. To let that happen, you can set the ‘dryrun’ option in the Dewifile:

set_opt('dryrun', 'true');

Now what if there are things we'd want to change in a file while it is being deployed? Like inserting actual passwords for placeholders (which you put into the files to be able to upload them to github).

Dewi has support for that. You can use the ‘filtered’ method and provide a filter in Perl, POSIX Shell or anything else.

Say, we got the following in our irssi configuration:

bitlbee = {
  type        = "IRC";
  autosendcmd = "/^msg -bitlbee &bitlbee identify mssp; wait -bitlbee 2000";
};

...where ‘mssp’ is my-super-secret-password for the bitlbee IM gateway. Obviously, that one cannot go into a public repository of the setup.

What we could do is, instead of ‘mssp’ put a ‘@@BITLPASSWD@@’ in there, and have some sort of filter replace it with ‘mssp’ when we say "make deploy". But where do we put the ‘mssp’? We can't put it into the filter, if we would, we couldn't publish the Dewifile or the global dewi setup publically. What we can do, is create a directory ‘~/.sensdata’ where we put all our sensitive data. And a file ‘~/.sensdata/irssi’ would contain the replaces for the irssi deployment:

@@BITLPASSWD@@:mssp

The filter can read that file and do the replacements accordingly. That way you'd only have to supply that ‘.sensdata’ directory via a safe channel. And if there's no such file, we could even refuse to deploy at all.

Dewi ships no filter code built-in, but this particular filter can be found in ‘examples/simple-filter’ in the project's source directory. Full instructions are provided within the file.

Dewi comes with a number of manuals, which document the system in detail:

dewi(7)     - This documents the system as whole; including what
              is needed to get a directory into dewi-mode.
dewi(1)     - This describes the ‘dewi’ maintenance program.
dewifile(5) - This is the reference documentation for Dewifiles and
              their special Perl API.

These are generated from txt2tags (*.t2t) markup files in the ‘doc/’ sub-directory of the project source tree.

dewi(7) is a must read. It is not terribly long, but it provides a description of the system and a full example, which starts at step one.

You should consult dewifile(5) as a reference when you want to know the exact behaviour of any of dewi's API, like the ‘register()’ function.

And finally, how to get ‘dewi’ and how to install the system?

It's pretty easy, actually. ‘dewi’ is kept in a public git repository at github: http://github.com/ft/dewi - So, in order to get the code, do:

% git clone git://github.com/ft/dewi.git

The recommended way to install ‘dewi’ is to install it system-wide. And that's as easy as this:

% make sys doc
# make install

The "make install" needs to be done with proper access (probably as ‘root’ or via ‘sudo’). Make sure you got ‘txt2tags’ installed before doing this. Otherwise you'll end up without documentation.

There are other ways to use ‘dewi’ aside from installing it to the system. But only use them if you know how they are supposed to behave.

This is all you really need to know. Well. Actually, that's not quite true. You still need to know how to get the system itself up and running in a directory tree. But that's all detailed in the ‘dewi(7)’ manual page. You should really read its ‘DESCRIPTION’ and ‘EXAMPLE’ sections. They are quite short and should give you an easy start into using ‘dewi’.

Posted Sun 24 Oct 2010 13:07:42 CEST Tags: