ftblog

Real admins probably use stuff like cfengine, lcfg or puppet to manage system
configurations in massive networks. But what about people's personal
configuration files? Like, your zsh setup?

Personally, I like keeping my setup files in ~/etc. That way, I only need to
bag that directory and transfer it to another host to have everything behave
the way I want to.

The simplest form is where a configuration resides in one file. Like with my
`git' setup. That would be `~/etc/git/gitconfig'. And it would be great if
that file could end up as `~/.gitconfig'. It could end up there in different
ways; as a copy of the original, as a hardlink or as a symbolic link to it.
Which of those probably depends on taste and needs of the user.

Things can - obviously - become way more complicated than that. Think of
people using emacs or zsh. They tend to have huge setups, often scattered
across many files. Like my zsh setup - which I'm in the process of cleaning
up. So let's discuss that and see what a deployment system would have to do
in order to help me.

Basically, my zsh setup starts in one file and that sources a bunch of
others in the process. That file would be `~/etc/zsh/zshrc'. It needs to
end up as `~/.zshrc'. So far, so easy. The files it sources are in a
directory `~/etc/zsh/zshrc.d'. The files files from there need to be put
into `~/.zshrc.d' so that the .zshrc can find them. There are quite a
number of files in there and I really don't feel like adding them all
by hand, so a proper system has to help me with that.

Now, there's also a directory `~/etc/zsh/functions' which contains
files that define functions (usually - but not always - one at a time).
That directory's contents need to be deployed into `~/.zfunctions'.

And then, zsh doesn't just load a .zshrc file. No, it can protentially
load a whole lot of files like that. I don't use all of them, but some;
like .zlogout. They need to be put into place, too.

So we need something that roughly does this:

~/etc/zsh/zshrc                       ->   ~/.zshrc
~/etc/zsh/zlogout                     ->   ~/.zlogout
~/etc/zsh/zshrc.d/*.z                 ->   ~/.zshrc.d/*
~/etc/zsh/functions/[_a-zA-Z]*[a-z]   ->   ~/.zfunctions/*

Before we go on with this, let's see how a system that can help us with
all that could look like.

Deploying all sorts of files is one thing. And it should be extremely simple
to trigger (a simple "make deploy" should be enough). But another thing is
to remove a setup for whatever reason. And it should be just as easy as
deploying. "make withdraw" should do the trick. Because of DEploy and
WIthdraw, the system is called `dewi'.

Why `make'? Well. a) it's available virtually everywhere (you just need
to try to avoid non-portable extensions). b) it does a great job of hiding
complexity from the user. and c) it allows for a great, "talking" interface
which a lot of shells can even assist you in, using (like in my zsh setup,
I can do "make <tab>" and see a menu of available make target and do
"make d<tab>" to complete to "make deploy").

Make has its limitations, though. Writing a system that can fulfill all
the requirements in pure portable make isn't possible without the author
and probably the users going insane.

The real work needs to be done by something entirely more powerful. The
obvious choice for me, was Perl. (And no, I won't even start defending
that choice against narrow-minded pissy little bitches who think every
Perl program is ugly as shit without knowing squat about the language
in the first place, just because some snake disciples said so on the
internet.)

Okay, now that we've lost the moronic part of the audience, why Perl?
a) Like `make' it's available everywhere. b) it's interpreted, so
bringing the actual code into place is very easy. c) it's interpreted.
And it's very easy to make it load other files containing Perl code, to
make it behave like we want to. d) with a reasonable and well-documented
API, such configuration files Perl are pretty simple to grok (actually,
with some examples, I'm fairly sure you don't need to know any of the
actual language).

So basically, the system would consist of a trivial Makefile, which defines
two (or more) simple targets (like `deploy' and `withdraw') which call a
Perl-script, that does the actual work. The Perl-script reads Perl-files for
customisation. These Perl-files obviously need a name. I chose `Dewifile' in
the tradition of the `Makefile' used by make.


Back to our examples. How would I tell our `dewi' system to deploy
that simple `gitconfig' file?

register('gitconfig');

...that could do it. It could default to making a dotfile out of the original
filename and create a thusly named copy in the user's home directory. That
would make handling such easy cases easy enough. So, let's do that - and why
copying by default? Well, you can copy on every OS and file-system.

But if stuff gets more complicated? Do we really want to go this way?

register('foo.rc', 'opt0', 'param1', 'anotheropt', 'and-so-on', ...);

How do you remember which option goes where? Do you really want to repeat
default values all the time? I think you don't. What you sometimes do
in Perl, if you'd otherwise create rather unclear APIs, is using a
hash reference as the sole argument to a subroutine.

A what to a what? A hash in Perl lingo is an associative array. In other
words something where you can link a key to a value:

{
    key_one => value_one,
    key_two => value_two,
    ...
}

The curly braces are important. The order of the keys does not matter. And if
done right, you'd only have to specify the mandatory keys and the ones where
you want to change default values.

"Reference" here more or less means that just an address to the hash is handed
to the subroutine. For the user, that doesn't make a difference.

Let's look at some complex register() calls to see how those hashref arguments
work:

register( { glob        => 'zshrc',
            method      => 'copy',
            destination => '~/' } );

If called with a complex argument, register behaves very generic. It will not
touch the given input file name. Therefore with the above call you would get
`zshrc' deployed as `~/zshrc' (not a dotfile). To fix that, you may provide
a `transform' parameter.

But that `transform' parameter would not be a string. Because we don't know
in what way you would want to transform a file name, you may write your own
transformation code and provide a reference to it in this parameter (Perl
programmers will know this as a coderef).

Don't worry, you don't need to write transformation Perl code to make a
dotfile out of a source-filename. `dewi' comes with predefined
transformations. What you need to know is how to create a coderef:

\&subroutinename

The dotfile transformation subroutine is called `makedotfile', so a correct
register call for the problem would look like this:

register( { glob        => 'zshrc',
            method      => 'copy',
            transform   => \&makedotfile,
            destination => '~/' } );

Dewi uses the `coderef' idea in more places if it wants to make hard things
possible. So get used to the idea.

What about the `zlogout' file? Copy that register call? Well, no. The
parameter is called `glob'. And the value is used by Perl's bsd_glob()
function from it's standard File::Glob module to generate file names. So, we
can do this (see File::Glob's manual for details):

register( { glob        => '{zshrc,zlogout}',
            method      => 'copy',
            transform   => \&makedotfile,
            destination => '~/' } );

We could have left the `method' and `destination' parameters out, because
they do not change any default values. But I left them in for demonstration.

Now you already know everything we need to, to deploy the files from the
sub-directories, too:

register( { glob        => 'zshrc.d/*.z',
            destination => '~/.zshrc.d' } );
register( { glob        => 'functions/[_a-zA-Z]*[a-z]',
            destination => '~/.zfunctions' } );

Why not just symlink to the sub-directories? That was done on purpose. For
multiple reasons. a) I do not want to litter my `~/etc/*/' directories. Like,
when I'd zcompile my setup, I'd end up with a truckload of `*.zwc' files. b) I
want to be able to inject code into my `~/.zfunctions' and `~/.zshrc.d'
directories without putting it in `~/etc/zsh'. Why? Because for me, the latter
directory is a git repository and when I want to use my zsh-lookup system, I
do not want to put it into my configuration's git repository. Those two
pieces of code do not have anything to do with each other. I could work around
that by putting more patterns into my `.gitignore' file. But it's unnecessary,
unclean and stupid. Oh and this is not a zsh-only thing. With emacs, it's
pretty much the same, and it's worse with other applications.

To cut a long story short: dewi deploys files. Not links directories. It makes
things cleaner and easier. Just get over it.

While writing or changing Dewifiles, you may not want dewi to always actually
do what you just configured. It should just print what it would do. To let
that happen, you can set the `dryrun' option in the Dewifile:

set_opt('dryrun', 'true');


Now what if there are things we'd want to change in a file while it is being
deployed? Like inserting actual passwords for placeholders (which you put into
the files to be able to upload them to github).

Dewi has support for that. You can use the `filtered' method and provide
a filter in Perl, POSIX Shell or anything else.

Say, we got the following in our irssi configuration:

bitlbee = {
  type        = "IRC";
  autosendcmd = "/^msg -bitlbee &bitlbee identify mssp; wait -bitlbee 2000";
};

...where `mssp' is my-super-secret-password for the bitlbee IM gateway.
Obviously, that one cannot go into a public repository of the setup.

What we could do is, instead of `mssp' put a `@@BITLPASSWD@@' in there, and
have some sort of filter replace it with `mssp' when we say "make deploy".
But where do we put the `mssp'? We can't put it into the filter, if we
would, we couldn't publish the Dewifile or the global dewi setup publically.
What we can do, is create a directory `~/.sensdata' where we put all our
sensitive data. And a file `~/.sensdata/irssi' would contain the replaces
for the irssi deployment:

@@BITLPASSWD@@:mssp

The filter can read that file and do the replacements accordingly. That way
you'd only have to supply that `.sensdata' directory via a safe channel. And
if there's no such file, we could even refuse to deploy at all.

Dewi ships no filter code built-in, but this particular filter can be found
in `examples/simple-filter' in the project's source directory. Full
instructions are provided within the file.


Dewi comes with a number of manuals, which document the system in detail:

    dewi(7)     - This documents the system as whole; including what
                  is needed to get a directory into dewi-mode.
    dewi(1)     - This describes the `dewi' maintenance program.
    dewifile(5) - This is the reference documentation for Dewifiles and
                  their special Perl API.

These are generated from txt2tags (*.t2t) markup files in the `doc/'
sub-directory of the project source tree.

dewi(7) is a must read. It is not terribly long, but it provides a
description of the system and a full example, which starts at step one.

You should consult dewifile(5) as a reference when you want to know the exact
behaviour of any of dewi's API, like the `register()' function.


And finally, how to get `dewi' and how to install the system?

It's pretty easy, actually. `dewi' is kept in a public git repository at
github: <http://github.com/ft/dewi> - So, in order to get the code, do:

% git clone git://github.com/ft/dewi.git

The recommended way to install `dewi' is to install it system-wide. And
that's as easy as this:

% make sys doc
# make install

The "make install" needs to be done with proper access (probably as `root'
or via `sudo'). Make *sure* you got `txt2tags' installed before doing this.
Otherwise you'll end up without documentation.

There are other ways to use `dewi' aside from installing it to the system.
But only use them if you know how they are supposed to behave.

This is all you really need to know. Well. Actually, that's not quite true.
You still need to know how to get the system itself up and running in a
directory tree. But that's all detailed in the `dewi(7)' manual page. You
should really read its `DESCRIPTION' and `EXAMPLE' sections. They are quite
short and should give you an easy start into using `dewi'.