“…done is Scheme.” was an idea I had when I started liking Scheme more and more. XMMS2 is my preferred audio player for a long time now. And I always wanted to write a client for it that fits my frame of mind better than the available ones.

I started writing one in C and that was okay. But when you're used to use properly extensible applications on a regular basis, you kind of want your audio player to have at least some of those abilities as well. I started adding a lua interpreter (which I didn't like), a Perl interpreter (which I did before in another application, but which is also not a lot of fun). So I threw it all away, setting out to write one in Scheme from scratch.

To interface the XMMS2 server, I was first trying to write a library that wrapped the C library that the XMMS2 project ships. But now I was back to writing C, which I didn't want to do. Someone on IRC in #xmms2 on freenode suggested to just implement the whole protocol in Scheme natively. I was a little intimidated by that, because the XMMS2 server supports a ton of IPC calls. But XMMS2 also ships with machine readable definitions of all of those, which means that you can generate most of the code and once you've implemented the networking part and have the serialization and de-serialization for the protocol's data types, you're pretty much set. …well, after you've implemented the code that generates your IPC code from XMMS2's definition file.

Most of XMMS2's protocol data types map very well to Scheme. There are strings, integers, floating point numbers, lists, dictionaries. All very natural in Scheme.

And then there are Collections. Collections are a way to interact with the XMMS2 server's media library. You can query the media library using collections. You can generate play lists using collections and perform a lot of operations on them like sorting them in a way you can specify yourself. For more information about collections see the Collections Concept page in the XMMS2 wiki.

Now internally, Collections are basically a tree structure, that may be nested arbitrarily. It carries a couple of payload data sets, but they are trees. And implementing a tree in Scheme is not all that hard either. The serialization and de-serialization is also pretty straight forward, since the protocol reuses its own data types to represent the collection data.

What is not quite so cool is the language you end up with to express these collections. Say, you want to create a collection that matches four Thrash Metal groups you can do that with XMMS2's command line client like so:

xmms2 coll create big-four artist:Slayer \
                        OR artist:Metallica \
                        OR artist:Anthrax \
                        OR artist:Megadeth

To create the same collection with my Scheme library, that would look like this:

    '() '()
    (list (make-collection COLLECTION-TYPE-EQUALS
              '((field . "artist")
                (value . "Slayer"))
              (list (make-collection COLLECTION-TYPE-UNIVERSE '() '() '())))
          (make-collection COLLECTION-TYPE-EQUALS
              '((field . "artist")
                (value . "Metallica"))
              (list (make-collection COLLECTION-TYPE-UNIVERSE '() '() '())))
          (make-collection COLLECTION-TYPE-EQUALS
              '((field . "artist")
                (value . "Anthrax"))
              (list (make-collection COLLECTION-TYPE-UNIVERSE '() '() '())))
          (make-collection COLLECTION-TYPE-EQUALS
              '((field . "artist")
                (value . "Megadeth"))
              (list (make-collection COLLECTION-TYPE-UNIVERSE '() '() '())))))

…and isn't that just awful? Yes, yes it is. It so is.

In order to reign in this craziness, the library ships a macro that implements a little domain specific language to express collections with. Using that, the above boils down to this:

(collection (∩ (artist = Slayer)
               (artist = Metallica)
               (artist = Anthrax)
               (artist = Megadeth)))

So much better, right? …well, unless you really don't like Unicode characters and the ‘∩’ in there gives you a constant headache… But worry not, you can also use ‘or’ in place if the intersection symbol if you like. Or ‘INTERSECTION’ if you really want to be clear about things.

If you know a bit of Scheme, you may wonder how to use arguments to those operators that get evaluated at run time. Since evidently, the Slayer in there is turned into a "Slayer" string at compile time; same goes for the artist symbol. These transformations are done to make the language very compact for users to just type expressions. If you want an operator to be evaluated, you have to wrap it in a (| ...) expression:

(let ((key "artist") (value "Slayer"))
  (collection ((| key) = (| value)))

These expressions may be arbitrarily complex.

Finally, to traverse Collection tree data structures, the library ships a function called ‘collection-fold’. It implements pre, post and level tree traversal with both left-to-right and right-to-left directions. So, if you'd like to count the number of nodes in a collection tree, you can do it like this:

(define *big-four* (collection (∩ (artist = Slayer)
                                  (artist = Metallica)
                                  (artist = Anthrax)
                                  (artist = Megadeth)))

(collection-fold (lambda (x acc) (+ acc 1))
                 0 *big-four*)

This would evaluate to 9.

The library is still at an early stage, but it can control an XMMS2 server as the ‘cli’ example, shipped with the library, will show you. There are no high level primitives to support synchronous and asynchronous server interactions. And there is not a whole lot of documentation, yet either. But the library implements all data types the protocol supports as well as all methods, signals and broadcasts the IPC document defines. The collection DSL supports all collection operators and all attributes one might want to supply.

Feel free to take a look, play around, report bugs. The library's home is with my account on github.

Posted Sun 15 Jan 2017 15:52:39 CET Tags:

Hello dewi users! (Yes, all three of you.)

I started developing dewi in 2010. Back then I did two releases in less than a week. And now the third release gets out more than six years later. There are a number of reasons for the amount of time this took: I made a few mistakes in dewi's design, and fixing them took some effort. Spare time got really scarce. I wasted some time implementing features, that should never have been part of dewi (because they were features, my previous solution to this problem had). I wanted lots of features, that took a while to finish. Because the new release would definitely break backwards compatibility, I was working on a separate branch called “onward”, that basically only I used. And finally, since the code in that “onward” branch worked for me (I knew its shortcomings and worked around them), I could not motivate myself to fix all the outstanding issues and add proper documentation for all that had changed. Therefore nothing happened.

In the meantime, other systems like vcsh, rcm or homesick have emerged. Really, there are tons of systems like these out there, these are just a few of them that get mentioned to me more often than the others. Some of these systems hard depend on a version control system — usually git — and even have it at their absolute centre of implementation. Others just made pretty weird design decisions, that make me feel uneasy. And yet others are just a little limited in what they can actually do (basically just symlink a file). I still prefer dewi over everything I have looked at in any sort of detail.

Earlier, I mentioned some mistakes in dewi's design. Let's take a look at some of them: First, I should have never depended on ‘make’: Dewi only used it to call the dewi script (from .dewi/bin) and because the "make deploy" looks pretty. But there's no feature in make that the basic dewi features actually need. And then you just end up with a bunch of downsides, like:

  • Superfluous dependency, that a target system has to satisfy
  • Proper portable make code is actually not that easy, and I always get issues on OpenBSD systems for example.
  • Using make required a Makefile in each dewified sub-directories to exist. Which means that you pollute the sub-directory's version control history with changes to that file each time dewi updates it.
  • If someone wants to use make as part of their deployment system, they can use it on top of a make-less dewi-system at will.
  • Needlessly complicates the whole system.

You can think of more. It was just wrong to use ‘make’.

And the second major mis-design was the case of keeping the code for dewi in the dot-dewi sub-directory. The idea was to ensure, that the deployment process doesn't ever break due to changes in dewi that might be in updated system version. That's still a moot point however, since nobody keeps someone from installing a fixed version of dewi into ‘~/bin’ and never ever change it. Dewi can be used from its source directory as well and its non-standard dependencies (dewi's implementation language, Perl, is installed on virtually all systems I have come across) are optional dependencies; no need to install anything. Instead, keeping the script in the ‘.dewi/’ directory made the system much less clear and much more confusing.

These issues are both fixed now. Dewi is now a single command line tool, that ships all its code. And where you previously did this:

% make deploy

…you are now doing this:

% dewi deploy

To discuss all changes in dewi since version 0.2 would completely go beyond the scope of a single blog post. Instead, let me list a few highlights:

  • You can now apply filters when deploying files (useful, if your configuration files carry sensitive information).
  • You can now use multiple input files to deploy into a single destination file.
  • Dewi now lets you run your own code while it works, because it lets you define hooks.
  • Dewi now has better support for deploying large, recursive trees of files.
  • You can now deploy files with full support of templates using Perl's Template module (available from CPAN).

Another thing that may be interesting for a few users is the trivial call of its register() function: That one takes a string argument, that is the name of a source file. Its effect would be to deploy that file using the copy method, after the source file name was turned into a dot-file-name. Copying was the default choice of deployment, because it absolutely works everywhere.

However, sometimes you may want to take changes you do to source files immediately after saving them to disk. You may think “symbolic links!” right now and you would be right. The trivial call style of register() now allows the user to supply string-options. So now the canonical example of a Dewifile for a git-configuration that should be symlinked to its source looks like this:

register '--symlink', 'gitconfig';

For all details, take a look at the CHANGES and UPGRADING files as well as the new and updated manuals dewi(7), dewi(1) and dewifile(5) (see below).

I am planning to release version 1.0 of dewi based on version 0.3 once it had time to settle a little and have remaining bugs fixed, if they come up, and to have its documentation be polished further. I consider dewi to be pretty feature complete right now, which is why after 1.0 it'll go to maintenance mode.

Thanks for listening. ☺

Related links:

Posted Fri 29 Jul 2016 01:31:03 CEST Tags:

You may have seen this already, but I thought I'd mention it anyway, because I like the level of detail some people put into their work. Like with this list of easter eggs that made it into episode one of season two of Mr.Robot. Once you make it the the website with the in-browser operating system with a working window manager, a small working shell and a cute impression of an IRC client, you may agree.

Posted Sun 17 Jul 2016 12:04:28 CEST Tags:

If you're using git with zsh, you may have seen this:

% git show HEAD^1
zsh: no matches found: HEAD^1

…and wondered what in the name of all that is holy is going on.

The reason behind this is, that zsh signals an error if you provide it with a pattern, that doesn't match any files. This is the sane thing to do. Especially in interactive shells. If your shell behaves differently, I am sorry.

Your reaction might be “What pattern are you speaking of?” — since there is no asterisk, question mark or similar construct in there. That is true, but zsh has pretty powerful extended patterns, if enabled. They let you to pull off lots of things you might resort to regular expressions for, in certain situations. Characters like “#”, “~” and “^” become special characters and must be quoted if their special meaning is to be suppressed.

So that is what's going on. And sometimes git users get annoyed.

With git, though, you almost never want your shell to do any globbing anyway. Seriously. On the one hand, that is because git more often than not interacts with its database rather than with files, so files don't come into play to start with. And on the other hand, even when you pass stuff to git that looks like a file pattern, often you want git to expand the pattern itself. Like with “git grep”:

% git grep 'foo.*bar' -- '*.h'

This finds all lines that match the regular expression “foo.*bar” in all files that match the pattern “*.h”. No matter in which subdirectory the header file is hidden in.

Git is really one of the most eligible commands to have globbing (the expansion of patterns in file names) turned off for. And zsh let's you do that:

alias git='noglob git'

Now when you execute commands like these:

% git show HEAD^1
% git grep foo.*bar -- *.h

…zsh will not expand any patterns for list of files and instead hands all arguments over to git unchanged. Which is exactly what you want almost every time.

In the odd case, when you really do want the shell to do globbing, you can circumvent your shell from using the alias you just defined: In case you've got the EQUALS option set (which is the default) you can do:

% =git add *.txt

Try “echo =git” to see why that works.

If you don't use that particular option, you can always turn off alias expansion temporarily by prefixing the alias name with a backslash:

% \git add *.txt

And that's that.

Posted Mon 20 Jun 2016 21:55:38 CEST Tags:

The previously commercial Scheme implementation (for more than 30 years) known as Chez Scheme from Cadence Research Systems (which was aquired by Cisco in 2012) was released as Open Source software using the Apache Licence 2.0.

This is a pretty cool move by R. Kent Dybvig and Cisco making this high quality implementation available to the general public. Chez has, among other things, support for compiling Scheme sources to native machine code. Something that my go-to Scheme GNU Guile doesn't feature (although there are long term plans into that direction, as far as I know).

In case you're interested:

Posted Sun 01 May 2016 00:37:40 CEST Tags:

Der OpenSourcePress Verlag hat scheinbar die Ohren angelegt.

In diesem Verlag ist das deutsche Buch zu Git erschienen. Die Verlagsrechte sind nun wohl an die Autoren zurück gegangen und die haben sich entschieden ihren kompletten Krempel unter CC-NC-SA Lizenz zu veröffentlichen. QuellenHTML Version des Buchs

Posted Sun 03 Jan 2016 12:24:38 CET Tags:

As it turns out “Christmas” is really this Christmas. I'm talking about Perl 6 of course.

Perl was the third programming language I learned (tied with C, which I started out with around the same time) and the first language I was comfortable using (before I was using Basic and Pascal, both of which I found rather lacking).

Perl is not a perfect language, not by a long shot — although, most of the criticism (especially the harsh ones) I hear comes from people who never really read or wrote any non-trivial Perl code. But since people on the internet said so, it's reasonable to blabber along. But since the default language behaviour (no "use strict" and no "use English") allows and invites — how do I put this — interesting code, even fans of the language cannot deny that. I have written some aweful stuff myself. I wouldn't want to work on my old programs. Not even for lots of money. Nobody is forced to write ugly ass code in Perl, though. And there are some beautiful examples out there.

However, this blog post is not about the Perl of old. No sir. This is about a language, that a number of smart people started to design in 2000: Perl 6 — A new incarnation of the Perl language:

For the first time, there would be a specification of the language. For the first time it would be clear how to parse Perl (with Perl5 the saying goes "Only Perl can parse Perl. And the tokenizer is black magic."). Perl6 is its own language in the tradition of previous Perls, but much more consistent and with less obscure defaults. “More Awesome. Less Crazy.” if you will.

Perl6 was also thought of as vaporware, as in "Nice idea, that is never going to happen". With respect to its release date, the designers used to say "We'll release on Christmas. We just don't know which Christmas it is going to be."

And like I said above, as it turns out, it's this Christmas. 15 years of development has not been four naught. Just browse Rosetta Code for some Perl6 examples. You'll notice the heavy use of operators in Perl6, of which I'm sure the askell fans will be fond of (Haskell is one Perl6's major influences anyway); also with Unicode characters like the MD5 implementation on Rosetta Code shows. You might also like how Perl 6 handles parallelism, asynchrony and concurrency

Don't give in to prejudices: Take a look at the examples on the Perl 6 homepage or Rosetta Code. Browse the Advent Calendars. Some of the features are amazing and if the community can grow into a decent size, so that the number of libraries grow into a useful lot, Perl 6 could be an amazing world to work in.

Posted Sat 26 Dec 2015 12:51:19 CET Tags:

So, they™ made one of my favourite horror franchises into a television series. …and it's glorious. It's quite gory (especially for television), trashy, not at all politically correct, fun, has a pretty decent soundtrack (Deep Purple, PJ Harvey, The Stooges… stuff like that) and features people like Bruce Campbell (the original Ash) and Lucy Lawless (yes, Xena). It's called “Ash vs. Evil Dead” and there has not been a dull episode, yet. ☺

Posted Sun 29 Nov 2015 22:29:38 CET Tags:

Suppose you are working on zsh's source code.

At some point in time you might want to try the code you're writing or changing. One way would be to just call the zsh binary ‘Src/zsh’ after compiling the project's code. The problem with that is, that zsh uses a module system to load features on demand, that are not part of the absolute core code of the shell. And it looks for its module libraries somewhere on your system, not however in your zsh source code repository. This is obviously a good thing, because you don't want attackers to be able to just drop a ‘zle.so’ file somewhere to get access to your system. ☺

There are two situations, where you can get away with just using ‘Src/zsh’ anyway: You're testing features that work in scripts and don't need to load any modules; or you've got the same zsh version installed on your system, that you're also working on and you don't actually change any of the module code, but just need to load a module to get to your test.

If you need to test code that touches modules, you could install the shell to your system. But that's cumbersome and takes a lot of time. Especially if you need to test lots of different builds, like you do when you use git bisect to chase down a bug.

…and that's what I did today. So I actually wanted to use the binaries I just built from the source. How to do that then? Well, you need to adjust at least ‘$module_path’, which you can't do via a parent process's environment for the security reason I mentioned earlier.

The way I did it was to use ‘$ZDOTDIR’ to provide ‘Src/zsh’ with a testing setup in the ‘.zshenv’ file in that directory, that basically did this:

if ! test -d _modules_; then
    mkdir -p _modules_/zsh || exit 1

cd _modules_/zsh || exit 1
for mod in "${root}"/Src/**/*.so; do
    test -h "${mod:t}" && continue
    ln -s "$mod" "${mod:t}" || exit 1
cd "$root" || exit 1

# Setup the shell's load paths for functions and modules:
fpath=( "${root}/Completion" )
module_path=( "${root}/_modules_" )

If you want to use zsh from its source directory, I've thrown together a small repository that contains the setup I used (well, a slightly more polished version): https://github.com/ft/zsh-test

Posted Sat 26 Sep 2015 21:56:46 CEST Tags:
ft SYSHH#5

Talking Heads - Speaking in Tongues - This Must be the Place (Naive Melody)

God. This song. ♥

Posted Sun 31 May 2015 23:17:51 CEST Tags:

In other news, this blog turned 10 last saturday (ftblog online!). Kind of missed that date myself. Post frequency is dwindling, but I guess I'll continue annoying you with my boring crap from time to time anyway. Cheers! ☺

Posted Sat 18 Apr 2015 22:24:32 CEST Tags:

So, The Prodigy released another record a few days ago.

In the past, I've been a massive fan of the band, buying everything I could get my hands on: Books, VHS Tapes, every single release in every version I could find, all the albums... Then the band took a bit time off (“The Fat of the Land” was released in 1997) and came back — seven years later — with “Always Outnumbered, Never Outgunned” in 2004. While it's a good release, it was the weakest when compared to the first three records. The next one (“Invaders Must Die” in 2009) was big again. I loved almost all tracks on that one.

And now in 2015 — after another six year waiting period — they release “The Day is my Enemy”. They put out a few things beforehand via their youtube channel (like “Nasty” and “Wild Frontier”); and nothing that I heard clicked immediately.

That's also why I put off buying the album right away. Today I did. And what can I say? Everything on this record works. Even the tracks I knew previously and dismissed work in the context of the album.

The Prodigy's sound doesn't change as much as it used to with the first three albums. But that's not necessarily a bad thing. They are probably converging towards how they are supposed to sound.

Don't get me wrong, this does not redefine which album is The Prodigy's magnum opus (that is and will remain “Music for the Jilted Generation”), but this is a versatile, 14-track, modern dance record that everyone who liked the two predecessors will love.

NP: The Prodigy — The Day is my Enemy — Destroy

Posted Sat 18 Apr 2015 21:33:02 CEST Tags: