ftblog

Over the years, I've seen people do very interesting things with gnus.

Gnus is - in its core - a Usenet News Client. And it is very good at it.
Its features seem boundless, really.

This always sounded a little like zsh to me. That is a unix shell and it
can be virtually be tailored down to behave *exactly* the way the user
would want it to. Problem is, zsh's default values aren't exactly what
a new user would hope them to be. So it takes time to tame the beast.
But after you do, the experience is just awesome.

With gnus, I feel pretty much the same. Frankly, its defaults *suck*. They
suck major butt. But after a handful of lines in emacs lisp, a whole lot of
the annoyances go away and you start to realize how useful gnus could be.
Potentially.

And if I would only want to use gnus for reading the Usenet, I could be
happy with gnus in - say - a day or so.

But then, there's not only the Usenet. There is also this medium you may
have heard of called `electronic mail' or `email' for short.

With email, my earlier sentence "Gnus is a Usenet News Client" comes
back to mind. And that's a problem really. Gnus handles emails very much
like it does news. And that will not work very well (read: not at all) if
you plan to work on your mailboxes with other tools, too. And for some
mail backends, I really found the length that gnus goes to piss me off
quite remarkable.

So what? Just forget about gnus handling mail and keep mutt for the job?
Well, we could. But after using gnus for news for a while, you start liking
a number of its features. I know I did. And luckily, there really is a
mail backend, which gets its behaviour pretty much right (with respect
to what you'd expect if you used *any* other mail user agent before).

And that backend is `nnimap'. That backend enables you to properly speak
to an IMAP server. Not just download stuff from there and then start being
annoying again but really speak to it and work with it like a real IMAP
capable client would.

"But my mail provider doesn't give me IMAP access to my mailboxes, and
 besides, I want all my mail locally on my machine. That solution just
 won't fly with me."

Yeah, I know. I said roughly the same. While I do have IMAP access to my
mailbox, I still want it locally. And there's a way to have both. "Local
IMAP server". Just install one. Specifically, install dovecot. It's pretty
easy to setup; it's very very quick and it's known to be pretty secure. It
handles maildirs very well. And so, as a bonus you can now use every client
that doesn't support maildirs but does support IMAP to read your mail. Like
claws mail, which except for its lack of maildir support I found to be the
by far most usable graphical MUA I ever tried. And I tried a lot.

So, a local IMAP server is something you'll likely want if you want to use
gnus to deal with email. If you don't want a local IMAP server, I suggest
you go and look elsewhere. IMHO, `nnimap' is the only sane way to let gnus
touch my mails.

While we're at it, what's the big picture with my mail/news setup?

We already know, that I use dovecot locally. I'm also running a local NNTP
server. Namely leafnode. Which rocks, too. It's far more lightweight than a
real NNTP server like inn2. But for a few users (in my case one - me) and
a few groups (around 25 at the time of writing) leafnode is a very sound
solution. I've been using it for years.

So, on the NNTP front, there's leafnode. Its `fetchnews' tool takes care of
getting news articles from my upstream NNTP server into leafnode's spool.
In my current setup it also considers a filter file, which I use to kill
off certain articles. So basically its my master-news-killfile. If you're
in there, you must have really pissed me off. Fetchnews also submits my
articles to the upstream NNTP host. Leafnode takes some time to setup, but
after that, you got a nice program suite that takes care of everything NNTP
for you. And you can pretty much switch clients as much as you like. :)

Now, email is a completely different kind of monster.
You got all sorts of agents, that do stuff. Some of these you desperately
want; others, you could do without. Then email has this tendency to,
from time to time, carry sensitive and/or personal information. The fact
alone, that it is used for 1-on-1 communication alone *requires* all
kinds to privacy facilities to be in place. To cut a long story short:
email tends to be a *lot* more complex to set up then news - at least from
a user's point of view.

On my system the setup looks like this:
First, sending mail. This can be become a cumbersome task if you're the
admin of a multiuser system. For me, I like to have something on my system,
that knows how to send mail to the outside. And it should better have an
interface that is known throughout the unix landscape. What comes to mind
is sendmail. And that's exactly what I want. *Everything* in unix knows
how to send mail via sendmail. Now, you don't need to use sendmail itself
(although you could - this isn't the 90s anymore...); about every MTA (one
of those agents - the Mail Transfer Agent) implements the sendmail
interface. Some people like qmail; others like postfix. Me - being a looong
time debian user, where this MTA is the default one for ages - I'm using
Exim. I got it set up to send mail via the smarthost of my mail provider
via encrypted authentication.

Now we need to fetch emails in order to read them. This can be done in
different ways. You could access the mail on your provider's server
(typically via IMAP, leaving stuff in a pop3 account is downright stupid)
or (if you're like me) you want to download the mail to one machine and
do whatever you want to it there.

Okay, downloading. There is another agent that takes care of this. The MRA
(Mail Retrieval Agent). Fetchmail is one of those; getmail is another.
I used both. I don't anymore. Moving on...

Now, assume you're getting a *lot* of mail. Many many mails. That's the
case for me. I'm subscribed to a number of mailing lists and that gets
me swamped in mail every day. To keep track of what is going on, I want
to sort mail into different mailboxes (maildirs, actually); one for each
mailing list usually. That task is performed by yet another agent. The
MDA (Mail Delivery Agent). Well known MDAs would be procmail (yuck!) and
maildrop.

I'm using a program that covers the jobs of both the MRA and the MDA. That
program is fdm; and I recommend everybody to take a look at it. It can do
pretty much everything that has to do with retrieving email from many sources
(pop, imap, other mailboxes, you name it) into mboxes (gzipped if you want
to) and maildirs. Oh and remember the scum of the earth that sends you all
those viagra and Rolex ads? Fdm allows the integration of a wide variety
of spamfilters, too. I'm using bogofilter for that (although the old
version of my config file that comes with fdm uses another one: spamprobe).
That tool is really nice, trust me. :)

Okay, so where are we? Ah yes, we're done fetching email from all my accounts
(five at the time of writing, different methods), it's spam filtered and
sorted into maildirs (e.g.: the mails from the zsh-workers mailing list
end up in ~/Mail/zsh-workers).

Now this is where dovecot comes into play. I'm pointing it to ~/Mail and
it makes those maildirs accessible via IMAP from localhost with whatever
and how many MUAs I'd like.

MUA would be the last agent for us, the Mail User Agent; mutt, gnus,
claws would be some of those.

So, MTA, MRA, MDA, spam filter, IMAP server and MUA. I said is was more
complex than news, didn't I?

We can now savely read and send email. Yay! But there's still one thing.
I need keep my maildirs from flowing over. One file per mail is really
nice for accessing mails. But not so much for archiving. Now I'm using
fdm again. Since it can use mail boxes as sources, you can use it for
archiving rather nicely. I'm just using fdm to get mails older than
30 days from each maildir below my ~/Mail and move them into gzipped
mboxes under ~/ArchivedMail into a file called
-q.gz (e.g.: zsh-workers-2005q3.gz).

We're almost done now. As I said earlier, email tends to carry sensitive
information now and then. Obviously we want to protect our privacy.
There's one way to do that (actually there are more but you should
really only care about this one). OpenPGP. Implemented by gpg. All you
need is a key pair and publish the public one to the world - like via
a keyserver on the internet.

Now that's nice. A local NNTP server and a local IMAP server, both
are automatically filled with new articles every ten minutes via cron
and both expire old articles automatically. So we just need to point
our mail and news clients to it.

Back to gnus. I know, gnus can do *everything* itself. Connect to the
upstream NNTP server, read and post there; read mail, sort it into
all sorts of mailboxes, send mail via smtp itself... *Everything*. But
I really prefer the setup I described earlier. It works very reliable
and it works the same for every client ever without having to reconfigure
anything. You may even be able to use multiple different clients if you
want to.

So why would I want to use gnus instead of mutt?
Well, who said I wanted to use it instead? I want to use it alongside.
There are some things mutt does very very good. Especially reading
mail. If there's *one* source, *one* account and *one* identity, mutt
is great (well for reading anyway there are other things I dislike
in it, too). So for pointing mutt to a mailbox and reading it, I want
to keep mutt. Browsing my compressed archives is one of the use cases
for that. Mutt is also nice for sending mail from scripts, so there's
another use case.

Okay okay! So why do you want gnus then?
That's not an easy question. Basically it's because gnus has every feature
any other message reader has; plus some crazy ones of its own. And, like
emacs - it's heavily extensible via elisp and a truckload of entry points.
Plus there are like a gazillion of options to tune gnus just to your
liking. It really kind of feels a lot like when I was heavily into
tuning my z-shell. It feels like there is nothing you couldn't get
it to do.

Also, you can make use of all those useful things emacs can do. And as
you might know, emacs can basically do everything.

But how can you use emacs for editing email? All those key combinations
should make you go nuts in no time... True actually. Emacs' input method
*is* inferior to vi's; let alone vim's enhancements. Modal editing just
beats the living crap out of everything else. I admit that and I embrace
it. Luckily, emacs knows a way out. That way is called viper mode. It
implements the original vi pretty much entirely. Everything you would
expect from vi plus all the stuff emacs can do if you'd like it to.
And if you'd want some of the nifty vim features, you should take a look
at `vimpulse.el' which implements quite some of them. So, the editor
just stopped being a show stopper. Emacs in fact has a nice mode for
writing messages called message-mode, which has all sorts of handy
features to ease writing messages.

In one sentence, the many many features that emacs implements can
nicely integrate into gnus giving it more or less infinite features
at the tip of gnus fingers.

Some features I particularly like:
 * I get some quality time with one lisp implementation (seriously (I
   always (wanted that. (kinda.))). :-)
 * group parameters; which let you set up stuff depending on which
   group was entered. mutt let's you do similar stuff via its hook
   system. But the way gnus does it feels a lot more structured and
   thought out.
 * advanced scoring (infinately complex patterns if you'd want to).
 * adaptive scoring (I'm not using it yet, but it sounds uber-useful).
 * different identities; in a usable way.
 * pretty cool gpg integration via emacs' pgg
 * article washing
 * topic mode
 * message mode


Let's break these down.

"Playing with Lisp"
I never could quite wrap my head around Lisp and it was one of the main
reasons I wasn't able to give emacs a serious spin... With gnus, you don't
get around lisp. It's not really deep Lisp. But here and there a few short
functions are needed and you shouldn't be afraid to get your fingers dirty.

"Group Parameters"
In gnus, many many things can be set up in a conditional manner with
respect to which context the program is in. Group parameters are one of
the most important facilities to do so. In short, if you want to change
settings depending on which group you're in, this is the way to go.

"Different Identities"
This is one of the things that pissed me of most in mutt. (The other big
issue I had with it was the handling of IMAP, especially with multiple
accounts - gnus excels in that field too; easily.) In gnus, posting styles
are a similar mechanism as groups parameters; they just handle switching
settings for creating new messages depending on the context.

"Advanced Scoring"
With simple scoring, you can usually alter scores by result of one
expression. Advanced scoring let's you put infinitely many of these
together with boolean operators. The gnus manual has some impressive
examples as to what you could do with this. As far as static scores go
this is as flexible as you can go.

"Adaptive Scoring"
When static scoring doesn't cut it anymore, you need adaptive scoring.
This feature changes scores of certain attributes depending on whether
you read a similar article in the past, or whether you just marked it as
read or if you deleted it; etc. etc. - As I said before, I'm not using this
feature yet, but this is one I'm looking forward to when I have more free
time again.

"GPG integration"
I specifically mean "when reading messages" here. While writing, it's damn
simple to use, too. But when reading, it's such a bliss. You get one line
above and below the encrypted or signed part of the message which shortly
tells you if the whole gpg business went well. If you need to know the
details of the process, you can un-collapse these lines into full reports.

"Article Washing"
This feature is something to get braindead articles into a readable form.
There are many many washing mechanisms, but one of the best (for me) is
this one: Ever got a mail from someone who just types and types and
types - he doesn't care about his readers - he just types. That results
in horrendously long lines, that your eye just cannot grasp. Now, with
gnus when I read such an article (and if it's a reply to a message I
wrote, I probably want to read it), I just press `W w' and gnus breaks
those lines ("fill" in emacs lingo) into 70-something character wide
lines that you can easily read. ...and as I said, that's just *one*
washing mechanism.

"Topic Mode"
You can put the group buffer of gnus into topic mode (in fact, I'm doing
that permanently). In topic mode, you can sort groups into topics. And you
may nest as must as you want to. You can also collapse topics so you don't
see the groups (and sub-topics) withing that topic.
To see is to believe here really. I made some screenshots of gnus running
my current setup and put them below. Topic mode can be seen there, too.
So, go and see for yourself.

"Message Mode"
This is the message composition mode for emacs that comes with gnus (you
know emacs has a mode for everything; and a mode is something that changes
the behaviour of emacs to fit a specific purpose). I'm using it with
flyspell (a mode that checks spelling on the fly using ispell, aspell or
hunspell - me, I'm going with aspell) and viper integration. And I love it.
I already liked editing mails in vim, but I think message mode with viper
and vimpulse has a slight edge.


There are more features I like about gnus, like the way you can just press
enter on links to send them to firefox, or integrating mail searching using
mairix... The list goes on and on. There's virtually no end to it.


As for tweaking gnus to your liking, let's take a look at some
customisations, which I am using:

Let's see, one of the things gnus does is hiding groups with zero unread
articles. That's a good thing and a lot of news-readers do that. With gnus,
you can fine tune what's hidden though - that includes not hiding anything,
which I had enabled for a time, but that really is damn useless. Anyway,
what I like is seeing my general email inbox all the time. Because generally,
when I want to read older mail, it's in there. The simplest way to get there
is to do this:

(setq gnus-permanently-visible-groups "\\(Inbox\\|INBOX\\)")

So there, now the inbox folder is visible all the time. Great. That also a
place you can focus on when there's not much else to see. "Focus?" I thought.
Wouldn't it be cool to make gnus move the pointer there for me every now and
then? Yes it would. When I say "check for new articles", it should just
move the point to the inbox group. So what I did is this:

(defun ft-gnus-group-go-home ()
  (with-current-buffer gnus-group-buffer
                       (gnus-group-goto-group "nnimap+localmail:INBOX" t nil)
                       (gnus-goto-colon)))
(defun ft-gnus-group-go-home-and-get ()
  (interactive)
  (ft-gnus-group-go-home)
  (gnus-group-get-new-news))

And then just bind `ft-gnus-group-go-home-and-get' to the `g' key, which is
usually bound to `gnus-group-get-new-news'. Done.

Now when gnus idles around, it could check for new articles every now and
then. I'm fetching mail and news every ten minutes when I'm connected to
the net. And then sometimes manually when I now something important is coming
my way. So I figured, gnus asking my local servers every two minutes wouldn't
be overkill. Now just checking would be too simple. Gnus should also conduct
the focusing business we discussed earlier. The solution is to add a `demon-
handler' to gnus. And so the code looks like this:

;; Check all sources for new article every 2 minutes.
;; If gnus-read-active-file is 'all, this may be annoying.
(defun ft-gnus-scan-news ()
  (gnus-demon-scan-news)
  (ft-gnus-group-go-home)
  (hl-line-highlight))
(gnus-demon-add-handler 'ft-gnus-scan-news 2 t)

The `hl-line-highlight' call is to make sure the current line is highlighted.
I'm not sure *why* is was needed, I'm just sure it is. Otherwise you end up
with point on the right line but with the line not being highlighted.

Now this "Focus-on-Inbox" thing has become an obsession and so I need gnus
to put point on the inbox after startup, too. Easy as pie:

(add-hook 'gnus-started-hook 'ft-gnus-group-go-home)

I wanted to show off more customisations I'm doing, but I'm getting kind of
tired. But rest assured I got a myriad of functions for thread-handling,
topic-navigation, group navigation, mail composing (in message-mode where
I'm using viper-mode, too) etc. The setup is approaching a thousand lines
at the time of writing this (not counting my colour-switching code, which
is part of my general emacs setup). This is really just the sharp edge of
the tip of the actual iceberg. :-)


I guess, now it's time for a conclusion.
Gnus is a very capable tool for dealing with messages. Be it news or mail.
If you're shy about lisp, don't bother. Look somewhere else. If you're not
willing to deal with the surroundings (like setting up a local IMAP server
to connect to via nnimap - which *again* is the only sane way to access
email with gnus, at least IMO), don't bother. Look somewhere else.

If all of that doesn't frighten you, come closer and give this beast a
shot. Even if you're a hardcore vi(m) user, don't be afraid. Viper and
vimpulse.el do quite a good job emulating your beloved editor. At least
for editing mail, you won't be disappointed.

Take it from someone who has used the mutt/vim combination in a lot of
different setups for slightly over a decade. I'm totally sold on gnus.


And now as promised, the screenshot slide show:

First, this is how gnus looks for me after startup (or after pressing `g'
in this group-mode). Point rests on Inbox, everything is in order.

Oh, these screenshots are taken in my default colour-set. It's perfect for
me when the surroundings are dark. Like in the early morning or the evening.
I got other colour-sets for when it's getting brighter outside.




This is the same as the previous one, but I un-hid the groups that don't
have new articles in them.




And here is a feature of topic mode. You can just hide entire topics; like
you would collapse chunks of code with code-folding.




The next shot shows how the `summary buffer' looks like for me. In mutt-
lingo, that would be the `index'. It only shows new articles by default.
You can change that if you really dislike it. And I did. But after some
time you realise that this is clearly the better way of handling a lot of
mail in huge mail boxes (only loading new articles is a *lot* quicker,
obviously).




Aaaaand, when you really really need to see some old messages, you can just
gnus to display them, too. No issue at all.




Now the next picture shows how an article is displayed to me. In article
mode, I can just hit enter on links and they're opened in a browser. I can
hide/un-hide quotations from certain people. Deal with attachments... It's
really pretty cool.




And finally, this is message-mode, the article-composition mode I'm using,
looks like for me. I got my viper-mode pretty customised in this mode to
easy dealing with different languages, email security and all that.




And that's that. Way too long, this article. :-)

(Comments: 0)