Gentoo: How to use Munin to monitor Mailman

Update 31/12/2012: Now this is a bit simpler, since bug 448414 has been fixed.

Before we start with the step-by-step procedure to get the mailman plug-in working, lets recall how to test a munin plug-in:

$ sudo munin-run mailman
posts.value 24
members.value 21892

The output above is what you should get once the mailman plug-in is correctly set-up. Continue reading “Gentoo: How to use Munin to monitor Mailman”

Gentoo: How to debug Exim

If you have some problem with exim, and the logs are not enough, edit the /etc/conf.d/exim file, and add the debugging option (-d):

/etc/conf.d/exim
# Command-line options for running exim
EXIM_OPTS="-bd -q15m -d"

Then restart exim:

# /etc/init.d/exim restart

The exim daemon will not be detached from the console, the debug information will be printed to the standard output. Good luck!

pygit2 v0.17.3 released

This should be the last release in the v0.17 series. The master branch has already been moved to be based in the libgit2’s development branch.

New in v0.17.3:

  • New Blob.size getter
  • New Repository.create_blob_fromfile method
  • Signature, now the time and offset parameters are optional
  • Improved diff support
  • Add pygit2.__version__
  • Optimize usage of Travis
  • Various fixes for the unit tests
  • Various documentation improvements

Thanks to Alex Chamberlain, Carlos Martín Nieto, Eric Davis, Eric Schrijver, Petr Viktorin, Ridge Kennedy and W. Trevor King.

CSS: Sticky footer

If you look around for a technique to get the footer stick to the bottom of the web page, you will find many variants on same theme (using absolute positioning, etc.). All have the same problem: the footer has fixed height, it does not grow with its content.

But there is one way to get a sticky footer of variable height, which is by the way much simpler than the other techniques. It involves using CSS tables. And here it is:

http://pixelsvsbytes.com/blog/2011/09/sticky-css-footers-the-flexible-way/

Have a nice read.

ikaaro 0.75 released

Almost two years have passed since the previous major release of the ikaaro web framework, and the itools library. The new release has tons of changes, so many I will not try to list them here.

This release is used by BePatient and some intranets.

I have labelled this release as beta, because the code to upgrade from the previous major release (0.62) is missing. Though, this is probably unfair since this version is already being used in production, and it is recommended to use it for new projects.

pygit2 v0.17.2 released

Just a week after the release of v0.17.1, here is v0.17.2, it fixes a single but rather important bug. The source distribution, as it was uploaded to pypi, was badly broken: it did not include fundamental files, so the package did not build at all. The problem was reported in issue 115

The bug was present forever actually, the MANIFEST file automatically generated by distutils did include everything, for instance test data files were missing. Things just get worse after the code refactoring that happened in v0.17.1

Now the setup.py file has been modified to generate the MANIFEST from Git itself, using git ls-files, so we don’t miss anything else again.

pygit2 0.17.1 released

pygit2 is a Python package that gives access to core features of the Git control version system. These features are implemented by the libgit2 shared library written in C, pygit2 is just a Python wrapper around this library.

The new version 0.17.1 of pygit2 adds several new features: support for diffs, the reflog and configuration files.

There has been too a huge code refactoring. Before the whole code was in a single C file of almost 4.000 lines of code, it has been split in several smaller files. Also pygit2 is now a package and contains a (for now empty) utils.py file, where we will add utility functions written in Python.

Other changes include improved support for Windows, and integration with Travis, a continuous integration service.

Several people has contributed to this new release, but Nico von Geyso deserves a special mention as the author of the code refactoring and of serveral new features.

Many thanks too to András Veres-Szentkirályi, Christian Boos, Martin Lenders, Petr Hosek and pistacchio for their contributions.

Gentoo: the HForge overlay

Just updated the ebuilds for itools and ikaaro in the hforge overlay, so I thought this is a good time to document how to use the hforge overlay.

Layman with Git support

If you have not installed layman yet, you have to, with the git use flag:

/etc/portage/package.use
app-portage/layman git

Then emerge layman:

$ sudo emerge layman

Tell layman about the HForge overlay

Edit the /etc/layman/layman.cfg file:

/etc/layman/layman.cfg
overlays : http://www.gentoo.org/proj/en/overlays/repositories.xml
https://raw.github.com/hforge/overlay/master/layman/list.xml

Add the HForge overlay

$ sudo layman -S
$ sudo layman -a hforge-overlay

Then edit the make.conf file:

/etc/make.conf
source /var/lib/layman/make.conf

Test by emerging itools

Now you can test for instance by emerging itools. First unmask the package:

/etc/portage/package.keywords
dev-python/itools ~amd64

Then emerge:

$ sudo emerge -p itools

Python: Running Valgrind on a C extension

This is how I use Valgrind to check for memory bugs in pygit2.

Glibc with debug symbols

The first issue I run into is that Valgrind refused to work if Glibc was not compiled with debug symbols.

This is how I did in my Gentoo notebook, I edited the /etc/make.conf file to enable the splitdebug feature:

/etc/make.conf
FEATURES="splitdebug"

Then re-emerged the glibc:

$ sudo emerge glibc

Then commented out the splitdebug feature to avoid emerging other packages with debug symbols (there is likely a better way to do this).

Valgrind 3.7

The big problem is that running Valgrind with a Python C extension raises tons of false positives. There are a number of things you need to do to avoid all these false positives.

The first one is to use latest version of Valgrind 3.7, because of some bug I forgot about present in version 3.6

Python, compiling for a memory debugger

Now, you need to install a version of Python to be used with Valgrind. There is one little change to do regarding an standard Python, edit file Objects/obmalloc.c and define Py_USING_MEMORY_DEBUGGER :

Objects/obmalloc.c
#define Py_USING_MEMORY_DEBUGGER

Now install as usual:

$ ./configure --prefix=~/Python-2.7.3-mem-debug
$ make
$ make install

The suppression file

Now you need to use the suppression file that you will find in the Python sources, at Misc/valgrind-python.supp. This is how I run the unit tests in pygit2 to find issues:

$ valgrind --trace-children=yes --suppressions=valgrind-python.supp \
~/Python-2.7.3-mem-debug/bin/python setup.py test

That’s it.

Gentoo: Mailman with Nginx & Exim

UPDATE 2017-05-11: Use https, with Let’s Encrypt.

UPDATE 2014-07-23: Forgot to give nginx access to mailman archives. Stop using sudo. Use service.

UPDATE 2012-12-10: the process has been simplified now that bug #37429 is fixed.

This is how I installed Mailman in a Gentoo server, using Nginx as the web server and Exim as the MTA.

Mailman

By default Mailman runs with the Apache user and group. First step is to configure Mailman so it runs with the Nginx user and group. To do so edit /etc/make.conf and add these lines:

/etc/make.conf
MAILMAN_CGIUID="nginx"
MAILMAN_CGIGID="nginx"

Now you can go ahead and emerge Mailman. Next comes to configure it.

Edit /etc/mailman/mm_cfg.py (replace the url and email hosts by yours):

/etc/mailman/mm_cfg.py
DEFAULT_URL_HOST = 'mailman.example.com'
DEFAULT_EMAIL_HOST = 'example.com'

# https
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
PUBLIC_ARCHIVE_URL = 'https://%(hostname)s/pipermail/%(listname)s'

# Let Mailman know that the MTA (Exim) needs no aliases setting
MTA = None

As the mailman user (add the cron jobs, create the site password, and add the main list):

# su - mailman

mailman $ crontab cron/crontab.in
mailman $ bin/mmsitepass
mailman $ bin/newlist mailman

Run mailman:

# rc-update add mailman default
# service mailman start

Nginx

The Mailman web interface works with the CGI interface. To get it working with Nginx start by emerging spawn-fcgi and fcgiwrap:

# emerge spawn-fcgi fcgiwrap

Create the configuration file and edit it:

# cd /etc/conf.d
# cp spawn-fcgi spawn-fcgi.fcgiwrap

These are the changes to the configuration file:

/etc/conf.d/spawn-fcgi.fcgiwrap
FCGI_SOCKET=/run/fcgiwrap.sock
FCGI_ADDRESS=
FCGI_PORT=
FCGI_PROGRAM=/usr/sbin/fcgiwrap
FCGI_USER=nginx
FCGI_GROUP=nginx

Now start the daemon:

# cd /etc/init.d
# ln -s spawn-fcgi spawn-fcgi.fcgiwrap
# rc-update add spawn-fcgi.fcgiwrap default
# service spawn-fcgi.fcgiwrap start

Now add the Nginx server configuration:

/etc/nginx/mailman.conf
server {
    listen 80;
    server_name mailman.example.com;

    location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name mailman.example.com;
    root /usr/lib/mailman/cgi-bin;

    ssl_certificate /etc/letsencrypt/live/mailman.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mailman.example.com/privkey.pem;
    # Redirect
    location / {
        rewrite ^ /mailman/listinfo permanent;
    }

    location ~ ^/mailman(/[^/]*)(/.*)?$ {
        fastcgi_split_path_info ^/mailman/([^/]*)(.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root/$1;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass unix:/run/fcgiwrap.sock-1;
    }

    location /mailman-icons {
        alias /usr/lib/mailman/icons;
    }

    location /pipermail {
        alias /var/lib/mailman/archives/public;
    }
}

Include the file into the main Nginx configuration file:

/etc/nginx/nginx.conf
include mailman.conf;

Add nginx to the mailman group, so it has access rights to the archive:

# gpasswd -a nginx mailman

And restart Nginx:

# service nginx restart

Exim

There is very good documentation on running Mailman with Exim: Using Exim 4 and Mailman 2.1 together.

The only thing I found missing from the docs is a reference to the mailman-loop address. Add an alias for mailman-loop to a routable address, edit the /etc/mail/aliases file, for example:

/etc/mail/aliases
mailman-loop: root