svnmailer - documentation

en

• svnmailer

• Code Documentation
• Change Log

General Comments

The svnmailer is an extensible subversion commit notification tool. However, its purpose in the first place is to create human readable commit messages. In order to accomplish this, the content may be recoded to fit the requirements of the MIME standard.

That means that you always get valid and readable mails, but you SHOULD NOT expect to be able to copy the diffs from the generated mails and apply them with the patch program. Use svn diff or viewcvs diffs for such tasks. The svnmailer is able to generate the proper viewcvs urls and place them quite handy near the diffs in the notification mails.

Installation

Before installing the svnmailer package make sure, that you meet the following requirements:

• Python 2.3 or later
• A POSIX compatible platform (Win32 is supported but not fully tested yet)
• The subversion bindings for python

The svnmailer is a pure python package, which is packed using distutils. So the installation on your system is fairly simple. First download the package and make sure you've checked the integrity of the downloaded file. Use the supplied PGP signature and/or the MD5 hash for this task.

After unpacking the archive file change into the svnmailer-0.9.9 directory and follow the typical python procedure:

Now there should be two things installed, the svn-mailer command line script and svnmailer package itself. The package is copied to the "proper" location, where python finds it by default. The location of the script depends on the OS and the python installation. For example, on linux it typically installs into /usr/bin or /usr/local/bin. For customizations please refer to the related python documentation.

The next step is to create a configuration file. After you've done that, you can configure your repository to let the svnmailer do its work.

Config File Design

In short: the configuration file controls, who gets the notifications for which path in which repository. Further it defines the basic script parameters like how to send the notification mail, which diff program to use (if any) and so on. It it supposed to be compatible to the config of the original mailer.py script.

Syntactical Elements

The file is a plain text file in an INI like format as defined by the standard python ConfigParser module. Basically it consists of several sections, that are started with a line containing:

and finished at the next [section] or the end of the file. Values are defined that way:

As you might have guessed, comments are preceded by the hash character (#). Empty lines (or lines containing only whitespaces) are ignored. Note that section headings, comments and value definitions have to start at the first column, because the configuration parser treats (non-empty) lines starting with spaces or tabs as continuations of the previous line (like in mail headers).

Semantics

The sections in your config file are processed by the svnmailer as follows: The [general] section contains the basic script parameters. The [defaults] section contains default values for the group sections. All values that are not defined in a selected notification group are taken from the defaults. [defaults] is optional. All other sections define notification groups. The names of these sections don't care (except for debugging purposes). They just have to be unique within the config and may not be named defaults or general, of course. if there is no separate group configuration, the defaults will be applied. So a minimal config is:

If you call the svnmailer with this config, it will generate diffs for every possible action at any path for any supplied repository and write a notification message to stdout.

Group Selection

When the svnmailer is called, it reads the config file and selects all groups, which should be notified of that particular event (commit or revprop change). Which groups are selected, is determined by the for_repos, for_paths, exclude_paths and ignore_if_other_matches options. Since the svnmailer tries to minimize the number of notification mails, each of these selected groups gets one mail at maximum per event. Furthermore, if the notifications generated for different selected groups are detected to be equal, these groups are merged and just one mail is sent to all of those groups. Currently this detection compares the list of modified paths (that were matched by each group) and several configuration options, which are in particular: commit_subject_prefix, propchange_subject_prefix, generate_diffs and viewcvs_base_url.

Note that there's a subtle difference to the original mailer script. If there are group configurations defined, the svnmailer will never consider the [defaults] section as an additional group to be notified.

[general] Configuration Section

The possible configuration options in [general]

Option Name	Type	Description
config_charset	string	The character encoding of the config file
diff_command	command line	The external diff program to use.
sendmail_command	command line	The sendmail compatible command line template
smtp_host	string	The SMTP host[:port] to use
smtp_user	quoted literal	The user used for SMTP authentication
smtp_pass	quoted literal	The password used for SMTP authentication
debug_all_mails_to	mail addresses	Mails should go only to these fixed addresses

Though the [general] section defines such basic parameters, it may be finally empty, because there are more or less useful defaults given for each option. However, to make sure, that you didn't forget it, the svnmailer requires at least the section heading to be present in the config.

The different options play together as follows:

• If the diff_command option is given and not empty, it is used to generate the diff output. Otherwise an internal differ is used. The internal differ always generates the unified diff format.
• If the sendmail_command option is given, it is used to send the mail.
• If the smtp_host option is given and sendmail_command is not given or empty, the former is used to open an smtp session to the specified host and deliver the mail via SMTP. If the server requires authentication, you have to supply both smtp_user and smtp_pass. However, if the server doesn't support authentication, you may not supply the smtp_user option (alternatively just leave it empty).
• If neither sendmail_command nor smtp_host is specified, the notification message is written to stdout (with a small header, which configuration groups were selected). This can be used for debugging purposes.

config_charset

The config_charset option defines, how the svnmailer should interpret the (bytes read as) option values in the config file. It defaults to us-ascii, which means, normally you can leave it just out. But note that there are exceptions. The options of type "quoted literal" are not charset decoded, but taken literally (possibly unquoted, however).

diff_command

The diff_command option defines, that you want to use an external diff program (instead of python's difflib module), where to find and how to call it. The option value is a template for the command line to call. The program has to write the diff information to stdout. Stdout and stderr are caught by the svnmailer and dumped into the mail.

As said, the value describes a template. There is a fixed number of substitutions defined for the diff_command option, in particular: label_from, label_to, from and to. from and to define the actual files to process (typically some scrambled temporary file names). label_from and label_to define, how the files should be labeled by the diff program. These substitutions are written as %(name)s and replaced by the svnmailer script. If you want a literal percent character somewhere in the command line, you have to duplicate it (i.e., write %% instead).

The command line template is split on spaces or tabs to separate the different arguments. If you want to have an argument contain such space characters, you have to enclose it in double quotes ("). Further if you want such a quoted argument to contain a double quote character, you have to escape it with a backslash (i.e., write \" instead). To complete the escaping mechanism, you have to duplicate backslashes inside a quoted argument. The surrounding quotes and the backslash escape characters are stripped by the svnmailer to get the final argument string.

Note that on POSIX systems no shell is called to execute the program, so shell metacharacters are not interpreted. However, on Win32 the shell (cmd.exe & Co.) is called, but the arguments are properly escaped.

For compatibility and convenience the diff_command option can also be written as diff.

sendmail_command

The sendmail_command option defines the command line template of the program that should be called for sending a mail. The program should expect the mail body on stdin. The stdout channel of the program is closed by the svnmailer, but stderr is passed through the caller of the svnmailer.

In contrast to diff_command there are no substitutions made on the arguments. Instead the final command line is constructed as follows:

1. The command template is split into its arguments following the rules described at diff_command.
2. To specifiy the mail sender the arguments -f and the sender address is attached to the argument list
3. The argument list is further extended with all recipient addresses.

This calling convention is compatible to, for example, sendmail or qmail's sendmail wrapper, hence the name sendmail_command.

As with diff_command no shell metacharacters are interpreted.

For compatibility reasons the sendmail_command option can also be written as mail_command.

smtp_host

The smtp_host option defines the SMTP server to connect in order to send a mail. This option is ignored if sendmail_command is defined and not empty. The option value is a hostname, optionally followed by a colon and a port. If the server supports authentication, you can supply the required credentials via the smtp_user and smtp_pass options.

smtp_user and smtp_pass

These two options are only used if the smtp_host is used by the svnmailer. They define the credentials to be used in the smtp session when attemting to send mail. If you supply smtp_user, you have to define smtp_pass as well. However, svnmailer just checks for the presence of smtp_user to know, if any credentials should be used.

The utilized smtp library supports the CRAM-MD5, PLAIN and LOGIN authentication mechanisms.

Because of the nature of those two options, the values are not considered to be charset encoded. They are sent literally to the smtp server. In order to make sure, that spaces and other possibly weird characters are taken literally, you can enclose the actual string in double quotes ("). For double quotes and backslashes inside the quoted string apply the same rules as for command line arguments. Have a look at the description of diff_command for details. Of course, the surrounding quotes and backslash escape characters are stripped before submitting the string to the SMTP server.

debug_all_mails_to

This one is a real debugging option. It specifies a fixed list of mail addresses, where all notification mails should be sent to -- regardless of the to_addr templates of the selected groups. The addresses of the overridden recipients are sent along with the mail using the X-Supposed-Recipients header.

Group Configuration Sections

The possible configuration options in group sections and [defaults]

Option Name	Type	Description
for_repos	regex	Matches the repository file path
for_paths	regex	Matches the virtual path inside a/the repository
exclude_paths	regex	Excludes paths that might be matched with for_paths
ignore_if_other_matches	boolean	Determines, whether the group should be ignored, if any other group matches this path
commit_subject_prefix	string	The mail subject prefix for normal commits
propchange_subject_prefix	string	The mail subject prefix for revision property notifications
from_addr	template	The sender addresses
to_addr	template	The receiver addresses
reply_to_addr	template	The reply-to address
generate_diffs	token list	The list of actions, which generate diffs
viewcvs_base_url	string	Base URL of the viewcvs installation

The options described here are all valid both in group configurations and in the [defaults] section. If a option in a normal group configuration is missing, its value is taken from [defaults]. If there is nothing defined, a hardcoded default is applied.

for_repos

The for_repos option defines a regular expression, which is used to match against the file path of repository, for example /var/svn/my-repository. The file-path, which is matched against is guaranteed to not have a directory separator at the end (slash or backslash). Note that the regular expression always matches from the beginning of the path, so your regex typically will begin with .*. This is, because the svnmailer uses the re.match function - see the python docs for further information.

If the for_repos option is not defined or empty, the particular group matches for any repository (which is the default). Named matches of this group are stored for later substitutions.

In the following example the group "sample group" will be selected only if the script is called for the "public" repository (e.g. /var/svn/repositories/public):

for_paths

The for_paths option defines a regular expression, which is used to match against one of the modified paths stored in the repository, but only if the group was preselected by repository (see for_repos). If the path matched against is a directory, it is guaranteed to end with a slash, so that matching by directory paths results in more simple regular expressions. As with for_repos, the match always starts at the beginning of the path, but without a leading slash.

If the for_paths option is not defined or empty, the particular group matches for any path inside the repository (which is the default). Named matches of this group are stored for later substitutions.

In the following example the group "sample group" will be selected only if the script is called for the "public" repository (e.g. /var/svn/repositories/public) and everything under the /site/ directory (e.g. /site/images/foo.gif, but not for /site-tools/buildsite.sh):

exclude_paths

Since regular expressions usually match positive, it's from time to time helpful (and better readable) to exclude substrings with a separate match. The exclude_paths option exists for that purpose. It matches exactly like for_paths, but the group is selected only if the supplied regex does not match (and has been preselected by for_repos and for_paths).

If the exclude_paths option is not defined or empty, nothing will be excluded (which is the default). Of course, named groups of the match will not be stored for substitution, because the group is not selected, if there is a match of exclude_paths.

In the following example the group "sample group 1" will be selected only if the script is called for the "public" repository (e.g. /var/svn/repositories/public) and everything under the /site/ directory (e.g. /site/images/foo.gif), but not for stuff under /site/tools/. For every change under the site/tools/ directory the group "sample group 2" will be notified:

Note that if the exclude_paths option was not given, every change under site/tools/ would generate a notification for both groups.

ignore_if_other_matches

Consider a main project, which consists of several subprojects. Every subproject has its own notification group:

The exclude_paths option could be matched easier, if the sub projects really would be named subdigit. But usually this is not the case. As you see, maintaining the exclude_paths regex grows to a nightmare the more projects are added. The ignore_if_other_matches option is supposed to help out of this ugly situation. If set to a positive value (e.g. yes), the group will not be selected for the matched path if there are any other groups that match the same path / repository. The above config could be rewritten as:

Note that there is a border case. If you use this feature for more than one group, it can happen, that finally the list of selected groups per path consists only of more than one ignorable groups. Theoretically these would unselect each other. Practically all those groups are selected, so that the notification is not lost.

The "boolean" values accepted by this option are yes, on, true and 1 for the "true" case and no, off, false, 0, none and the empty string for the "false" case. The default is false.

For compatibility reasons, convenience and better readability this option can also be written as suppress_if_match or fallback.

commit_subject_prefix and propchange_subject_prefix

These options define the subject prefix of the generated mails depending on the described event. If a string is supplied, it's included with a space between the prefix and actual the subject line. commit_subject_prefix defines the prefix for normal subversion commits (files, directories and versioned properties). propchange_subject_prefix defines the subject prefix for unversioned property change notifications. The default prefixes are empty.

from_addr, to_addr and reply_to_addr

from_addr, to_addr and reply_to_addr define address templates for the mails to be sent. Both from_addr and to_addr accept space or tab separated lists of address templates, while reply_to_addr takes just one address. The semantics should be quite clear: from_addr defines the sender addresses (but usually just one), to_addr the recipient addresses and reply_to_addr the address, where answers to the commit mail should be sent to. If groups are merged during the selection process, there can be any number of senders, receivers and even reply-to addresses in the mail (which conforms to RFC 2822, if you care about such things). Duplicates in the address lists are filtered away. In the case of more than one final sender address, the svnmailer generates an additional Sender: header with the first item of the sender address list (which is more or less random, but they should be all valid, right?).

If there are no sender addresses given, it uses the string no_author (as the original script does). That may lead to an error while mail sending, so the best is to supply a valid from_addr in the [defaults] section.

If there are no recipients, the svnmailer simply doesn't send the mail (this will be useful in the future, when there are more notifier types like news etc). If this is not, what you want, you have to supply functioning to_addr options.

All those addresses may contain substitution patterns in the form %(name)s. The list of values to substitute is determined for each notification group dynamically using the for_repos and for_paths regular expressions. The substitution name author is always defined. If not overridden by one of the regular expressions, it contains the author of the change (or the string no_author if no author could be determined).

All address templates described here are empty by default. For compatibility reasons the reply_to_addr option can also be written as reply_to.

generate_diffs

The generate_diffs option defines which actions diffs are generated for. It takes a space or tab separated list of one or more of the following tokens: add, modify, copy, delete, propchange and none.

If the add token is given and a new file is added to the repository, the svnmailer generates a diff between an empty file and the newly added one. If the modify token is given and the content of an already existing file is changed, a diff between the old revision and the new revision of that file is generated. The copy token only worries about files, that are copied and modified during one commit. The delete token generates a diff between the previous revision of the file and an empty file, if a file was deleted.

If the propchange token is given, the svnmailer also takes care of changes in versioned properties. Whether it should actually generate diffs for the property change action depends on the other tokens of the generate_diffs list. The same rules as for files apply, except that the svnmailer never generates property diffs for deleted files. For example:

Now svnmailer generates diffs, if:

• A file is added, modified or copied (and modified)
• A property is added or modified (properties cannot be copied)

If a file or property is deleted, it's written as action information into the mail, but no content diff is generated. The default value for generate_diffs contains all possible actions. Mistyped tokens are ignored. If the token list is empty, svnmailer falls back to the default. If you really don't want diffs to be generated, use:

viewcvs_base_url

If the viewcvs_base_url option is defined and not empty, the svnmailer generates URLs for the viewcvs repository browser. One URL for the whole revision is placed on top and for every generated file diff a conrete URL is written before the actual diff output. The default value is empty.

And here is an output of a change in a sample repository (revision 5):

Substitutions

Substitutions are a powerful feature that can simplify some configurations very much. They base on named python format strings. The format strings are similar to the ones you may already know from the printf() function of C or perl, except that the particular formats are not determined by order, but by name. To give a particular format a name, you just write %(name)s instead of %s (for a string). The name can be any sequence of characters.

As usual, when dealing with such format strings, if you want to express a literal %, you need to duplicate it (%%). Note that you should limit your formats to strings (i.e. the s format), because that's what the svnmailer always supplies. Here is a real life example:

Well, where are the format names and values taken from? Depending on the option they are either fixed or determined dynamically. The former -- simpler -- variant is used by the diff_command option. For this option the svnmailer defines, which format names and values are used (label_from, label_to, from and to). For the actual meaning of these format names have a look at the diff_command description.

The dynamic definition of format names and values is a bit more complex. This is used by the address templates. There the list of format names (and values) is taken from previously matched regular expressions (default for_repos, default for_paths, group for_repos and group for_paths -- in that order, later definitions override earlier ones). This relies on another python feature: named matching groups. These are like normal storing parentheses, but you can give them a name. Instead of (to-match), you write (?P<name>to-match) in your regular expression. Such a name has to look like a valid python identifier. However, after a matching regular expression is executed, the svnmailer stores such named groups for later use separatly for each notification group. Currently these values are supplied by the mentioned address templates. A typical use case of this feature is:

Configuring the Repository

Now, after you've created a config file, you can hook the svnmailer into the repository. Subversion uses so called hook scripts to perform customized actions at certain stages of a change event.

Configuring For Commit Messages

In order to run the svnmailer for normal commits you need to call it at the post-commit stage. If your post-commit hook isn't customized already, change into the repository/hooks directory and copy the post-commit.tmpl template to post-commit It's generally a good idea, to keep the template file for later reference.

After you've copied the template open the newly created hook script with your favorite editor. You will see a lot of comments describing the purpose of the hook and giving some hints about file system permissions etc. After you've read these comments, there follows a small shell script. Typcially there are sample scripts activated, you may not want to run. If so, comment them out or delete the entries. Finally to activate the svnmailer, add the following:

Of course, the REPOS and REV variables are only available if you left the definitions below the comments. Anyway, that's it. After you saved the file and closed your editor, you only have to make it executable and subversion will execute it after every commit. It is, however, a good idea to test the hook script as the user who runs it, before doing the next commit (That is only possible, if there are already revision stored in the repository):

Now you should receive a notification. If not, you should get a descriptive error message, what went wrong. By the way: a typical error is a non-readable configuration file.

Configuring For Revision Property Changes

Configuring for a revprop change message is mostly equal to commit messages. If you do not allow to modify revision properties, you don't need to care and can stop here. If you want to allow for revision property changes, you should read the related subversion documentation regarding the pre-revprop-change and post-revprop-change hooks first.

You can hook the svnmailer into post-revprop-change to get a notification containing the new property value.

After opening the hook script the same warnings as for post-commits apply: remove all stuff, you don't want there. After that add the following to the script:

The REPOS, REV, USER and PROPNAME should be still defined, of course. Close the file, make it executable and test it:

Congratulations, you're done. Happy Hacking!

Perlig - Projects - Publications

Imprint