Over the last few weeks I've had the displeasure of dealing with an imploded SVN FSFS repository. In the process of learning what was wrong and how to go about fixing it (or working around the problem) I discovered that I could have mitigated some of this problem by monitoring my svn repositories for problems with svnadmin verify.
This wouldn't of course help me fix the problem, but it would have identified it long before I did, making the overall data loss that occurred far smaller. As it stands we lost about 25 revisions from one of our major SVN repositories, as we had to remove everything at and after the damaged repository and recommit it on top of the undamaged remnants. This means we lost alot of history but not necessarily alot of code. Still it was an extremely time consuming fix that would have been better simply avoided.
As such I've written a little python script that's simply designed to catch any errors generated by the verify command that is meant to be inserted into a crontab for execution. As such there are a number of things you'll have to edit before it'll be truly useful to you. It's been written this way to avoid many of the major pitfalls associated with running scripts from cron, which has a severely restricted execution environment.
Specifically you must update a few variables to reflect what you would like them to do or point at.
The first and most obvious is updating the /usr/bin/python, to point at your actual version of python. I strongly recommend typing which python and just pasting the results in the place of the above string, if you just put python in there instead your beholden to the path available to crond.
After that it's a simple matter of updating the variables under the Setup comment. I will list each one and explain what you need to put in that position.
svnadmin_path: this is the string returned from which svnadmin, this should be an absolute path, and the line under it will make it into such for good measure.
repository_path: this value serves one of two functions, either it's the location of the repository (singular) that you wish to keep an eye on. Or it's the location of the root of several repositories you wish to keep an eye on. In either case the path should be absolute.
recursive_path: this is a bool flag with possible values of True or False, if it is set to True the program will treat the repository path as the root of a group of repositories instead of as the target repository itself.
verbose: this is another bool flag with the same possible values. If set to True it will output short status notes at specific segments throughout the program.
I recommend running the program every day, more often is possible but seems like overkill unless your committing at a rapid pace. As such, my crontab entry:
0 0 * * * /root/verify_svn.py
First and foremost, this is by no means a 'perfect' piece of software. First, it has no error checking what so ever, so it doesn't fail gracefully, on the other hand the worst its going to do is attempt to do comparisons on things that make no sense and generate some nonsense output. Second, it relies on the fact that if you generate stdout output in a crond execution that output is captured and emailed to the owner of the crontab (in my case root, who also owns SVN and the repositories).
That all said, it works and pretty well if a bit slow. The svnadmin verify command is by no means a quick command even on a small repository. On a large one it takes as long as ten minutes to execute. However it does it's job which is to run a verify on all of our repositories every night and notify us if there are any issues (of a nature that verify can detect). Two possible areas of improvement that I've considered up till now but haven't done anything with is allowing command line arguments and making it work with Nagios somehow. Command line arguments are pretty easy to get up and running, but undo the whole compact one place to find all the info feature I like about the current implementation. On the other hand it will be a must if I want Nagios support. Nagios on the other hand is no nearly so easy, I'm not sure how I would even go about it, but it would prove supremely useful in our overall monitoring strategy where I work.
Entire code: (this code can be downloaded here as well.)