As to why they did it this way, I can’t say I understand.

The reason for this is that Mercurial’s ConvertExtension is somewhat of a low-level tool. (In other words, although reliable it is not too bright). Browsing the internet you may get the impression that it’s an automated conversion system: it isn’t. It does fully automated migration only for straight SVN repositories, but for the rest it’s more like something to use in your migration script. It seems to do its primary purpose – converting revisions from one repository format to another – quite well but the rest of the tool is not so intelligent and it needs help. So, lesson number one: if you have a complex repository, don’t take the migration lightly.

A small disclaimer is in order: this post is not intended to be a complete step-by-step guide to migration. Rather, it’s something to fill in the blanks left by what little is available on the internet. I’ve done a complex migration and I want to do a brain dump for my future reference or for “whomeverother it may concern”.

Between the two alternatives I perceived as most promising, hgsubversion and the convert extension, i chose the latter. Hgsubversion was claimed by some to be the better tool for this job, but it was somewhat troublesome. The problem with hgsubversion was that it had a memory leak and broke easily in the middle of the conversion (note that this happened a couple of months ago: things may have changed in the meantime). The solution, they say, was to do hg pull repeatedly until it finishes. I wanted to do a hg clone with a filemap, but when the import broke I was in trouble because hg pull doesn’t accept filemaps. (It could be that the filemap was cached somewhere inside of the new repository and my worries were unfounded, I don’t really know). I may try that in the future. One other way around it would be to do a straight clone of SVN – no branches or anything – into an intermediate mercurial repository and then split that into separate final repositories. In that case, hgsubversion could be a viable solution, maybe even better than the conversion extension. I had more success with the conversion extension so this is what we’ll talk about here.

Part 1 – splitting import by revision

The repository in question here – that is, a folder within the SVN repository – started as part of another project and only a thousand revisions later was moved into its separate folder – thankfully, at that point I at least created the proper trunk and branches folders. So, I had one part of history where everything was trunk but it moved around the repository, and another part where nothing moved but I had a trunk and a couple of branches. Luckily, I had a buffer zone of a couple of hundred revisions where everything was trunk and nothing moved so I didn’t have to pinpoint the exact revision on which I had to split the import.

Let’s say that the early version had a structure like this:

Crm/Fwk1
Crm/Fwk2
Crm/Other unimportant folders (that is, not to be imported)

At revision 1000 the first two folders were moved to

Framework/trunk/Fwk1
Framework/trunk/Fwk2

The first branches appeared at revision 1200 in Framework/branches.

So, it was to be done like this:

• Step one, import everything up to the revision 1100 into the default branch. Include the Crm/Fwk* folders and Framework/trunk/Fwk* folders. As I said, in this revision the Framework/branches folder was empty so we don’t lose anything.
• Step two, import the rest but tell the conversion extension that the branches are in the Framework/branches folder so that it picks them up properly.

Sounds simple? Note that I had to perform a serious research of my repository: had a branch been created earlier than revision 1000 or had I made a branch at the same moment I created the trunk, things would have been more complicated. That is to say, I would have probably had to split the import into more steps and repeat some operations and do more testing to see on which exact revision I should stop the first step. Lesson number two: know thy repository.

The first step of this import is not so hard. I used the –rev 1100 argument to stop it at revision 1100: the convert extension purportedly remembers what it imported so far and when called again continues at that point (well, not exactly… read on).

"c:\Program Files\TortoiseHg\hg.exe" convert d:\data\Subversion Framework --rev 1100
  -s svn --filemap=fwkmap_step1.txt 

Note that I have access to a local SVN repository – for some reason, the convert extension didn’t want to access the local repository using an svn:// url (possibly the firewall had something to do with it).

The only thing left is to make a good filemap. Something like this:

include Crm/Fwk1
include Crm/Fwk2
include Framework/trunk 

rename Framework/trunk .

What we want to do here is to include the old folders – this is the first pair of lines: the folders exist only in earlier revisions since they were removed (that is, moved) later. I was under the impression that it would have been sufficient to include just “Framework/trunk” and that the convert extension would somehow detect that where this path originated from and include the full history, but it didn’t work out. On another repository I tried I was surprised to see that it actually did something like that, but it may have been a coincidence (a combination of other includes, possibly). In any case, it doesn’t hurt to specify the filemap as precisely as possible since you may have to fiddle with various parameters and do repeat runs. Make the filemap as tight as possible so that nothing unexpected leaks through it and eliminate any uncertainty.

The last line of the file map – “rename Framework/trunk .” tells it to make the trunk folder root. This is to make sure the structure of the folders is the same as it will be in the second step, where we use different parameters and import a completely different structure into the same folders.

Always keep in mind that the filemap (probably as well as everything else) is case sensitive. I spent hours debugging my imports because I didn’t notice the difference in case. Also, if you have a folder (or file) whose case has changed through history, it may be wise to add a rename statement in your filemap to make it consistent so that the conversion logic understands that it’s the same file/folder in different revisions (otherwise I’m not sure it would?).

In step two, we tell it where the trunk and branches are, using the –conf –config convert.svn.trunk and convert.svn.branches parameters. I’ve come to the conclusion that this changes the game for everything: the convert extension regards trunk and each branch as a root folder so a filemap like the one from the first step wouldn’t work. I haven’t tried it with pathnames relative to trunk and/or branches, though, and it may be worth investigating. In this case I didn’t need a filemap because after revision 1100 everything was done “by the book” in the Framework/trunk and Framework/branches folders.

So, when I ran both steps I got two distinct revision lineages: one that started at 0 and finished at 1100 and included the first run, and another that had revisions from 1000 onwards, but there was no connection between the two, each ended with its own head. And – oh, yeah, I got the branches the way I wanted them in the second part. But, how to connect the two parts?

The thing is, when running an import from a local repository (be it SVN or another HG repo), a file called SHAMAP is stored in the .hg folder of the destination repository (if you import from a remote repository, there’s an equivalent file the name of which I forgot – I believe it’s stored somewhere in .hg/svnsomething). The SHAMAP file contains pairs of revision hashes/numbers so that it knows which source revision was converted into which destination revision. For SVN import, it contains a GUID for the repository and a revision number, in the format of “SVN_REPO_GUID@SVNREV”. I’m also under impression that revisions stored here won’t be imported again on subsequent repeated conversions – this is (as far as I know) wrong because filemap include/exclude may cause a partial import of a revision and other parts of it may need to be updated again in the following steps. In such cases it is you who needs to help by supplying your own revision mapping file, and that’s probably what the convert extension authors also thought because it can be done by supplying the REVMAP parameter to hg convert. Remember what I said about it being a low-level tool? This is it. You need to write your own script to do the import properly, and hg convert is a tool used in the script. The bottom line – at the end, you should know what you’re doing. You can (and probably will) learn as you go, though, so don’t be afraid to experiment. (And while we’re at it: if you’re doing a time-consuming import in multiple steps, test each step separately and when you’re satisfied with it, zip the resulting repository so that you don’t have to repeat that step while testing the next one).

But I digress… Back to SHAMAP: the fact that it remembers revisions already imported and doesn’t allow repeated imports didn’t bother me here because I don’t have overlaps – that is, I don’t need to import the same revision (but different files) in multiple passes. The Crm/* folders disappear long before revision 1000 and at that point I only need Framework/trunk, which is also true in the second step that comes in after revision 1100.

Ok, but I did get duplicate revisions. It imported Framework/trunk up to rev 1100 in the first step and then again imported Framework/trunk from its inception to the end. Looking at SHAMAP shows why: the revisions were registered in a different way here in the second step. Instead of the “SVN_REPO_GUID@SVNREV” format, it stored something like “SVN_REPO_GUID/Framework/trunk@SVNREV”. Why? I’m not sure, it may have something to do with treating the trunk and branch folders as roots. It’s probably an attempt to prevent the problem mentioned above, when a revision needs to be imported multiple times. But this is far from complete, because in that case the filemap also needs to have similar influence on the SHAMAP so that it reflects both filtering and renaming policies set by it. (Mission impossible, I know… That’s probably why the convert extension is badly documented – when you need to explain something like this you risk receiving questions like “so why didn’t you make it better to prevent this problem?”).

One solution for this could be the splice map: it’s a file wherein you can define which revision needs to be connected to which during import. I tried this without success (I suspect that I didn’t pick the right revisions – probably the two spliced revisions need to be identical) but found a hack that produced immediate results: I opened SHAMAP and did a quick find/replace of “SVN_REPO_GUID@” with “SVN_REPO_GUID/Framework/trunk@”. This converted the revision hashes into the format the second step used, so it understood them and connected them correctly.

Here’s the command used for conversion.

"c:\Program Files\TortoiseHg\hg.exe" convert d:\data\Subversion Framework -s svn
  --config convert.svn.trunk=Framework/trunk --config convert.svn.branches=Framework/branches
  --config convert.svn.tags=Framework/tags --branchmap=fwkbranchmap.txt

The fwkbranchmap.txt file has one line (I’m not sure why it is needed anyway, I supposed that the convert extension understands that “trunk” in SVN is “default” in Mercurial):

trunk default

So this is one way to do it. I thought I would need to investigate the things further for the import of other repositories, but came up with a different strategy. So it’s left at this state, a bit unpolished but usable.

Part 2 – splitting import by trunk and branches

For the second repository, I had three projects of which two were partial branches of the third one which from now on is to be considered the trunk. So I thought I could import them one by one: the trunk has moved a bit through the repository, and the branches have stayed mostly in place. This is what it looks like:

trunk – up to revision 2000:

Crm/Crm1  Crm/Crm2
Crm/Fwk* which were imported in part 1 and need to be ignored now.

trunk – after revision 2000:

Crm/trunk/Crm1
Crm/trunk/Crm2
Crm/branches - which we will ignore to make things simpler, as they are obsolete anyway

branch for client1:

Client1/trunk/Crm1
Client1/trunk/Crm2
Client1/branches – ignored for simplicity

branch for client2:

Client2/trunk/Crm1
Client2/trunk/Crm2
Client2/branches – ignored for simplicity

What do we do now? Split the repository vertically – import the trunk folders into default branch in step 1. Then import the first client in step 2 using branchmap to move the default branch into a new “client1” branch and repeat for client2 branch. We won’t use the convert.svn.branches parameter but import each branch explicitly. This we can do even with a straight Mercurial-to-Mercurial conversion, which I tried to do: I imported the full SVN repo into a Mercurial repo and then did the next conversion from it. I thought it would be faster: it wasn’t. Also, the difference in speed between importing from an SVN repository folder and importing from a local SVN server is not significant.

In a case like this, you need to exert full control over import. Treat the convert extension like it doesn’t know much and tell it all the details about the conversion. From what I’ve seen, its logic is somewhat counter-intuitive: it would make sense for it to reconstruct the revision history by following each file through its revisions, whether through branching or moving around in the repository. In that case it could reconstruct a file’s history from its creation till today, and all you would need is to tell it where the file is today. But it doesn’t do it like that: instead, you give it a bunch of include/exclude filters to tell it which files to retrieve, and these filters are applied at any point in history. If you made an accidental move of a folder at some point in the past, make sure you include that path also or your revision history will stop at that point.

We will possibly be importing each revision multiple times (in case multiple branches were committed to SVN in a single commit – which is improbable but possible), but each time with a different include/exclude filter in the filemap – for this, we have to make sure the filters don’t overlap unless necessary. Here we come to a new problem: it seems that the conversion extension doesn’t want to do repeated conversions of old history. It seems to remember what was the last revision imported and only imports newer ones. When convert.svn.branches or convert.svn.trunk is used, it views the revisions differently (relative to a different root) and doesn’t mind importing them again. But we won’t use those here, at least in this case, and it wouldn’t have made much difference anyway – I think it would just help with this particular problem and nothing else.

We solve this by using an empty REVMAP file. An empty REVMAP will replace the SHAMAP and make it look like nothing was imported yet. And, better still, we can also use it to get rid of the splicing problem: put in it the mapping for revisions where the source was branched so that the two branches connect at the proper point. Otherwise, the conversion will create branches that aren’t connected. How do we do this? After the first step – in which we import the trunk with all of its history from day one (and here I assume that all branches originate from it), we should have all junction points in the repository. The next step would be to view the history for each branch and note at what revision in the trunk it was branched. We find that revision in our SHAMAP file (which was updated at step 1 – trunk import – but won’t be used afterwards) and add this line in our REVMAP file for that branch. It may happen that multiple revisions from the source repository were imported into this one in our new trunk, in that case I put all of them in the REVMAP, just in case.

One important thing to note here is that we have full power over the outcome. If we miss-connect the branches, we may get odd results – I don’t expect the source to be screwed up, but the history may become a bit strange. In fact, the complicated combination of filemap filtering, splicing and the rest may produce some odd contents in the repository. I managed to get a folder that was deleted years ago reappear in the latest version of the repository. It seems that it was deleted on the trunk after a branch was split from it, but the branch didn’t include it in the first place – it probably happened that the filtermap filters for the branch made it ignore the folder completely, and since it wasn’t mentioned anywhere in the branch history (primarily not as being deleted), it appeared in the branch as it was at the point where the branch was created. Strange stuff, but instead of trying to tune everything to get it correctly imported (and risk getting more unneeded garbage in the process), I simply deleted the folder and committed this change in the destination repository.

There’s another lesson learned here: in order to make the import for this part easier, I tried moving folders around in the SVN repository to recreate the canonical trunk/branches/tags structure. If you want to do your conversion without convert.svn.branches – that is, convert each branch separately, don’t do it. It will only make your life harder because you will have also to include that folder in your filemap, and probably to rename its contents to become root. I myself stripped this revision from the destination repository as if it never happened.

The command sequence looks something like this. First the trunk import:

"c:\Program Files\TortoiseHg\hg.exe" convert Full Crm -v --filemap=crmfullmap_step1.txt 

(I added the “-v” switch so that I can see what’s going on, remove if the output is too verbose… This switch is useful because it causes the printout of all files included, so you can check to see if there’s anything suspicious – it won’t be of too much help determining whether something’s missing but you’ll be able to see if there’s anything you don’t want).

At this point we need to look at the SHAMAP file generated and create REVMAP files for each branch, as described above. Since the file will be overwritten by the import logic, you may want to make a revmap template file and copy it to the real file used each time this step is run (I zipped the repository after the first step so that I can repeat the second part as many times as needed until I get it right). It looks something like this:

del crmfullrevmap_step2.txt
  copy "crmfull revmap template step 2.txt" crmfullrevmap_step2.txt
  "c:\Program Files\TortoiseHg\hg.exe" convert Full Crm --filemap crmfullmap_step2.txt
  --branchmap crmfullbranchmap_step2.txt crmfullrevmap_step2.txt

Note that here I have a Mercurial copy of SVN in the folder named “Full”, and I import from it. The same thing could probably be done directly from SVN, only the REVMAP file would look different.

The “crmfullrevmap template step2.txt" file:

d379848121be332d162b1df014670558e2fa8dd4   be7726e1e2c98b3694b0c28ca5f058769a382018
3d753bea786bdbc1d25747c93e2554aa4134dd0c   be7726e1e2c98b3694b0c28ca5f058769a382018
d3c9339835df79c8d6cb10d3e9e589669cd993bd   be7726e1e2c98b3694b0c28ca5f058769a382018

Here be7726e1e2c98b3694b0c28ca5f058769a382018 is the hash ID of the revision at which the branch and trunk join. I got this number by going to the original SVN history to see what revision it was joined at, then went to my destination repository (imported at step 1) to find the equivalent Mercurial revision – luckily, the SVN revision numbers are kept even in svn-to-mercurial-to-mercurial conversion. Just to be sure, I put in all the lines from SHAMAP file where this revision appears in the right column.

The branch file crmfullbranchmap_step2.txt is a one-liner to move everything into the appropriate branch, called “client1”:

default client1

This process is then repeated for other branches.

Conclusion

The end result here is that we have working repositories in Mercurial, have been using them for a couple of months now (yes, this post is a bit old but hindsight is also worth something) and all seems right. I haven’t noticed losing any of SVN revisions (although I may have) and the imported Mercurial repositories behave just like any others – they even exhibit Mercurial’s flaws (like problems with unicode comments) the same way in the imported and newly created revisions. So, this procedure may be far from perfect but it did the job. If someone creates a better and more automated one, I’ll be sure to try it since I have a couple of low-key projects still left in SVN and awaiting migration.

Now that I think about it, there was one crucial reason to abandon SVN and upgrade to something more powerful: merging. My company now has a lot of parallel development (several customized instances of an application), and SVN’s support for this is not as good as it should be – that is, not as good as the competition’s. In SVN you can separate a development branch and keep it parallel to the main one, and merge-reintegrate it back. SVN will remember what revision it merged so that it doesn’t try to do it again (which would otherwise produce merge conflicts). But that seems to be the limit of its capabilities: a bit more complicated structure and merge produces so much conflicts that it looks the same as when everything is done manually. Contrast that to a more recent VCS like Git or Mercurial, where the essential feature is the ability to manipulate changesets, stack them one upon another or restructure their connections: if you can merge a set of changes from one branch to another, then continue developing both of them, do a back-merge (even though you shouldn’t), and the system doesn’t produce an enormous amount of conflicts, it gives you more power.

Of course, there are additional advantages that both Git and Mercurial give over SVN, but in our case that was just a bonus, not a necessity. I believe that many developers think likewise. SVN is good enough at what it does, the problem is when you need something that it wasn’t designed to do.

So which one to choose? It seems that Git, Mercurial and Bazaar are the most popular. People say that of the three Git is the most powerful because it was written by Linux kernel developers who do complicated merges every day. Ok, so I seemed natural to choose that one.

Git

The first impression that I got is that Linux kernel developers are guys who talk between themselves in hexadecimal and to the rest of the world in mnemonics. Git had some documentation but not nearly enough explanations on how things work. The error messages the tools produced were cryptic – and not only that, they were formatted to be viewed in the console so when they are displayed by GUI tools they tend to get messed up. Ok, so Linux kernel developers think GUIs are lame… While we’re at it, error messages are also misleading: I spent a lot of time trying to diagnose what’s wrong with my repository because the clone operation kept producing a warning about the remote head pointing to a non-existent revision or branch (and yeah, it was a fatal warning – a new term, at least for me – because the command didn’t do anything but gave only a warning). It turned out at the end that I was using a wrong URL for the repository – that is, I was trying to clone a nonexistent repository. I agree, in a nonexistent repository the nonexistent head points to a nonexistent revision, but it’s a bit of a roundabout way to say it, isn’t it?

But these are the things that can – and most probably will – be addressed as the tools mature. Since there are so much Git users, it’s surely reliable enough. I don’t mind some rough edges, I’ve been there with SVN and it was worth it (it was much better to cope with occasional bugs in SVN than to start with CVS and then migrate to SVN).

Ok, so how do we get Git to work on a Windows server? The answer that seemed complete enough hinted that I should simulate a Linux environment, install the shell, something like a SSH daemon and everything else. In other words, it’s not supposed to run on Windows but it can. Ok, I tried – there are several variations to the theme and each one had a glossed-over part that needs to be researched (something like “at this point you need to import the created certificate into Putty” – well, Putty doesn’t want to read this file format). And it didn’t help that Git itself doesn’t always tell you the real reason something doesn’t work, as I already mentioned. Moreover, it was for some reason allergic to empty repositories – unless I commit something locally, it won’t work. And the repository got easily screwed up while experimenting – that is, it got into a state where I didn’t know what to do with it and it was easier to create a new one.

At this point it was clear that the client tooling also leaves a lot to be desired – there was TortoiseGit that locked occasionaly on clone/pull (and it never displayed the progress of the operation so you never really knew if it was doing something), there was Git GUI that was a bit more stable, and there was Git Bash that was the most reliable. (One interesting sidenote is that at one point I managed to get three different error messages for doing the same – clone – operation with these three tools). One thing, though, the Bash in Git Bash is probably the best part of the package, I had almost forgotten the comfort of working in a Unix shell. Command prompt is light years behind it.

I did get the server to work, though, after a week of running around in circles and repeating the same stuff with slight variations. At the end I was able to create a new repository, clone it, make changes, commit, push, pull… Everything worked until I restarted the server. Or so it seems – when I tried to clone the same repository afterwards, it started again producing weird error messages. I didn’t know what else changed except for the restart (which shouldn’t have affected it – and probably didn’t, but what else? It’s an isolated environment). If I didn’t have the cloned repository with revision history I would have doubted I actually succeeded in doing it… Ok, so it’s also fragile.

Then I tried to find an alternative: there’s a couple of PHP scripts that wrap Git server functionality, but they don’t seem to work on Windows (I tried them in Apache). There’s Bonobo Git server that is written in .Net – well, I never looked at it seriously, how’s a .Net Git server going to work when their own true-to-the-original-idea SSH configuration doesn’t? But it does work – it also needs a bit of tinkering (you have to Google-translate a page in Chinese to get the info on how to really do it, WebDAV etc.) but the installation is amazingly painless: it takes a couple of hours, which is nothing compared to a week wasted for the previous experiment.

So, on to the next step: migration. I migrated a test folder from SVN with no trouble. Tried a bigger one, something under a 1000 revisions – well, it finished in a couple of days, I suppose it’s tolerable. Finally, tried to convert the central project – and when after two weeks of non-stop import it managed only 2000 of the repository’s 5000 revisions, I gave up. Back to Google: why is it so slow, how to speed it up? Turns out that the Git client is developed on Windows XP and that it should probably work well there. As it did: it managed to get all 5000 revisions in a couple of hours. Ok, now this I didn’t like. How can a serious tool not work on newer Windows versions? They said, it’s slow because of the UAC (introduced on Windows Vista), the UAC slows everything down. Well, it’s not like Vista was released yesterday. If this problem exists for years, should I expect a solution ever to appear? More research hinted that Linux kernel programmers think Windows users are lame. So – Git was slow on Windows newer than XP. TortoiseGit seems to execute the same Git shell commands, so it’s the same. I found Git Extensions in the meantime, which is supposed to be independent – but it didn’t even handle HTTP repositories.

In the meantime, I tried cloning the big repository I converted – big as in 200 megabytes big – and it was, as expected, slow. But, I don’t really know which one was to blame here – seems like Bonobo server choked on the large repository since it nailed the CPU at 100% and produced around 40 bytes per second to the client (possibly it was just some kind of sync messages and no data at all). Ok – Bonobo is open source and was built around GitSharp (or something like that) Git library written in .Net. What if I tried myself to update the library and compile the server? Well – GitSharp is discontinued at version 0.3. They’ve all gone to some new library written in C.

Ok, that was enough. After three weeks completely wasted, I gave up on Git.

(Update: Bonobo was since upgraded to version 1.1. I looked at the change log hoping to see a note about moving away from GitSharp, but it didn’t seem to happen. So as far as I know, this performance issue may still be present – nevertheless Bonobo seems the most promising solution for a Windows Git server).

Mercurial

So? Should I look at Mercurial? The Mercurial supporters seem a bit shy – that is, compared to Git fanatics who shout GIT! GIT! GIT! GIT! at each possible occasion, there occasionally appears one that says “or, you could try Mercurial”.

Well, the first look revealed the difference: I can download everything I need from a single web page. Mercurial basic installation – right there, server support included (as a script that is installed in a web server). TortoiseHg, right below it – even it has an ad-hoc server built in! Python extensions – do I need this? So I thought – no, this is too easy. Let’s try something unheard of in Git – RhodeCode. It’s a server implementation that is in fact a full-featured web application. Seems very nice, but due to some incompatibility in a crypto library, very hard to get installed on Windows: it took a lot of workarounds, I ended up installing Visual C++ Express 2008 (it has to be 2008, 2010 doesn’t cut it) and another kind of simulated shell environment (MingW32) to try to get the installer to compile it from source but it was impossible. That is: impossible on Windows 2008 R2, 64-bit. The RhodeCode developers say they’re working on making it more compatible with Windows (and for one thing changing the crypto library), and I found that I believe them, so I’ll be coming back to it. (In the meantime, they’ve released a couple of versions with windows-specific bugfixes, it might be worth it to check it out again).

In the vanilla Mercurial instalation there’s a script called hgweb.cgi that contains the full functionality needed to get a server running. A bit of tinkering is needed to make it run inside IIS – and there are a couple of slightly outdated tutorials on how to do this. I found out that the best combination is to download Mercurial for Python – so, no Mercurial or TortoiseHG on the server. This download says in its name for which version of Python it was written, and that version of Python is the second thing needed. Once both are installed, it is sufficient to put the hgweb.cgi in a virtual server in the IIS, add a hgweb.config and a web.config file, configure the IIS (basic authentication and what have ya), and set permissions on the folders – including the repositories. It took less than one day, research included, to get it up and running.

The client tooling seems better than SVN. TortoiseSVN (in fact, TortoiseCVS) was a breakthrough idea – a VCS integrated into windows, allowing you to version-control any folder on your disk. Well, TortoiseHG went one step further and actually improved the user experience. It has its bugs – more than SVN – but also has a lot more features. The whole thing was written in Python and seems to have a good API because a lot of plugins have been written for it, and TortoiseHG includes the most important ones. At this point I had to install TortoiseHG on the server because that’s the only way to get the subversion converter plug-in. The other way would be to install the Python module, but it cannot be done: first of all, they’ll tell you to install Subversion for Python (which is quite simple, there’s a pre-packaged downloadable setup), but when you do and get an error from the convert extension, you’ll find out that you don’t need that package but something called SWiG. But SWiG doesn’t have anything to do with Subversion – you have to download the Subversion package from Subversion site which moved to Apache leaving the Python part behind and the best you can do is find a source somewhere and compile it, but nobody says how it’s done.

On to converting the repositories – for one thing, it’s speed is normal, on any Windows. As fast as Git on XP, maybe even faster. So I was encouraged to do the thing I never even got around to thinking about with Git – and that is splitting the repositories into individual projects. It did take a week – the details of it will be a subject of a future post – but in the end it produced much less pain then Git.

Conclusion

Looking at it now, I think that Mercurial is unfairly underrated, and this seems to be due to the loud chanting of Git proponents. They say Git is blazingly fast – well, if you calculate the average performance on all operating systems, I think that on the average it’s either very slow or Windows users don’t use it at all. On Windows XP, Mercurial is at least as fast as Git, and on newer Windows versions Git is usable only for small repositories. Git is largely badly documented (which is getting better but Mercurial is way ahead – suffice it to say that I understood some Git concepts only when I read the Mercurial documentation). Git tooling, generally speaking, sucks – it was designed to be used in a Unix shell and nowhere else – and on Windows it is total crap. On the other hand, Mercurial tooling on Windows is, after this experience with Git, impressive. My conclusion is that for Git I would have to require special skills when employing programmers – “C# and Bash knowledge required”, how sane is that? Ok, I’m joking but it’s not far from truth: there has to be at least one Git specialist on the team when ordinary developers get stuck. With Mercurial, the usual SVN-like skill level should be enough for all because it’s not that easy to get stuck. And, after all this, I’m inclined to think that the story about Git being so great should be taken with a grain of salt since everything I’ve seen so far from it seems to tell exactly the opposite.

I tried searching for a solution on how to modify the import process and found nothing. Tried to add a new comment to the repository with a non-ASCII character and got a Python error (“expected string, QString found”). Some said that I should change my Windows’ default system encoding (which is English(US)), and that solved the problem but I would have liked a simpler solution, since changing the default encoding used to cause other problems in the past. I managed to find a couple of workarounds that solve the problem of console display and involve setting environment variables… Would it work for Tortoise? Actually: it does. The solution is simple: go to (this is on Windows 7) Control Panel – System – Advanced System Settings – Environment Variables, add a new variable called HGENCODING and set it’s value to either “utf-8” or your code page (mine is “cp1250”). TortoiseHg respects this. There’s a slight difference in the two values, though, because the diff viewer doesn’t really like “utf-8”, it prefers the concrete code page. There may be other components that behave like this, so I suppose that setting the code page is the optimal solution.

The problem is this: I have a script that backs up my database server. It creates one folder for each database and puts a new file into it every day. I want to rar each file (with a password), delete the original and delete all files in the folder except the two newest.

Here’s the script – you may recognize parts of it from other scripts, but unfortunately I can’t remember the URL’s where I picked these pieces up, sorry… Google around for solutions to this problem and you’ll probably find them.

#Powershell Script to recurse input path looking for .bak files, move them into a rar archive
# and delete all archives in each folder except the newest two. 

$InputPath = $args[0] 

if($InputPath.Length -lt 2)
{
    Write-Host "Please supply a path name as your first argument" -foregroundcolor Red
    return
}
if(-not (Test-Path $InputPath))
{
    Write-Host "Path does not appear to be valid" -foregroundcolor Red
    return
} 

$BakFiles = Get-ChildItem $InputPath -Include *.bak -recurse
Foreach ($Bak in $BakFiles)
{
  $ZipFile = $Bak.FullName -replace ".bak", ".rar"
  if (Test-Path $ZipFile)
  {
      Write-Host "$ZipFile exists already, aborted." -foregroundcolor Red
  }
  else
  {
    & "C:\Program Files\WinRAR\winrar.exe" m -m1 -pyourpasswordhere "$ZipFile" "$Bak" | Out-Null 

    if(Test-Path $ZipFile)
    {
      # Keep two newest files in the directory based on creation time
      $path = split-path $ZipFile -Parent
      $total= (ls $path).count - 2 # Change number 2 to whatever number of files you want to keep
      $path = $path + "\*.rar";
      ls $path |sort-object -Property {$_.CreationTime} | Select-Object -first $total | Remove-Item -force
    }
  }
}

But it could be interesting to generalize the “add-on” controller pattern and make it work in other situations as well. One interesting area of application could be controlling data: if you use an Object-relational manager (ORM) to read your data (and if you don’t, you should), you already have your data in the perfect form for this – that is, as objects. Likewise if you use a traditional data-access layer and then create business entities as object-oriented structures. Usually, these classes contain much of the business logic inside and this is quite natural to do because business logic represents the behaviour of the data. But it’s not good for reusability: whether you combine the data access and business entities or implement them separately, the business logic is tightly coupled with data/hardcoded inside data so it’s not easy to customize. It would be nice if we could externalize at least some parts of it, but keep it as close to data as possible. Even better would be if we could split the logic and implement different aspects as different components.

If we attached controller objects to our data, they could take the responsibility of providing its behaviour. We could have different controller types for different types of behaviour (status changes, automatic calculations etc.), and if we instantiate them using some kind of configuration engine (or dependency injection framework), they will be easy to replace and thus make your application highly customizable.

Of course, these “data controllers” sound similar to what rules engines do, only more lightweight. But how much of an overhead would they add? As always, it depends on what we do, but I would say not too much. They would contain the code we already have, with probably one layer of isolation because it would be split into distinct classes. Would the instantiated rules waste memory? Probably some: but when you consider that the ORM instantiates each record as an object and has a complex infrastructure for tracking its changes; that for each data binding there is a couple of objects created in the interface; that there can possibly be a control (which is quite a large object) bound to each field… It seems that it shouldn’t make a big difference.

The most sensitive bit is collections: there we can have hundreds of records with one control (a data grid) bound to them. In this case, the overhead of the rules engine could be significant compared to the overhead of the interface – but not in comparison to ORM overhead. Again, this depends on how we implement the rules: the rule could be optimized to utilize a single instance for the whole collection – this can be done using an IBindingList implementation where the collection transmits events for everything happening inside it… Of course, here it is the collection which incurs the overhead, but let’s say we would use it anyway for data binding purposes.

Obviously, there’s a lot of technical issues here. This is partly because the base framework doesn’t have built-in support for what we need. But, it is to be expected that having an ORM (which can be interrogated for data structures and from which we can receive notifications when data is loaded/saved) generally helps.

Of course, I’m still talking hypothetically here: but I’ve already made a couple of small steps in the direction of having a living prototype (in fact, I’m testing an minimal working something as we – uhm, speak). More on that in another post.

This is the first in what we hope to be a series of posts on various subjects related to IT, e-business and application development. We will concentrate on a broad range of topics, some of them highly technical, but for the most part we will try to provide information that is otherwise hard to find, at least in a readable and concise form.