Migrating a SVN repo to Git, part deux: SubGit to the rescue

  1. Migrating a SVN repo to Git: a tale of hacking my way through
  2. ➤ Migrating a SVN repo to Git, part deux: SubGit to the rescue

To improve is to change; to be perfect is to change often.
— Winston Churchill

In my previous post about SVN→GIT conversion I’ve described steps to convert a nested SVN repo to GIT using svn2git, svndumpfilterIN, SVN::DumpReloc and some manual editing of SVN dump files.

This process worked fine for smaller repos, but after some threshold I’ve hit the wall: final conversion with svn2git for one of the larger repos was taking 5 days and was never quite finished because of Windows version of Git crashing in the middle of process. Those crashes were related to Cygwin’s implementation of fork which requires some address space to be reserved for Cygwin heap and 5 days long run was exhausting those addresses.

After a couple of attempts to convert a repo (which took about 2 weeks!), I’ve realized that I need a more robust and preferably faster solution. And that’s when I finally found SubGit!

SubGit is a tool for a smooth, stress-free SVN to Git migration. Create writable Git mirror of a local or remote Subversion repository and use both Subversion and Git as long as you like. You may also do a fast one-time import from Subversion to Git or use SubGit within Atlassian Bitbucket Server.

SubGit is a commercial closed-source Java application. Fortunately, it’s free for one-time conversions and mirroring for repos with up to 10 Git and SVN users. It also has time-trial version that will mirror repo with any amount of users for one month. If you’re daring enough, you can also use EAP or interim builds. Note that it seems that interim builds don’t have any time/user limits whatsoever.

With SubGit, I was able to convert abovementioned SVN repo to Git overnight without any extra steps, using this simple command:

subgit import --svn-url http://server/svn/my/nested/repo --authors-file .\authors.txt .\repo.git


SubGit also can filter/map SVN branches to Git ones which saves you from filtering SVN repo when you need to sync/migrate only parts of SVN repository. To do so, however, you need to install SubGit into the target SVN repository and configure it by modifying subgit.conf file.

Since SubGit requires JRE and I’m reluctant to install it system-wide, I’ve created a set of scripts to use Subgit with portable JRE. You can find them in this GitHub repository: SubGit-Portable.

Here is a step-by-step walkthrough of installing and configuring portable SubGit with JRE. Please note that SubGit can work in two modes: local and remote. This guide assumes local mode, when SVN sever, SubGit and Git (repo and executables) are on the same server.

Create portable SubGit installation
  • Clone or download and unpack SubGit-Portable repository
  • Get latest SubGit release and unpack it to the SubGit directory inside the repo
  • Get latest Oracle JRE/Server JRE and put it to the JRE_Installer directory inside the repo. You can use either exe installer or tar.gz archive.
  • Double-click unpack_jre.cmd in JRE_Installer folder. It will unpack all exe installers/tar.gz archives to JRE_Portable folder.
  • Edit subgit.cmd to set the name of the folder with portable JRE to be used. E.g: set jre_dir=jre-8u40-windows-x64

Now you can use subgit.cmd as if you were using original subgit.bat:

C:\Path\To\SubGit-Portable> subgit --version
SubGit version 3.1.1 ('Bobique') build #3448
  (c) TMate Software 2012-2016 (http://subgit.com/)

Install SubGit into the SVN repo
  • Run this command to install SubGit binaries into the target SVN repo: subgit configure X:\Path\To\SVN\Repo. You should see something like this:
C:\Path\To\SubGit-Portable> subgit configure X:\Path\To\SVN\Repo
SubGit version 3.1.1 ('Bobique') build #3448

Detecting paths eligible for translation... done.
Subversion to Git mapping has been configured:
     <root> : X:\Path\To\SVN\Repo\.git

CONFIGURATION SUCCESSFUL

To complete SubGit installation do the following:

1) adjust SubGit configuration file, if necessary:
    X:\Path\To\SVN\Repo\conf\subgit.conf
2) add custom authors mapping to the authors.txt file(s) at:
    X:\Path\To\SVN\Repo\authors.txt
3) run SubGit 'install' command:
    subgit install "X:\Path\To\SVN\Repo\" 

  • Create authors.txt that to map SVN commiters to to Git authors. You can use my New-GitSvnAuthorsFile script for this. See my previous post for details.

  • Setup a mapping of projects in SVN repo to corresponding Git repositories. In my case, SVN repo held multiple projects and I needed to create Git repositories only for some of them. SubGit can detect nested repositories and by default will add all of them to the configuration file. So I had to edit conf\subgit.conf in SVN repo folder to remove unneeded ones. Here is example of configuration section for a single nested project:

[git "my/nested/repo"]

    # Path within Subversion repository to the root of trunk/branches/tags structure.
    translationRoot = my/nested/repo

    # Path to the Git repository.
    repository = X:/Git/repo
    pathEncoding = UTF-8

    # Options below (trunk, branches, tags, shelves) define correspondece between Subversion
    # directories and Git references. Depending on the actual Subversion project layout and whether
    # all or only some of the branches have to be mirrored, these options might need to be adjusted.

    trunk = trunk:refs/heads/master
    branches = branches/*:refs/heads/*
    shelves = shelves/*:refs/shelves/*
    tags = tags/*:refs/tags/*

  • Start SVN→Git translation process: subgit install X:\Path\To\SVN\Repo

C:\Path\To\SubGit-Portable> subgit install X:\Path\To\SVN\Repo
SubGit version 3.1.1 ('Bobique') build #3448

Subversion to Git mapping has been found:
     /my/nested/repo: X:\Git\repo

Translating Subversion revisions to Git commits...

    Subversion revisions translated: 54321.
    Total time: 6543 seconds.

INSTALLATION SUCCESSFUL

If something goes wrong, files named subgit-COMMAND-DATE-TIME.zip in the Subgit-Portable folder and X:\Path\To\SVN\Repo\subgit\logs\ contain detailed operations log and error messages.

Basically that’s all you have to do to get SubGit up and running!

Updating SubGit

SubGit is actively developed and from time to time you’d need to update your existing SubGit installation to the latest build.

  • Get latest SubGit build
  • Delete all files in Subgit-Portable\SubGit folder and unpack new build here
  • Reinstall SubGit to the target SVN repo: subgit install X:\Path\To\SVN\Repo

For example, this is how upgrading from EAP build looks like:

SubGit version 3.0.0-EAP ('Bobique') build #3262
This is an EAP build, which you may not like to use in production environment.

Subversion to Git mapping has been found:
    /my/nested/repo: X:\Git\repo

About to shut down background translation process.
Background translation process is not running.

SubGit binaries have been upgraded (3.0.0-EAP#3141 > 3.1.1#3448).
Information on previously encountered errors is cleared.

Processing '/my/nested/repo'
    Translating Subversion revisions to Git commits...

    Subversion revisions translated: 54321.
    Total time: 6543 seconds.

INSTALLATION SUCCESSFUL

I’ve been using SubGit with custom hooks to enable one-way Git mirroring to Visual Studio Online for a quite a long time and it works perfectly. So if you ever find yourself in need of painless SVN→GIt translation, give it a try!

Advertisements

One thought on “Migrating a SVN repo to Git, part deux: SubGit to the rescue

  1. Pingback: Migrating a SVN repo to Git: a tale of hacking my way through | IT magician with a knack for automation

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s