Merging Vendor Drops with Custom Branches

Merging Vendor Drops with Custom Branches

During a refactor of our cms, it became apparrent that manual merging of Silva releases with our custom modifications was too painful a process.  This document is an attempt at summarizing the process of creating vendor drops in our subversion repository and merging against those.

From the "Vendor Branches" section of "Version Control with Subversion":

A vendor branch is a directory tree in your own version control system that contains information provided by a third-party entity, or vendor. Each version of the vendor's data that you decide to absorb into your project is called a vendor drop.

Vendor branches provide two key benefits. First, by storing the currently supported vendor drop in your own version control system, the members of your project never need to question whether they have the right version of the vendor's data. They simply receive that correct version as part of their regular working copy updates. Secondly, because the data lives in your own Subversion repository, you can store your custom changes to it in-place—you have no more need of an automated (or worse, manual) method for swapping in your customizations.

First, I will outline the steps to initially create the vendor drop.  The steps after should be used whenever merging to a new vendor release.

Creating Vendor Drop

Start with the release you're basing your changes on (e.g. Silva 1.3.0). 

1. Create a directory having the name of the package, e.g. Silva.
2. Unpackage Silva 1.3.0 in Silva.  Rename the directory to current.
3. Change directories so you're outside of Silva.

4. Run the command:

   svn import Silva http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva"
   

   This imports the files and folders of Silva into the repository.  It does not create a working copy!

5. You will also want to tag the current directory as a release:

   svn copy http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/current \
            http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/1.3.0 \
            -m "silva vendor drop 1.3.0 release"

6. Create a working copy of this vendor drop by issuing the following command.  N.B.: you first need to remove the Silva hierarchy, or change directories.

   svn co http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/current Silva

You will use this working copy to merge the changes with the new release of Silva.

Creating a local branch vendor drop

Before you can start making changes to this vendor drop, you need to create a branch of it for your own project.  On the filesystem, create a hierarchy for your Silva branch.  The standard subversion project layout is this:

Project/
  trunk/
  tags/
  branches/

Trunk is where main development occurs.  Releases of your code go in tags.  Sometimes you need to create a branch to develop some feature or make major changes without affecting main development.  These branches go in the branches directory. A branch can be merged back into the trunk when finished.

For our Silva local branch, create the following directory structure:

Silva/
  tags/
  branches/

Then, in the parent folder of Silva, run an svn import command:

svn import Silva http://svn-web.its.bethel.edu/Zope-Projects/Silva -m "initial Silva local branch import"
svn copy http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/1.3.0 \
         http://svn-web.its.bethel.edu/Zope-Projects/Silva/trunk

Notice trunk was created via svn copy, so it contains the 1.3.0 release of the Silva vendorr drop.  You can now do your Silva development in a working copy of trunk.

Merging a new vendor release with your vendor drop

After some time, hopefully the vendor will release a new version of your project.  On the grand scale, this is a two step process.  The first step is to upgrade the vendor drop, and then do a three way merge with the vendor drop and the local branch.

Upgrading the vendor drop

This process outlines the general procedure I follow to upgrade a vendor drop.  There are certainly other ways to do so.

1. Checkout a working copy of the current directory of the vendor drop, and name it Project-svn (e.g. Silva-svn):

   svn checkout http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/current Silva-svn

2. Unpackage the new release in the same directory.

3. Get a list of the files in each release.  For the working copy, you'll need to omit any paths containing '\.svn':

   [zope@cirdan Silva-svn]$ du -a .|awk --source '{print $2}' > ../1.3.0.diff
   [zope@cirdan Silva-svn]$ cd ../Silva
   [zope@cirdan Silva]$ du -a .|awk --source '{print $2}' > ../1.3.1.diff
   [zope@cirdan newsilva]$ egrep -v '\.svn' 1.3.0.diff > 1.3.0.diff.nosvn

4. Remove (or rename) the new release directory.
5. Rename the working copy to it's original name (e.g. 'mv Silva-svn Silva')

6. Unpackage (tar -xvf, probably) the new release

   over

   the working copy.  If the working copy has the same name (e.g. Silva), the unpackaging should automatically do this.

   
   

7. Compare the two file lists, looking for files that are in the old release but not in the new.  You can use the unix 'diff' command, or your favorite diff program, like emacs.
   For files that are in the old release but not the new, delete them from the working copy.  For example, if views/edit/macro_nosilva.pt was not in the new release:
   

   [zope@cirdan Silva]$ svn remove views/edit/macro_no_silva.pt

8. When finished, doing svn status should reveal only files flagged with M or A (modified or added).  If there are any marked with C (there shouldn't be), you'll need to resolve those conflicts manually.

9. Finish up the project and tie up the loose ends:
   

   [zope@cirdan Silva]$ svn commit -m "Silva 1.3.1 vendor drop upgrade"
   [zope@cirdan Silva]$ svn copy http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/current \
                                 http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/1.3.1
                            -m "Silva vendor drop 1.3.1 release"

Merging vendor drop and local branch

Now for the tricky part.  Subversion will be able to merge most of the changes automatically, but some files that have been modified to heavily will fail.  These will be flagged with a C when running svn status.  But first we need to do the merge.

1. In the working copy of your local branch, make sure you've committed all changes.  Then run the following command:

   
   

   svn merge http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/1.3.0
             http://svn-web.its.bethel.edu/Zope-Projects/vendor/Silva/current
             .
   

   NB: It is important that merges are always done against current and not latest tagged release!

2. Merge output will inform you of all changes made (via flags

   A, M, C

   ).  For all files marked with

   C

   there was a conflict during the merge.  You'll need to manually resolve these conflicts.  See the SVN Book for an explanation of this process.

3. When all merging is finished (and you can also test your code), commit the changes:

   
   

   svn commit -m "Silva local branch merge to 1.3.1"