Kevin's Weblog 2006-11-25

Subversion Migration — Binary File Difficulties

2006-11-25 14:10 in /tech
By now, I’ve switched most of my version control working copies over to subversion. The one major exception has been my websites. The obstacle was that when I compared a new subversion working copy to an old CVS working copy, I found that all my binary files had been corrupted.

The explanation seems to be that while SVN does have a binary file detection heuristic, somehow cvs2svn bypasses it. Supposedly if you have ‘-kb’ set on your binary files it does the right thing, but I’m usually too lazy to bother. The reason this causes problems in SVN is that in addition to keyword expansion, subversion also does end-of-line translation, which CVS doesn’t. Clearly, this will cause badness if applied to binary files.

It seems like I might have been able to avoid this with appropriate config file voodoo for cvs2svn, but it’s not really documented. Alternatively, had I known, I could have set the binary flag in CVS before I did the conversion. Alas, at this point, short of running the conversion again from scratch, I was stuck fixing things up somewhat by hand. In essense, this is just a matter of running something like:
```
for f (**/*.jpg) ; do
  svn propdel svn:keywords $f
  svn propdel svn:eol-style $f
  svn propset svn:mime-type image/jpeg $f
  cp ~/old_cvs/$f $f
done 
```
but, in practice, I had a bunch of other media types that also needed to be dealt with, and also I had to deal with some pain related to case-insensitivity on my old (Mac) server vs case-sensitivity on the new (Linux) server. Overall, not a fun couple of hours. But, now it’s done, and I can declare the migration complete.
permalink · 3 comments

Comments

Ben Collins-Sussman wrote:

Actually, you've got it backwards. CVS automatically does end-of-line translation on *all* files unless you *explictly* mark them with the -kb switch. In other words, CVS's default behavior is to corrupt files, unless you happen to remember to tell CVS that something is a binary file. (Subversion is the opposite; it never ever changes files contents unless you explicitly mark a file as tweakable.)
So the file corruption you're seeing was actually in CVS to begin with; try checking out a working copy from CVS and see if your binray files aren't end-of-line-munged already. :-)
cvs2svn simply assumes that you've been a good CVS admin, and have already marked binary files as -kb. It tries to ease the transition pain by marking text files (those without -kb) as 'ok to tweak'.

(2006-11-29 07:16)

Michael Haggerty wrote:

Ben, you are being too quick to dismiss these problems.
I believe that your suggested experiment has a different result than you expect: as long as you check the binary file out under Unix, and as long as it doesn't contain a string that looks like a keyword, the binary file will *not* be munged even if it was checked in as text.
Of course, a properly-maintained CVS repository should have -kb set on binary files. But under Unix, unless the file happens to have a keyword string it it, the user won't have any obvious problems if he forgets -kb. And if problems arise because a keyword string is in the file or because somebody wants to check out the file under Windows, he can set the -kb option on a file *retroactively*. Since CVS stores the original text of the file, I believe that the retroactive fix clears things up.
SVN, on the other hand, does not allow a file to be committed in text mode if it has mixed line-end conventions. Therefore, if cvs2svn decides to treat a file as text, it is *forced* to munge the file contents before storing it into the repository. This munging is *irreversible*, even if the file is later changed to binary mode. In this sense, SVN + cvs2svn are much less forgiving than CVS to sloppy administration of the text/binary distinction.
Several people have run into this problem, therefore, we will probably change cvs2svn's default behavior to convert files as binary even if CVS didn't include the -kb option. This default will be able to be overridden so that non-sloppy CVS administrators don't lose out.

(2006-12-11 03:52)

Kevin wrote:

Thanks for your comments, Ben and Michael. I'm quite certain that CVS is not doing end-of-line translation on my images, since I commit them on a Mac and serve them from Linux and all my photos have looked just fine. However, the versions in a SVN checkout (from my cvs2svn import) were clearly corrupted. New images added to the subversion repository don't have any problems, because the binary file detection heuristic is applied successfully in that situation.
At any rate, I'm not trying to avoid any blame on myself for this issue, but I do think it would be nice if cvs2svn at least mentioned this in the documentation.

(2006-12-11 09:01)

Leave a comment

Please use plain text only. No HTML tags are allowed.

Comments are closed for this story.

Trackbacks are closed for this story.

About

Reading

Current Reading

Recently Completed

The Deadline, by Tom Demarco
Crypto, by Steven Levy
Emotional Design, by Donald Norman
The Hallowed Hunt, by Lois McMaster Bujold
Time-Out For Parents, by Cheri Huber

In Queue

Unconditional Parenting, by Alfie Kohn
Bhagavad-Gita, by Swami Prabhavananda (translator)
Introduction to Tantra: The Transformation of Desire, by Lama Yeshe
Visual Explanations, by Edward R. Tufte
A People’s History of the United States, by Howard Zinn
The Visual Display of Quantitative Information, by Edward R. Tufte
The Puzzle Palace, by James Bamford
Body of Secrets, by James Bamford
A Pattern Language, by Christopher Alexander
The Language Instinct, by Steven Pinker
Words and Rules: The Ingredients of Language, by Steven Pinker

Kevin's Weblog

2006-11-25

Subversion Migration — Binary File Difficulties

Comments

Leave a comment

About

Reading

Current Reading

Recently Completed

In Queue

Blogroll