Removing the mime-type of files in Subversion with SvnQuery

If you add files to subversion they are associated with a mimetype. SvnQuery will only index text files, that means files without an svn:mime-type property or where the property is set to something like “text/*”. At work I wondered why I couldn’t find some words that I know must exist. It turned out that subversion marks files stored as UTF-8 files with BOM as binary, using svn:mime-type application/octet-stream. This forces the indexer to ignore the content of the file.

I used SvnQuery to find all files that are marked as binary, e.g. t:app* .js finds all javascript files and t:app* .cs finds all C# files. With the download button at the bottom of the results page I downloaded a text files with the results. Because svn propdel svn:mime-type [PATH] can work only on one file (it has no --targets option) I had to modify the text file to create a small batch script like this:

svn propdel svn:mime-type c:\workspaces\javascript\file1.js
svn propdel svn:mime-type c:\workspaces\javascript\file1.js
svn propdel svn:mime-type c:\workspaces\javascript\file1.js
…

After this change indexing worked again. I now run a daily query that ensures that no sources files are marked as binary.

What means “Skipped” in the subversion merge log?

I have seen many people who tend to ignore the “Skipped” messages that are sometimes displayed when you do a merge with subversion:

 

The “Skipped” message means that the merge operations wants to create or modify a file but that file already exists in your working copy and is not under version control (added to the repository:This can easily happen if you do a forward merge from trunk to a brunch and revert your local changes to the working copy (maybe because some manual merges went wrong). The revert command will leave new files as orphaned files in the working copy.

Don’t ignore those warnings! If you commit a working copy with “Skipped” warnings and do a backward (reintegrate) merge from branch to trunk, subversion will delete the formerly skipped files from the trunk! As a rule, you should never ignore “Skipped” warnings and if they occur fix the root problem then repeat the merge on a clean working copy.

Always “Compare whitespaces” when merging

I am working in an environment where we develop features in branches, using Subversion and TortoiseSVN. Every day we do a forward merge from the trunk to branch. When the feature is complete the branch is merged back to the trunk and the branch gets deleted.  Recently I noticed a lot of whitespace issues, like wrong indentations. It turned out that the reason for this is one commonly misunderstood option in the merge dialog of TortoiseSVN: “Compare whitespaces”

 

The default settings of this dialog are the correct ones but many developers automatically click on one of the "Ignore whitespace" options. Why? Because they are using this option in their diff tools all the time. But what is good when comparing is actually a very bad thing when merging. Imagine that someone surrounds some existing code with an “if” or “try-catch” block. That would indent the existing code which is just a whitespace change. The new indentation will not be merged to the branch if you choose one of the ignore options! The same is true if you changed tabs to spaces or inserted some empty lines. And now comes the worst problem: You chose to ignore whitespaces on forward branches. But you didn't ignore them when merging back to the trunk. Now you actually have erased all whitespace changes from the trunk between the time you created the branch and the time you merged it back!

<Insert picture here>

So remember one thing: Always choose "Compare whitespaces" in the merge options!

Speed up your Subversion repository

Normally, I am satisfied with the performance of Subversion. I used to work on a 20GB repository with ca 20000 revisions, 20 developers in a not so fast local network. Daily business operations like update, commit were done in a few seconds.

Now I work with 30 developers on 4GB repository with ca 18000 revision in a super fast network. And the performance sucks, compared to my former experiences. A checkout of a 120MB working copy takes more than 10 minutes. Updating (without any changes) takes at least 40 seconds.

So what’s the problem here? I have some guesses but are not yet sure about the real reasons. But here are my suggestions:

• Use the FSFS mode (I hope nobody is using Berkeley DB anymore?)
  
• Use Sharding. This affects the way how the repository is stored on the server. At most 1000 files are put in one directory then another directory will be created. This helps a lot if the underlying file system performance worse with an increasing number of files per directory.
  
• Pack your repository regularly. Normally, each revision is stored as a single file. If you have a sharded repository then svnadmin pack will convert all full shards into one big files. This saves space and helps the OS to reduce I/O and improve caching.
  
• Use the svn:// protocol. The http and https protocols are just a tunnel for WebDAV delta-v which is a quite chatty protocol. For each file you need a full roundtrip from the client. On high latency networks this could be a real bottleneck. The svn protocol is much faster and consumes less cpu cycles. On my test machine a complete checkout of 120MB working copy took on average 5min 20s over https:// and only 4min over svn://.
  
• Check your commit hooks. Perhaps you have installed some expensive commit hooks. On Windows, try using RunDetached to prevent subversion from waiting for the hook to finish.
• Beware of Polling Build Agents. If you are Continuous Integration there will be some kind of mechanism in place to detect changes in the repositories. The most efficient one is a post commit hook, but for example CruiseControl.NET and TeamCity use an inefficient polling approach that is basically doing an “svn log” and parsing the output. Doing this over https every second from forty build agents can easily bring up the load on the repository server to 100%. A more efficient polling mechanism would be to store the latest revision and query only the newest revision of the repository. This is only implemented by some custom plugins.
  
• Monitor I/O load. Still, when people think of performance they think of CPU performance. But for the subversion repository I/O is the limiting resource. An update or checkout operation will do many small reads on different areas of the repository. Therefore the average access time is the most important factor. If your repository is running on a virtual machine make sure that the repository is located on a physical drive that is exclusively reserved for this purpose. Use a SSD or at least a 10,000 RPM drive.
  
• Store your repository on a  Intel Solid State Disk. The disk should be exclusively reserved for use by the repository. No other application should touch it. This is the simplest and most effective way to improve performance.
  
• Optimize your working copy. Change the layout so you can do partial updates. Try to use svn update –-svn-depth -exclude to exclude parts you don’t need in your day-to-day work. Remove files you don’t need. 

Note: All tips are written at the time of Subversion 1.6 and increase the server performance. Subversion 1.7 will improve the local working copy which theoretically should also increase the client performance.