Using rsync to obtain the e-smith CD ROM image

We highly recommend using rsync as the method of choice for obtaining the e-smith CD ROM image. We have found it to be the most efficient and least bandwidth-intensive method for downloading any large file(s). In a nutshell, rsync only downloads the differences between a file at the source location and a file of the same name at the destination. When dealing with a 300+ MB file like the e-smith CD ROM image, this can result in download savings upwards of 80%.

This document provides links to information about rsync, and instructions on how to use it to obtain the e-smith server and gateway software.

Who should use rsync?

In the broadest terms, anyone using with an Internet-connected Linux machine should seriously consider using rsync. Even if bandwidth savings don't mean anything to you individually, the bandwidth savings that can be achieved using rsync benefit everyone connecting to the download resource.

What is rsync?

First, rsync is not by nature Windows software. While there are documents online that explain how to build rsync on Windows 95 and Windows NT, only advanced users who are comfortable compiling their own software should even try.

rsync does run on Linux and many Linux-like operating systems *grin*. You can find binary versions for 10 or so different operating systems and Linux distributions at http://rsync.samba.org/ftp/rsync/binaries/.

From the rsync home page:
"rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License."

The following is taken from the rsync man page:

rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speedup file transfers when the destination file already exists.

The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

Some of the additional features of rsync are:

How do I learn about rsync?

The first place to start learning about rsync is of course the man(ual) page. Simply enter man rsync at your linux command prompt.

You can look at some of the examples at http://rsync.samba.org/rsync/examples.html.

There's a very good tutorial on how to use rsync located at http://www.eunuchs.org/linux/rsync/. [NOTE: This site seems to be offline right now.]

Finally, for the more technically inclined, the rsync resource pages contain a number of other useful links.

How do I use rsync to obtain the e-smith CD ROM image?

Follow these steps to download the e-smith CD ROM image:

  1. If you have an older copy of the e-smith 4.1 ISO image (for example a beta or release candidate) you can use it to prime the destination file. Copy the old CD ROM image to a new name. For example, if you have an image of e-smith 4.1 beta 2, you would type the following:
    cp e-smith-4.1b2.iso e-smith-4.1.1.iso
    
  2. Next, select the rsync mirror nearest you. See the download page for a listing. In the following example, we'll assume you've chosen e-smith's site. Be sure to check for a mirror closer to you, though.
       rsync -avv --stats \
    rsync://ftp.e-smith.org/e-smith/e-smith/current/iso/e-smith-4.1.1.iso .
    

    Note that the backslash character at the end of the first line indicates that the second line is actually a continuation of the first.Note also that there is a dot ('.') character at the end of the second line.

    The output from this command should look something like this:

    
       Number of files: 1
       Number of files transferred: 1
       Total file size: 329203712 bytes
       Total transferred file size: 329203712 bytes
       Literal data: 50102272 bytes
       Matched data: 279101440 bytes
       File list size: 52
       Total bytes written: 117913
       Total bytes read: 50176944
    
       wrote 117913 bytes  read 50176944 bytes  179945.82 bytes/sec
       total size is 329203712  speedup is 6.55
    

    In the example above, the total bytes written is tiny compared to the actual size of the ISO image. The speedup of 6.55 (i.e. rsync was 6.55 times faster than a straight download) means that the bandwidth and time taken was about 15% of what it might have been!

    Important Note: Be prepared to have space temporarily for two images in your local directory during the rsync.

    Additional command-line options to consider

    -z (compress):
    While we have had some intermittent problems with the -z (compress) option with rsync and the large iso images, we do recommend trying it. Be sure to check the file size and the MD5Sum before burning the ISO to CD though. A corrupt CD image can potentially do damage to your server.

    -P --partial:
    The --partial option instructs rsync to leave any partial file(s) that might have been created during an interrupted download. This is especially useful for those with intermittent or unreliable connections to the Internet, because it means that they can resume their download from the place they left off. See the rsync man pages for more details on how to apply this option.