Using rsync to obtain the e-smith CD ROM image

We highly recommend using rsync as the method of choice for obtaining the e-smith CD ROM image. We have found it to be the most efficient and least bandwidth-intensive method for downloading any large file(s). In a nutshell, rsync only downloads the differences between a file at the source location and a file of the same name at the destination. When dealing with a 300+ MB file like the e-smith CD ROM image, this can result in download savings upwards of 80%.

This document provides links to information about rsync, and instructions on how to use it to obtain the e-smith server and gateway software.

Who should use rsync?

In the broadest terms, anyone using with an Internet-connected Linux machine should seriously consider using rsync. Even if bandwidth savings don't mean anything to you individually, the bandwidth savings that can be achieved using rsync benefit everyone connecting to the download resource.

What is rsync?

First, rsync is not by nature Windows software. While there are documents online that explain how to build rsync on Windows 95 and Windows NT, only advanced users who are comfortable compiling their own software should even try.

rsync does run on Linux and many Linux-like operating systems *grin*. You can find binary versions for 10 or so different operating systems and Linux distributions at http://rsync.samba.org/ftp/rsync/binaries/.

From the rsync home page:
"rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License."

The following is taken from the rsync man page:

rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speedup file transfers when the destination file already exists.

The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

Some of the additional features of rsync are:

support for copying links, devices, owners, groups and permissions
exclude and exclude-from options similar to GNU tar
a CVS exclude mode for ignoring the same files that CVS would ignore
can use any transparent remote shell, including rsh or ssh
does not require root privileges
pipelining of file transfers to minimize latency costs
support for anonymous or authenticated rsync servers (ideal for mirroring)

How do I learn about rsync?

The first place to start learning about rsync is of course the man(ual) page. Simply enter man rsync at your linux command prompt.


    
      You can look at some of the examples at http://rsync.samba.org/rsync/examples.html.
    
    
      There's a very good tutorial on how to use rsync located at
      http://www.eunuchs.org/linux/rsync/. 
      [NOTE: This site seems to be offline right now.]
    
    
      Finally, for the more technically inclined, the rsync resource pages
      contain a number of other useful links.
    
    
    How do I use rsync to obtain the e-smith CD ROM image?

    
      Follow these steps to download the e-smith CD ROM image:
    
    
      
	If you have an older copy of the e-smith 4.1 ISO image (for example a beta 
	or release candidate) you can use it to prime the destination file. Copy the old CD 
	ROM image to a new name. For example, if you have an image of e-smith 4.1 beta 2, 
	you would type the following:
cp e-smith-4.1b2.iso e-smith-4.1.1.iso

      
      
	Next, select the rsync mirror nearest you. See the download
	  page for a listing. In the following example, we'll assume you've chosen 
	e-smith's site. Be sure to check for a  
	mirror closer to you, though.
   rsync -avv --stats \
rsync://ftp.e-smith.org/e-smith/e-smith/current/iso/e-smith-4.1.1.iso .


	
	  Note that the backslash character at the end of the first line indicates that the 
	  second line is actually a continuation of the first.Note also that there is a dot ('.')
	  character at the end of the second line.
	

	
	  The output from this command should look something like this:
	

   Number of files: 1
   Number of files transferred: 1
   Total file size: 329203712 bytes
   Total transferred file size: 329203712 bytes
   Literal data: 50102272 bytes
   Matched data: 279101440 bytes
   File list size: 52
   Total bytes written: 117913
   Total bytes read: 50176944

   wrote 117913 bytes  read 50176944 bytes  179945.82 bytes/sec
   total size is 329203712  speedup is 6.55


	
	  In the example above, the total bytes written is tiny compared to 
	  the actual size of the ISO image. The speedup of 6.55 (i.e. rsync was 6.55 times faster
	  than a straight download) means that the bandwidth and time taken was about 15%
	  of what it might have been!
	

	
	  Important Note: Be prepared to have space temporarily 
	  for two images in your local directory during the rsync.
	

    Additional command-line options to consider
    
      -z (compress):

      While we have had some intermittent problems with the -z (compress) option with
      rsync and the large iso images, we do recommend trying it. Be sure to check the 
      file size and the MD5Sum before burning the ISO to CD though. A corrupt CD image can 
      potentially do damage to your server.
    
    -P --partial:

      The --partial option instructs rsync to leave any partial file(s) that might have been created
      during an interrupted download. This is especially useful for those with intermittent 
      or unreliable connections to the Internet, because it means that they can resume their
      download from the place they left off. See the rsync man pages for more details on
      how to apply this option.