We highly recommend using rsync as the method of choice for obtaining the e-smith CD ROM image. We have found it to be the most efficient and least bandwidth-intensive method for downloading any large file(s). In a nutshell, rsync only downloads the differences between a file at the source location and a file of the same name at the destination. When dealing with a 300+ MB file like the e-smith CD ROM image, this can result in download savings upwards of 80%.
This document provides links to information about rsync, and instructions on how to use it to obtain the e-smith server and gateway software.
In the broadest terms, anyone using with an Internet-connected Linux machine should seriously consider using rsync. Even if bandwidth savings don't mean anything to you individually, the bandwidth savings that can be achieved using rsync benefit everyone connecting to the download resource.
First, rsync is not by nature Windows software. While there are documents online that explain how to build rsync on Windows 95 and Windows NT, only advanced users who are comfortable compiling their own software should even try.
rsync does run on Linux and many Linux-like operating systems *grin*. You can find binary versions for 10 or so different operating systems and Linux distributions at http://rsync.samba.org/ftp/rsync/binaries/.
From the rsync home page:
"rsync is an open source utility that provides fast incremental file
transfer. rsync is freely available under the GNU General Public
License."
The following is taken from the rsync man page:
rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speedup file transfers when the destination file already exists.
The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link, using an efficient checksum-search algorithm described in the technical report that accompanies this package.
Some of the additional features of rsync are:
- support for copying links, devices, owners, groups and permissions
- exclude and exclude-from options similar to GNU tar
- a CVS exclude mode for ignoring the same files that CVS would ignore
- can use any transparent remote shell, including rsh or ssh
- does not require root privileges
- pipelining of file transfers to minimize latency costs
- support for anonymous or authenticated rsync servers (ideal for mirroring)
The first place to start learning about rsync is of course the man(ual) page. Simply
enter man rsync
at your linux command prompt.
You can look at some of the examples at http://rsync.samba.org/rsync/examples.html.
There's a very good tutorial on how to use rsync located at http://www.eunuchs.org/linux/rsync/. [NOTE: This site seems to be offline right now.]
Finally, for the more technically inclined, the rsync resource pages contain a number of other useful links.
Follow these steps to download the e-smith CD ROM image:
cp e-smith-4.1b2.iso e-smith-4.1.1.iso
rsync -avv --stats \ rsync://ftp.e-smith.org/e-smith/e-smith/current/iso/e-smith-4.1.1.iso .
Note that the backslash character at the end of the first line indicates that the second line is actually a continuation of the first.Note also that there is a dot ('.') character at the end of the second line.
The output from this command should look something like this:
Number of files: 1 Number of files transferred: 1 Total file size: 329203712 bytes Total transferred file size: 329203712 bytes Literal data: 50102272 bytes Matched data: 279101440 bytes File list size: 52 Total bytes written: 117913 Total bytes read: 50176944 wrote 117913 bytes read 50176944 bytes 179945.82 bytes/sec total size is 329203712 speedup is 6.55
In the example above, the total bytes written is tiny compared to the actual size of the ISO image. The speedup of 6.55 (i.e. rsync was 6.55 times faster than a straight download) means that the bandwidth and time taken was about 15% of what it might have been!
Important Note: Be prepared to have space temporarily for two images in your local directory during the rsync.
-z (compress):
While we have had some intermittent problems with the -z (compress) option with
rsync and the large iso images, we do recommend trying it. Be sure to check the
file size and the MD5Sum before burning the ISO to CD though. A corrupt CD image can
potentially do damage to your server.
-P --partial:
The --partial option instructs rsync to leave any partial file(s) that might have been created
during an interrupted download. This is especially useful for those with intermittent
or unreliable connections to the Internet, because it means that they can resume their
download from the place they left off. See the rsync man pages for more details on
how to apply this option.