Soon after ZMODEM was introduced in 1986, protocols claiming to outperform ZMODEM appeared almost monthly. Most have faded into obscurity, but a few have found niches in specialized applications and proprietary environments.
Columbia University designed Kermit for the sixties' generation of mainframes and the pioneering networks that supported them. These design tradeoffs earned Kermit a reputation for slow file transfers. Indeed, it was slow XMODEM, YMODEM, and Kermit file transfers that prompted GTE Telenet to fund the development of 1986 ZMODEM.
Since then, advances in computing and networking have largely supplanted the legacy mainframes that needed Kermit. With interest and funding waning, it was time for Columbia's Kermit group to address different markets.
Columbia changed the Kermit specification to allow hackers to tweek the protocol for speed. Permission for developers to include Columbia code in their programs was quietly withdrawn. Columbia's Kermit group had become a nonprofit business unit.
Columbia revealed this change in its July 1993 issue of Kermit News. This promotional issue began with an announcement that Columbia University now accepts Credit Card orders. Under the heading "Kermit Saves You Money" Columbia University invited businesses to switch from other vendors' shareware and commercial software to Columbia's product. Columbia used its non profit mail permit to distribute this promotional literature to a large mailing list. Their taxpayer subsidized mail permit gives Columbia a crucial advantage over competitors.
In The Truth about Kermit File Transfer Performance Columbia University proudly announced that Columbia's Kermit software "outperforms X-, Y-, and ZMODEM protocol transfers every time" (Kermit News p.14). To lend credence to this assertion, Kermit developer Frank da Cruz supplied results of "True-Life Benchmarks" performed at Columbia University.
One would expect benchmarks published by Columbia University to be fair and accurate. Unfortunately, these benchmarks leave much to be desired.
The documentation for sz 3.24 states:
LICENSE AGREEMENT This is a copyrighted shareware program. Commercial use of this program is subject to licensing conditions detailed in the rz.c source code. "Commercial Use" includes use of this program to send files to any commercial or shareware program not published by Omen Technology INC.Using sz to send files to Procomm and Telix (both commercial programs) for the specific purposes of writing a promotional piece to increase market share is as commercial as one can get. Had Columbia University honored Omen's Copyright, Omen Technology could have advised on the benchmarking, ensuring a fair and accurate report instead of the misinformation that was published.
XMODEM and YMODEM
transmit file data verbatim, without any control character
prefixing or other modifications.
Data patterns do not
affect X/YMODEM throughput on direct connections.
Under these conditions, X/YMODEM throughput is affected only by the file length. The time required for start-up and end of file processing is relatively less important for longer files. Deviations from the trend of these data points are small. In the absence of modem compression, line noise, and other perturbations, a stable protocol runs as smoothly as clockwork.
Columbia's data plotted above show relatively large random variations. These variations, which are not a function of file size, show an experimental error of more than 20 per cent.
"The results given in the article are exactly as reported by the software, and each trial was run three or more times. Theory is fine, but in the real world all sorts of factors can cause variations."But 20 per cent error in controlled benchmark tests? These tests did not involve any modems or telephone lines! How real is a 20 per cent jitter in real life testing? I once ran a series of downloads of b17mh.gif. I was testing Omen Technology's Professional-YAM running in a Windows 3.11 DOS Box. A total of 100 megabytes were sent in 291 downloads over a 16 hour period. During this time Omen's Unix system performed its normal activities, including netnews processing, information utility downloads, database processing, and a 1.2 gigabyte system backup. Even with all this activity there was only a 6 per cent difference between the fastest and slowest download. Excluding the top and bottom one percentile, the difference was only 3 per cent. Columbia suggests that all sorts of unstated external factors can cause the large variations only they see.
Some explanation is in order here. Based on recommendations by Kermit's authors, I prepared a C-Kermit initialization file to replicate Columbia's benchmark tests. Initialization files are vital to Columbia Kermit; without a customized initialization file Kermit will not provide fast and reliable file transfers.
set terminal bytesize 8 set parity none SET BUFFER 30000 30000 ; Allocate packet buffers set send packet-length 5000 set receive packet-length 5000 set window-size 6 set block 3 set transfer cancelation off set file type binary set delay 0 set control unprefix all set control prefix 0 1 129 define asend echo START, apc rec, send \%1, echo DONE, exitI timed the C-Kermit/MSKermit combination with the shell script "tks":
kermit quit >/dev/null 2>&1 time kermit -C "asend $@"
The first line in the shell script ensures that C-Kermit and its initialization file are resident in the disk cache for instant access for the timing run. Compared to sz, Unix C-Kermit is a massive program that takes some time to load. Frank da Cruz doesn't believe this real-world overhead should be counted in "True-Life Benchmarks", so I agreed to handicap the race for Kermit's benefit.
When the command "tks uuencode" is given, MSKermit immediately switches to its file transfer screen.
Receiving: In progressAfter some seconds data transmission starts. After transfers complete, MSKermit returns to terminal emulation. A further second or two passes before the "DONE" message appears as Unix Kermit finishes. Using Columbia's uuencode benchmark file, MSKermit 3.13 reports a file transfer time of 4.62 seconds. Timed with a stopwatch, 13.7 seconds pass between the appearance of the Kermit file transfer screen and the "DONE" message. The Unix time command reported 13.87 seconds. Columbia would have you believe the 4.62 second figure, never mind the fact that Columbia Kermit really takes 14 seconds to get the job done.
When this issue was raised on the Usenet comp.dcom.modems newsgroup, Kermit disciples acknowledged this overhead. They could not suggest ways to get rid of this Kermit overhead. They merely suggested we ignore this Kermit overhead to make the tests "fair" for Kermit. This "True-Life Benchmarks" mind set is not a real world attitude.
Considering this measured Kermit overhead, the claimed 3 to 9 second Kermit file transfers reported in Tables 2-5 must be viewed with suspicion.
To avoid cheating, file transfer times should be measured with the Unix "time" command and verified with a stopwatch.
Some of Columbia's results were simply so preposterous I could not resist having a wee bit of fun with them. We noticed that cutting the communications speed from 19200 to 9600 and introducing noise actually increased the speed of a Columbia Kermit file transfer from 2648 to 2736 characters per second (Kermit News, p.13, Tables 2 and 5). This is the moral equivalent of lowering the speed limit by half and installing speed bumps every block only to find the same trip takes less time than before. When I announced the "Columbia Stochastic Telepathic Kermit Hyperprotocol" to "explain" Columbia's "True-Life Benchmarks" the Kermit faithful rather lost their sense of humor. They flamed about the "news release" but could not explain how Columbia came up with such crazy results. Frank's response: "This is indeed what happened. Explain it any way you like."
Selecting the uuencode Sun Sparc binary for a benchmark took some chutzpa. Uuencode is a tiny program stored in a 24576 byte file. All but a few thousand bytes are nulls. (Before you complain to Sun about the inefficiency of their executable format please note that this is an atypical file!)
Naturally this preponderance of empty nulls does wonders for Kermit transfer speeds. Had Columbia's protocol boffins read the sz 3.24 documentation and followed its instructions, they would have discovered that ZMODEM compression does even better. On direct connect transfers, Kermit gets 320 per cent efficiency on this file; ZMODEM compression gets an even better 363 per cent efficiency. A separate compression program does even better at 646 per cent, twice as good as Kermit.
Less egregious, but still atypical, is the selection of kermit.exe as a benchmark file. As the above chart shows, kermit.exe is more compressible than stsplus.exe, whose compressibility is typical of DOS executables.
For fairness, I included ZMODEM-90TM compression in the chart above. To quote Frank dz Cruz, "The results speak for themselves."
So why is Kermit not the fastest when it only prefixes a few control characters? Because Kermit prefixes more than you tell it to. Several other characters are quoted by Kermit itself. Columbia claims that only 0, 1, and 129 need to be prefixed, but this doesn't work sending to MSKermit 3.11. The following recommendations made by Frank da Cruz do work:
SET CONTROL UNPREFIX ALL
SET CONTROL PREFIX 1 13 129 141
This makes C-Kermit prefix 8 characters, 0, 1, d, 7e, 81, 8d, a3, and fe (hex). One can understand why YMODEM-g is fastest since it does not prefix ANY control characters. MobyTurboTM ZMODEM is next with one control character prefixed, followed by vanilla ZMODEM which prefixes 5 control characters (7 when dialing out). Columbia's Kermit-94 not only prefixes 8 characters, but suffers from the startup and shutdown delays mentioned above.
What is not reasonable is the incredibly low 66% efficiency Columbia reported for an ASCII ZMODEM transfer. Standard ZMODEM is 99% efficient transferring ASCII files. ZMODEM does not prefix any of the characters appearing in ASCII files, and the CRC-32 protocol overhead is about 6 out of every 1024 characters (0.6%). A standard ZMODEM single file transfer involves 5 exchanges for a total delay time of about one second. So how did Columbia get ZMODEM to take more than a minute longer than expected to transfer IKCKER.DOC than expected? Columbia reported "the connection is clean and no errors occurred". But their results are definitely in error.
Contrary to Columbia's claims, ZMODEM-90TM automatically detects 7-bit paths. Few applications involve true 7-bit paths, but ZMODEM-90TM works well here also.
ZMODEM-90TM optimizes performance with two 7-bit submodes:
The most impressive test at the ShootOut, and the likely reason nobody from Columbia University chose to appear, was Kermit's spectacular total failure in the noise test. After all Frank da Cruz has published about alleged ZMODEM reliability problems it would have been interesting to witness his embarrassment when Kermit decisively failed Frank's very own test, correctly executed.
Columbia's "True Life Benchmark" noise test consisted of a noise burst every two seconds on a 9600 bps modem link. This can be caused by a "clock slip" between telephone central offices. BBS users rarely see such line noise on 9600 and faster connections because today's high speed modems provide link level error correction. With today's modems, line disturbances manifest themselves as slow throughput and disconnects, not gibberish on the screen. But Columbia University lives in a different time period; Frank da Cruz claims clock slips are a common problem on New York's phone lines.
The Kermit News "True-Life Benchmarks" specified a packet length of 5000 to minimize Kermit's high per packet overhead. When we attempted to replicate Columbia's 9600 bps plus noise test, Kermit failed every time. When Kermit uses the specified 5000 byte packets with noise bursts every 2 seconds, no data packets get through. Why? It takes 5 seconds to send a 5000 byte packet at 9600 bps. Under Columbia's test conditions, every packet would suffer at least 2 noise bursts, and none could get through. Unlike ZMODEM, Kermit cannot resend a packet with a different size, so Kermit has to resend the same 5000 byte packet over and over until it gives up. This is a fundamental limitation of the Kermit protocol.
Columbia's Frank da Cruz has not been able to explain why his private tests showed Kermit working well under these mathematically impossible conditions. He still insists Columbia's tests were honest and fair, but neither he nor Columbia University have agreed to repeat them in public.
Assuming Columbia's tests were not completely faked, the only explanation is that their results are merely an artifact of the two-line terminal server and flawed assumptions used in Columbia's "True Life Benchmark" tests. This lightly loaded server buffers Kermit packets at 10000000 bps Ethernet speeds instead of the stated 9600 bps modem speed. Frank da Cruz has admitted that the noise bursts Columbia used in their tests were not injected into the modem's analog data stream as a real world clock slip would be. Instead the noise bursts were injected into the buffered Unix data stream. Because Unix C-Kermit buffers packets and outputs each in a single atomic operation, the Unix buffering forced the noise bursts to go BETWEEN Kermit packets where they would not cause errors very often. This artificial redistribution of noise bursts also explains the better than expected XMODEM and YMODEM performance Columbia reported. (Normally, ZMODEM transfers do not benefit from such artificial conditions.)
I repeated Columbia University's "True Life Benchmark" tests according to the description published in Kermit News. I was not able to replicate Columbia's results because the the noise bursts I added were at equal intervals as they would be with a real world two second clock slip. Kermit failed this test every time the test was run according to Columbia's published conditions.
Few users see such conditions today. With higher speed error correcting modems selling for less than a speeding ticket, real world conditions bear little resemblance to Columbia's tests.
Although Frank da Cruz did not read the sz documentation carefully enough to give ZMODEM a fair chance, he did read a 1988 ZMODEM Protocol Overview carefully enough to complain about the document's comments on Kermit. This old document mentions that ZMODEM is not optimized for the rare situations where delays and error rates are both high. It is no surprise Frank attempted a supremely unrealistic test that simulates conditions rarely seen today.
In contrast, ZMODEM is optimized for the real world.
I made an informal survey of the guests at the Protocol ShootOut. As so many have noted, today line impairments manifest themselves mostly as interrupted calls. When this happens ZMODEM's Crash Recovery is useful, allowing resumption of interrupted file transfers. Many users think Crash Recovery is ZMODEM's most important feature. Crash Recovery was an integral part of ZMODEM since its introduction in 1986. Since then Omen Technology has refined Crash Recovery to a high degree of reliability. Since Kermit was not designed to support Crash Recovery, Columbia chose not to discuss it, instead relying on bogus tests incorrectly simulating conditions rarely seen today.
What ZMODEM detractors don't know or won't tell you about file transfers may hurt you.
Since these points were first brought up, Columbia University's Frank da Cruz has repeatedly attacked the speed and reliability of ZMODEM in his Internet postings. He continues to reference Columbia's defective "True-Life Benchmarks".
In return, I have challenged him to allow me the same courtesy I offered him, to monitor a repeat of Columbia's "True-Life Benchmarks". This would seem a reasonable request, considering Columbia started this controversy by mailing misleading sales literature to many thousands of purchasers at public expense. At each challenge, Frank either disappears, changes the subject, or offers a lame excuse. I suspect he knows exactly who would be embarrassed by a fair and honest repeat of the so-called "True-Life Benchmarks".
Computer Bits Magazine
March 1994 Computer Bits, available by sending $3.00 to Computer
Bits Magazine PO Box 329 Forest Grove Or 97116-0329 or call Paul
at Computer Bits Magazine 503-359-9107.
The Daily Oregonian
An article and picture appeared shortly afterwards in the Daily Oregonian.
The article concentrated on the personalities in attendance.
The Daily Oregonian An article and picture appeared shortly afterwards in the Daily Oregonian. The article concentrated on the personalities in attendance.