Measuring Japanese machines in Tokyo tells nothing about their US, European models

When asked to comment on these claims, several industry sources made a number of similar points. First of all, measuring Hitachi and Fujitsu boxes in Tokyo is of limited value since Amdahl, Hitachi Data and Comparex systems have different chips, cycles and even different microcode from the Japanese base machines. Secondly, IBM’s methods may have been developed over 15 years, and the methodology may be documented in LSPR G1 ZZ05-0466, but that card is for internal use only and not published for comment and analysis. Consequently, the results may be available in LSPR documentation, but since it is not generally available, competitors and analysts are unable to comment on their credibility. Also, it is workload that dictates performance, and workloads have changed radically over the past few years, both for technological and business reasons. If IBM is using workloads that are 15 years old, it may be using workloads that are more representative of the 1970s than the 1990s.

The limitations in extrapolation seem manifest when applied to a four-way machine…

Next on the Dublin agenda was Large Systems Performance Reference MVS measurement history. IBM has measured all of its processors plus selected ones from other vendors, and this is an ongoing process with LSPR numbers for the new ES/9000 9021-820, 860 and 900 machines coming in the third quarter. IBM says that when it measures an n-way box like Amdahl’s 5990-1400, it gets four-way, three-way, two-way and one-way results by varying off-engines. However, other sources say that multiprocessors usually have better system control unit resources to assist in overcoming system conflicts, and varying one engine off removes a processor, but not the extra resources. Consequently, the one-way will have four-way resources like interleaving, and it will run more quickly. So, if IBM then measures a four-way at normal performance, the MP ratios will be lower than other vendors would anticipate. The presentation included a slide that provided details on the processors that have been measured, as well as operating systems, workloads and dates. Something that is not included in the reference card. The Hitachi measurements were especially interesting since none of the processors – 9080, XL60, VL60 and XL 100 – appear to have been measured under ESA. In the same order, the systems were measured in April 1985, November 1986, November 1987 and August 1988. Consequently, IBM seems to have discounted the effects of ESA and has yet to measure the EX 100. The speaker is believed to have advocated caution when discussing these benchmarks with installed Hitachi users, and he acknowledged that there is a similar exposure with the low-end EX 33 and EX 44 machines. As regards Amdahl and Hitachi performances, he said that there is often a disparity between Amdahl’s announcement claims and AmPEC measurements. Also, he claimed that while the company occasionally publishes reports on its own processors, he has yet to see a an AmPEC comparison of Amdahl and IBM systems. He suggested that Amdahl announcement claims are for batch only, but most customers buy machines to run on-line systems. On Hitachi Data and Comparex, there is no evidence that either company benchmarks its own processors or those of IBM, although Hitachi Data does have a benchmarking centre in Santa Clara. Further, there is no publicly available document that describes their benchmarking methodology and results, and the UK arm relies heavily on Gartner Group’s MIPS tables which are based on the Hitachi Data anouncement claims. Using the EX 100 as an example, the speaker concluded that LSPR measurements are just below the bottom-end of Hitachi Data announcement claims. As the LSPR numbers are for commercial batch, perhaps Hitachi Data, like Amdahl, makes announcement claims for batch workloads. Other sources say differently. First of all, they point out, both Hitachi Data and Comparex would claim that they do benchmark their own processors. Secondly, the G

artner Group constantly changes its MIPS tables, but Hitachi Data and others do not reposition processors accordingly. Thirdly, why use the EX100 as an example when IBM has yet to measure the machine?

and is it all a clever marketing ploy to dismiss plugs as batch pushers?

And finally, is this a clever marketing ploy by IBM to categorise the plug-compatible manufacturers as providers of batch, not interactive, systems? No industry source disputes IBM’s right to produce reference documents that compare its processors with competitive offerings, and it is fully entitled to distribute these findings within the company. However, they all expressed similar reservations about distributing such documentation outside the company. It is marked for internal use only, and by putting it into the public domain, IBM is raising questions about the status of such marking. More importantly perhaps, the document implies a substantial degree of accuracy, but there is no indication of which processors have been measured and when the benchmarks took place. Is IBM entitled to extrapolate? Again, the consensus is that extrapolation is legitimate, but only if the assumptions and underlying conditions are clearly stated. The overwhelming concern is that IBM will present these figures to third parties without necessary background information and explanation, which amounts to a potentially misleading course of action. Mainframe users have gone on a buying strike in droves – in the US, in Europe, in Japan it’s the same story, with the UK as usual in the vanguard of new trends, the most threatening to the likes of IBM being the temptation to downsize to open systems. So that where a contract is up for grabs, competition in the IBM-compatible mainframe market has never been so fierce as it is now, where for all parties involved, what’s at stake is coming close to their very survival – certainly a large number of jobs are on the line, as John Akers has been drumming into the IBM workforce, and never has the wise injunction caveat emptor been more appropriate. – Janice McGinn