IBM UK is to hold a meeting for its large customers on July 11 and sources say that controversial benchmarks conducted at British Gas South Eastern will form a major plank of the presentation, called Quo Vadis?. The benchmarks showed an 18% difference between Amdahl Corp’s 5990-1400 and an IBM ES/9000-720, which resulted in one of two 1400s being removed from the Mitcham data centre and replaced with a 720. Other commentators say that the performance difference is below 5%, and British Gas South Eastern is now re-evaluating the benchmarks, having invited Amdahl to participate in further analysis. The performance of Amdahl’s top-gun 5995-8650M, rated at 300 MIPS and due in the second quarter of next year, is closely linked to the performance of the 1400 machine. If the 1400 has to be downgraded by 18% from 113 MIPS to 95 MIPS, then the multiplier between the 1400, the 4550M and the 8650M is also downgraded.
Shogun
Amdahl operates on a 1.9 multiplier, but if that falls to 1.6, then the the 4550 mid-point is 163 MIPS and the 8650M midpoint is 276 MIPS, instead of 326 MIPS from a 113 MIPS base for the 1400 machine. Accordingly, the benchmarks, analysed by Camberley-based consultants Coveford Data Systems, seem to be designed to convince users that the six-processor 900 can be compared with Amdahl’s eight-processor 5995-8650M. As part of the deal with British Gas, IBM has permission to present the benchmarks at 20 user sites and at an internal sales and marketing meeting. A meeting is planned to take place in Dublin between this week and it will focus on IBM’s aggressive Shogun programme which emphasises reasons for choosing an IBM mainframe when IBM does not have the lowest price. British Gas South Eastern had two 5990-1400s installed and for the purposes of the benchmark, they were running at 30% to 50% busy. The TP 1400 had two Domains – on-line production for the CICS/MVS/ESA system and sandpit for the system programmers development system. The Batch 1400 had three Domains: the first was batch for development and production batch TSO MVS/ESA; VM for the VM/HPO systems; and UTS for Unix support. Both Multiple Domain Feature and PR/SM were excluded from the measurements despite the company’s use of Domains on both 1400s and probable use on the 720. The benchmarks are said to have a number of limitations, not least of which is Internal Transaction Rate-based ratios and low utilisation effects. ITR is the number of transactions processed per CPU second and said to be favoured by IBM, while Amdahl favours calculations based on External Transaction Rate, the maximum number of transactions processed per wall-clock seconds.
The controversy over benchmarking of IBM systems looks set to come to a head after what is being seen in some quarters as a determined campaign by the company to neutralise the threat from Amdahl’s forthcoming eight processor machine before it even ships: Janice McGinn reports.
A speedier processor encourages the use of heavier transactions since response time can remain the same even though more instructions are being executed. On the basis of ITR calculations, the CPU content of each transaction increases and so the ITR drops. A slower processor has the opposite effect and encourages the use of light transactions. Consequently, ITR-based calculations favour slower engines at low utilisations, and low utilisation effects can affect measurements of processor power. Many would argue that it is only when a processor is being driven to capacity that full effects can be measured, and most users buy processors to do exactly that. Driving a processor at 30% to 50% provides performance figures for that range only, and extrapolation is not an accurate means of determining capacity and performance at higher loadings. There have been other criticisms of the benchmarks, including the charge that only partial validation took place and many of the validation criteria were ignored. Variations in workload and response time effects were also ignored, and IBM’s direct involvement in the measurement analysis could be seen
as improper. Other sources say that IBM has tried to run the same benchmarks at several companies including Daimler Benz AG in Germany and the Dutch airline KLM Royal Dutch Airlines, and it has been suggested that KLM came up with a 5% performance difference, as opposed to the 18% claimed for the British Gas South Eastern trial. If IBM presents these benchmarks at the Quo Vadis? seminar, apparently with help of Mr Duncan MacPherson of British Gas South Eastern, it will be the second attempt to publicise its claims, having failed to do so at the UK Computer Measurement Group’s Sixth Annual Conference up in Birmingham earlier this month.
Garden path
According to Dave Claridge, UKCMG vice-chairman and responsible for conference programming, IBM submitted its paper – Large Systems Performance Update – only hours before the seminar was due to take place. Claridge says that by comparing products and claiming that the IBM offering is faster, ergo better, IBM was breaking rules that have been in place since the user group’s formation several years ago. Accordingly, the Measurement Group programming committee had no option but to pull the offending piece. So why did IBM deem it acceptable to present the paper? The highly-respected IBMer, Noel Bradbear, was meant to be presenting the update paper, but he was substituted by Roger Mansfield, reportedly a member of the Basingstoke-based Shogun team. Shogun has been a successful and aggressive marketing tool in the UK, and there are reports that it has been adopted in Germany, with a trial run in the Wiesbaden area. Both IBM and British Gas South East were asked to comment on the the benchmarks, but neither company responded by the time of going to press. A final thought for those customers planning to attend IBM’s Quo Vadis? presentation. Quo Vadis means whither goest thou, and an appropriate answer could well be up the garden path.
