You might think that supercomputer companies pick Minnesota because the HVAC bills are lower, and maybe they do. For whatever reason, Minnesota has always been a hotbed of supercomputing, and Big Blue is continuing in that tradition by opening its first Blue Gene utility computing center in its Rochester, Minnesota, facilities.
The Rochester labs are where IBM’s minicomputers, 3.5-inch disk drives, and early 64-bit PowerPC processors were all created. It is also where IBM manufacturers its OS/400 and AIX servers these days, and where a lot of the design work in its Power servers gets done. Suffice it to say, IBM Rochester has no shortage of smart nerds, and that is why the Blue Gene/L supercomputer is built there and that is also why, says Mark Solomon, the man behind the supercomputer utility centers at IBM, the company decided to locate the first Blue Gene utility in Rochester.
Technically, the facility is called the Deep Computing Capacity on Demand Center, and it is starting out modestly if you are looking at rack counts, but pretty impressively if you count teraflops. The center will have a single rack of the eServer Blue Gene servers.
That doesn’t sound like so much, right? Well, each rack has 1,024 dual-core processors running at 700 MHz (based on a stripped down 32-bit PowerPC 440 embedded processor), and that rack delivers a 5.7 teraflops of processing power. That is a lot of computing power to cram in one space, which is why IBM designed Blue Gene in the first place. IBM’s first customer for Blue Gene is the US Department of Energy, which is installing a 64-rack, 360-teraflops Blue Gene/L system at Lawrence Livermore National Laboratory to help it simulate and manage the country’s stockpile of nuclear weapons.
The Blue Gene/L machines run a stripped down Linux kernel created by IBM Research on its processing nodes (that’s what the /L stands for in Blue Gene) and run Novell’s SuSE Linux Enterprise Server 9 on I/O and management nodes in the cluster. IBM’s list price for a rack of Blue Gene/L servers is $2m, including the cost of Linux licenses but any external disk storage. At that price, Blue Gene/L is about an order of magnitude less expensive (just under 30 cents per flops) than parallel supercomputer clusters based on RISC/Unix servers and high-speed switch interconnects.
Back in March 2004, IBM was showing off a prototype deskside Blue Gene/L machine that had 64 Blue Gene server nodes (128 cores) in a deskside box, and the company was suggesting that this machine would be sold to developers to give them a chance to play around with the new Blue Gene architecture.
It is unclear if IBM is pushing ahead with the commercialization of Blue Gene in this manner, but the new Rochester utility is also clearly being positioned as a means for developers to test their code on a much more powerful Blue Gene box without having to acquire anything but time on the Blue Gene/L utility.
IBM says it has created a highly secure virtual private network to give customers access to the utility, which is particularly important since the kinds of military and commercial simulations that run on supercomputers are by their very nature something that organizations want to keep secret.
Even with the impressive price/performance of the commercialized Blue Gene servers, a couple of million bucks is still a lot to pony up for what is still largely an experimental supercomputer. While IBM has been pushing Power and X86 clusters in its On Demand centers in Poughkeepsie, New York, in Houston, Texas, and in Montpelier, France (which together have over 5,200 processors available for utility-style use by commercial customers), these architectures are well known and therefore customers know what they are buying into.
Blue Gene is a new and exotic technology, and On Demand gives people a reasonable way to get going, explains Solomon. He says that IBM is making a 32-node partition (just under 180 gigaflops) the minimum piece of the utility that customers can buy, and they have to buy it for at least a week. After that, they can buy capacity in daily increments.
By the way, he says that in the Blue Gene’s architecture, the toroidal interconnect doesn’t kick in until 512 nodes are clustered together, which means all of the nodes in the rack are linked using a flatter hierarchical scheme. Over time, perhaps by the end of the year depending on customer demand, IBM will expand the Blue Gene utility to four racks and the toroidal interconnect can come into play.
Solomon said he was not at liberty to say what the computing capacity cost is on the Blue Gene utility, except to say that it is significantly lower than for on-demand capacity for X86 clusters. And while that cost savings is important on the front-end of the deal, Solomon says that after customers use these utilities, they realize something else is more important.
Customers figure out that they can do things they could not do before, he says. Being able to have ten times as much computing capacity, even for a short time, allows companies to compete on a level that they could not compete at before. This is the real reason customers love this.
In fact, IBM didn’t have to go looking for its first Blue Gene utility customer. QuantumBio, a provider of drug discovery software and services for big pharmaceutical companies based in State College, Pennsylvania (and loosely affiliated with Penn State), came to IBM because of the Blue Gene architecture and said it wanted to buy utility capacity on the box.
