Dodgy benchmarks
Sunday, August 19th, 2007For anyone following JavaBlogs, you’ve probably skimmed over a bunch of Sri Lankan shirtlifters cooing recently about a highly dodgy benchmark that somehow proves that Axis2 is faster than XFire.
What’s interesting about this benchmark is that it manages to compare apples with oranges, and yet has its proponents drawing some truly perplexing conclusions.
There are some, ahem, issues with this benchmark. Like any vendor driven benchmark desperate to compete, It manages to compare wildly different things, just to find a case whereby they ‘win’. Instead of comparing the stacks fairly, they focussed on comparing the data binding parts. What they proved in fact is that ADB (a proprietary Axis2 data binding framework) is faster than JAXB (data binding specification). OK, not so surprising. If you wanted to be fair and picked the fastest data binding for XFire too, you could test Axis2/ADB against XFire/JiBX (JiBX being a frightfully fast OSS data binding framework).
In fact, buried in the so called benchmark, you’d find the results themselves prove that XFire is faster than Axis2 when the comparison is fair and uses the same binding framework. Somehow though, this rather crucial fact is lost in the conclusions.
What’s more impressive is that no sooner than the company issues this highly dodgy report, its employees all start blogging about it and bragging that they finally managed to concoct a benchmark whereby they beat their competition.
I thought that with JBoss being bought out and the old cabal being effectively neutered, we’d seen the end of the snide slimey comments by OSS against OSS, but our Sri Lankan friends seem keen to assume the throne of corporate fuckwittery in the name of OSS.
I hope this benchmark is taken as what it is, a company defending its own product against what is obviously a superior OSS solution. Friends don’t let friends use Axis2. Do you want to base your webservices infrastructure on a product from a company desperate to take your money, written as a student project, with a bunch of incompetent weasels at the helm trying very hard not to look like used car salesmen? Do you want something that integrates and embeds well withint your existing infrastructure, or do you want something that hijacks it instead?
Still, good thing we have the goodship Apache; that thriving haven of open communities, that final refuge for corporate flotsam and jetsam, that last bastion against whored solutions foisted on an unwilling and uninterested public, eh?
