As I mentioned in my last post I've made my way to Microsoft HQ with a Visit to Canada on the way; this meant flying out through Heathrow Terminal 5.

T5 got off to a bad start, and we made contingency plans for lost luggage, delays etc. The reality: the easiest taxi drop-off I've seen at a UK airport, very short queues to check in in, lower security hassles than most airports, and as pleasant a place to wait for my flight as I've found; and baggage arrived successfully at the other end. The story is doing the rounds that the initial problems weren't just teething troubles, but were down to BA not training its staff on what was new... I understand that they showed up for work on the morning the terminal opened and had to work out where they were supposed to park and go on from there...

Now one of the things that has come up in the meetings I have been in is that we've done a lot of survey work around Vista. 94% of PCs sold via retail have Vista on them. In our recent financial statements to Wall street we said we'd sold 180 Million licences. Corporate customers are buying Vista licences faster than they are deploying, but the same happened with XP, and the rate of deployment of Vista is about the same as XP was at the same point in its life. But Vista is not getting the good press it deserves.

2 years ago, at every meeting I went to someone would try it plug their laptop into a projector to show their slides. And it wouldn't work "Press Function F5" someone would call out "No, this one's a Dell , F5 is for the HP" another would say. "Oh... try F8" .  People would slip out of the room to get a coffee or make a quick call. By the time it had been sorted out 5 minutes had passed. It's a very conservative estimate to say I lost 10 minutes a week this way. Over a year that's a whole working day. 

Then Vista came along and it has the Mobility Center. Press [Window key] [X] and up it Pops. Click connect to display and WHOOSH the presentation is on the screen. That's a day saved: and knowing what Microsoft consulting Services used to charge for my time, that's worth more than cost of the upgrade licence, and the deployment cost and so on.

I brought this up in the group meeting this morning when someone said something about "people not being able to use PowerPoint because of Vista".. The betas of Nvidia's Vista driver 6 months before launch didn't work when a monitor was plugged in, but 3 Months before launch that problem went away. I must have sat through upwards of 500 PowerPoint presentations since Vista came out, and I've never seen a problem related to Vista. If the battery fails in someone's slide clicker what I hear is "That's vista for you", if the projector won't focus "It's vista". That's wearing a bit thin. What shocked me was someone at the same table leant over after I'd said all that and asked "What was the key combination for that ?"

Earlier in the session, a senior Microsoft person said she'd really valued the training everyone received on XP, and how she missed that with Vista. Anyone can switch from XP to Vista without re-training. You get security benefits, a better network stack, easier deployment  but it takes a little time to show people what can be done. How you get the best out of search, use tagging and previews in explorer, turn off the sound for one irritating application without turning them all off. The list goes on. We didn't teach people those things. And not everything about a brand new OS is positive.  Some hardware isn't up to the task - I will tell the story of my Home PC another time, but the short form is it needed me to spend £40 on RAM and the 5 year old graphics card doesn't support Glass; it works better and does more than it did under XP. Some things won't have drivers in the early days. The number of drivers available at launch was a creditable 30,000 but that's increased by over 150% (77,000 at SP1). Some applications don't work on a new service pack never mind a new release: not everyone has got to grips with the Application Compatibility Toolkit which has helped to provide an environment where some real horrors can be persuaded to run. And user account control annoys IT Professionals like the seat-belt alarm in my car annoys drivers. I don't need to be told I'm manoeuvring the car with my belt off. I do want to know if my children undo their belts when we're driving down the motorway. I hope the safety parallel is obvious.

We didn't give our own people the skills to talk about these things. And our own gleaming new state of the art product had a "Terminal 5" experience. SP1 for Vista was a milestone which gave people the feeling they could go back and have another look at Vista (in fact almost all of key changes had happened before SP1). I wonder how long it will be before people stop trying to avoid T5.

Update. Fixed several typos. Jet lag. Grrrr.

There's a saying in politics People will fall more easily for a big lie than for a small one*. I think someone at VMware has picked this up because they're trying a really big lie.
VMWare doesn't need memory.

Or to quote the post more precisely: VMware Infrastructure's exclusive ability to overcommit memory gives it an advantage in cost per VM the others can't match.**

They were able to start 40 instances of Windows XP to achieve the 40 VMs, with 512MB of memory on a machine with only 4GB of RAM - a 5 times over commitment ratio.  Of course they didn't actually run anything in them, because if you and I fired up Outlook, and IE (with our own mail boxes and choice of pages)  you open word and I open PowerPoint very few memory pages will be sharable (I've got 47 pages open in IE right now, and it's using over 300MB of RAM, almost all for data). That means a lot of paging will have to happen in the virtualization stack. Brace yourself for really poor performance.

When I read they were using Windows 2000 in the server test I got suspicious, why use an OS which is out of support ? to get better results of course. (As I understand it they don't support Windows Server 2008 yet). To get as much page overlap as possible they ran identical OS, and application on all the VMs, and gave them only a little memory - 512MB for a server is pretty meager. That's important because it ensures memory is occupied by OS and programs  rather than data. Programs will have a lot of sharable pages of memory: data and caches won't.  Suppose they had done the same over commitment with 1.5GB machines and got 4  on their system to 2 on Microsoft Virtualization.  Spare memory will get used as a cache and this will have almost no sharable pages. Data is in from disk to fill memory which isn't there and has to be paged out in the Virtualization stack - that's two sets of disk I/O for nothing.

The writer then cooks the figures: a $6000 for a  2-way server with 4GB of RAM ? I went onto Dell's web site, and got a quote for 2xDual Core Processors, 16GB of RAM and 2x500GB drives in an SC1435 for under £2,000. Given that the UK prices for hardware are always higher than US prices I'm very dubious of that server price. But let's allow that. According to the VMware post their software costs an extra $5,750 (note that's without support). Again I'm working from the memory upgrade prices on Dells web site, an upgrade from 4 to  32 GB upgrade adds £2,720 to the price. At today's Exchange rate, and ignoring the cheaper prices in the US, that's $5,440. So according to VMware a $17,000 system based on their software would have 4GB of ram and run 14VMs. On 4GB the system would run 7 VMs on Microsoft software. But the same money using Microsoft Software would buy 32GB of RAM, the extra 28GB which allows you to run 56 more VMs. That's 63 VMs on $17,000 of system with Microsoft ~ that $270 per VM for Microsoft, against $1268 for per VM for VMware. And with physical RAM for every byte the VMs see, the Microsoft system will perform better.
(To save smarter readers from pointing it out, a $6,000 server might run out of steam between the 7 VMs initially supported, and 63 VMs. The VMware work assumed there would enough CPU, DISK and network resource and the system would be RAM limited. If 14 VMs is where CPU becomes a limitation then we'd be looking at a 4GB upgrade costing £260 /$500, so the Microsoft system would come out about $5,250 cheaper for the same VMs - that's about $375 per VM.)

That's the argument ? Over $5000 of VMWare software is a good investment because it saves $500  on RAM ?

Further Cooking the figures. If you have a Microsoft support contract Microsoft Virtualization Software is covered. One of my colleagues tells me that VM platinum support for 3 years, would add $3666 to the total cost. I haven't checked this numbers but divided over 14VMs that's $261 per VM for VMware support... yes you read the numbers correctly support for VMware costs nearly as much per VM as the hardware, software and for the Microsoft solution. That's before you allow for training, maintaining a separate management infrastructure for the virtual world, and so on.

If someone can show me a customer who is running, in production, a VMware VI3 Enterprise system with a 2:1 memory overcommit ratio on all the VMs, where spending the cost of VMware on RAM wouldn't remove the need to use overcommitment then I'll give... lets say $270 to their choice of charity. ***

Update. Someone asked me to remove a term which is OK in British usage, but tends to offend other English speakers.

* Foot note 1. Yes I know who wrote that, and I'm not going to suggest that VMware have been reading him. Like me, someone there might collect books of quotes and aphorisms.

**Footnote 2. As one of the commenters to the post points out VMware's published recommendation is  "Make sure the host has more memory than the total amount of memory that will be used by ESX plus the sum of the working
set sizes that will be used by all the virtual machines." I.e. Don't use this feature, if you want decent performance.

*** One offer only. Void where prohibited. Over 18s only. My cause drowsiness; if affected to not operate virtual machinery. etc.

Unless this is the first post you've read here you'll know that I like Windows Vista, not because I work for Microsoft but on it's own merits. (I'm sure I've said before I work for Microsoft because I like the products and the thinking behind them , not the other way round).

You'll also know that I'm convinced that a lot (not all, but a lot) of negative perception about Vista is down to ignorance, and problems of a bad press.

Someone had a bright idea. Lets take people who don't know Vista and ask them about their view of it. Then lets show them Vista and see what they think; but lets not tell them it is Vista. Lets tell them they're looking at a future Microsoft OS codenamed .... Mojave.

You can see the results of this experiment here http://www.mojaveexperiment.com/ 

CNET had a teaser about this story last week , but the website has only gone live in the last few hours. IT Wire has the story, so does Daily Tech.

It's quite amusing to see the VMware blogger (Citrix's blog names him as Eric Horschmann) has come back with a screen full of algebra to try justify it the post I commented on yesterday

They look convincing, don't they ?  He's right only if you fixed the amount of RAM in the server.  But when was that ever a fixed point ? either you want to run 14 VMs on the box, or the budget per box is $17,748. Who ever yelled "Screw the budget, Screw the workload. Keep the memory constant !" ?

Needless to say if your product is adding cost, you want the other figures beefed up to keep the proportion due to you as small as possible. I said the cost of the box was suspiciously high. And would you run 0.5 GB VMs (was that why they chose an out of support OS) . If you were only running 1GB VMs you'd get 3 on the box and use the cheaper Enterprise edition of Windows rather than Datacenter. That would reduce the cost of the Microsoft system, perhaps to the point where it was as cheap to run two small Microsoft boxes as one larger VMware one. 

So lets try those figures again shall we ?  This time we'll build a server to support 14 VMs. The only cha

Who's cheapest now  ?

Now if the customer is prepared to $5,750 (plus support, training, and extra management tools) on VI3 enterprise... what would they get if they spent that on RAM

Who's cheapest now ? Oh look it's Microsoft again.

Now, I mentioned the screenful of algebra and of course this is based on the same fallacious axiom that memory is the same for the two systems. This contains the same error as the first post
one line explaining the terms. M_H  Physical server memory which of course is held to be the same on both systems. In practice, the Microsoft system would have more RAM and therefore better performance. I've reworked the equations allowing for M_HV  and M_Hm  the memory on the VMware and Microsoft systems respectively , but keeping everything else the same.

C_H Cost of server hardware
C_M Cost of memory per GB
C_VMW Cost of VMware virtualization software
C_MS Cost of Microsoft virtualization software
C_OS Cost of operating system software
M_HV Physical server memory, GB – Microsoft
M_HM Physical server memory, GB – VM ware
M_V Memory per VM, GB
r Memory overcommit ratio

The cost of the Microsoft system is C_H + C_M M_Hm + C_MS +C_OS
And the VMware one costs C_H + C_M M_HV + C_VMW +C_OS

If C_MS is zero and the systems cost the same then  C_M M_{Hm =} C_M M_HV + C_VMW

Solving this for memory we get  M_{Hm =} M_HV + (C_VMW / C_M )
(In English if the cost of VM ware is $5000, and Memory costs $100 per GB, then a Microsoft system can have 50GB more memory for the same money)

The Number of Virtual Machines on VMWare is  r M_Hv / M_V

And on Microsoft it is  M_Hm / M_V

So to run the same number of VMs: M_Hm = r M_Hv
(in English if VMWare can run an overcommit ratio of 2 Microsoft needs twice as much memory, for the same VM count. In reality a ratio of 1.25 is more realistic, so Microsoft would need 25% more memory.)

Now we have two equations M_{Hm =} M_HV + (C_VMW / C_M ) And : M_Hm = r M_Hv

So the break even point for VM ware comes when M_HV + (C_VMW / C_M ) = r M_Hv

M_{Hv =}(C_VMW / C_M ) / (r -1)

if the cost of VM ware is $5000, Memory costs $100 per GB and r =2, then the VMware system needs to have 50GB of RAM and the Microsoft one 100GB of RAM: above that VMWare is cheaper, below that Microsoft is. If you think my figure of r=1.25 is closer to the real world, then the VMWare system would need 200GB of RAM (and in Eric's scenario that means 400 VMs). Just remind me again what the memory and VM count limitations are with VMware....

Three closing points. First there are scenarios which do benefit from being able to overcommit memory (for example if you're setting up training machines and need 100MB more memory than you've got)- we may have to cap the level at which it can be used to prevent customers getting themselves into trouble. But Bob Muglia has said internally and externally that the feature is needed. Whether "needed" means for tick-in-box feature comparisons or real-customer-need is open to interpretation, either way the feature will come back (it was in the Alpha versions).
Secondly Validation.Back at IT forum we pre-announced the "Server Virtualization Validation Program" personally I'm hoping that before we give customers the ability to overcommit memory or dynamically add CPUs and memory to running VMs we validate applications to give them confidence that these abilities are safe to use. If memory which a service has asked to be allocated from a non-paged pool is being paged by the virtualization stack, what will the impact be.
And finally. People tell me that VMware customers are using over commit rates of 2 or more in production the offer I made yesterday still stands. show me a customer who is running, in production, a VMware VI3 Enterprise system with a 2:1 memory overcommit ratio on all the VMs, where spending the cost of VMware on RAM wouldn't remove the need to use overcommitment then I'll give... lets say $270 to their choice of charity.

Technorati Tags: Windows Server 2008,Windows Server,Virtulization,Hyper-v,VMware,Fud

I've managed to get a lot of my concerns about privacy down to a simple statement. "Databases of everything" worry me. Where we've been, what we've bought, who we've associated with. I alluded to a conversation I was in last week where we talked about the information that could be gathered by Live ID -  during that conversation someone made the observation that people stop worrying about privacy when they see utility.  Even with my paranoia I'm fairly happy for Amazon to tell me things I might like, because the know what I've bought in the past. I haven't bought many things - and some gifts I've bought lead to odd recommendations. But I don't use a supermarket loyalty card because (or even use the same credit/debit card each time I shop) because that's somehow the wrong side of the line.

I thought there might a place where everyone would draw the line... ?  For example implanting RFID into people is pure sci-fi, right ? Wrong: I thought when I read that doctors were talking doing just that to track patients with Alzheimer's - the technology comes from Verichip makers of "VeriGuard™ "the first radio frequency identification (RFID) security solution to combine access control [with] VeriChip's patented, human-implantable RFID microchip. "  

The BBC has previously reported on surveillance uses of RFID tags and last Friday they reported how RFID can be used in combination with Wifi : 
'Angelo Lamme, from Motorola, said tracking students on a campus could help during a fire or an emergency. "You would know where your people are at any given moment," he said. ' 
Yes. You'd know where they are every moment of every day - a classic "database of everything".  1.8 Million people signed the Downing Street Petition against tracking every vehicle movement for road-pricing - clearly this didn't offer enough utility to offset the loss of privacy. But the Motorola representative thinks Emergency protection does.

As I said, we were chatting informally about the Utility/Privacy trade-off and was it acceptable for Windows Live to be a database of everything ? Around the same time, Google's CEO, Eric Schmidt was telling to the press he has grander ambitions in that direction.. To quote the FT he said

Gathering more personal data was a key way for Google to expand and the company believes that is the logical extension of its stated mission to organize the world’s information. Asked how Google might look in five years’ time, Mr Schmidt said: “We are very early in the total information we have within Google. The algorithms will get better and we will get better at personalization. “The goal is to enable Google users to be able to ask the question such as ‘What shall I do tomorrow?’ and ‘What job shall I take?’ " 

Worrying for privacy or great utility ? The next day a piece by Mark Lawson in the Guardian was introduced with the words "Anyone stupid enough to do a computer's bidding is not losing civil liberties so much as their marbles" Over at ZDnet Andrew Keen really had a swing at Eric. He calls him "the Chauncey Gardiner of Silicon Valley" (twice) and "Google’s Chief Eccentric Officer" (also twice) ouch. "Eric" he says "I thought you were a businessman rather than a looney". I remember Eric's time in charge of Novell, so I've got a view on which he is. Andrew's colleague on ZDNet, Donna Bogatin - who posted a summary of my post on Google's stance on T-shirts - calls him "Harmless" with links to explanatory posts.

Plainly I'm not the only one worrying about databases of everything. It doesn't matter who it is. What I wonder, and would love your comments on, is just what privacy will people give up for utility ?

One of the ways to divide I meet is break them into the "Stallers and Installers". I meet plenty of people who have installed are running Windows Vista. No OS is perfect, but those who have made the move to Vista seem to be happier there than they were with XP. The others divide into the very few who tried it and didn't like it, a few more who couldn't make it work with a specific app or device. Device support has come a long way in the last year: the present nVidia drivers work well but the initial ones were simply dreadful. With SP1 just around the corner the time is probably here for those who had a bad experience

What does surprises me is number who haven't even bothered to look at it, and the excuses I hear. I've had people whose companies have bought the right to move up to Vista telling me they're not doing it because of the cost (they're looking at the retail cost for one copy on the high street) without realizing they've paid for it. A couple of people at the BETT show last week said Vista wasn't stable, so they wouldn't try it. Monthly service packs cause my laptop to reboot about every other month; it goes into sleep or hibernate a couple of times a day but it's been an age since I initiated a reboot. Outlook and Internet explorer hardly ever close - I have 35 pages open in one instance of IE right now and 28 windows open. This workload would bring XP to it's knees. Unstable indeed !

Share this:

In my quest trying to find something for an earlier post, I found this image which was too good not to share.

In the Credits was a link to Janina Köppel's Excellent SP-Studio site. It's done in flash, which proves that although flash is often put to bad uses, that's not always the case. 

We had a lot of fun at home doing members of the family, characters from Robin Hood, and finally other our the team - and that includes George while we're on the Road (her picture is rather too flattering after the one she posted of me).  

Last year I was trying to draw some cartoons for this blog ... Now at the weekend I was looking through a book of Banksy's work, in which he poses the question "Many artists are prepared to suffer, but why are so few prepared to learn to draw ?" This might see the cartoons making a comeback

*Thames Valley Park is the address of Microsoft in the UK

Aargh. I just broke one of the cardinal rules. Never say anything you don't beleive to be true.

I love to say "I told you so".

I read Chris Case's break-up note to his I-phone with great amusment.

I wonder if Ben (who has left a void in a village somewhere to become my blog stalker) will berate him for the violence of saying "that it's best to sever ties now while we still both have our dignity and I have not smashed you on the sidewalk in frustration"

We're two dates into our roadshow and I've twice been asked to do a comparison of VMware and Microsoft in the high availability area.

So lets go back to basics a second. Microsoft is involved in lots of areas of software, covering: Operating Systems, several different kinds of Virtualization, server Applications and Management software (a lot of customers are keen to Manage VMware with System Center Virtual Machine Manager - which is probably worth it's own post). We've got a history with high availability. Back in OS/2 LanManager days we had domains where any one of several machines could validate a logon. When we introduced WINS and DHCP in NT 3.5 we supported multiple servers being able deliver the same service to the same client. We have Network Load Balancing - Office Communications Server is designed to leverage it, and IIS in server 2008 is designed to play better with it. We introduced fail over clustering 10 years or so ago, and we're up to our 4th generation of it with Server 2008. Exchange, SQL, file shares and virtual machines can all be clustered. Clustering at the application level is THE only way to provide high availability over a wide range of problems. If the hardware fails, if the OS running the server application fails, if the application itself fails... application level clustering saves the day. If an application is critical of itself and can be clustered there is no excuse for not clustering it. 

We see the main task of Hyper-V Servers as running a reasonably static collection of Server workloads. That's not to say workloads never move between servers: but they tend to stay put. It's not to say we never run client workloads using Virtualization; but usually Terminal Services is a better way to run many identical "virtual desktops"  Running many clients as VMs has a much bigger disk, memory and CPU overhead: but in some cases it is still the best way to go. Companies who can sell you the same solution based on Terminal Services, or Client OS virtualization (ourselves or Citrix) will tend to go the TS route: patching and application deployment is simpler that way too. VMware don't offer that choice.

I talked about applications which are critical of themselves: over on the virtualization blog Jeff talked about consolidating applications which aren't critical individually, but move 5, 10, 20 such apps onto one server and that server becomes critical. If it fails unexpectedly your job's on the line. So, to allow VMs to live on shared storage and be failed over to another machine, VMware have their "HA" option and we use the clustering of Enterprise/Datacenter builds of Windows  A by-product of clustering is the ability to migrate VMs from one box to another - this is quick but not "live" it involves a brief interruption of service. 

This is the area where VMware have their major differentiator, VMotion. We know that some customers want to be able to move machines around with no downtime, and we've talked about it for a future version. I want to avoid getting into any criticism of the feature itself - with Microsoft not having it today that would have the tang of sour grapes to it. I don't think it is controversial to say VMware's software costs substantially more  than Microsoft's nearest equivalent; to stay in business they need to offer features which justify that cost. VMotion is just such a feature, the problem is that VMotion is touted as the cure for all ills: which it is isn't. It lets you say "Move this machine", it copies the machine's memory to another host and switches over in under a second. But VMotion doesn't help with unplanned downtime (Jeff gives chapter and verse on VMware's HA document here). So Vmotion helps with planned downtime - patching or upgrading the host. As Jeff points out in a third post we think most customers - even the ones who  have a live migration solution still warn people the system will go down and do the upgrade during off hours. If both host and guest are running Windows there is the possibility to patch the guests and take them down, patch the host, and then bring everything back up together.

One other thing about VMware's approach is that they make a feature of "sweating" the hardware to a higher level than we do - whether the workloads are client or server ones (See the argument about over-committing memory ). This means dynamically allocating resources and being able to move VMs from an overloaded box to an underloaded one. It's really a kind of "grid" computing where the workloads (VMs) float from host to host, cost makes it necessary and VMotion makes that possible. In the Microsoft world we tend to say spend the money you save from cheaper software on more hardware, so you don't have to sweat it as much; and workloads don't need to hop from box to box as frequently.

A couple of Stories have been doing the rounds on our internal Virtualization discussions. One was headed "HOLY COW HYPER-V VIRTUALIZING MICROSOFT.COM!!!!!!" (and before anyone wonders if this is breaking something internal to the world, it's already been described in detail on by Rob Emanuel on the Windows Server blog ). The MS.COM operations team have also produced an article on how they virtualized Technet and MSDN

Now... Microsoft.com is not your average home page. The statistics are staggering: 1.2 Billion hits per month, on average that's over 4,000 every second, but at busy times it peaks at 4 times that. Its 7 million pages take up 300GB of space. It used to need 80 servers to deliver it, but the Ops team migrated it to Server 2008 and newer hardware and saw the opportunity to reduce the number of servers. What became apparent with modern CPUs and RAM sizes, the servers were disk bound Simply throwing more CPU cores and more RAM at the servers wasn't going to reduce the number of boxes needed. Redesigning the site so it used more disk spindles would help - but the quickest win would be to take bigger servers and use Hyper-V to partition them, with each Virtual Machine getting it's own disks that would double the the number of disk IOs available without breaking the site into parts on different disks. But could they use virtualization with out Hyper-V itself becoming the bottleneck. Deploying new servers into the array is involves sync'ing 7 million pages: would virtualizing the servers - even if they ran one VM per box - help deployment ? Even if Hyper-V could scale and wasn't a drag on management and deployment , would it be reliable ?  And would running one Mega site on Hyper-V give the Microsoft.Com folks confidence to consolidate some of the smaller machines they mange. ... incidentally Blogs.technet.com where this page is hosted is run for us by a third party.

If you read the post you find the detail behind why the answers to all these questions turns out to be yes.

I think (at least in the short term) most of the deployments of Hyper-V will be consolidating 5-20 servers into a single box.  It's perfectly capable of running many more VMs than that - indeed we demonstrated hundreds of VMs on the old Virtual Server product - (more than VMware will support) but my own view is that the typical VM requirement, and the typical hardware capacity leads to a typical ratio of 10:1 (it could be 8, or 12 but I'm using rough orders of magnitude here) and the greatest most deployments will fall within half and double that. That's not scientific, but that's how I get to my own "gut-feel". I say servers, because I'm not a great believer in virtualizing the desktop OS - a thin client with a fatter server running your desktop as a VM doesn't reduce hardware costs compared with, rich client and skinnier server architecture. It doesn't reduce power consumption (in fact it probably increases power and A/C costs) and delivers an inferior service; don't try video conferencing company events, or deploying a Voice technology from the the PC. Don't try working off line either. Yet it has the management and licensing overhead of having many machines.  If there isn't really a requirement for a PC , just one or two PC applications, then a terminal Service way of working is usually better.

So, I see Hyper-V most running servers and this case of running a single workload on under Hyper-V - even running multiple identical instances of the same workload is unusual. But if anyone tries to tell you Hyper-V doesn't scale to take on the biggest workloads... well you know different.

In the same vein, QLogic announced Hyper-V can do 180,000 IOPs. That's not a typo. It's vast number of I/O operations per second. In fact some people find it unbelievable, Chris Wolf posted a critique of the test on his blog , the comments are interesting and I felt the need to join in. Chris actually sent me a nice mail afterwards, so I'll repost what I said on in my comment.

The purpose of this benchmark is to prove - if it can be proved - that Hyper-V is not an I/O bottle neck. I read the numbers and said "What the hell kind of system can do 200,000 IOPs per second" it was plainly not the kind of system which is going be installed in many environments. It allows Microsoft people to shout "B.S." at the top of their lungs if anyone from VMware claims to have drivers which are much better than Windows ones. It also kills any suggestion that Hyper-v and Windows drivers are OK in small systems but don't scale.

You're right that if a Microsoft benchmark says "runs at 90% of the speed of RAW hardware" the intelligent question to ask is "is that better, worse or about the same as the competition". Is it "Faster than any previous benchmark on virtulization" because it got a better percentage of the hardware or because it kept on scaling when the hardware improved ? Either would be a win for Microsoft. Just saying "ya boo sucks ... we're faster than you " isn't.

Would VMware spread disinformation ? Sure they would. These are the people who can title a section "VMware ESXi – The Most Advanced Hypervisor" and in the very next sentence say "VMware ESXi 3.5 is the latest generation of the bare-metal x86 hypervisor that VMware pioneered and introduced over seven years ago.". So a design that's more than 7 years old and wasn't designed to exploit the latest Intel and AMD technology is also the most advanced ? These are the people who can claim "Many VMware ESX customers have achieved uptimes of more than 1,000 days without reboots." which is pretty remarkable when you look at impendent analysis of VMware's patch history. (follow the link and you'll find a quoted interval of every 19 days... 50 sets of missed patches ! Don't tell the boss). 

The Xen and Microsoft architectures rely on routing all virtual machine I/O to generic drivers installed in the Linux or Windows OS in the hypervisor’s management partition. These generic drivers can be overtaxed easily by the activity of multiple virtual machines

When I challenged VMware to find a customer who was in production with the over-commit ratios they claimed, they could only produce one who was thinking about it. So I think I don't think I'm being unfair in calling it BS. Interestingly the post I linked to above repeats that claim.  So I really don't feel bad calling their pronouncement on drivers BS. (I'll wait to see if someone from VMware comes up with a reason why the sum of activity many small VMs is different from one big one. )

I think everyone knows the saying an optimist says the glass is half full, and a pessimist says it is half empty.

• A cynic says it doesn't matter what you call it, it still contains the same amount.
• A PR person will tell you it matters a lot whether people think is it half full or half empty.
• An Engineer will tell you the glass is twice as big as it needs to be
• A Sales person will try to persuade you to re-fill the glass
• A Consultant will tell you how much you have in your glass.
• An Evangelist will tell you if you looked from a different...

I've been doing the evangelist job for a year now. I spent 6 years working in Microsoft Consulting Services. For most of 1990s I wanted to be in consulting and I'd have loved to work for Microsoft, so it should have been my dream job. Eventually I had to admit that I was a Microsoft person but not an MCS person. There's nothing wrong with MCS, and there's nothing wrong with me, we just don't go together well. The culture is different working in evangelism, and it suits me very well – though my old friends who are MCS people wouldn't like it.

This week I've been doing some telephone interviews for people wanting to join the team and that's had me thinking about the good and bad of working here. I've mentioned this list before and I'll share it now for the benefit of anyone who is thinking of going to http://www.microsoft.com/uk/careers/ - life at Microsoft isn't perfect, but we have more than half a glass.

Bonus Link

You might have picked up from the last point and a post I made earlier this week that I'm not a big fan of "process". When I meet some which is ridiculous I want to make seem all the more so. Last night in on the drive home I heard Mark Thomas on the Radio. Mark decided to poke fun at the "Serious Organised Crime and Police Act" – or strictly it's definition of what needed to go through the process of getting a permit to hold a demonstration. Rather than break the law Mark decided to apply for authorization for a lot of demonstrations. On being presented with 20 applications the Police officer who handles permit applications complained about the work he had to do "I share your pain" said Thomas "look at the first cause I'm demonstrating for." The office turned to the application "Cut police Paperwork".
One of several places in the story I nearly drove off the road. If you have Real Player installed you can play the whole programme from the BBC website.

A few minutes before yesterday's race in the Brazil began, I had a bad feeling about how the Formula one world championship would end. I know my F1 history (Recently I correctly dated each car in a display of F1 cars from the 1980s and 90s:  to the shock of some colleagues who were with me, and the chap who was showing us the cars). When ITVs commentator said this was the first time since 1986 that 3 drivers had a chance at the last race, I knew that was the year that Englishman Nigel Mansel went to the last race of the season expected to come home as champion. Over the season his team had insisted they were being fair to him and his team mate - Nelson Piquet, who had been champion twice before and expected number one status. (Some also suspected that Honda who were backing the team wanted Piquet.) Would Piquet snatch it ? In the end a failure on Mansel's car put him out of contention, Piquet couldn't do enough to take the title and Alain Prost came through and won.

Change the names and 2007 was the year Englishman Lewis Hamilton went to the last race of the season expected to come home as Champion. Over the Season his team had insisted they were being fair to him and his team mate -Fernando Alonso, who had been champion twice before and expected number one status. (would McLaren sponsor Banco Santander be happier seeing a Spaniard win ?) Would Alonso snatch it ? In the end a failure on Hamilton's car put him out of contention, Alonso couldn't do enough to take the title and Raikkonen came through and won.

With [a] England's football team looking like they won't make it to the next European finals and [b] the Rugby team proving that whilst you can beat a better team, but not if you give them more chances to kick penalties than they give you (ignoring officials who take very strange views) it made up a very nasty hat-trick, as a couple of this morning's papers have pointed out.

Since I mention officials, there is a ghost of chance that cars ahead of Hamilton will be disqualified - since both Williams and BMW had chilled their fuel to an illegal degree. Of course that would require the FIA to do something which worked against Ferrari. Since they didn't take points away when Ferrari were found to have run an illegal car in he first race the chances of that happening is infinitesimal. Indeed whilst I can name every world champion back to 1968 from memory, I can't recall the FIA ever punishing Ferrari, please post a comment if you can.  Pete at demotivate.org sent me a link to his Ferrari parody T-shirt after my last post about the organization lots of people think of as "Ferrari International Assistance". Looks like I might end up buying one.

I’m taking a breather from re-recording the voice track for a Video on Live Migration in Hyper-V. When it’s done it will end up on YouTube.

Now YouTube is giving me pause right now: it is certainly the easiest way to put up videos so that people can find them. I’m viewing it as an experiment because anecdotal evidence suggests that the audience I’m trying to reach (IT Professionals) don’t really think of it as a good source of content. And also because you-tube is all very well for bits of fun (like this one or these ) but if you have a serious message isn’t it a bit needy ?

Consider the case, for example, that you have had things pretty much your own way for years, but now it seems everyone who hasn’t deserted you yet is flirting with the other side, the people who pay your wages are even beginning to question the money they pay you. This may sound like Gordon Brown’s attempts to woo the public – although derided by his colleagues* at least he has some more control over YouTube than press photos - but in fact I was referring to VMware.

There’s a video on You tube which starts with a factual error about Technet and MSDN. First off, Technet and MSDN themselves didn’t fail, the failure was in the download site for Windows 7, Server 2008 R2 and Hyper-V Server R2. Demand for these greatly outstripped predictions – there just wasn’t enough hardware capacity, as explained here. I don’t have the stats on how many of the downloads were for Hyper-V server R2 or people wanting to test Hyper-V on Windows Server 2008 R2, we’d been averaging in excess of 100,000 downloads per month of the first release of Hyper-V. Live migration was missing from that release (the main reason that customers chose VMware) but its in R2 - even with the free Hyper-V server product (hence my video). These must be scary times at VMware…  but I digress.

Last year I ripped a hole in VMware’s dishonest pricing examples, this year one someone thought it would be a wheeze to post footage of Hyper-V blue screening on YouTube. He kept quiet about who he was, but it didn’t take long for Virtualization Review to reveal “the root cause of the VMware FUD: Scott Drummonds.”  as they go on to say “his job basically is to look at the competition and spread the word about VMware superiority. Unfortunately, Drummonds doesn't identify himself on the Hyper-V crashing video. Why not? Cynics might say because the video would have less impact if they knew it came from Microsoft's chief virtualization competitor”.

Drummonds confesses he made the post and gives some blather about using two virtual disks (in VMware IDE performance isn’t much good, so they run the test from SCSI disks. In Hyper-V IDE performance matches SCSI yet they wanted to run the test from SCSI disks, which conveys a degree of ignorance of hyper-V and a lack of scientific method - what effect does doubling the number of disks  have on the validity of the tests ?). 

Jeff Woolsey demolishes the Video – since it showed the STOP error at the blue screen he went digging and found from our 750,000 downloads of hyper-v “we've had 3 reports of crashes under stress and with the same error code as seen in the video bugcheck (0x00020001). The solution in all three cases was to upgrade the server BIOS which solved the problem”. VMware have seen similar things incidentally. I love a good demolition, so Jeff’s follow-up post makes good reading; in particular he points out that to have any merit a test has to be repeatable, there’s no published methodology, no statement of what is in the VMs being tested, what the hardware was etc. Jeff points out that VMware prohibit publication of benchmarks unless they have approved the way in which they are carried out, because as they put it Benchmarking is a difficult process fraught with error and complexity at every turn. It's important for those attempting to analyze performance of systems to understand what they're doing to avoid drawing the wrong conclusions or allowing their readers to do so." 

As the question of disks made clear they don’t understand they are doing with hyper-v  and anyone doing a serious test for publication would put in a call to Microsoft support and get a problem like this solved. Who says “A-ha ! a problem with the competitor let’s not try to fix, just video it and put it on YouTube.” …  well Scott Drummonds, obviously. But you can decide for yourself if VMware – at least Scott – were allowing their readers to draw the wrong conclusions or deliberately leading them astray.

Oh and Scott, if you’re reading, anyone with a good knowledge of testing windows will tell you that SlMgr.vbs –rearm will stop that “You may a victim of counterfeiting” message spoiling your videos.

* I had to go to the source of Hazel Blears’ comment “You tube if you want to” because it sounded like there was something missing it’s obviously meant to echo the famous Thatcher quote “To those expecting a u-turn I say You turn if you want to … the lady’s not for turning”. The full quote is actually “I’m not against new media. YouTube if you want to. But it’s not substitute for knocking on doors”.

Update. Fixed some typos and bad edits.

Information week published an article last week “9 Reasons why enterprises shouldn’t switch to hyper-v”. The Author is Elias Khnaser, this is his website and this is the company he works for.  A few people have taken him to task over it, including Aidan . I’ve covered all the points he made, most of which seem to have come the VMwares bumper book of FUD, but I wanted to start with one point which I hadn’t seen before. 

Live migration. Elias talked of “an infrastructure that would cause me to spend more time in front of my management console waiting for live migration to migrate 40 VMs from one host to another, ONE AT A TIME.” and claimed it “would take an administrator double or triple the time it would an ESX admin just to move VMs from host to host”.  Posting a comment to the original piece he went off the deep end replying to Justin’s comments , saying “Live Migration you can migrate 40 VMs if nothing is happening? Listen, I really have no time to sit here trying to educate you as a reply like this on the live migration is just a mockery. Son, Hyper-v supports 1 live VM migration at a time.” . Now this does at least start with a fact : Hyper-V only allows one VM to be in flight on a given node at any moment: but you can issue one command and it moves all the hyper-v VMs between nodes. Here’s the PowerShell command that does it.
Get-ClusterNode -Name grommit-r2 | Get-ClusterGroup |
  where-object { Get-ClusterResource -input $_ | where {$_.resourcetype -like "Virtual Machine*"}} |
     Move-ClusterVirtualMachineRole -Node wallace-r2             
The video shows it in action with 2 VMs but it could just as easily be 200.  The only people who would “spend more time in front of [a] management console” are those who are not up to speed with Windows Clustering. System Center will sequence moves for you as well. But… does it matter if the VMs are migrated in series or in parallel ?  If you have a mesh of Network connections between cluster nodes you could be copying to 2 nodes of two networks with the parallel method, but if you don’t (and most clusters don’t) then n copies will go at 1/n the speed of a single copy. Surely if you have 40VMs an they take a minute to move it takes 40 minutes either way…  right ? Well no... Let’s use some rounded numbers for illustration only: say 55 seconds of the minute is doing the initial copy of memory, 4 seconds doing the second pass copy of memory pages which changed in that 55 seconds, and 1 second doing the 3rd pass copy and handshaking. Then Hyper-V moves onto the next VM and the process repeats 40 times. What happens with 40 copies in parallel ? Somewhere in 37th minute the first pass copies complete - none of the VMs have moved to their new node yet. Now: if 4 seconds worth changed in 55 seconds – that’s about 7% of all the pages - what percentage will have changed in 36 minutes ?  Some won’t change from hour to hour and others change from second to second – how many actually change in 55 seconds or  36 minutes or any other length of time depends on the work being done at that point, and the memory size and will be enormously variable. However the extreme points are clear (a) In the very best case no memory changes and the parallel copy takes as long as the sequential. In all other cases it takes longer (b) In the worst case scenario the second pass has to copy everything – when that happens the migration will never complete.  

Breadth of OS support. In Microsoft-speak “supported”  means a support incident can go to the point of issuing a hot-fix if need be. Not supported doesn’t mean non-cooperation if you need help – but the support people can’t make the same guarantee of a resolution. By that definition, we don’t “support” any other companies’ software – they provide hot-fixes, not us - but we do have arrangements with some vendors so a customer can open a support case and have it handed on to Microsoft or handed on by Microsoft as a single incident. We have those arrangements with Novell for Suse Linux and Red Hat for RHEL, and it’s reasonable to think we are negotiating arrangements for more platforms: those who know what is likely to be announced in future won’t identify which platforms to avoid prejudicing the process. In VMware-speak “supported”, has a different meaning. In their terms NT4 is “Supported”. NT4 works on HyperV but without hot-fixes for NT4 it’s not “Supported”. If NT4 is supported on VMware and not on Hyper-V exactly how is a customer better off ? Comparisons using different definitions of “support” are meaningless. “Such and Such an OS works on ESX / Vsphere but fails on Hyper-V” or “Vendor X works with VMware but not with Microsoft” allows the customer can say “so what” or “That’s a deal-breaker”.

Security.  Was it hyper-v that had the vulnerability which let VMs break out of into the host partition ? No that was VMware. Elias commented that "You had some time to patch before the exploit hit all your servers" which makes me worry about his understanding of network worms. He also brings up the discredited disk footprint argument; that is based on the fallacy that every Megabyte of  code is equally prone to vulnerabilities, Jeff sank that one months ago and pretty comprehensively – the patch record  shows a little code from VMware has more flaws than a lot of code of Microsoft’s.

Memory over-commit. Vmware's advice is don't do it. Deceiving a virtualized OS about the amount of memory at its disposal means it makes bad decisions about what to bring into memory - with the virtualization layer paging blindly - not knowing what needs to be in memory and what doesn’t. That means you must size your hardware for more disk operations, and still accept worse performance. Elias writes about using oversubscription, “to power-on VMs when a host experiences hardware failure”. In other words the VMs fail over to another host which is already at capacity and oversubscription magically makes the extra capacity you need. We’d design things with a node’s worth of unused memory (and CPU , Network, and Disk IOps ) in the other node[s] of the cluster. VMware will cite their ability to share memory pages, but this doesn’t scale well to very large memory systems (more pages to compare), and to work you must not have [1] large amounts of data in memory in the VMs (the data will be different in each), or [2]  OSes which support entry point randomization (Vista, Win7, Server 2008/2008-R2) or [3] heterogeneous operating systems. Back in March 2008 I showed how a Hyper-v solution was more cost effective if you spent some of the extra cost of buying VMware on memory – in fact I showed the maths underneath it and how under limited circumstances VMware could come out better. Advocates for VMware [Elias included] say buying VMware buys greater VM density: the same amount spent on RAM buys even-greater density. The VMware case is always based on a fixed amount of memory in the server: as I said back then, either you want to run [a number of] VMs on the box, or the budget per box is [a number] Who ever yelled "Screw the budget, Screw the workload. Keep the memory constant !" ? The flaw in that argument is more pronounced now than it was when I first pointed it out as the amount of RAM you get for the price of VMware has increased.

Hot add memory. Hyper-v only does hot-add of disk, not memory. Some guest OSes won’t support it at all. Is it an operation which justifies the extra cost of VMware ? . 

Priority restart - Elias describes a situation where all the domain controllers / DNS servers on are one host. In my days in Microsoft Consulting Services reviewing designs customers had in front of them, I would have condemned a design which did that, and asked some tough questions of whoever proposed it.  It takes scripting (or very conservative start-up timeouts) in Hyper-V to manage this. I don’t know enough of the feature in VMware to know how sequences things not based on the OS running but all the services being ready to respond. 

Fault tolerance. VMware can offer parallel running - with serious restrictions. Hyper-v needs 3rd party products (Marathon) to match that.  What this saves is the downtime to restart the VM after an unforeseen hardware failure. It’s no help with software failures if the app crashes, or the OS in the VM crashes, then both instances crash identically. Clustering at the application level is the only way to guarantee high levels of service: how else do you cope with patching the OS in the VM or the application itself ?     

Maturity: If you have a new competitor show up in your market, you tell people how long you have been around. But what is the advantage in VMware’s case ? Shouldn’t age give rise to wisdom, the kind of wisdom which stops you shipping Updates which cause High Availability VMs to unexpectedly reboot, or shipping beta time-bomb code in a release product. It’s an interesting debating point whether VMware had that Wisdom and lost it – if so they have passed through maturity and reached senility.

 Third Party vendor support. Here’s a photo. At a meet-the-suppliers event one of our customers put on, they had us next to VMware. Notice we’ve got System Center Virtual Machine manager on our stand, running in VM, managing two other hyper-V hosts which happen to be clustered, but the lack of traffic at the VMware stand allows us to see they weren’t showing any software – a full demo of our latest and greatest needs 3 laptops, and theirs ? Well the choice of hardware is a bit limiting. There is a huge range of management products to augment Windows – indeed the whole reason for bring System Center in is that it manages hardware, Virtualization (including VMware) and Virtualized workloads. When Elias talks of 3rd party vendors I think he means people like him – and that would mean he’s saying you should buy VMware because that’s what he sells.

The last person I punched was a schoolboy. Before you report me for child cruelty, I should point out that I was a schoolboy myself at the time, and that makes it at least a quarter of a century ago. As befits someone who went on to work in IT punching people was something I wasn't very good at, which may explain this long period of non violence. Only the presence of some very large policemen, a timely phone call and a man called Rod Green prevented that record coming to an end on Friday night.

Over the next week or so Microsoft employees descend on Seattle for the "Tech ready". This brings me back into  contact with the Microsoft Travel portal. Steve has talked about,and I I said of it "this tool is to Web Usability what Scrapheap Challenge (or Junkyard wars in the US) is to automotive styling". To add to its other faults I discovered it dies if the IE7-Pro add in is installed. I don't know what acceptance tests we did on it but it won't book travel to Seattle.

So I called the people I know as "American Excuse",  The agent asked if I was going to tech-ready "We can't book that  you have to e-mail another office". So I did that. They sent me a form to complete with my contact details (all of which are in my e-mail signature) and booked me on Friday's  BA flight, I'd rather fly SAS but hey.

I decided to leave my car at home and take the train to the airport. But on Friday we had more rain than even a British summer is used to. Seconds after my wife left me at Didcot station I found that the subway leading to the London bound platform was flooded. Station staff, were using a wheel chair to ferry passengers from the last train which was allowed to stop through the floods. Passengers for London were told to go 25 miles the other way change trains at Swindon and come back. But by the time I'd traveled 50 miles to avoid what was basically a large puddle the rest of the journey into London was looking a uncertain. I told one of the staff: if their use of the wheelchair was remarkable, then the response to this was astonishing. "if you hold on a moment while we sort out a form, we'll pop you in a Taxi". Major customer service brownie points for First Great Western.

Two minutes into the taxi ride my phone buzzed; I love Exchange push mail and Windows Mobile 6.  The message read.

From: British Airways Customer Services 
To: James O'Neill 
Sent: 20 July 2007 13:37
Subject: Cancellation - BA0053 LHR to SEA on 20 Jul 2007 : ref [deleted] 
 
Dear Customer,
We regret to inform you that flight BA0053 from Heathrow (London) 
to Seattle on 20 Jul 2007 has been cancelled.
Please accept our apologies for any inconvenience this may cause.
We have sent you this information as quickly as possible by automated 
email and hope it reaches you in time to allow alternative arrangements to be made.
Please do not respond to this email as we will be unable to reply.
It may be possible to view your options and rebook or cancel your flights on ba.com. 
To check please click the link below:
http://www.ba.com/mmb
Alternatively, please call us on:
- United Kingdom 0800 727 800
We suggest you also contact your booking agent for any further changes to your travel plans.

We park the taxi while I call BA - busy.  Call the Microsoft travel emergency number - a recorded message tells me to it's an out of hours only, and to redial the normal number. I do (why not do the re-routing for me ?). "Oh" says the agent "We can't access that booking from here You need to call the group people", at which point we get cut off. I use the Search Server feature of Windows Mobile 6 to find the mail with their number-it's not in a dialable format.  Mobile 6 - unlike my previous phone - lets me fix it. The agent at the group bookings place promises to ring BA right away and get back to me. Twenty minutes later I call again. Another agent says they've compiled a list of people on the same flight and will be calling BA soon. I try to access the BA's web site from my phone but it never leaves a "please wait" screen. The taxi driver and I decide to head for Heathrow airport even if we end up turning round and coming home. I forward the mail to my wife who tries to rebook the flight on BA's web site. Errors Occur. The travel agent calls back and says they can't rebook my flight (why not ?), and to continue to the airport. They suggest rebooking my flight at Terminal 1 rather than 4 where I was due to depart. 

The situation at T1 merited the description of  "a large crisis" hours before I got there:141 flights canceled, with BA hardest hit. The rebooking queue snakes round the terminal, and most of the way along the link to Terminal 2. Heathrow's own website has some harsh things to say T1 and T2 so chose to wait in T4 - though it turned our I'd be waiting for 5 hours in a tent outside; things are so bad so often that BA have two Marquees permanently set up outside. My phone's call history tells me the call I made when I joined the queue was at 15:55. Its Internet connection sharing let me try the BA web site from my laptop, it can't handle substitute routes (via New York, Chicago, Atlanta etc)  never mind a change of airline are beyond it,and direct flights are all full. I try BA's executive club who answer and sound helpful, but can't change route either. Aargh. An American next to me needs to be in Paris, and has no cell phone - so we get onto Eurostar to confirm the trains are running. It's 16:45 and we work out that he can probably get to Paris by 20:20 London time - before he'll be at the front of the queue.

I mail the cancellation note to Steve who blogs about things to do if stuck at Heathrow.  These assume you are checked in and not stuck outside in big tent. Things get so bad that BA bring us some sandwiches; and people eat them. I don't want to talk to anyone round me so I start on the book I've brought  and plug my headphones into the phone and put some music on. I can't help laughing when Monty Python's "Always look on the bright side of life" comes on.

I'm not laughing at 21:10. Two of the metropolitan Police's biggest firearms officers stroll through our tent as nonchalantly as is possible when you have a semi-automatic weapon strapped across your chest. Someone from the airport (not BA) announces that after 5 hours 15 minutes waiting, we won't be "processed", because the BA staff want to go home.  I shout at him - why didn't someone figure that out hours ago and save hundreds of us some of the wait. He doesn't give me a proper answer and I want to punch him.  He's only the messenger. I bet those policemen were sent by an Inspector who's seen this before. "Remind 'em that you're their to provide protection from passengers as well as protection of passengers". Rage rolls in and I know why people call it a red mist. What I want most in the world is to punch someone from BA, I'm imagining the punch (a right hook) I wonder if other passengers will join in. The policemen make me hesitate- I picture myself defying them, and more (only later do I realize getting into a fight while traveling on company business could get me fired). Then the phone rings and the moment for punching people is gone.

I hang up the phone call and find myself asking a BA chap who I complain to - they didn't have to wait so long before telling us they were going to stop processing. The man's name badge bears the name Rod Green. He's volunteered to come down from the BA office to do what he can. 20 or 30 like him could have made a noticeable difference in the last 5 hours. They've had half a dozen. Rod makes a phone call, and finds me a substitute flight which works. If he worked for Microsoft I'd get him a customer service award.

BA has a problem. They have plenty of people who could do what Rod did, but when they needed them to show some passion for customer service, all but a very few went home.

Now I had to get home. Avis have earned my loyalty with a couple of great bits of customer service, their people really have justified their "We try harder" badges. A sign at their desk said get straight on the courtesy bus... Across the airport - a customer is shouting at an Avis rep who is trying to stay calm."I'm not arguing with you" he says, "I'm the customer. You have to argue with me" comes the reply. The lady dealing with me makes that slight sucking sound when I say I haven't reserved a car. I know that noise - and I picture myself losing it - in the style of Steve Martin in the film "Planes Trains and Automobiles". Then she said "I can't offer you much choice...". And for the first time since "Always look on the bright side of life" I smile.

I thread my way home avoiding places I know are flooded.The neighbouring village to mine was badly hit, with the fire brigade pumping out the pub. We had our drains rebuilt after being flooded with "foul water" twice around 2000, and the system held up. It only takes a small amount of water to cause chaos. I draft this post and go to sleep. In the morning I read Hugh has been flooded; grim. We had clean-up people out the next day, but they're going to maxed out for a while.

.