Michael Platt's WebLog : Stories

Business Rules

Michael Platt — Sun, 04 Jul 2004 02:04:00 GMT

Nowadays there is a lot of pressure on the middle tier of the three tier stack from both the database and the smart client. If technologies such as Yukon allow all the code to run in the databases is the business tier of logic required? Whilst the actual physical implementation of the business tier as a separate element may not be necessary the functionality of the business logic is vital as this business logic provides key data validation and business rules checking. I was given a graphic illustration of the importance of business rules checking when I first started work many moons ago.

I was doing chip design for IBM and wanted a chip carrier to put the chip in for testing when it came back from manufacturing. These were large and complex chips with many pins and so required a fairly specialist chip carrier which cost about $50. The carrier itself was only about 1 inch square so was a pretty small item; we did not have any in the lab and so I ordered a couple. IBM at the time (and may still have for all I know) had 10 digit part numbers for everything so I dutifully looked up the part number, filled in the Purchase Order and sent it off to purchasing.

As was usual nothing happened for about a month and then I got a phone call one day from goods inwards. They said that one of the parts had been delivered but as they had only had one in stock in the factory the other was on back order until they started up the production line again. I said I would be down to pick up the part and the dispatcher said “Ok, but can you get down here soon as it’s blocking the loading bay”. This rang alarm bells as the loading bay at an IBM plant is huge, being able to take up to 10 large trucks simultaneously. It was probably 150 feet long, so it was unlikely a 1 inch chip carrier was going to block the whole bay! I said I would be down immediately and the dispatcher said “You had better bring the heavy gang”. The heavy gang were the group that moved major items of plant and machinery such as big compressors or major cooling systems, again it seemed unlikely that they would be needed for a part that I anticipated being able to put in my pocket.

So say that I ran down to goods inwards with trepidation would be an understatement, there had obviously been a major foul up. Sure enough when I got into the loading bay there was a huge wooden packing case about 30 feet by 10 by 10 blocking a vast area, blotting out the light from outside. I gulped and found the dispatcher who pointed to the case and said “There it is”. Visions of my managers face and my intemperate departure from IBM floated before my face. “What the **** is that” I asked. The dispatcher checked his paperwork and said “it’s an MG set”. Worse and worse, MG sets are the combined electric motor, generator, water pump and distribution control system for a very large water cooled mainframe and cost about $500,000 each. That was going to put a dent in the department budget and how was I going to hide it! I grabbed the paperwork from the dispatcher and ran back to my office with my heart in my mouth and his cries of “You can’t leave that here!” floating over my shoulder.

Back in the office I scrabbled through the old paperwork in my drawer and found the original purchase order with trembling hands. Please oh please let the part number be correct! I am very bad at these sort of details so the chances were that it would not be and that would be the end of my career. Alleluia! My original Purchase Order had the correct part number and the dispatcher’s paperwork had one digit in the ten digit part number wrong, hence ordering a 30x10 foot $500,000 MG set rather than a 1X1 inch $50 chip carrier. Purchasing must have made a mistake when entering the details from the Order.

A very quick phone call to the purchasing department to confirm, sure enough they had the correct details on my paperwork but the wrong number in the ordering system. First thing that they said was “Oh, we had better stop them setting up that manufacturing line in France then” The next thing was to ask me if I could take delivery anyway as it was in goods inwards. Uncontrollable and rather hysterical laughter from me gave them the answer to that one! I could just see me asking my manger to sign off a $500K mistake! Anyway I pointed out that it was their problem and they had better sort it out before I put the phone down pronto. I never heard any more about what happened to the MG set but I made do without the chip carrier, I never had the nerve to reorder it.

Anyway the moral of the story was that if the system had business rules checking such as does the order value exceed the budget of the group or any other criteria then this would not have happened. Nowadays of course all systems do have this sort of checking in place but 30 years ago many didn’t, allowing these sorts of mistakes to occur.

It certainly instilled in me a life long appreciation of the value of business rules checking, hammered home with real terror, the very best teacher!

Electricity as a Managment tool

Michael Platt — Fri, 25 Jun 2004 17:07:00 GMT

Physical phenomenon can not only be used as a valuable architectural training aid but they are also useful for management training, especially handy when trying to free up budgets!

I was talking yesterday about designing power supplies for TV monitors and some of the hazards associated with that. Testing 25Kv power supplies and CRT’s was a hazardous job involving a lot of Perspex, nylon string (to pull switches), rubber gear (not really necessary but when was the last time you got to buy rubber clothing on expenses?!) and standing a long way back (it’s amazing how far a 25Kv spark will jump). To this end we had a big test rig which we could plug the CRT’s into.

I was working with a new, very small CRT tube designed to go into a head up display (very cool for games) on the test rig and got a nasty shock despite the rubber gear. I always had a standard process to go through when something nasty like this happened, starting off with a sit down with a cup of strong tea. After a lot of thought I decided that there was something wrong with the tube itself which was causing a leakage to the test ring and hence to me. Donning even more rubber (!) and returning to the lab I reran the test, sure enough I got another belt. Clearly we needed to get a new CRT. The snag was that this was a hand built prototype which had taken ages to make and cost the earth. Off to see my manager.

The first thing my manager said was: “Why are you dressed like that?” I had forgotten to take off the rubber gear. Next thing, after I explained the problem, was “We can’t afford a new tube, it must be something you are doing wrong” which was an interesting piece of managementese, so breathtaking in its illogicality that it’s hard to argue with. I suggested that he tried the same test himself and so off he went to the lab. I trailed along to see the fun. Sure enough he ran up the test rig, there was a bang and he was thrown back across the lab (the rubber gear was a good idea after all). After I picked him up he said “order a new CRT and hang the budget, I’m going home” which he proceeded to do.

So whilst I wouldn’t recommend electrocuting your management to get a budget increase it certainly works!

Danger as an architectural teaching mechanism

Michael Platt — Thu, 24 Jun 2004 22:01:00 GMT

Yesterday I postulated that an engineering training is valuable in architecture because it teaches cross domain skills, and in some cases reinforces these with physical harm or danger; a very effective teaching aid!

A good example of this was an experience I had soon after I graduated, well before the PC era. I was working on the EHT power supply design for the first ever colour display (an IBM 3279 for the consegnenti) and we were testing the first few units. The design of EHT power supplies providing highly stabilised 25Kv with huge load changes is well understood now but in those days was very challenging. In particular one of the strange things that occurs with CRT’s is that every now and again, for no apparent reason, there is a “flashover” of the 25Kv to the anode pins and hence to the low voltage and digital circuitry. Most chips don’t like 25Kv on their input pins so protection circuitry had to be included and then tested.

This testing of the protection circuits was a problem because we had to have a flashover for the test. The cause of the flashovers was unknown and only happened once a year; clearly waiting for natural ones wasn’t going to work. We built a device called the mousetrap which connected the 25Kv supply to the anode pins but the spike characteristic wasn’t the same as a natural flashover in the CRT itself. We played with lasers (fun!) trying to ionize a path in the CRT but there was not enough gas in the evacuated tube to ionize with the low powered lasers we then had. We were keen to get a mega powered laser, mainly to see what we could burn holes through, but that was blocked by management (whining about budgets!). Clearly we had to come up with another mechanism.

So I had the bright idea of adding some gas to the CRT so that the laser could ionise it. General agreement ensued and so we broke the sealing nipple where the tube had been evacuated with a pair of pliers and immediately slapped some epoxy putty on the hole before too much air rushed in. A highly scientific implementation; hey we were engineers! We then ionised the gas with the laser and applied the 25Kv supply to the cathode.

Ok, so who can see the problem here?

Well what happened next was a small bang followed by a very large implosion of the tube which lifted all the ceiling tiles in the lab and brought people popping out of offices all down the hall like gophers from holes. It would have been funny if we hadn’t frightened ourselves witless; have you ever heard a TV tube implode? Very noisy and a lot of bits of glass too!

So what happened? Have you worked it out? The answer is in the feedback.

Anyway this was a case where some basic thought in the chemistry domain would have avoided an explosion and change of pants in the physical domain. Cross domain analysis driven home with a Bang!

Complexity

Michael Platt — Sat, 21 Feb 2004 08:02:00 GMT

Thinking again about the writing course I was on the other day we were told that one of the most important things you can do to make writing more understandable is to simplify things. I think that is a good principle in general, I have been to architectural reviews where the proposed solution is so complex I doubt anyone could understand it.

I learnt about complexity of systems a long time ago when I was heading up the hardware design team for a very advanced mainframe attached parallel processor for very compute intensive applications. My team completed the hardware design and got it working on schedule and under budget but the software team doing the OS were miles adrift. Eventually they got the code to work but it was huge, some 100K lines on each of the 64 processors (in those days that was a big program!). I have to say that they were very smart guys and good programmers but the OS architecture and implementation did seem overly complex to me.

Anyway we went into integration test and the system kept crashing the mainframe. On investigation we found that it was running so slowly that sometimes the mainframe timed out. I was told it my fault and to improve the hardware performance so I wrung every nanosecond out of the design and doubled the hardware speed but it was nowhere near enough. Everyone kept saying it was the hardware causing the problem so in the end I wrote a hardware test harness which just ran the mainframe interfaces. This ran for a week without a failure so I pointed out it must be the software. I must admit to a certain amount of gloating over this, after all I had been accused as the bad guy for about 6 months. Sure enough the test shut the software guys up a treat however I didn’t think about all the ramifications.

About a week after the completion of the hardware verification test I caught flu and was at home in bed feeling dreadful when the phone went. It was my boss to say that a very important customer to whom the system had been promised had phoned up our VP and given him a roasting about non delivery. This of course had been passed down the management chain at high speed until it reached my boss who decided to fire the software guys and get me to write the code instead! After all I had managed to get it to work in test. I pointed out that I had just written a hardware test harness not a complete multiprocessor, multichannel, parallel, high performance operating. That argument rather went over his head and I got told to get a team together and write the OS in a month! The original estimate had been six months and the experienced software team of ten had been at it for over a year so it seemed like a big challenge to me, especially as I didn’t have any programmers and did have flu. Life can be so like Dilbert.

Anyway I dragged myself into work and got hold of a couple of interns that had been doing some of the hardware test coding for me. I didn’t tell them the magnitude what we were trying to do but explained the architecture of the system. I also got a couple of programmers from the customers who had been doing acceptance testing and roped them in too. We set to work with a beta date two weeks later. After two weeks of intense coding we had the code finished and fired it up. Needless to say it hung big time. Debugging OS’s, especially multiprocessor ones is very difficult; every time you try to measure something it changes the dynamics of the system and so the failure mode. In the end we spent three intense days with the listings, a white board and a lot of coffee and finally cracked the problem. Back to the system, reload the code and it all worked brilliantly. We just had one small glitch so the system hung very occasionally but we knew what that was and how to fix it.

Time was up however and we had to go into customer acceptance test. Sod’s law cut in and the glitch showed up just at the wrong moment. However because we had the customers programmers on the team and they knew we had a handle on the problem we got a sign off to ship.

There was great relief and celebrations all around. The interesting thing was that the code was one tenth the size of the original software and ran 100 times faster, it was just much simpler. We fixed the final glitch and installed the processor in the customers DP centre. My boss got a big promotion and I got a certificate. Someone please tell me this doesn’t always happen!

An interesting footnote to this story happened five years later when out of the blue I got a letter from the customer. They had been keeping availability statistics for all their IT equipment and in the whole of the five years my system had not crashed once, giving over 99.999999% (8 nines) availability. It was the most reliable system they had (including the mainframes) and they wanted to give me an award. When my management heard about this they asked me to write a white paper about architecting highly available multiprocessor operating systems. It was a very short white paper, it just said:

Keep it Simple, Stupid.

Of course it never got published, it was far too simple!

Architectural Infrastructure and Reliability

Michael Platt — Mon, 16 Feb 2004 17:42:00 GMT

I really like architecting highly available systems; I think they are the most architecturally demanding. In fact serious performance and scalability issues often result in poor reliability or availability. The standard way to build a highly available system is to cluster however I prefer to start with a queued approach and then cluster as appropriate. The nice thing about this approach is that if something does fail then the system will restart without data or transaction loss.

To give an example of this the European Football system I talked about earlier was built with a transactional application but of course the infrastructure it ran upon was critical to the reliability of the system. As I mentioned there were about 10 servers per city and 10 cities in the system. Each server had dual Lans and was clustered, additionally there were dual WAN’s to each city and the central system (a Digital Alpha) was also clustered with a hot standby (it was a hub and spoke model system).

Basically everything was duplicated everywhere and so when I insisted on queues between all the systems the design team thought I was mad. I made lots of threats about what would happen to them if the system failed and so eventually they included queues between all the servers.

On the day of the cup final when the systems had to be running it was very stormy (nothing unusual for UK weather) however we were more interested in the system performance than the match conditions.

Monitoring from the central hub the systems were all running perfectly when we suddenly lost all the communications links! Panic! We picked up the phone but that was out too. Using a mobile phone we got through to the telecom carrier and it transpired that a lightening strike on the phone carrier’s exchange in the central city had taken out all their lines. I didn’t have a clustered exchange! I was so glad we had included the queues as the remote cities continued to process independently, writing the transactions to the queues. Things were a bit tense as we phoned around the cities monitoring the queue length and increasing it where necessary.

After a few minutes the exchange systems came back on line and the queues started to flush. Immediately the central server went to 100% and stayed there for a couple of minutes whilst the queues all cleared down, teaching me the need for throttling in queued systems very quickly. Luckily the central server kept running and the system then settled down.

So the nice thing about using queues is that they will keep your system running even when something unforeseen happens but, as is the case in most failure analysis, it’s the recovery which is the most difficult part of a high availability system. Needless to say I am a great fan of messaging and queues!

Being a Architectural Hero

Michael Platt — Fri, 13 Feb 2004 15:51:00 GMT

Architecture is great because it is so easy to look like a hero. I talked about earlier in this blog about looking like a hero with a poorly architected application but there are lots of other ways that bad architectural decisions can make you look like a hero.

In particular uninformed consultants are a rich seam of ego boosting; I have to say that sometimes they are assisted by our product groups. To give you an example in SQL Server under the admin tab we used to have a button labelled “Boost Performance”. This boosted the performance of the SQL sever by increasing its thread priority, a great idea if the was nothing else on the system.

In a production system however it was a very bad idea because there is another user of the CPU on a server of course, it’s the operating system! When the database came under heavy load, such as when there were a lot of users, then the OS started to get starved of CPU. Alas when there are a lot of users it means that the networks stacks (clearly part of the OS) are also heavily loaded. In the resulting battle SQL Server, with its “boosted performance” won, took all the CPU so the network started to fail.

So a typical assignment for the caped architect would start with a frantic phone call from a customer saying that their network kept crashing on load. I always like those calls because it is typically an architectural problem and indeed you can normally tell where the issue is by the number of users online when the crash occurs (and someone tells you “Windows doesn’t scale”… gurr!).

For instance SQL Sever will support about 5 to 10 users with the application taking a Database lock (amazing but true). So if someone has a system crashing with 10 users the first question you ask is what locks they are taking out (to which the normal response is “what’s a lock”!). One day I will write out a list of the architectural issue associated with a particular level of scalability.

With boost performance set on then typically the system will crash with about 60 users, so if the system used to work and suddenly starts to crash with network errors at sixty users then the first question to ask is “have you had a consultant in recently”. The normal response is “Yes, we had a chap in last week to tune the system and it was much faster, but he worked on the database and the problem is in the network”

I know straightaway I am about to make someone very happy which is a nice feeling and sometimes leads to a free lunch! I would just ask the person to go into the SQL Administration panel and see if the “boost” button was on and if it was to turn it off. They would come back 10 minutes later, amazed and incredulous, to say that it was on, they had switched it off and now everything was working perfectly! I could then bask in the admiration of my technical abilities. Too easy!

I used to call the “boost” button the “Hero” button because it always made me look like a Hero. Alas in later versions of SQL Server Microsoft hid the button so people who didn’t understand what it did can’t find it; so another opportunity to shine was lost!

Architectural Skills

Michael Platt — Wed, 11 Feb 2004 00:59:00 GMT

So if Architecture is a skill that can be taught then can it be lost? I think it can in a very strange way, not through the passage of time but by a radical change in environment. I came to this conclusion after working with a customer in Spain.

This customer built event management systems for major sporting events. They had been doing this very successfully for many years on a CICS / Cobol mainframe platform. The team was some 15 very smart Spanish people who had been writing these sorts of applications for 10+ years and really understood how the programs worked and how CICS worked. It was great to work with such clever and bright guys except for one thing; they spoke no English and I spoke no Spanish.

The project was to build a event management system for a major event (actually it was the European football championships) on the Microsoft platform (VB4 and SQL 6.0 running on NT 3.51 at the time.. gulp!). This had to be very highly reliable and support up to 150 operators in 10 locations in 10 cities with ticketing level integrity and performance e.g. less than 1 sec response time (Double gulp). Finally if the system was down then tickets couldn’t be issued and if the software held up the cup final then riots were likely. One of the few systems I have architected which has had “no deaths” in the reliability requirements.

I was to do the architecture and SQL training and a tall and very bright Scot from Glasgow called John was going to do the VB and developer training and so off we went to Barcelona.

First let me say what a wonderful place Barcelona is, beautiful weather, fantastic architecture (the building kind), I even stayed in a Hotel which was designed by Gaudy, dined in the restaurant were Picasso ate when he was young and paid for his bill with sketches which were still on the walls. A truly memorable experience.

Work was also memorable because trying to move the development team to VB from CICS was a challenge. They were used to rigorous flowcharting and modular development with a 4 – 6 month analysis period before starting to program. They were also used to 24x80 green screen alpha input. They spent 3 days looking at a copy of Expedia we had bought in amazement and voluble Spanish. No matter what we did we could not get them to start programming so in the end John just started to write VB and they stood behind him and watched.

It was a sight I don’t think I will easily forget. A tall, skinny Scot pounding out VB code interspersed with swear words surrounded by a group of vigorously puffing Spaniards (they all chain smoked). By the evening you could hardly see through the smoke but could still hear the swearing; I think it was a couple of days before the Spaniards realised that F*** wasn’t a piece of VB syntax!

Suddenly the development team cottoned on. Their Lead said to me (via interpreter) “So you just write code”. I agreed that was pretty much how you wrote VB (and this before agile was even thought of J). I did point out however that they still had to be careful about locking and transactions to vigorous nodding from the team. I later found out that the interpreter may have known Spanish and English he didn’t know anything about computing so what he said that they agreed to I have no idea.

The team dispersed to their workstations, I set up the databases and servers and peace and harmony reigned for 2 weeks. Looking through the source I was concerned that I couldn’t find any locking or transactional code and quizzed the Lead. He nodded a lot so I assumed they were going to put it in later. Finally the day of the stress test arrived and we got to 10 users before the whole system ground to a halt. Consternation all around and I was given a Spanish computing publication which had an article about how “Windows doesn’t scale”; in Spanish of course. I think I could probably recognise that phrase in any language.

An all hands meeting was called and I asked: “where is the locking and transaction code?” Blank stares. After much discussion it transpired that they had not put any in. “Why not?” I asked, after all this was a crack team used to high throughput transactional applications. The lead (via interpreter) explained to me that as VB did all the screen handling in a magical (to them) way they thought it would do the same for the transactional code. Not so I explain. Dawning recognition of the problem on the teams faces and back they went to their workstations. Another two weeks elapsed (note how long it took to make the code scalable rather than just functional, although retrofitting transactions was not a joy) and now the application streaked through the stress test.

So the moral of the story is that you can forget your architectural training if you are working in a new environment although it is pretty easy to get it back, certainly a lot easier than it is to teach new programmers about transactions!

Architectural thinking

Michael Platt — Fri, 06 Feb 2004 19:33:00 GMT

In my last blog I hypothesised that Architectural analysis is slightly different from developer analysis and so needs a subtly different skill set and way of thinking. To demonstrate what I mean let me describe a real life example of an architectural problem and different solutions.

I was called up late one Friday afternoon (why is it always Friday these things happen?) by a distraught business manager who’s biggest customer had a problem (I wont say who it was but they are household name in the UK). They had decided to provide a new product over the telephone and so had built a customer / order processing system for a maximum of 200 telesales operatives using Microsoft Products. They were going live on the Monday when the new product launched (and that wasn’t going to be easy to stop!). They had been in stress test for 3 weeks and when they took the load up to 50 users the system crashed. They had tried to fix the problem and couldn’t so it must be Microsoft’s fault, after all they had read in the press that Windows didn’t scale and here was proof! The Business Manager wanted me at their office (a 3 hour drive) asap, not so much to fix the problem but more to show that we were doing something. It seems to be a common misconception that putting technical people in cars or trains is a valuable use of their time which I vigorously dispute, I feel that most problems can be solved more quickly over the phone. There was a short discussion about efficient problem solving techniques, he spoke to my manager and I was in the car. Why is it my life is so like a Dilbert cartoon?

Getting to the customer a disaster scene met my eyes, paperwork everywhere, empty coffee cups, red eyed technical people, irascible managers, phones ringing, you know the sort of thing. The technical people just wanted me to say it was our technologies fault so they could go home. Managers wanted to nail me to a whiteboard and take turns with the whip; fun all around, however as I am not into S&M I insisted on looking at the application first (so maybe I am, just not that sort!).

It was a simple 3 tier app, smart client, business tier doing some business processing and a database with some simple stored procedures isolating the data access; nice and simple. There was however one strange thing; they had a second server running a piece of the business logic alongside the main business server. I asked why this was and it transpired that they had profiled the application (nice but unusual in my experience) and found one piece of code which was doing some simple customer validation and generating a GUID was taking about 30% of the CPU. They were concerned that it would become a bottleneck so had come up with the idea that, as the application was very well modularised, they could put that code on a second server and so distribute the load. They knew all about scale out.

The problem was that when the load got to 50 users the network stack on the server overflowed and so the system crashed. They had been on to product support and got patches to increase the network stack size (something I didn’t even know you could do!) but of course that didn’t fix the problem. Because it seemed to be something in the network layer they had spent ages in network tuning, putting in faster Ethernets and hubs etc. They were now convinced that it was an OS problem and Windows wasn’t scalable so why didn’t I admit it and let the blame fall on MS.

This is not a great career move at Microsoft and anyway I thought I knew what the problem was. I suggested a quick rebuild of the application with a simple change and then a retest whilst I went and moved the car (I had left it on double yellows). By the time I got back they had done the modifications, stress tested and were able to meet the 200 user criteria easily (which either shows how productive our platform is or how difficult it is to find a parking place in the UK!). Congratulations all round, techies treating me like a guru, senior managers fetching me coffee and a much relived business manager who carried my bag to the car, sometimes I love this Job!

So four questions:

1 What was causing the problem?

2 Waht was the fix?

3 How should it have been architected for scalability in the first place?

4 Why do I hate marketing messages?

Answers in the feedback