The way I see it...

What is Code Complete? Simply put, Code Complete is the point in which for the given milestone or checkpoint, that the features planned are completely coded. One might think that once a software application hits Code Complete (CC), that it is time to ship it out to the customers. Well, I'm sure that happens somewhere but not here does that happen. What happens once we hit CC is the next phase of the product cycle, testing to get to ZBB. Yes, there's testing done during the coding phase as all of the different features get coded to confirm that their functionally working but it isn't until after all of the features are coded, hence, Code Complete, that the Test Team can fully test the product.

Getting to CC on a large scale product like Microsoft Exchange, takes a lot of coordination and scheduling between the various teams and components. Due to its large scale, we have more than one milestone with a period of time specified when our developers primarily code the features for the new release.

Between Coding Start and Code Complete, the developers main focus is on the new features but they also fix bugs found along the way. The bugs come from the Test Team testing and through our eating dogfood. This bug fixing is needed to stabilize the code and to assist us in getting new builds into dogfood and to hit CC. Now as you may have already guessed, bug fixing can impact hitting CC. This is where having bug bars set properly help but above all, having high coding quality standards is even more important as this helps cut down the number of bugs found.

High coding quality standards includes having properly defined feature specifications, design documentation, test documentation, and Threat Models reviewed and completed prior to code start. It also includes using industry coding best practices with code reviews and testing called "smoke" testing, prior to checking the code into the main source code.

There are a number of ways to track the progress that developers make towards CC. We use what we call, Work Items, to track our developers progress. A work item is the lowest breakdown of a complete component. Work items can take from one day to a number of work days to complete the feature. Each work item is given an estimated time of check in date or Check-in ETA, which creates a rough schedule. Tracking the closing of the work items against the work item's Check-in ETA is a good way to see how well things are going against the product's schedule. Along the way, each work item's Check-in ETA should be adjusted to keep a more accurate picture of what is going on.

Once all the work items are completed, you are Code Complete. Hitting Code Complete is a terrific achievement for a team as there is a lot of extra work, stepping up and sacrifice made by all working on the product to get everything just right to make it happen.

When talking about cutting and cuts, one thinks of hurting and hurts if the cutting and cuts are being done by knifes or other sharp objects. Looking up "cutting" in a thesaurus gives you such words as, wounding, hurtful, unkind, spiteful, and harsh while looking up "cut" gives you slash, hack, and sever. Just using the words cutting and cuts in a sentence could make someone shiver a bit inside in the context above. We however, are not going to talk about making cuts and cutting things with knifes or other sharp objects. We are going to talk about cutting features from a software point of view and how those cuts come about and why they are needed and some of the caveats surrounding cutting software features of software under development.

Although we are not talking about cutting and cuts as done with knifes and other sharp objects, we can make someone shiver inside and generate some strong reactions from others. Mention cutting and making cuts to a Program Manager (PM) or Developer (Dev) about one (or more) features and you could get wounding, hurtful, unkind, and spiteful looks your way while being called names like slasher and hacker while the conversation gets severed short but not until after some really harsh words are spoken to you first. This reaction to cutting features from software I've seen and have experienced first hand and sometimes by a PM or Dev I least expected it to come from too!

So, why would features need cut from a software application under development? The two primary answers to this question are, 1) the feature doesn't fit within the schedule and/or 2) it doesn't meet the needs of the majority of the software program's customers. Sometimes a time comes in software development that every feature originally planned to be included in the software just doesn't fit into the schedule like it did when the planning was done. The cost to keep the feature would hinder the proper completion of the schedule and as such it needs cut. There again, often due to changes in customer demographics or lack of customer input in the first place, the feature doesn't meet the needs of the majority of customers. It could be a combination of both where due to the time it will take to finish the feature, the customer need for the feature will not exist at the feature's completion date and so it needs cut.

One needs to make sure that the feature proposed cut doesn't break any key scenarios, makes the code base less stable or creates security issues if it becomes a cut. A key part about making cuts is to have a plan ahead of time to follow for making cuts if and when the time comes to start cutting features. Having a plan thought out well ahead of the time the cuts may be needed helps to not overlook the areas the feature touches and thus helps to not break key scenarios and make the code base less stable. You may want to consider having whole scenarios described so that if needed, the whole scenario and its related features and work items are known so that if it does come down to making cuts, you have the most information at hand available to know the impact made by the cut being proposed.

A couple of the biggest things to consider about the proposed cut if looking to save the schedule is, will it really save time and where will it save time? A cut for example, could save dev time but cost more in test resources and time and make the schedule go out even further. Again, this comes back to detailed planning prior to having to make cuts to help understand how much and where the time will be saved when making cuts. The best cuts save time overall but these are cuts that are sometimes difficult to find and often even harder to make.

The best things to do to help lessen the hurt of cutting follows: Have planned ahead of time to understand how the features work into scenarios and work items. Find out how the features, scenarios and work items interact with and w/o each other. Set priority orders on scenario, features and work items including the cut order and estimated amounts of time saved and where the time is saved if the cut is made. Understand when a cut can be made and when it cannot and the impact of making the cut at different times throughout the development cycle.

It's in our nature to want to finish something and to not want to cut it once we've planned to do it but, if we plan ahead and understand the cut, it could be the best thing to do to get things back on track. So remember, take cutting software features seriously even if it isn't done with knives or other sharp objects because there just may be more to it than you think and it could come back and hurt you if you don't.

Growing up in East Tennessee, I owned American Bull Terriers, German Shepards and hound dogs but at no time did I ever eat any dogfood nor did I ever even taste it… however, I did see a girl from down the road try it one day but that's a different story and I own a cat now. The dogfood that I'm talking about eating today is when a team uses its own software bits during the software product cycle prior to Beta or release to see how it works or as is sometimes the case, how it doesn't work.

Dogfooding can yield great results that one cannot get in a test lab. During dogfood, you are actually using the software product as part of your daily routine. In the case of Microsoft Exchange, our dogfooding encompasses thousands of people across multiple groups/divisions here at Microsoft using email on newly created backend services. You might be thinking that in a lab, you could have more email test accounts that could stress and work the software to its limits and be able to inject various emails with and without attachments, yada, yada, yada, etc. Well, that is true and prior to dogfooding one needs to have the bits properly tested and as stable as possible. The real world usage in dogfood however, allows one to understand the product more deeply and find customer pain points that a test environment just cannot do.

The feedback and issues obtained during eating dogfood comes from the dogfooding customers of the software. For example Microsoft Exchange has as our dogfooding customer Microsoft IT deploying the dogfood bits and maintaining the dogfood servers. MSIT gives us a great feedback on how our setup/install and configuration documentation is, of how well our setup program works, what it is like maintaining the dogfood servers, etc. Then there's the dogfood customers using these dogfood servers to send/receive email. (I need to mention that at Microsoft, we live and work by email.) If the dogfood Exchange servers are down for some reason and our dogfood customers cannot send/receive email, well, let's just say we hear about it very quickly. We also quickly investigate and fix the issues as fast as possible. This feedback is unique to dogfood because the issue most often could have been found in a test lab within a reasonable period of time and cost nor possibly even at all given the real world randomness.

Now as you may expect, your dogfood customers have to have patients as the issues arise and while you are working to fix the issues found in dogfood. It is along with you that all involved in dogfood get to taste the bitter sweet success that only eating dogfood brings from first hitting and finding the issues and then from fixing them. Remember all along that in the end, you will learn how to build better software and that your customers using the released software will benefit from the pain experienced while you were eating dogfood.

I suggest that you give eating dogfood a try even if it is a small application you are working on and if it would only involve a few people. The real world feedback you get could just make the bits you are writing even better!

At this time, I would like to thank all of those people here at Microsoft who are supporting us in our Exchange Dogfood efforts of Exchange Server in our process of making Microsoft Exchange better than ever! Nailing the production issues before our customers do is what dogfooding is all about! Again, thanks all of you who are supporting us in our Exchange Dogfood efforts, you are the greatest!

For more on

Security testing is required prior to shipping any software these days as there are so many threats coming from so many angles against so many entry points. Buffer Overruns, Integer Overflows, Cross-Site Scripting, Race Conditions, Driver Reliability, Security Operations, Validation, Alternate Cod Paths, Bad Assumptions, Assertions and thousands of other problems and quirks can give leave the door open for mischievous and/or malicious attacks.

There are various security testing tools to use and testing methods to follow but just using tools and testing methods alone can miss vulnerable areas. A good way to figure out where your vulnerabilities are, is Threat Modeling.

Threat Modeling is a process of identifying and documenting the vulnerable areas and issues of the system. A system is anything that exposes functionality to an end user. A system can describe a single feature and anything all the way up to a full blown multi-tier web application including all of its various tiers and supporting infrastructure.

Threat Modeling is methodical and complete, describes the system's threat profile, characterizes the security of the system, and often finds many vulnerabilities not normally thought of using normal testing methodology. Using Threat Models (TMs) at Microsoft is a required process for any release and is often done multiple times throughout a release cycle prior to shipping.

There are people out there I know who believe that hitting a Code Coverage goal of 90% is a good reason to stop testing. Now given that I have to admit that hitting a 90% code coverage rate is good, it however should not be used as the sole reason to stop testing.

Using code coverage as a sole measurement to know when to stop testing is wrong because using code coverage numbers alone doesn't indicate how well the tests used to obtain the coverage results executed the code and whether or not the code was executed in the proper scenarios required for hitting the release criteria. It could be that the 10% missed is the code that 80% of your customers need to use and it wasn't executed at all. Or it could be that the 90% hit was not executed in the proper order of execution for the scenarios that 80% of your customers will execute it.

So, it is very important in the planning stages that PM, Dev and Test properly communicate and set code coverage goals for the milestone that the team can drive towards and measure progress by. It is equally important to define and prioritize the key scenarios used by your customers and to insure that the test cases and scenarios that execute the code cover the scenarios used by at least 80% of your customers otherwise, you will not know if the proper code was executed in the correct manner… you will just be looking at code coverage results.

Spec reviews at Microsoft are handled differently across the various product groups and teams but they all have some commonalities. In my 10 year tenure at Microsoft, I have participated in spec reviews for Windows, Word, Excel, Access, SQL, SQL Reporting Services, Visual Studio, numerous MSN offerings, various other Microsoft products in limited fashion and now Exchange Edge. Each of the spec reviews that I've been in have had their own flare, personality and style. Some spec reviews that I have been in were dominated by one person thus stifling the creativity and input from the others there and rendering the usefulness of the review to nil. I have also been in spec reviews which were non structured with seemingly no primary agenda other than just being able to check off the exit criteria of having a spec review.

The most productive spec reviews that I have attended have giving way to some of the best products that Microsoft has to offer including the spec reviews here in Exchange Edge. These spec reviews have active dialog discussions between the participants; a focused direction to achieve the goal of clarifying the functionality of the features, resolving open issues, defining the feature set while keeping the customer's best interest in mind.

A productive spec review cannot happen unless there is a well thought through written functional specification (spec) that has been created with feedback from the feature's architects leads and management from test, development and program management. After the spec includes the feedback from the team's leadership roles, feedback from partners and dependencies need included once the spec has all of the major details and issues covered relating to them. This interaction of the spec owner with the team's leadership roles, partners and dependencies keeps the key people involved and active in the creation of the spec and is essential to the review of the spec being productive and successful. The review of the spec will be randomized without having a well written spec with feedback from the key people. This is because discussions during the spec review will not come from a sense of ownership with prior knowledge with understanding of the spec's content and direction being taken.

When a team is working to hit ZBB at the end of a milestone, that team has at various points gone through a number of bug bar level changes limiting the quantity of bugs to those bugs directly impacting the milestone's most critical release criteria. During the timeframe right before ZBB is hit, there is a lockdown period where developers must get permission to check in their bug fixes.

During this period of lockdown, each new bug coming in gets questioned to its importance for the milestone's release criteria. If the bug does not meet the milestone's release criteria, that bug is put into the next milestone. This is where it is very important to have the proper release (or exit) criteria defined for the milestone prior to reaching the end of the milestone.

Without having the proper criteria defined ahead of time for each milestone, it is impossible to properly hit ZBB due to the constant incoming flow of bugs albeit at a slower rate as the bits stabilize. So, plan your exit criteria ahead of your milestones and match the release criteria to the features being coded during the milestone and don't forget about your previous milestones either to have a better path to hitting ZBB as scheduled.

Opening the DCR is the easy part… the harder parts comes when the development team has to triage it, figure out whether or not to take the change requested, its customer impact, priority, resource costing PM, Dev, Test, & UE, what the schedule impact is, any security implications, risks associated, dependencies, affected DLLs, etc… and the list can go on and on dependent upon the given DCR.

As a Release Manager for my team, I'm the one to come up with the process to use for handling DCRs. This may sound like an easy task to you but, there are many things to take into consideration such as, making sure that everyone has the same understanding of what a DCR is and what a DCR is not across to the team and getting them all to agree that, that is what a DCR is and isn't. There's also the, "What do we do with a DCR after it is accepted?" question that needs answered to everyone's understanding so that they agree with it and actually follow it.