Executives have often asked what are my thoughts on measuring the performance of their Infrastructure architects. So here are some of my thoughts. As this profession begins to mature, I'm sure there will be many more new ideas and refinements. I look forward to your insightful thoughts and suggestions as we examine this issue for Infrastructure Architects.
Architects (just like all business leaders) must be personally responsible for the architecture they commit the organization to. Their salaries must reflect the continuous tracking of past initiatives and how successful these systems operate and provide value to the business. Of course, this should be done within reason. If this is going to happen, then the architects will need some significant business authority to make successful decisions.
As Architects are good communicators and must understand the complexity of relationships within an organization, a good peer 360 review with all of the diverse groups the architect works would help the professional find useful suggestions on his or her growth, strengths and weaknesses and help the executive understand how to develop the architect effectively in the future.
As architects must understand a large range of technical knowledge, the architect must prove data center architectural skills in a variety of areas. This includes (but not limited to) computational system design, data management design, communications (networking) design, systemic quality methodologies, architectural process methodologies, technical risk analysis, mapping business metrics to systemic quality modeling, capturing and absorbing synthetic knowledge (information from the industry, partners, conferences, etc..), and development and communication or architectural organic knowledge (information obtained from working in their specific organization and role). People take a wide range of techniques to measure such capabilities and development strengths: peer review, publications, peer review and analysis, post-mortem architectural analysis, position papers, etc... No one currently, has a perfect metric that measures every aspect of this capability. The metrics are often qualitative in nature (which, when done correctly, good qualitative measurements can generate far better knowledge and understanding than quantitative metrics: my little rant)
Of course, this becomes more challenging since architects must sometimes design solutions with technologies that are still quite new and unproven. Architects must be capable of forecasting the capitalization of new technologies for their businesses. At the same time, Architects must demonstrate an ability to capitalize on existing technical investments. Architects who like to design 15 separate authentication mechanisms that could be effectively managed by three is not capitalizing on existing investments well and hence, driving operational costs higher for the company. This becomes especially true for core data center services (example: authentication, DNS, remote access, DHCP, NAT, authorization systems, monitoring and management, patch management, system provisioning, etc…)
Architects are business leaders. Therefore, they must demonstrate good qualitative and quantitative business skills. One should hold a data center architect to the same level of business competency as they would any other business leader professional in their organization. Furthermore, their past architectures must demonstrate good business metrics. TTM, Operational and Acquisition Cost, Business Complexity Reduction, enabling future business models, etc... I believe the operational (continuous) cost should be measured on an on-going basis and reflect back to the architect's overall performance. If the Data Center architect saved a lot of acquisition cost through a design which resulted in a ten fold increase in operational cost and it increased future modification costs by three fold, then that project would not be viewed with as much success and the architect's current performance should reflect this experience.
Architects should understand the business and technical relationships between processes, team culture and skills and technologies utilized within information system solutions. This should be a large metric for architects to continuously demonstrate over and over again. Good peer review and post mortem architectural peer review (just like doctors and lawyers) develop good skills in this capacity. It is one of the fine points that separate the great from the average.
Architects should know many process methodologies well and demonstrates an exceptional capability to execute on established processes. This is critical. It also separates the star performers from the rest. A great data center architect will develop solid architecture models for the business that doesn't require heroics and hack and rack, myopic designs. A business cannot survive on heroics for long term success and neither can an information systems solution. Architects know and execute on process.
This could be measured with peer review, process teams, business process groups, process certifications, etc...
Architects should understand people. This includes operational, business and development team skills, cultures, politics and personal dynamics. This includes communicating appropriately to different audiences, developing trust between different groups and generating a sincere perception of contributing to the success of the organization. 360 degree reviews, communication coaches and evaluators are some examples to use to help develop here.
Anyway, here are my random thoughts. As the profession matures and businesses realize the value of data center architects, then the metrics and development plans for the architecture profession should mature as well. I look forward to your thoughtful insights.