Whites Papers

Applying a Three Tier Measurement System to Measuring and Maximizing Network Stability

Mon, 30 May 2011 15:01:00 GMT

The traditional IT management measurement system serves two important functions. It provides the current status of a problem and it indicates in which direction that problem is trending. For the purpose of this document we will refer to that measurement system as a one tier system. A one tier measurement system is an internal informational system.

The first tier of a three tier system performs the same functions as a one tier system, but because that first tier is linked to its second and third tiers, the three tier system has a far greater informational value to IT management. This three tier system also has the added advantage of causing most measured problems to be resolved by the vendors whose products and services are often the cause of those problems.

R+2 is an example of a three tier system that is used to measure the stability of the corporate network system.

Tier One

Each day the first tier of RPI's three tier system (R+2) gathers the count of all network server machines operating on the network and the count of the interruptions caused by each of those machines. Each month from those daily counts the Interruption Rate and the Uptime Percentage rate of the network are computed. These monthly rates enable IT management to track the trend of the stability of its network. In addition to the monthly network stability rates, six additional sets of rates are produced that are used to increase IT management’s understanding of which components on the network produce the higher levels of interruptions. Those additional rates are computed for the two network server platform options (mainframe and traditional servers), for the several operating systems that support the machines on these two platforms, for the several hypervisors that facilitate virtualization, for the individual vendor’s hardware that run those machines, for the hardware maintenance providers that maintain that hardware and for the two operational states, virtual and non-virtual.

Tier Two

The second tier of RPI’s three tier system compares those internal stability rates to industry standards to provide added context and to give guidance on which products and services IT management should choose in order to maximize the stability of the network. Measuring the current stability of the network and knowing if the stability is increasing or decreasing is important, but knowing how that stability compares to the average stability in the industry, knowing how it compares to the best stability in the industry and knowing where that stability ranks in the industry are also important. The first tier tells management where its stability is and in what direction it is trending. The second tier tells management where its stability should be and where it could be. A one tier system uses the IT organization as its own standard of what is normal. The second tier of a three tier system uses the industry as a whole as the measure of what is normal and what is possible.

Tier Three

The third tier promotes competition between those vendors whose products and services are the source of most of the interruptions that undermine the stability of the network. The third tier is predicated on the concept that those that create the problems on the network and have the resources to solve those problems should be provided with access to the measurements that clearly show their culpability for those problems. A three tier system is based on the concept that the best measurement system is one that not only has the power to inform management of the status of a problem, but also has the power to resolve the problem. Each month RPI sends a report to each vendor organizations that provide products and services that can impact the stability of the corporate networks. Included in that report are the comparative interruption rates of that vendor’s products and services and the rates of its competitors. History tells us that the vendor with the best rates will see to it that those rates are presented in every sales situation where its products or services are being considered and will do everything it can to keep those rates higher than its competitors. Conversely the vendors with less than the best rates will either have to accept that their market share will decline significantly or their rates will have to increase significantly. Ignoring the rates is not an option. It is not the good will or the good intentions of the vendors that will provide an interruption free network, but their desire to remain competitive in the marketplace. The third tier is designed to take what has up to now been viewed as an IT management problem (interruptions of the network) and transform it into a vendor management problem (loss of market share).

A three tier measuring system is not a theory

The original R+ measurement system introduced in 1970 and eventually used for about thirty years by more than two thousand IT data centers in the western world was the industry’s first experience with a three tier measurement system. Tier one measured the reliability of mainframe hardware in each of the participating IT organizations. Tier two compared those measurements to the average and to the best reliability in the industry. It then ranked each individual data center’s hardware based on its reliability by model and vendor. Tier three provided each hardware vendor organization the result of those monthly measurements. Some vendor organizations could not sufficiently increase their reliability and were forced out of the marketplace. The remaining vendors competed over the next three decades to be the best and as a result of that competition the problem R+ was measuring virtually disappeared (see Remembering R+ and what it accomplished).

Remembering Reliability Plus And What It Accomplished

Tue, 16 Nov 2010 20:39:00 GMT

In the mid-90s R+ was literally measuring nothing, which was ultimately what it was always intended to do. To understand what R+ accomplished, go back in time to the late 60s, early 70s when abnormal end of production jobs, or abends, were the scourge of the IT industry. In the 60s, most IT organizations did not measure the failure rates of their hardware, so they had no way of knowing if those failures were increasing or decreasing. Had they been measuring those rates and found that those failures were increasing, it is not clear what they could have done about it. Then we introduced R+ in 1970, and IT management learned firsthand the power of an industry-wide management measurement system…the industry’s first “collective measurement" system. The biggest change that R+ pioneered was the idea of getting first dozens, and then hundreds, and finally thousands of data centers to measure their abends using the same measurement protocol. Then we collected and pooled the results of all those measurements into an industry data base against which the measurement of any one data center and any one vendor’s hardware could be compared. This is what we termed “collective measuring”.

Within a year of the introduction of R+, Telex, the largest of the plug compatible vendors was on its way out of the industry because the R+ collective measurements showed the reliability of its products were substandard. Within a few more years CDC and Memorex also departed the industry for similar reasons. But the loss of these three vendors did not lessen competition. The surviving vendors started making major improvements in the reliability of their products in response to the R+ "collective measurements" and rankings. Over time, vendors began to build hardware specifically designed to prevent abends. The team that pioneered RAID technology acknowledged that one of their major reasons for their design was to do well in the R+ rankings by making sure that their hardware never caused abends.

R+ demonstrated to hardware users, that being big did not necessarily mean being good. Some of the largest IT organizations in the US found themselves ranked by R+ in the bottom half of the industry. One of the largest financial organizations in the US found that despite a policy of only using IBM hardware, having its CEO on IBM’s board and having IBM’s CEO on its board, its hardware reliability, when ranked by R+, was among the worst in the industry. Two of the largest IT organizations in California learned in the same week that their size did not protect their hardware from also being ranked in the bottom 10% of the industry.

The original R+ demonstrated that combining internal measurements collected from thousands of data centers using the same measuring protocol could achieve extraordinary results that internal measurements used only internally would never have accomplished. In the mainframe world, abends went from millions a year to virtually none. Now RPI proposes to focus that "collective measurement" concept on the events that disrupt the network system with the goal of reducing those events from hundreds of millions a year to none. Internal IT measurements can tell IT management where it is and where it has been, but only RPI’s "collective measurements" can tell IT management where it could be and explain how to get there. Only collective measuring can trigger the kind of competition between vendors of goods and services that will lead this industry to an interruption-free network system.

The Value of Shared Statistics

Fri, 04 Mar 2011 12:12:00 GMT

The concept of IT organizations sharing statistics through an industry wide database was pioneered by RPI to great success in a measurement facility known as Reliability Plus or R+, from 1970 to 1999. For all of its success, R+ remains until today the only example of this kind of consistent long term data sharing among large numbers of IT organizations that was focused on achieving a specific management objective. R+ gathered statistics monthly from approximately 2,000 large data centers throughout the industry and provided each participating organization with monthly reports on the reliability of all mainframe hardware. What R+ proved in 1970 thru 1999 was that shared statistics achieved the power necessary to accelerate the growth of reliability far beyond what would have happened otherwise. The following are some of the reasons why data sharing is far more effective than relying only on internal statistics:

Shared statistics are available sooner. Hundreds of IT organizations can generate a statistically significant measurement in one month that would take one organization hundreds of months to generate.

Shared statistics report on all components. One organization can only monitor the components it has. Shared monitoring allows each IT organization to see the stability of all components used in the industry, or at least all those components used by those IT organizations participating in R+2.

Shared statistics are not easily dismissed by the vendors. Vendors can easily dismiss the data collected by a single IT organization, regardless of its size, as being atypical of the industry as a whole. Shared statistics collected from hundreds of mid to large size organizations throughout the industry are more difficult to dismiss as being atypical.

Shared statistics have more clout in the marketplace. If one of the purposes of measuring stability is to get the vendors to aggressively compete in their products' ability to maximize network stability, then it is reasonable to assume that hundreds of organizations collecting and using the same statistics will have significantly more clout than will just one.

Shared statistics send a loud message. That message is that the stability of the network matters. When hundreds of senior IT management join together to measure the stability of their networks that message is heard by the IT workers who build and protect those networks, by corporate workers who depend on those networks, by top corporate executives who are ultimately responsible for those networks, and by vendors whose products and services impact the stability of those networks. When IT organizations measure the stability of their networks using R+2 that stability will become far more stable because no vendor organization wants to be known as the company that undermines that stability and no IT manager or corporate manager wants to find out that its corporate network is one of the least stable in the industry. R+2's measurement not only inform, they motivate those that can improve network stability, to do so.

Welcome

Thu, 24 May 2012 11:13:00 GMT

The Power of the Monthly External R+2 Reports

The R+2 monthly external reports provide three capabilities that will enable all senior IT managers to better manage their IT organization. Those capabilities are:

The ability to once a month look outward and see where their IT organization ranks in the industry.
The ability to once a month look inward to see how their various data centers compare to each other.
The ability to once a month focus on which vendors’ products and services enhance the stability and availability of the corporate network system and which ones don’t.

For fifty years IT organizations as a whole have gone virtually unmeasured with the result that senior corporate management and senior IT management have no idea where their IT organizations rank in the industry. Conventional wisdom has historically considered IT organizations too complex, too multifaceted, too technical and too fast changing to be measured. But RPI believes that there is one IT function that can be measured and, depending on the data used, becomes a microcosm of the IT organization or a microcosm of any of the data centers managed by that IT organization. This means that both the IT organization and its data centers can be compared and ranked externally among all major IT organizations. It also means that each of the IT organization’s data centers can be compared and ranked internally.

To qualify as a microcosm of the IT organization a function would need to have four important characteristics. The function would need to have a high percentage of the traits of what it is representing, in this case, the IT organization. It would need to be a function that was associated with all IT organizations. It would need to require most, if not all, of the management skills to manage, as managing the IT organization does. And lastly, this function would need to be capable of being measured in such a manner as to allow for comparisons across extended time periods and across all major IT organizations.

The IT function that meets all four of these characteristics is the function of managing the stability of the corporate network system, which also happens to be the function of managing the stability of the corporate server machines.

Managing the stability of the network is nearly as essential to the success of the IT organization as managing the organization itself. Managing that stability is a responsibility that all major IT organizations share. Managing that stability requires a level of management skill approximating that required to manage IT, which includes the ability to select hardware, application software, operating systems, hypervisors, operating procedures, monitoring software, backup procedures, and security software. And lastly, managing that stability can be measured in such a way as to be comparable over extended time and across all major IT organizations.

Using this measurement to improve managing the IT organization

R+2 enables senior IT managers to look outward from their organization and see where the stability of their corporate network ranks in the industry and, by extension, where their organization ranks in the industry. A senior IT manager can see how large the stability gap is between his organization and the best organization in the industry and how large the gap is between his organization and the worst organization in the industry.

Because R+2 also measures six component parts of the network, senior IT management can see how these components contribute to its network stability and see how the different vendors’ products and services support the stability of the network and these six component parts.

Because R+2 is designed to be used by hundreds of IT organizations in the industry, how it ranks the stability of the network, the six component parts of the network, and the products and services used on the network will influence what those IT organizations acquire. That means that it will encourage vendors to maximize their efforts to produce products and services that enhance network and component stability. With no other effort than participating in R+2, IT senior management will cause platform vendors, operating system vendors, hypervisor vendors, hardware vendors and maintenance vendors to improve their offerings or risk losing business.

Many IT organizations claim they use “Best Practices” in support of maximizing network stability. However, they do so with no apparent knowledge of where the stability of the networks of those IT organizations that use those practices rank in the industry. It would seem logical that the “Best Practices” are the practices of the organization with the best stability. IT organizations can both learn what to do from the best organization and what not to do from the worst organization. R+2 facilitates this learning process by examining what makes the best network the best and the worst network the worst and then sharing those findings (without revealing those organizations) with all participating organizations including the organizations that have the best and worst network stability.

The first half of the R+2 monthly reports enables the senior IT manager to look outward from his organization to see where his organization ranks in the industry and to learn from those organizations whose stability ranks higher. The second half of the monthly R+2 reports enables the senior IT manager to look inward to see how his different data centers and their managers compare in terms of their contribution to corporate network stability and their contribution to the IT organization as a whole.

Many major IT organizations operate multiple data centers, each with its own manager, its own history and shaped by its unique geographical influence. The inward looking report provides the Interruption Rate and the average downtime minutes of the server machines for each of those data centers. It then compares and ranks those data centers against each other on the basis of their Interruption Rate. It also ranks each of the six components (platform, virtual versus non-virtual, operating systems, hypervisors, hardware, and maintenance options) of each data center based on their Interruption Rate. This enables the senior IT manager and the data center managers to see the strengths and weaknesses of each of the organization’s data centers as they compare internally and then see how those data centers compare against the data centers of other major IT organizations in the industry. For example, if there are 300 participating IT organizations in this month’s R+2 reports and those organizations average 5 data centers each, then each of the data center managers in all of these 1,500 locations are ranked from first to last so that each knows how well he is doing in comparison to his peers, and so that each senior IT manager knows where the managers of its data centers rank in the industry.

The two views (looking outward to the industry and inward to their individual data centers) offer each senior IT manager a unique view of its management challenge never before available. It motivates and enables that senior IT manager, its data center managers, and all vendors to do a better job. RPI believes that R+2 will be as important to the way IT is managed as email is to the way IT communicates.