Control the risks of living with untestable and unmanageable architectures

Dec 6, 2000
Gartner

By N. Jones

Message-oriented, distributed, asynchronous architectures demand new skills of the IS organization.

Gartner Research Notes

Gartner Research Notes offer valuable data that is compiled and written by analysts at Gartner, a business technology advisor based in Stamford, CT. (TechRepublic is a subsidiary of Gartner.) Every week in IT Consultant Republic, you'll find a new Research Note like this one, which will provide you with a snapshot of information about a particular mission-critical topic. To see more Gartner research, click here.

Key issue

Which technologies and vendors will provide effective support for construction and maintenance of applications architectures?

Strategic planning assumptions

Through 2004, application development costs related to implementing, testing, and supporting message-oriented, distributed, asynchronous architectures will cost at least 50 percent more than the equivalent functionality implemented on a monolithic single-processor architecture.
By 2004, the architects of complex, distributed, message-oriented systems will be forced to adopt statistically based simulation techniques to predict system performance.

Gartner research on TechRepublic

As you know, Gartner is the world authority for IT research. Because of our relationship with Gartner (TechRepublic is an independent subsidiary of Gartner), TechRepublic is your online resource for Gartner research and analysis. To see more Gartner research, click here.

Enterprise application architectures are quietly acquiring two unpleasant characteristics:

No one owns the entire system. No single person or organization owns end-to-end system responsibility. Increasingly, elements of an application are hosted outside the organization in trading-partner and customer environments. New applications (e.g., involving mobile phones and wireless PDAs) will include both client devices and infrastructure not owned by the organization. Effective end-to-end system monitoring and management is impossible. The system "architecture" is effectively a set of interfaces and service-level agreements.
The paradigm has changed. A quiet paradigm shift is occurring. At a "city planning" level, the enterprise architecture is evolving toward a set of autonomous components that communicate via messages. This is fundamentally different from "traditional" components, which were interfaced using a procedure call paradigm. Messaging is essential for three reasons. First, it supports necessary business requirements, such as ad hoc notification. Second, it can operate in networks with intermittent connections and unpredictable latency. Third, it facilitates global-class application architectures that can be flexibly "scaled out" instead of being "scaled up," with message routing being used to achieve load balancing and redundancy. The challenge implicit in the messaging paradigm is that complexity increases, and the system components typically operate asynchronously.

Factors affecting enterprise application architectures

"City planning" architecture—Some key principles of architecture (such as componentization and a focus on interfaces) are fractal and can operate at many levels of granularity. "City planning" refers to the architecture applying to large functional services that may sometimes have interfaces crossing enterprise boundaries. Examples might include access to customer information, interfaces to logistics suppliers, and credit card authorization. Similar issues arise at both finer and coarser levels of granularity. For example, organizations must also consider "national planning" architectures, dealing with the relationships between networks of business peers, and "room planning" architectures, dealing with the components of individual applications.
Scaling up vs. scaling out—Scaling up refers to increasing capacity by moving the application onto a larger processor. Scaling out refers to increasing capacity by adding more processors to a pool that shares execution of the application. Scaling out may be attractive to organizations that cannot easily predict application volumetrics, because it can be accomplished quickly using commodity hardware—but it implies a more complex architecture. Scaling out is also encouraged by vendors, such as Intel and Microsoft, that wish to motivate customers to buy more machines, rather than larger machines.
Asynchronicity—"Asynchronous" means that the system consists of a number of independent processes executing either concurrently or at different times, often on independent processors.

Systems that involve asynchronicity and potentially unreliable message queues are very hard to design. They tend to have more starts and more failure modes than simple sequential systems and are not amenable to static analysis. Few developers are used to designing them, and few design or development tools provide significant support for these challenges. The complexity of such systems means that their behavior is highly nondeterministic. Testing is much less effective than for monolithic applications.

Message-based architectures

Message-based architectures have the advantage that, as messages are passed asynchronously, they can be stored and forwarded if a communications link is temporarily unavailable. This makes asynchronous, message-based architectures very appropriate for use with unreliable communications links like the Internet and wireless networks. However, application design is more complex because errors, such as nondelivery, may occur long after the message was sent.

What should organizations do?
Returning to simple, monolithic systems is not an option, as they cannot support the business or technical requirements of global e-business. The application development (AD) and operations organizations must therefore work to address the risks that such new architectures imply. This will require a package of measures including:

Training. Staff must be trained in appropriate design techniques and synchronization concepts and understand the new failure modes of networked message-oriented systems.
Making the individual components of the system more robust and fault-tolerant. The goal is not to achieve high reliability of every component, but a perception that the overall system has sufficiently high availability. This must be achieved by giving individual components the ability to perform a useful (if degraded) function if some of their peers are unavailable.
Implementing devolved system management and service-level monitoring. System components must be able to report on their operational status to other components (which might be outside the organization), alert operations staff when problems occur, collect the information required to monitor and maintain their service levels, and verify service levels of their peers.
Considering operations and support issues, such as system management, monitoring, and recovery after failures at the application design stage. This will require organizations to take a wide view of cost-of-ownership that encompasses both development and operations costs.
Adopting more sophisticated testing and system-analysis techniques, considering approaches used by real-time systems-design methodologies that already address such issues. This will require using simulation tools to predict system performance.
Changing training, design, system operation, and testing standards to address the above issues.

As our strategic planning assumptions point out, the implications are that AD organizations will require new techniques and must expect higher costs to develop and support such architectures.

Robustness vs. reliability

The goal of IT reliability is generally to provide a guaranteed level of user availability. When reliability cannot be guaranteed or even predicted, the goal must change to providing a guaranteed level of perceived availability. In essence, this requires increased spending on the robustness of individual components so that they provide a useful (if perhaps degraded) function when some of their peers are not available.

Bottom line
Distributed, asynchronous, message-oriented architectures will be more difficult to design, develop, and operate than a monolithic application performing the same function. Resistance is futile, however, as business and technical demands mean that it will be impossible to build the new generation of distributed e-commerce applications as monolithic systems. Organizations must therefore adopt processes and technologies that control the risks of living with such untestable and unmanageable architectures.

This report was originally published by Gartner on Oct. 31, 2000.

How will you adapt?

What do these new asynchronous architectures mean for you and your clients? Post a comment below or send us a note.