Published in the Proceedings of Middleware'98,
the IFIP International Conference on Distributed Systems Platforms and Open Distributed Processing
September 15-18, 1998
The Lake District, England
Rodrigo Vanegas
John A. Zinky
Joseph P. Loyall
David Karr
Richard E. Schantz
David E. Bakken
BBN Technologies
GTE Internetworking
10 Moulton St., Cambridge, Massachusetts 02138 U.S.A
Email: vanegas@bbn.com, jzinky@bbn.com, jloyall@bbn.com, dkarr@bbn.com, schantz@bbn.com, dbakken@bbn.com
Most distributed applications are brittle; they work in a limited environment and cannot adapt to changes in this environment. Making these applications less brittle is a complex engineering task that is hard for specific application areas and even harder to generalize. The Quality Objects (QuO) project offers a framework for creating applications that adapt to different Quality of Services (QoS) offered by the underlying resources. QuO offers several layers of tools for creating adaptive distributed applications using the familiar CORBA development process of code generators and runtime support libraries. In this paper we describe the QuO Runtime and the components it uses to maintain and adapt to QoS. The QuO Runtime is the base functionality on which QuO code generators specialize for a specific adaptive application. The QuO Runtime handles much of the complexity of collecting, organizing, and acting on changing QoS, thus reducing the burden for application developers.
Keywords: QoS, quality of service, distributed object computing, adaptive programming, CORBA
The QuO system is a framework for making distributed applications that can adapt to changing quality of service. The goals of QuO are threefold: create more robust distributed applications, enhanced control over how applications adapt, and to support reuse of adaptive mechanisms. Let's examine each of these goals and how QuO achieves them.
The first goal is to create applications that run effectively over a wider range of operating conditions. Finding a single adaptive mechanism that works over a wide range of operating conditions is hard, but several specific mechanisms usually exist which work over limited operating regions. The QuO architecture supports the notion that wide range adaptation can be achieved by identifying the current operating region and dynamically selecting the most appropriate specialized mechanism.
The second goal is to give more control over the application's adaptive behavior during its entire life-cycle. Many different sources of information are available to help decide how a system should adapt. Unfortunately this information is spread out over multiple places, times and people, hence it is hard to gather and usually ignored. The QuO architecture helps organize and the move this information between the parties to enhance the effectiveness of adaptive decisions. For example at runtime, a remote method call conceptually just moves the methods parameters and results between the client and the object, but how this is done involves many subsystems (Figure 1). As we will see in later sections, the QuO Runtime adds many components help make this operation more adaptive, such as System Conditions to gather runtime information, Contracts to summarize operating regions and trigger adaptive behavior, and Delegates to execute alternative behaviors. QuO also partitions the task of creating adaptive application into many roles which create and use these components.

Figure 1: A remote method invocation in a QuO application
This paper is one in a series of papers about the QuO system and concentrates on the first runtime implementation of QuO and our design tradeoffs. The QuO architecture is described in (Zinky, 1997) and the QuO description languages and code generators are described in (Loyall, 1998a). We are also using QuO to integrate QoS adaptive mechanisms into CORBA applications, such as fault tolerance (Cukier, 1998) and resource reservation, and survivability. In this paper, we describe the implementation of the QuO runtime system in detail. Section 2 introduces the QuO runtime components and their interaction by tracing the execution of a QuO application. Section 3 describes the QuO runtime system and its components. Section 4 describes how to design and develop a QuO application in the QuO environment. Section 5 describes work related to our QuO research. Finally, Section 6 provides the current status of QuO and future directions.
Figure 2: Example remote method call in a QuO application
Figure 2 illustrates the steps that can occur
during a remote method call in a QuO application. QuO replaces the
CORBA binding of a client to a remote object with a connection through
a series of delegates to the object. This is transparent to the
client, because the remote object and the delegate have the same IDL
interface.
When the client calls a method on the remote object, it is actually calling that method on the local delegate. The delegate optionally triggers contract evaluation, which grabs the current value of all system condition objects measuring aspects of the system's state relevant to the contract. The system condition objects return their values immediately with their current estimate of some system property. The work needed to determine the values is done continuously in anticipation of its need. That is, the updating of system condition object values is performed asynchronously with respect to contract evaluation, delegate execution, and client execution.
The contract consists of a set of nested regions which describe the relevant possible states of QoS in the system. Each of these regions is defined by a predicate on the values of system condition objects. The contract evaluates the predicates to determine which regions are active (i.e., their predicates are true) and passes the list to the delegate. This list of active regions represents the current state of QoS in the system with respect to this contract. If contract evaluation caused a region transition, i.e., caused a change in the list of active regions from the last time the contract was evaluated, it might trigger transition behavior. Transitions are behaviors associated with moves from regions, into regions, or between regions, and consist of callbacks for notifying the client or calls to methods on system condition objects for controlling QoS.
The delegate code includes a dispatch statement, i.e., switch in Java and C++. The active regions cause branches of the dispatch statement to be chosen, each of which specifies local delegate behavior alternative for the method call. The delegate's alternatives are specified by the QoS developer using QuO's description languages. In many cases, especially those in which the contract indicates that the QoS desired by the client is being satisfied by the system, the delegate will pass the method call through to the remote object. In others, especially when QoS requirements are not being met, the delegate can choose an alternate method on the remote object, might block or buffer the method call, or might throw an exception.
The delegate performs similar processing upon a method return, i.e., it evaluates the contract to obtain the current QoS regions and selects a behavior based upon the current regions. The default is usually to pass the return value to the calling client. Note in Figure 2, we show the connection consisting of only one client-side and object-side delegate, but the connection consist of an arbitrary call graph that can change dynamically at runtime.
Contract evaluation can also be triggered by changes in some system condition objects, i.e., those that are observed by the contract. Other system condition objects, especially those whose values change frequently, are non-observed and do not trigger contract evaluation. Regardless of how contract evaluation is triggered (by a method call/return or change in a system condition), a transition from one active region to another can trigger transition behavior, which consists of client callbacks or method calls on system condition objects.

Figure 3: Components of a QuO application
The first release of QuO splits the runtime into a delegate that runs in the application and contracts and system conditions that run in an independent QuO Kernel. The delegate communicates with the kernel through a CORBA interface. This split helps flexibility is several ways. First, the code generators only have to support multiple target languages for the delegate generator, the contracts can all be output in the kernel's implementation language. Second, the asynchronous behavior of system condition, contracts and transitions, implies separate threads. By grouping these components into an independent subsystem, the concurrency constraints are vastly simplified. An alternative implementation of including the contract evaluation as inlined code in the delegate was prototyped and rejected for now, because of its inflexibility, but does offer the possibility of higher delegate performance in the future.
The object factory creates and initializes contract and system
condition objects. Currently all of these objects are created at the
startup of a QuO application, although future QuO applications might
require dynamically created contracts or system condition objects. The
factory is implemented using reflection in Java, since it is difficult
to anticipate at compile time the classes of contract or system
condition objects that will be used by the runtime system and how many
instances will be instantiated by the delegates. A delegate makes a
CORBA call to the QuO kernel factory to instantiate an object, i.e.,
either a contract instance or a system condition object, passing the
class of the object as a string. The factory locates the corresponding
class file and loads it dynamically if it has not already been
loaded. The factory then creates and initializes an instance of the
class and returns the object reference to the delegate. The kernel
maintains a record of the objects used by each delegate, so that it
can clean up when applications quit or delegates disappear.
The contract evaluator schedules the evaluation of contract objects in
the QuO runtime for a given application. A QuO application might
involve any number of contract objects, whose evaluation can be
triggered in the two following ways:
The QuO kernel is flexible about its environment. It can run in its
own process or in the same process as the application. It can run on
the same or a different host as that of the application. Finally, any
number of applications can share it whether or not they are on the
same host. The kernel also includes a built in GUI for displaying the
current state of contracts and system condition objects.
One would hope that the mixing of such disparate sources
(notwithstanding their variable languages) could be implemented with
some kind of multiple implementation inheritance. But even if all
sources could be expressed as classes in some common language, it
doesn't seem that inheritance of any kind could satisfactorily produce
the delegate we want. Our solution is to provide description
languages for different aspects of QoS, e.g., QoS contracts and
adaptation, and code generators that compile all the description
languages and produce as output a single source program with each
contributing element in the desired place. We see a basis for this
solution in the ideas of aspect-oriented programming (Kiczales 1996a,
1996b) and adaptive programming (Lieberherr, 1995).
One important feature of the QuO design is that delegates can be
stacked, nested, or arranged in any rooted configuration, each with an
association to its own or a shared contract object and each with its
own local adaptation. The client establishes a connection only to the
root delegate, whose connect() method recursively invokes those of the
delegates below it. In other words, instantiating a delegate and
calling connect() may in turn cause other delegates (of potentially
different types) to be instantiated and have their connect()'s be
called. Each will establish connections to a contract object and the
necessary system condition objects. References to kernel objects can
be passed to connect(), so that contract objects can be shared by
different delegates, and system condition objects can be shared by
different contract objects.3.1 The QuO Kernel
The QuO kernel is a library of services, implemented in Java,
providing the basis for the QuO runtime system, contracts, and system
condition objects. The kernel library provides two essential services:
an object factory and a contract evaluator.
Even though there might be many contracts concurrently ready for
evaluation, we expect the QuO kernel to consume a small overhead of
CPU cycles relative to the application, therefore QuO does not
concurrently evaluate contracts. Instead, triggering simply marks a
contract object as ready for evaluation. The contract evaluator is a
single high-priority thread that performs a round-robin check of all
contract objects in the runtime. If a given contract object is marked,
the evaluator evaluates it before proceeding to the next contract. If
evaluation of a particular contract is triggered more than once before
the evaluator gets to it, the contract is only evaluated
once. Contract evaluation is described in more detail below.3.2 QuO Delegates
The function of the QuO delegate is to wrap a remote object with local
behavior that can adapt to dynamic changes in QoS. The delegate code
is generated from IDL and QDL. The IDL specifies the delegate's
external interface to the client and internally specifies, at the
least, how many distinct methods can be mediated to implement adaptive
behavior. The contract description further constrains the
implementation of adaptability by specifying how many and which
distinct QoS regions the system can be in. Each region represents a
possible state of QoS to which the delegate can adapt. Finally, the
structural description specifies the selection mechanism and the
alternative behaviors, such as selecting between methods, selecting
between remote object references, or executing local code. QDL and the
code generation process is described in more detail in (Loyall 1998a,
1998b). The generated delegate is implemented in the language of the
client. Currently C++ and Java are supported.3.3 Bind-Time Configuration
Although the delegate provides an interface identical to the CORBA
object it wraps, the application's client code requires a slight
change to establish a binding to the delegate through QuO. We provide
a facility similar to the bind() call provided by some
commercial ORBs, such as Visibroker and Orbix. A QuO delegate's
connect() call performs the functionality of the bind(), i.e.,
it instantiates a CORBA proxy to a remote object. In addition, it
establishes the delegate's interface to the client and instantiates
and establishes connections to the necessary QuO runtime
objects. First, it either locates an existing QuO kernel thread or
creates one. It then calls the kernel's factory object to locate or
create the necessary contract and system condition objects. Finally,
connect() initializes and links all of the runtime objects as
necessary. In addition to initializing all necessary support within
the QuO kernel, the configuration procedure may also make initializing
calls to external quality of service brokers such as traders, or other
control mechanisms as intended by the QoS designer.3.4 Contract Objects
Contract objects collect data from the system condition objects and
analyze it to determine the current state of QoS in the system, as
captured by a set of current regions. Contract evaluation is triggered
as described in Section 3.1. The evaluation
of a specific contract begins with the collection of all the most
current system condition object values relevant to this contract. We
call this a snapshot to emphasize that it should appear
instantaneous to the application, though of course the values are read
serially. The values are read quickly, with minimal blocking, and all
values are cached for subsequent use during the evaluation. Each
region of the contract is defined by a predicate over the values of
some system condition objects. If a region's predicate is true, that
region is active. The QuO runtime evaluates the regions of a contract
in one large conditional statement of the form
The first predicate to evaluate as true determines the contract's current region, even if subsequent predicates also would evaluate to true. The kernel also includes a built in GUI for displaying the current state of contracts and system condition objects.
If the current region is different from the previous one, there is a region transition. If transition behavior has been specified for the particular transition, it is called at this time. Typically transition code includes either calls to methods of system condition objects or external mechanisms (to try to affect QoS), asynchronous calls to application callback objects (for notifying the client of changes in QoS), or both. Transition code can run synchronously or asynchronously. If it is specified to run synchronously then it is ensured that it will have finished before returning the current region to the delegate. This is useful if, for example, the delegate which triggered the evaluation should not act until certain external mechanisms are available. Asynchronous execution of transition is sometimes preferred since it reduces the overhead introduced by the QuO runtime in a remote method invocation.
Regions within contracts can be nested. In addition to helping organize the space of QoS regions, this adds flexibility in coding region transition behavior. For example, assume that there are two contract regions A, with nested region C, and B, with nested region E. If there were a region transition from E to C, the contract evaluator would run transition code associated with exiting region B, entering region A, going from B to A, exiting E, and entering C. Transition code can be associated with any or all of these in the QDL contract (Loyall, 1998a).
Each system condition object is implemented as a Java class which implements a simple Java interface common to all system condition objects. This interface must provide a getValue() method used by contract objects to query system condition objects for their values. Some simple system condition objects, e.g., those which are internal to a contract, are completely specified if they implement the getValue() method. All other system condition objects must be CORBA objects and as such must inherit from an existing hierarchy of system condition CORBA objects. This requirement makes it possible for the delegate to instatiate the system condition objects (using the QuO kernel's factory).
Since system condition objects can publish their IDL interface, they can provide control and access to their values outside the QuO runtime, effectively allowing system condition objects to function as interfaces between QuO and system resources, mechanisms, managers, etc. external to QuO. Each system condition object may also run any number of threads exclusive to either each instance or to a class. The freedom to implement system condition values with asynchronous control does not interfere with the general control flow within the kernel and facilitates the implementation of certain kinds of objects such as timers or polling resources. A growing library of system condition classes that we provide and that can be composed or inherited from also facilitates the implementation of new system condition objects.
The following examples give a sense of the expected granularity of system condition objects. Of the five system condition classes that follow, the first four are part of the standard system condition library and the fifth can be trivially implemented in terms of the first.
In this section we will address the degree of effort needed by programmers to write adaptive applications. The goal is that the large number of inexperienced programmers can reuse adaptive functionality created by fewer experts. We expect more users of adaptive than creators of adaptive objects and even fewer creators of underlying QoS mechanisms. We break the task of creating an adaptive application into several loosely defined roles:
The object designer may choose to modify an existing QoS contract, e.g., to add new QoS regions, new system condition objects, or different adaptation strategies. The edited QDL code is compiled to create Java classes for the contract object and a new delegate. Note that the object designer can easily change the external interface to the contract, by editing references to new system conditions and callback objects. Currently, there is no feature such as contract inheritance for managing contract modification and reuse.
The ReTINA project is developing a distributed processing environment for telecommunication applications (Bosco, 1996) that complies with the Telecommunications Information Networking Architecture (TINA) standard. ReTINA provides real-time audio and video and network QoS guarantees.
TAO is a real-time CORBA ORB that provides end-to-end QoS by vertically integrating CORBA middleware with OS I/O subsystems, communication protocols, and network interfaces. TAO is the first real-time ORB supporting end-to-end QoS over COTS platforms and ATM networks (Schmidt, 1997).
Monteiro, et al describe a technique for specifying QoS for congestion in networks. The technique involves selecting a set of quantifiable parameters that comprise the QoS (Monteiro, 1995). Then the upper and lower bounds for a correct operating interval are selected, followed by the upper and lower values of an interval in which service is degraded, but acceptable. These values are combined into a matrix that defines a service contract between a client and a server.
In contrast, QuO is a general-purpose framework for incorporating QoS in new and existing distributed applications, where QoS is defined as any non-functional property of an application. This definition of QoS is often called "ilities" because it includes properties such as availability, dependability, reliability, and so on.
BeeHive shares QuO's goals of providing QoS in the broad sense, although its approach is different. BeeHive provides a set of service-specific application programming interfaces (APIs), through which objects can request QoS from a resource manager (Stankovic, 1997). They are currently developing APIs for real-time, fault-tolerance, and security requirements. Each API allows an object to request a requirement in application terms, e.g., Mean Time to Failure for fault-tolerance. The resource manager translates each request into low-level resource requests.
The QuO framework borrows and builds upon the concepts of open implementation, aspect-oriented programming, and adaptive programming. Open implementation (Kiczales, 1996a) allows programmers to access and control non-algorithmic parts of an application's implementation, such as access to system resources, alternate implementations of an algorithm, insight or control over choice of data structures, or control of scheduling or other operating system functions. QuO allows programmers to access implementation details that are normally hidden behind IDL interfaces in distributed object applications, but are necessary to satisfy the QoS requirements of many applications.
QuO also provides support for Aspect-Oriented Programming (AOP) (Kiczales, 1996b) using its specification languages, as described in detail in (Loyall, 1998b). In AOP, a program is divided into aspects of concern, each of which is programmed separately in a language suitable for expressing the particular aspect. The application is constructed by weaving (using code generators) the aspects together into a single, executable application. QuO allows an application developer to design and develop functional behavior using IDL and QoS contracts, alternate implementation, and adaptation strategies using QDL. QuO's code generators weave these together into a single application. All of these aspects would ordinarily have to be interleaved throughout an application, making design, development, maintenance, and portability more difficult.
The QuO work also parallels research in adaptive programming (Lieberherr, 1995) and its offshoots, such as variation-oriented programming (Mezini, 1997). Adaptive programming is the creation of software that adapts itself automatically to the contexts in which it is used, including changes in behaviors, structures, etc.
Cukier, M., Ren, J., Sabnis, C., Henke, D., Pistole, J, Sanders, W., Bakken, D., Berman, M., Karr, D., and Schantz, R. (1998) AQuA: an adaptive architecture that provides dependable distributed objects. Seventeenth Symposium on Reliable Distributed Systems (SRDS-17), IEEE, October 1998.
Kiczales, G. (1996a) Beyond the black box: Open Implementation. IEEE Software January 1996.
Kiczales, G., Irwin, J., Lamping, J., Loingtier, J., Videria, C., Lopes, Maeda, C., and Mendhekar, A. (1996b) Aspect-Oriented Programming ACM Computing Surveys, 28(4es).
Lieberherr, K. (1995) Adaptive object-oriented software: the Demeter Method with propagation patterns, PWS Publishing Company, Boston.
Loyall, J., Schantz, R., Zinky, J., Bakken, D., (1998a) Specifying and measuring quality of service in distributed object systems, Proceedings of the First International Symposium on Object-oriented Real-time distributed Computing (ISORC '98), April 20-22, Kyoto, Japan.
Loyall, J., Bakken, D., Schantz, R., Zinky, J., Karr, D., and Vanegas, R. (1998b) QoS aspect languages and their runtime interactions, (to appear in) Lecture Notes in Computer Science, Springer-Verlag.
Mezini, M. (1997) Variation-oriented programming beyond classes and inheritance, Ph.D. thesis, University of Siegen, Germany.
Monteiro, E., Boavida, F., Quadros, G., and Freitas, V. (1995) Specification, quantification and provision of quality of Service and congestion control for new communication services. Proceedings of the 16th AFCEA Europe Symposium, AFCEA (Association for Communications, Electronics, Intelligence & Information Systems Professionals)/IEEE COMSOC/IEE, October 18-20, Brussels, Belgium, 58-68.
Object Management Group (1996), CORBA 2.0, July 96 revision, OMG Document 96-08-04.
Schmidt, D.C., Bector, R., Levine, D., Mungee, S., and Parulkar, G. (1997) An ORB end system architecture for statically scheduled real-time applications. Proceedings of the Workshop on Middleware for Real-Time Systems and Services, IEEE, San Francisco.
Stankovic, J., Son, S., Liebeherr, J. (1997) BeeHive: global multimedia database support for dependable, real-time applications Computer Science Report CS-97-08, University of Virginia, April 21, 1997.
Zinky, J., Bakken, D., and Schantz, R. (1997) Architectural support for quality of service for CORBA objects. Theory and Practice of Object Systems, 3(1).