Goal Achievement Evaluation in the NFR Framework

by Sam Supakkul and Lawrence Chung

Introduction | Evaluation Catalog under Closed World Assumption | Evaluation Catalog under Open World Assumption |


Using goal-oriented modeling, an analyst represents stakeholders' hard (functional) and soft (non-functional) goals, then refines the goals and explores alternatives to achieve them. When using the NFR Framework [book (draft), paper (draft)] to model non-functional requirements (NFRs) as softgoals, the analyst explores goal operationalization alternatives and selects the desirable operationalizations by labeling them as Satisficed (denoted by a check mark). The label of each selected softgoal is evaluated to determine how it impacts the achievement of its parent goal. This evaluation is based on the evaluation procedure as depicted in the catalogs in Fig. 6 or 7, depending on the appropriate world assumption (closed or open).

Under the closed world assumption, "The implicit representation of negative facts presumes total knowledge about the domain being represented."[Reiter78], while under the open world assumption, "gaps in one's knowledge about the domain are permitted" [Reiter78]. Take Fig. 1a for an example, only TakeElixir is assumed under the closed world assumption to be the only knowledge about how to achieve StayHealthy goal (denoted by MAKE/++). When TakeElixir is considered invalid and thus labelled as Denied (denoted by a cross mark), StayHealthy goal is in turn evaluated as a result to be Denied because there is no other known ways to stay healthy (under the closed world assumption). On the other hand, in Fig. 1b, TakeElixir is assumed to be one way, but not the only way, to StayHealthy. When TakeElixir is denied, StayHealthy is not also considered denied because there may be other ways to stay healthy that are not represented. Fig. 1c and 1d show additional examples of label evaluation with negative contributions where SmokeCigarette is known to be the only way that greatly hurts StayHealthy goal (denoted by BREAK/--) under the closed world assumption, but considered to be only one way that breaks StayHealthy goal under the open world assumption. When SmokeCigarette is denied, we can consider that StayHealthy is WeaklySatisficed (W+) under the closed world assumption as the only known health hazard (SmokeCigarette) has been denied while we cannot make the same conclusion under the open world assumption as SmokeCigarette is not the only health hazard.

Figure 1. Comparisons of Label Evaluation Between Closed and Open World Assumptions

The closed assumption has been the key assumption for information systems such as in database systems [Reiter78]. However, it may not be intuitive for some cases of world or organizational level modeling, especially for stakeholder goals modeling, as it deals more with stakeholders' perspective, rather than system's perspective, of the world. The open world assumption could provide a better world assumption. For instance, the examples in Fig. 1b and 1d above are more intuitive than examples in Fig. 1a and 1c from life experience perspective.

To support the goal achievement evaluation as described above in a goal graph, the general rule is that Denied labels are evaluated under the closed world assumption and are omitted from goal achievement evaluation under the open world assumption. For example, when considering Password and RetinaScan alternatives for achieving Authentication of Account access (Fig. 2a), Password may be chosen over RetinaScan because the latter is more costly to collect and maintain biometric samples. To reflect the decision, the Password alternative is labeled as Satisficed to indicate the selection and RetinaScan is labeled as Denied to reflect the rejection. Under the open world assumption, the Denied label of the RetinaScan is not evaluated and the Satisficed label is evaluated and propagated to WeaklySatisficed label for the Authentication[Account] goal over the weakly positive Help(+) contribution. On the other hand, in Fig. 2b under the closed world assumption, both Satisficed and Denied labels are evaluated and propagated. Individually, Satisficed label would be propagated to WeaklySatisficed over a Help(+) contribution as shown in Fig. 2a and Denied label propagated to Denied label over a Help(+) contribution as shown in Fig. 1a, resulting in an Undecided label as shown in Fig. 2b. Having evaluated only the positive labels makes the goal achievement evaluation in this goal graph seem more natural and intuitive.

Figure 2. The Open World Assumption is Useful in General Goal-Oriented Modeling

However, in some cases, the closed world assumption can be useful in goal modeling, for example when modeling obstacles to goal achievement and the respective countermeasures. This type of problem modeling is common in security engineering where emphasis is often on mitigating known threats. Using Fig. 3a as an example, "Hacked passwords" is a problem to be avoided, which can be realized by "Dictionary attack" and "Hack passwords repository" specific problems. An organization may mitigate the problems by using "Strong passwords" and "Encryption" respectively, the mitigation relationships are represented by BREAK/-- contributions. "Strong passwords" may be further And-decomposed to "Non-dictionary words" and "Frequently changed passwords" countermeasures. Let's suppose the stakeholders agree with the mitigation techniques as denoted by the Satisficed label (checked). The Satisficed labels are evaluated and propagated to Denied labels on "Dictionary attack" and "Hack passwords repository" respectively over the BREAK/-- contributions. Under the closed world assumption, the Denied labels would be propagated to a Denied label on "Hacked passwords" across the MAKE/++ contributions. Using the closed world assumption is desirable as it is difficult to know and foresee all potential problems. Therefore, it is only practical to consider the top-level problem mitigated if known threats are sufficiently mitigated. To further illustrate the point, let's suppose the goal model is evaluated using the open world assumption. The Denied labels on "Dictionary attack" and "Hack passwords repository" would not evaluated. As the result, the label of "Hacked passwords" remains unknown, an inconclusive status despite that all known threats have been mitigated.

The notations for problems (e.g. Hacked passwords) and stakeholder level problem-oriented modeling are discussed more in detail here.

Figure 3. The Closed World Assumption is Useful in Problem-Oriented Modeling

The video below shows the differences and impacts of closed and open world assumptions on goal satisficing evaluation in action using the evaluation catalogs presented below. A larger video is available here.

Uniform Label Evaluation

Label evaluation and propagation has been discussed in terms of goal-to-goal propagation so far (see Fig. 3), which is also illustrated in the catalogs in Fig. 6 and 7. However, the label evaluation procedure in fact treats softgoals and contribution links generically at a higher level of abstraction as propositions. Therefore, the propagation illustrated in the catalogs are applicable for all proposition-to-proposition combination, including goal-to-goal, goal-to-contribution, contribution-to-goal, and contribution-to-contribution propagations. Figure 4 shows examples of goal-to-goal and goal-to-link propagation swhere the Satisficed label of Auditing is propagated to a Satisficed label on Accuracy, while the Denied label of the MAKE (++) contribution link between StayHealthy and TakeMagicalDrug is derived from the Satisficed label of "Not scientifically proven" Claim that is reversed from Satisficed to Denied across the BREAK (--) contribution according to the catalogs illustrated below.

Figure 4. Examples of Label Propagation (a) Goal-to-Goal and (b) Goal-to-Link

The Effect of Contribution Link's Satisficing on the Label Propagation

Notice that, in Fig. 4, the label of Accuracy is evaluated but that of StayHealthy is unknown despite their respective offspring goals (Auditing and TakeElixir) are both satisficed. The label of a goal is evaluated and propagated only if the contribution towards its parent goal is satisficed. For instance, in the case of Make(++) contribution, the label evaluation is defined as:

"If the offspring is satisficed when the interdependency itself is satisficed then the parent is satisficeable" [book p.67]

or more formally,

[lecture p.69]

Figure 5 shows additional examples of the effect of the contribution on the label propagation over a BREAK (--) contribution. In Fig.1a, the label of Op1 goal is not evaluated because the BREAK(--) contribution is Denied as a result of a label propagation of the Satisficed label of Claim1 over a Break(--) contribution. However, if the Break(--) contribution of Claim1 is countered by Claim2 in Fig. 5b, the label of Claim1 would not be evaluated. As a reslt, the Break(--) contribution of Op1 towards NFR1 is not denied, therefore, the label of Op1 is evaluated and propagated as normal as if Claim1 did not exist.

Figure 5. Effect of the label of MAKE(--) contribution on label propagation

The rules for other contribution links (e.g. HELP/+, HURT/-) have a similar clause that in effect defines that the parent goal's label be evaluated only when the contribution in question is Satisficed. This is reflected in the catalogs in Fig. 6 and 7 where the propagation are as illustrated only when the links are Satisficed as denoted by a check mark on the links.


A Catalog of Label Evaluation Using the Closed World Assumption

This catalog illustrates the label propagation procedure under the closed world assumption [Reiter78], as defined by the NFR Framework [book (draft), paper (draft)]. The evaluation rules defined in the catalog are implemented in the RE-Tools.

Figure 6. A Catalog of Label Evaluation Using Closed World Assumption


A Catalog for Label Evaluation Using the Open World Assumption

This catalog illustrates the label evaluation under the open world assumption as supported by the RE-Tools. This is an extension to the label evaluation procedure as defined in the NFR Framework [book (draft), paper (draft)] that assumes the closed-world assumption.

Figure 7. A Catalog of Label Evaluation Using Open World Assumption



© 2007-2010 Sam Supakkul
Updated Nov. 10, 2010