REGISTER  |  SIGN IN
United States Patent5603029
Aman , ; et al.February 11, 1997
Title
System of assigning work requests based on classifying into an eligible class where the criteria is goal oriented and capacity information is available
Abstract
Apparatus and accompanying methods for use preferably in a multi-system shared data (sysplex) environment (100), wherein each system (110) provides one or more servers (115), for dynamically and adaptively assigning and balancing new work and for new session requests, among the servers in the sysplex, in view of attendant user-defined business importance of these requests and available sysplex resource capacity so as to meet overall business goals. Specifically, systems and servers are categorized into two classes: eligible, i.e., goal-oriented servers running under a policy and for which capacity information is currently available, and candidate, i.e., servers which lack capacity information. Work requests for a client application are assigned first to various eligible systems and eligible servers thereon based on their current capacity to accept new work and in a manner that meets business goals inherent in a sysplex policy; followed, if additional servers are requested by that application, to candidate systems and candidate servers thereon. As to session placement, first those system(s) are selected that have lowest utilization, at a target importance level, but with sufficient available capacity at that level. Competing servers on the selected system(s) are then evaluated based on their corresponding session count data to yield a single resulting server. Thereafter, identification of multiple servers and their corresponding weights are returned to, e.g., a client application, for eventual routing of work requests to those servers, or the identification of a single server is returned to that application for establishing a new session therewith.
Inventors:Aman; Jeffrey D. (Wappingers Falls, NY), Cotner; Curt L.  (Gilroy, CA), Dillenberger; Donna N. T.  (Yorktown Heights, NY), Emmes; David B.  (Poughkeepsie, NY)
Assignee:International Business Machines Corporation (Armonk, NY)
Appl. No.:476157
Filed:June 7, 1995
Current U.S. Class:718/105 707/200 710/39 
Field of Search:395/200.01,200.03,600,650,826 364/242.94,229,230.3,280,281,284.4,902
U.S. Patent Documents
4481583November 1984Mueller
5241677August 1993Naganuma et al.
5341477August 1994Pitkin et al.
5537542July 1996Eilert et al.
Other References
D Sitaram et al, Issues in the . . . Load Skew, Parallel and Distributed Information Systems, pp. 214-223 1993. .
Lin et al., "A Dynamic Load Balancing Policy with a Central Job Dispatcher (LBC)", Proceedings of the Eleventh International Conference on Distributed Computing Systems, Arlington, Texas, 20-24 May 1991, pp. 264-271. .
Geise, "Dynamic Load Balancing in a Cluster of Loosely Coupled Systems", IBM Technical Disclosure Bulletin, vol. 37, No. 7, Jul. 1994, pp. 243-244. .
Herrin II, et al., "The Benefits of Service Rebalancing", Proceedings of the Third Workship on Workstation Operating Systems, Key Biscayne, FL, 23-24 Apr. 1992, pp. 104-110. .
Menditto et al., "Single System Image And Load Balancing For Network Access To A Loosely Coupled Complex", IBM Technical Disclosure Bulletin, vol. 34, No. 9, Feb. 1992, pp. 464-271..~
Primary Examiner: Lee; Thomas C.
Assistant Examiner: Chen; Anderson I.
Attorney, Agent or Firm:Kinnaman, Jr.; William A. Michaelson; Peter L. Michaelson & Wallace
Claims

We claim:
1. In a multi-processing environment having a plurality of computer systems, each of said systems having an operating system instance resident thereon and providing at least one application server, a method for assigning work requests among individual ones of the servers in order to meet business goals inherent in a policy governing the environment, the method comprising the steps of:
in response to an incoming request to route work,
classifying each active one of the systems either as an eligible system that then exhibits at least a minimum pre-defined capacity utilization, at a lowest one of a number of pre-defined business importance levels among all the systems, over a first pre-defined time interval, or as a candidate system so as to form first and second sets of eligible and candidate systems, respectively; each eligible system being one of the systems that is goal-oriented and runs under the policy for which current capacity information is available, and each candidate system being any remaining one of said plurality of systems other than all the eligible systems; and
forming, in conjunction with said classifying step, third and fourth sets of eligible and candidate servers, respectively, such that said third and fourth sets contain identifications of all active ones of the servers residing on each of the systems in said first and second sets, respectively;
assigning, in response to said classifying step, a system weight to each one of the systems in the first and second sets, the system weight representing an amount of total available capacity utilized at the lowest one of the business importance levels, by said each one system in the first and second sets;
determining, in conjunction with said system weight assigning step and for each different one of the systems in said first and second sets, a corresponding server weight for each different server residing on said each different one system;
forming, in response to said third and fourth sets, an output server set populated first by identifications of eligible servers, and their associated server weights, taken, successively and in order of descending server weights, from said third set followed, by identification of servers, and their associated server weights, taken successively from said fourth set; and
routing work requests to each one of the servers identified in the output server set wherein, of a total number of work requests to be routed to all of the identified servers, an amount of the total work requests is routed to each one identified server in the output server set in proportion to the server weight associated therewith.

2. The method in claim 1 wherein the output server set forming step comprises the steps of:
writing identifications of successive ones of the eligible servers, if any, and their associated weights, from the third set into the output server set until the output server set is full or identifications of all the eligible servers in the third set have been taken; and
if the output server set is not full, writing identifications of successive ones of the candidate servers, and the associated server weights, from the fourth set into the output server set until either the output server set is full or identifications of all the candidate servers in the fourth set have been taken.

3. The method in claim 2 wherein the system weight assigning step comprises the steps of:
ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system, at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of said eligible systems.

4. The method in claim 3 wherein the system weight assigning step further comprises the steps of:
in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, setting the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, setting the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

5. The method in claim 2 wherein the server weight determining step comprises the steps of, for said eligible servers residing on each one eligible system:
where the number of eligible servers residing thereon exceeds the system weight therefor, setting the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
where the number of eligible servers residing therein is less than the system weight therefor,
selecting one of the eligible servers residing on said each one eligible system as a selected eligible server;
assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

6. The method in claim 5 wherein the server weight determining step comprises the steps of, for said candidate servers residing on each one candidate system:
where the number of candidate servers residing thereon exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
where the number of candidate servers residing therein is less than the system weight therefor,
selecting one of the candidate servers residing on said each one candidate system as a selected candidate server;
assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

7. The method in claim 6 wherein the server weight determining step further comprises the steps of:
if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero-weighted eligible servers is to be written into the output server set:
writing the identifications of each successive zero-weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken; and
assigning a pre-defined fixed server weight to the output eligible server; and
if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
assigning a pre-defined fixed server weight to the output candidate server.

8. The method in claim 2, wherein the multi-processing environment is a sysplex, further comprising the steps of:
receiving, from a network connected to the sysplex, the incoming request in a routing node, the routing node being one of the plurality of systems; and
undertaking the classifying, third and fourth set forming, assigning, determining and output set forming steps by the operating system instance residing in the routing node.

9. The method in claim 8 further comprising the steps of:
communicating, through the network, the output server set from the routing node to a client application executing in a client computer connected through the network to the sysplex; and
said work requests routing step comprises the step, performed by the client application, of routing each of the total number of work requests, through the network, to a corresponding one of the servers in the sysplex and identified in the output server set.

10. The method in claim 9 wherein the system weight assigning step comprises the steps of:
ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system and at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of aid eligible systems.

11. The method in claim 10 wherein the system weight assigning step further comprises the steps of:
in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, setting the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, setting the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

12. The method in claim 11 wherein the server weight determining step comprises the steps of, for said eligible servers residing on each one eligible system:
where the number of eligible servers residing thereon exceeds the system weight therefor, setting the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
where the number of eligible servers residing therein is less than the system weight therefor,
selecting one of the eligible servers residing on said each one eligible system as a selected eligible server;
assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

13. The method in claim 12 wherein the server weight determining step comprises the steps of, for said candidate servers residing on each one candidate system:
where the number of candidate servers residing thereon exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
where the number of candidate servers residing therein is less than the system weight therefor,
selecting one of the candidate servers residing on said each one candidate system as a selected candidate server;
assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

14. The method in claim 13 wherein the server weight determining step further comprises the steps of:
if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero-weighted eligible servers is to be written into the output server set:
writing the identifications of each successive zero-weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken; and
assigning a pre-defined fixed server weight to the output eligible server; and
if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
assigning a pre-defined fixed server weight to the output candidate server.

15. The method in claim 14 wherein the pre-defined utilization equals at least 5% of total capacity provided by said each active one system and the pre-defined fixed weight is one.

16. The method in claim 9 wherein the server weight determining step comprises the steps of, for said eligible servers residing on each one eligible system:
where the number of eligible servers residing thereon exceeds the system weight therefor, setting the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
where the number of eligible servers residing therein is less than the system weight therefor,
selecting one of the eligible servers residing on said each one eligible system as a selected eligible server;
assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

17. The method in claim 16 wherein the server weight determining step comprises the steps of, for said candidate servers residing on each one candidate system:
where the number of candidate servers residing thereon exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
where the number of candidate servers residing therein is less than the system weight therefor,
selecting one of the candidate servers residing on said each one candidate system as a selected candidate server;
assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

18. The method in claim 17 wherein the server weight determining step further comprises the steps of:
if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero- weighted eligible servers is to be written into the output server set:
writing the identifications of each successive zero-weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken; and
assigning a pre-defined fixed server weight to the output eligible server; and
if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
assigning a pre-defined fixed server weight to the output candidate server.

19. The method in claim 18 wherein the system weight assigning step comprises the steps of:
ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system and at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of said eligible systems.

20. The method in claim 19 wherein the system weight assigning step further comprises the steps of:
in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, setting the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, setting the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

21. The method in claim 20 wherein the pre-defined utilization equals at least 5% of total capacity provided by said each active one system and the pre-defined fixed weight is one.

22. The method in claim 8 further comprising the steps of:
receiving, at the routing node, the total number of work requests from a client application executing in a client computer connected through the network to the sysplex; and
said work requests routing step comprises the step, performed by the operating instance residing in the routing node, of routing each of the total number of work requests received thereat to a corresponding one of the servers in the sysplex and identified in the output server set.

23. The method in claim 22 wherein the system weight assigning step comprises the steps of:
ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system and at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of said eligible systems.

24. The method in claim 23 wherein the system weight assigning step further comprises the steps of:
in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, setting the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, setting the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

25. The method in claim 24 wherein the server weight determining step comprises the steps of, for said eligible servers residing on each one eligible system:
where the number of eligible servers residing thereon exceeds the system weight therefor, setting the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
where the number of eligible servers residing therein is less than the system weight therefor,
selecting one of the eligible servers residing on said each one eligible system as a selected eligible server;
assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

26. The method in claim 25 wherein the server weight determining step comprises the steps of, for said candidate servers residing on each one candidate system:
where the number of candidate servers residing thereon exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
where the number of candidate servers residing therein is less than the system weight therefor,
selecting one of the candidate servers residing on said each one candidate system as a selected candidate server;
assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

27. The method in claim 26 wherein the server weight determining step further comprises the steps of:
if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero-weighted eligible servers is to be written into the output server set:
writing the identifications of each successive zero-weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken; and
assigning a pre-defined fixed server weight to the output eligible server; and
if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
assigning a pre-defined fixed server weight to the output candidate server.

28. The method in claim 27 wherein the pre-defined utilization equals at least 5% of total capacity provided by said each active one system and the pre-defined fixed weight is one.

29. The method in claim 28 wherein the server weight determining step comprises the steps of, for said eligible servers residing on each one eligible system:
where the number of eligible servers residing thereon exceeds the system weight therefor, setting the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
where the number of eligible servers residing therein is less than the system weight therefor,
selecting one of the eligible servers residing on said each one eligible system as a selected eligible server;
assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

30. The method in claim 29 wherein the server weight determining step comprises the steps of, for said candidate servers residing on each one candidate system:
where the number of candidate servers residing thereon exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
where the number of candidate servers residing therein is less than the system weight therefor,
selecting one of the candidate servers residing on said each one candidate system as a selected candidate server;
assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor; and
assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

31. The method in claim 30 wherein the server weight determining step further comprises the steps of:
if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero-weighted eligible servers is to be written into the output server set:
writing the identifications of each successive zero-weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken; and
assigning a pre-defined fixed server weight to the output eligible server; and
if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
assigning a pre-defined fixed server weight to the output candidate server.

32. The method in claim 31 wherein the system weight assigning step comprises the steps of:
ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system and at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of said eligible systems.

33. The method in claim 32 wherein the system weight assigning step further comprises the steps of:
in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, setting the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, setting the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

34. The method in claim 33 wherein the pre-defined utilization equals at least 5% of total capacity provided by said each active one system and the pre-defined fixed weight is one.

35. In a multi-processing environment having a plurality of computer systems, each of said systems having an operating system instance resident thereon and providing at least one application server, apparatus for assigning work requests among individual ones of the servers in order to meet business goals inherent in a policy governing the environment, the apparatus comprising:
means, responsive to an incoming request to route work, for classifying each active one of the systems either as an eligible system that then exhibits at least a minimum pre-defined capacity utilization, at a lowest one of a number of pre-defined business importance levels among all the systems, over a first pre-defined time interval, or as a candidate system so as to form first and second sets of eligible and candidate systems, respectively; each eligible system being one of the systems that is goal-oriented and runs under the policy for which current capacity information is available, and each candidate system being any remaining one of said plurality of systems other than all the eligible systems; and
means, operative in conjunction with said classifying means, for forming third and fourth sets of eligible and candidate servers, respectively, such that said third and fourth sets contain identifications of all active ones of the servers residing on each of the systems in said first and second sets, respectively;
means for assigning a system weight to each one of the systems in the first and second sets, the system weight representing an amount of total available capacity utilized at the lowest one of the business importance levels, by said each one system in the first and second sets;
means for determining, in conjunction with said system weight assigning means and for each different one of the systems in said first and second sets, a corresponding server weight for each different server residing on said each different one system;
means for forming, in response to said third and fourth sets, an output server set populated first by identifications of eligible servers, and their associated server weights, taken, successively and in order of descending server weights, from said third set followed, by identification of servers, and their associated server weights, taken successively from said fourth set; and
means for routing work requests to each one of the servers identified in the output server set wherein, of a total number of work requests to be routed to all of the identified servers, an amount of the total work requests is routed to each one identified server in the output server set in proportion to the server weight associated therewith.

36. The apparatus in claim 35 wherein the output server set forming means comprises:
means for writing identifications of successive ones of the eligible servers, if any, and their associated weights, from the third set into the output server set until the output server set is full or identifications of all the eligible servers in the third set have been taken; and
means for writing, if the output server set is not full, identifications of successive ones of the candidate servers, and the associated server weights, from the fourth set into the output server set until either the output server set is full or identifications of all the candidate servers in the fourth set have been taken.

37. The apparatus in claim 36 wherein the system weight assigning means comprises:
means for ascertaining the total available capacity utilized as total capacity utilized by all of the eligible systems in the first set at the lowest one of the business importance levels which has sufficient capacity; and
means for specifying, for each one of the eligible systems in the first set, the system weight therefor as a function of a ratio of capacity utilized by said each one eligible system and at the lowest one of the business importance levels which has sufficient capacity, divided by the total available capacity utilized at a target importance level among all of said eligible systems.

38. The apparatus in claim 37 wherein the system weight assigning means further comprises:
first means for setting, in the event identifications of more candidate systems than identifications of eligible systems are to be taken from said second and first sets, respectively, the system weight for each different eligible system to be taken from the second set to a pre-defined fixed weight; and
second means for setting, in the event identifications of more eligible systems than identifications of candidate systems are to be taken from said first and second sets, respectively, the system weight for each different candidate system to be taken from the first set to a minimum of either an average and a median of the system weights of all the eligible systems to be taken from the first set.

39. The apparatus in claim 36 wherein the server weight determining means comprises:
means for setting, where the number of eligible servers residing on each one eligible system exceeds the system weight therefor, the server weight for each of the eligible servers residing on said each one eligible system equal to the system weight therefor divided by the number of eligible servers; and
means, operative where the number of eligible servers residing therein is less than the system weight therefor:
for selecting one of the eligible servers residing on said each one eligible system as a selected eligible server,
for assigning the entire system weight for said each one eligible system to said selected eligible server as the server weight therefor, and
for assigning zero as the server weight to each remaining one of said eligible servers residing on said each one eligible system.

40. The apparatus in claim 39 wherein the server weight determining means comprises:
means for setting, where the number of candidate servers residing on each one candidate system exceeds the system weight therefor, setting the server weight for each of the candidate servers residing on said each one candidate system equal to the system weight therefor divided by a number of candidate servers residing thereon; and
means, operative where the number of candidate servers residing therein is less than the system weight therefor:
for selecting one of the candidate servers residing on said each one candidate system as a selected candidate server,
for assigning the entire system weight for said each one candidate system to said selected candidate server as the server weight therefor, and
for assigning zero as the server weight to each remaining one of said eligible servers residing on said each one candidate system.

41. The apparatus in claim 40 wherein the server weight determining means further comprises:
means, operative if one or more eligible servers identified in the third set each has a zero server weight and the identification of one or more of said zero-weighted eligible servers is to be written into the output server set:
for writing the identifications of each successive zero- weighted eligible server into the output server set as an output eligible server until the output server set is full or the identifications of all the zero-weighted eligible servers have been taken, and
for assigning a pre-defined fixed server weight to the output eligible server; and
means, operative if one or more candidate servers identified in the fourth set each has a zero server weight and the identification of one or more of said zero-weighted candidate servers is to be written into the output server set:
for writing the identifications of each successive zero-weighted candidate server into the output server set as an output candidate server until the output server set is full or the identifications of all the zero-weighted candidate servers have been taken; and
for assigning a pre-defined fixed server weight to the output candidate server.

42. The apparatus in claim 36, wherein the multi-processing environment is a sysplex, further comprising: means for receiving, from a network connected to the sysplex, the incoming request in a routing node, the routing node being one of the plurality of systems.

43. The apparatus in claim 42 further comprising:
means for communicating, through the network, the output server set from the routing node to a client application executing in a client computer connected through the network to the sysplex; and
said work requests routing means comprises means for routing each of the total number of work requests, from the client application through the network, to a corresponding one of the servers in the sysplex and identified in the output server set.

44. The apparatus in claim 42 further comprising:
means for receiving, at the routing node, the total number of work requests from a client application executing in a client computer connected through the network to the sysplex; and
said work requests routing means comprises means, situated within the operating instance in the routing node, for routing each of the total number of work requests received thereat to a corresponding one of the servers in the sysplex and identified in the output server set.
Description
CROSS-REFERENCE TO RELATED APPLICATION

This application describes and claims subject matter that is also described in co-pending United States patent application of Jeffrey D. Aman, David B. Emmes and David W. Palmieri entitled "APPARATUS AND ACCOMPANYING METHOD FOR ASSIGNING SESSION REQUESTS IN A MULTI-SERVER SYSPLEX ENVIRONMENT"; filed concurrently herewith; assigned Ser. No. 08/488,374, now pending and which is also assigned to the present assignee hereof.

BACKGROUND OF THE DISCLOSURE

1. Field of the Invention

The invention relates to apparatus and accompanying methods for use preferably, though not exclusively, in a multi-system shared data (sysplex) environment, wherein each system provides one or more servers, for dynamically and adaptively assigning and balancing new work and for session requests, among the servers in the sysplex, in view of attendant user-defined business importance of the requests and available sysplex resource capacity so as to meet overall business goals.

2. Description of the Prior Art

Prior to the early-1980s, large scale computing installations often relied on using a single monolithic computer system to handle an entire processing workload. If the system failed, all processing applications in the workload were suspended until the failure was remedied. While a resulting processing delay was tolerated at first, as increasingly critical applications were processed through the system, any such ensuing delays became increasingly intolerable. Furthermore, as processing needs increased, the entire system was eventually replaced with a new one of sufficient capacity. Replacing systems in that manner proved to be extremely expensive and very inefficient. However, at that time, few workable alternatives existed, to using monolithic systems, that appreciably eliminated both these outages and an eventual need to replace the entire system.

To efficiently address this need, over the past several years and continuing to the present, computer manufacturers are providing processing architectures based on a multi-system shared data approach. Through these architectures, multiple large-scale computer systems, each of which is often referred to as a computer processing complex (CPC) or a central electronic complex (CEC), are inter-connected, through, for example, a coupling facility or other inter-processor communication mechanism, to permit each such system to gain read-write access to data residing on one or more shared input/output devices, such as a direct access storage device (DASD). The resulting inter-connected computer system is commonly referred to as a "sysplex". In a sysplex, as with a typical multi-processing environment, a processing workload is generally distributed among all of the inter-connected computer systems such that each computer system is responsible for processing a portion of the entire workload. Conventionally then, each of these systems executes its own portion of the total workload independently of that undertaken by any the other such systems. Owing to the inherent high reliability and highly cost-efficient expansion potential of a sysplex architecture, sysplexes are particularly attractive in handling so-called critical business support applications that involve real-time transaction processing and can tolerate essentially no downtime.

Generally, within a sysplex, separate copies (so-called "instances") of an application are resident and simultaneously active on more than one of the computer systems, each henceforth referred to as a "machine" to differentiate the physical hardware therefor, and, based upon, e.g., the processing capacity required for the application, often on all or most of these machines.

Certain currently available machines that can be readily incorporated into a sysplex, such as illustratively the Enterprise System/9000 (ES/9000) Series manufactured by the International Business Machines (IBM) Corporation, can each support, if appropriately configured, multiple actively and simultaneously executing copies of various operating systems (OS) to implement separate corresponding individual and unique application processing environments. (Enterprise System/9000 is a registered trademark, and ES/9000 is a trademark, of the International Business Machines Corporation.) Each of these environments utilizes a separate copy of the operating system, such as the MVS/ESA (henceforth simply "MVS") OS, which forms a so-called OS "image", along with an instance (s) of corresponding application program (s) and a dedicated storage area (typically a logical partition--"LPAR"). (MVS/ESA is a trademark, and IBM is a registered trademark, of the International Business Machines Corporation.) As such, each such environment thus constitutes a separate "processing system" (henceforth referred to, for the sake of brevity, as simply a "system"). Each application instance that executes on any such system constitutes a separate application server (henceforth referred to as simply a "server" or "real instance") to service a portion of the total workload presented to the overall application on the sysplex. A system, based on its processing capacity and that required by the corresponding applications, can implement one or more corresponding servers.

A recurring difficulty in using multiple servers has been how to effectively balance the current processing load across the servers. Traditionally, operating systems, such as the MVS OS, relied on a totally static approach to allocating available sysplex resources, such as available servers, processing time, and processor storage, to each current work request. To accomplish this, system administrators utilized historic performance measurements of past workload processing to project just what sysplex resources would then be available as each new work request was presented to the sysplex and how these available resources should be allocated to handle that request. The overall goal of the administrator in allocating these resources to the current work requests was simply to keep each system maximally busy, i.e., to utilize as many available clock cycles thereon as possible, in effect keeping that system "pegged" and hence maximizing its throughput.

For a sysplex, historic averaged performance measurements were made over a variety of intervals and in relation to a variety of causes: e.g., on a day-by-day basis, on an hour-by-hour basis, and by each individual application, as well as in relation to other time or usage-related metrics. Based on this data, a system administrator determined, from projections made from this historic data: how current work requests should be assigned to individual servers, a dispatching priority for each one of these requests that would be queued on each server, i.e., the order in which these requests were to be executed on that server, and the amount of resources at that server to allocate to each new work request presented thereto. Once these determinations were made for an expected sysplex workload in view of the goal of maximizing throughput of each system, the administrator simply instructed the operating system at each server accordingly. Through this effort, the administrator strove to distribute the total workload, as he or she then foresaw it, across all the servers as evenly as possible consistent with maximizing the throughput of all the servers.

Unfortunately, dispatching relationships existing between different work requests queued for execution in a sysplex tend to be extremely complex. Not only were accurate predictions of workload and resource allocations across multiple servers extremely tedious and difficult to create, but also such allocations were based on static, i.e., fixed, workloads having concomitant demands for each server that were not expected to change over time. Unfortunately, in practice, workloads do change, often significantly with time. Predictions predicated on static workloads simply could not accommodate subsequent changes in sysplex workload. Hence, each time a new application or a change in arrival patterns or demand for existing workload was to occur in a sysplex, the administrator had to totally re-formulate (re-iterate) the predictions and accordingly change the work request assignments and resource allocations therefor in order to accommodate the additional work request. Doing so would, of necessity, involve determining whether any processing conflicts would arise by introduction of the new work request vis-a-vis existing requests then being processed and then resolving all such conflicts. Moreover, not only did each subsequent iteration consume substantial effort, but a static prediction assumed that future work requests, even for the same application, would behave as past work requests therefor did. Since this assumption often failed to account for sudden increases, i.e., spikes, in processing demand by an application, such as a surge in users and/or transactions therefor, these static workload predictions, coupled with fixed work request assignments and pre-determined sysplex resource allocations, simply could not efficiently accommodate dynamic changes in workload. Hence, imbalances between systems frequently arose through which one or more systems would be heavily loaded while others would be lightly loaded. Consequently, work requests that then had a high degree of business importance, and either could not wait or could tolerate only minimal delay, might nevertheless be queued on the former systems for relatively some time awaiting dispatch for execution, while queued work requests of much lesser business importance would be dispatched far more quickly on the latter systems. Hence, the sysplex, due to inter-system processing imbalances resulting from static work assignment and pre-defined resource allocation, was often unable to meet its business goals, i.e., its total current processing demand was not met and accompanying processing results were not provided in a manner temporally consistent with the business importance of the underlying application(s).

While the art teaches several approaches for providing improved workload balancing in a sysplex, or generally a multi-processing environment, all these approaches suffer drawbacks that limit their attractiveness and general utility.

Specifically, an early attempt at balancing workload across multiple systems involved physically connecting a certain number of users, on a pre-defined basis often in terms of physical wiring or other such interconnections, to each system and thereafter routing all work requests, incoming over a network and originating from those users, to only that system to the exclusion of all other systems. For brevity, we will refer to this approach hereinafter as "connection based" balancing. The user assignments, specifically the interconnections, were initially established such that an approximately equal number of users would be connected, at any one time, to each system. Under this approach, once a user, through a physical connection to a given system, established a terminal session thereat, all the work requests for that session were routed solely and directly to that given system. Unfortunately, significant inter-system processing imbalances frequently occurred. In that regard and at one extreme, one or a small number of users using one common system but having a collectively large demand for processing, could overwhelm that system to the detriment of all the other users executing applications thereat; while a large number of other such users, such as those having sessions with relatively little activity, connected to one or more other system(s) might collectively present relatively light processing demands and all receive quick dispatching of all their work. At another extreme and prior to networked systems, users simply choose the particular system they logged onto. Consequently, a large number of active users could utilize a given system(s) thereby causing a significant imbalance between that system(s) and the others, which were then much less loaded. Furthermore, since user assignments were established through pre-defined hardware connections, users could well be connected to systems that were not then available and hence receive no application processing whatsoever, thereby further exacerbating workload and session imbalances among the systems and hence once again resulting in an overall failure of the sysplex to meet its business goals.

A later attempt, commonly referred to as "session placement", provided increased flexibility in terms of balancing workload in view of system failure(s). Session placement relied on assigning and connecting each user, then seeking to establish a terminal session, on a balanced session count basis to the next available system. This user assignment and connection was generally accomplished through some type of network interconnect facility--such as an IBM Virtual Telecommunications Access Method (VTAM). (VTAM is a registered trademark of the International Business Machines Corporation.) While this approach precluded session assignment to a failed system and thus accorded improved inter-system workload balancing, it still proved deficient. Specifically, the interconnect facility simply had no knowledge, a priori, of the amount of work any one session entailed or, for that matter, the business importance of that work vis-a-vis other work then queued or executing on the sysplex. Here too, as with connection based balancing, a system could be overwhelmed by a relatively small number of users with collectively heavy processing demands, thus leading once again to workload and session imbalances.

Moreover, while VTAM maintained knowledge of which systems were available at any one time, each of these systems, simply by virtue of their own processing hardware, could well provide radically different processing capacity than the others: some of these systems might have substantially more processing power relative to others having much less. VTAM had no knowledge of these capacity differences, which could, if recognized and utilized, tend to skew the number of work assignments towards the larger capacity systems. By failing to successfully exploit these capacity differences, workload imbalances were exacerbated in sysplexes having systems of widely differing capacity. In contrast, with connection based balancing, increasingly large systems frequently accommodated correspondingly increased numbers of physical connections and hence users and thus, to a certain extent, successfully exploited these capacity differences.

Unfortunately, session count balancing, as well as certainly connection based balancing, failed to account for the business importance of the various work requests that constituted this workload. Thus, both of these approaches were often unable to meet processing demand in a manner temporally consistent with the current business importance of the underlying application(s) to be processed. For example, by concentrating on maximizing throughput of processed work, no attention was paid, during dispatching, to the relative business importance of the individual work requests, thereby often causing relatively important work to be delayed at the hand of other such work of much lesser importance with a concomitant failure to meet overall business goals.

Given the deficiencies inherent in distributing sessions on a simple balanced session count basis, the art has attempted to remedy these deficiencies by modifying the session count balancing approach to accommodate work request transfers among systems--hereinafter referred to as the "session count balancing with transfer" approach. Specifically, once sessions are assigned to given systems by VTAM, then, in the event of a workload imbalance between systems, heavily loaded system(s) could then transfer individual work requests, on a request-by-request basis, to any other system that then had sufficient idle capacity. Accordingly, if session count balancing resulted in relatively poor session placements, i.e., "bad" choices which caused or exacerbated a current workload imbalance, then, to a certain extent, these bad choices could be subsequently alleviated by subsequent work redistribution among the systems themselves. While at first blush, this appears to be an attractive solution; unfortunately, it can result in significant cost. Specifically, the process of communicating and transferring work requests is heavily dependent on the inter-system communications fabric, incorporated into the sysplex, and available processor resources. Not only must the sysplex contain sufficient communication links, providing high-speed bandwidth to enable such a transfer at any time, but also each such transfer consumes a certain amount of system instructions, expended both at a transmitting system and a receiving system, such as on the order of, e.g., 50K instructions/work request. If the work request is relatively large, then, the resulting processor overhead needed to implement the transfer may be small or even negligible as compared to the processing demands of the work request itself, thereby readily justifying the cost, in terms of system overhead, of the transfer. On the other hand, a work request that consumes a relatively small number of instructions, such as on the order of, e.g.,
100K or so, would be simply be too expensive, again in terms of system overhead, to transfer to another system. Unfortunately, rarely, if ever, will a system have a priori knowledge, immediately upon its receipt of a work request, as to just how much processing that request entails, i.e., just how many instructions that request will ultimately consume. Once a system starts processing a request and is then able to possibly estimate its size, it is then simply too late to transfer the request. Thus, given the lack of insight as to the ultimate size of any processing request, the session count balancing with transfer" approach can still produce "bad" choices that result in workload imbalances among the individual systems in the sysplex.

An alternate approach taught in the art for workload balancing, applies in a network context where a network can route a work request from any user to any system in the sysplex. Similar to connection based balancing, this approach involves returning a list of routers, from a network type OS in the sysplex, and then, routing through the network, a current work request to one of these servers. This server is identified in a fixed manner through directories, by the network OS, such as in a round-robin fashion, as the next successive server in the directory or as simply the first server in the directory. Unfortunately, this approach relies on a customer, particularly the sysplex administrator, to define a directory, i.e., an installation table, of all the servers. This table changes whenever a new server is installed or removed. Furthermore and similar to the other approaches described above, the network-based routing process disadvantageously has simply no knowledge of the business importance of the work requests, both those currently executing as well as those that are competing for service, or what other tasks, other than routed work requests, are being executed at each of the available servers and their respective levels of importance. Hence, work requests are frequently assigned and ultimately dispatched totally inconsistent with their actual business importance. Moreover, owing to a lack of knowledge as to actual server loading or availability, a server can be overloaded or taken out of service, but, no information thereof will be immediately passed back to the network routing process to prevent the network from attempting to send a work request to any such then non-available server. As such, whenever a server becomes non-available--because, e.g., it is overloaded or taken out of service, the network is forced to wait for an appropriate response, more likely a lack of response after a given time interval has elapsed, to signify that a given server is not then available. As such, once this time interval has elapsed, the network must then reroute the work request accordingly to the next server listed in the directory. However, this delay disadvantageously postpones both the dispatching and the ultimate processing of this work request--possibly contravening the importance underlying the request and hence causing the sysplex to once again fail to meet its overall business goals.

A recent attempt at allocating system resources to work requests based on attaining one or more pre-defined end-user oriented goals, such as execution velocity or response time, is described in co-pending U.S. patent application "Apparatus and Method for Managing a Data Processing System Workload According to Two or More Distinct Processing Goal Types", Ser. No. 08/222,755; filed Apr. 4, 1994; now U.S. Pat. No. 5,473,773, assigned to the present assignee hereof and incorporated by reference herein. This attempt represents a significant advance inasmuch as here OS software, rather than a system administrator, takes over the responsibility for allocating system resources in a manner that attempts to satisfy the end-user goals. However, this attempt still relies on a system administrator to assign all the work requests, based on a static workload prediction, to the individual servers in the sysplex and, only after this assignment has been made, allocates the available system resources to attain the goals. As a result of this static workload allocation among the servers, imbalances in workload and/or session placements, as discussed above, can disadvantageously still arise.

Therefore, a need currently exists in the art for a technique, including both a method and accompanying apparatus, that can be used in a multi-system environment, such as illustratively a sysplex or other multi-processing environment, for effectively balancing session placements and/or work requests, across all the servers therein, in view of attendant user-defined business importance thereof and available sysplex resource capacity. By doing so, this technique would be expected to utilize these available resources to balance workload and/or session placements in a manner that properly satisfies the overall business goals of the sysplex. This technique should not merely rely on static predictions of workload and/or session placements but rather should dynamically react and adapt to changing workloads and session requirements, as well as current server availability, and also effectively accommodate capacity differences existing among the various available systems. In addition, by not just relying on static predictions or fixed network based routing schemes, this technique should avoid making any "bad" choices as to session and/or work request placement, thereby obviating the need and cost that might otherwise be incurred to subsequently remedy such choices.

SUMMARY OF THE INVENTION

Advantageously, we have developed a technique, including both methods and accompanying apparatus, for use in illustratively a multi-system (sysplex) processing environment, for balancing work requests and/or session placements among the servers therein that substantially, if not totally, eliminates the deficiencies that currently exist in the art.

In accordance with our present invention and with respect to work request assignment, systems and servers are categorized into two classes: eligible and candidate. As discussed below, work requests for a client application are assigned first to various eligible systems and eligible servers thereon based on their current capacity to accept new work in a manner that meets business goals inherent in a sysplex policy; followed, if additional servers are requested by that application, to candidate systems and candidate servers thereon. Eligible systems are those goal-oriented systems running under the policy and for which current capacity information is known; candidate systems are those for which no current capacity information is known.

As our invention specifically teaches, in response to a routing selection request from a client application, a list of appropriate systems is first fabricated. This list is populated first by selected eligible systems and then, if space remains in the list, by selected candidate systems. Those eligible systems selected for inclusion in the list are those then exhibiting a pre-defined minimum level of capacity utilization at a lowest business importance level. Weights are assigned to each of the eligible systems based on the actual capacity utilized at these lowest levels, over a pre-defined time interval, illustratively three minutes, with respect to total capacity utilized at that level across all eligible systems. A server weight for each of the eligible servers (that are part of a collection of common servers which support the client application) on each eligible system is then calculated by dividing the weight for that eligible system by the number of active application servers thereon. In the event that the number of such servers exceeds the system weight thereby otherwise resulting in a fractional server weight, then the system weight is assigned to one of these servers on that eligible system; zero to all the others residing thereon. Thereafter, if candidate systems are to be selected, then weights are assigned to each candidate system and active candidate server thereon. If only candidate systems are to be selected, then the weight of each candidate system is set to one. Alternatively, if eligible servers are to be selected as well, then, if more eligible systems are to be selected than candidate systems, the weight of each candidate system is set as a minimum of the average and a median of all the system weights for all the eligible systems. In a similar fashion as with the eligible servers, the weight of each candidate server is calculated by dividing the candidate system weight by the number of candidate servers thereon. Here too any individual candidate server weight can not be fractional. Hence, in the event that the number of candidate servers exceeds the weight for the corresponding candidate system thereby resulting in a fractional server weight, then the system weight is assigned to one of these servers on that candidate system; zero to all the others residing thereon.

If non-zero weight eligible servers exist, then an output client server list is populated with identification of these servers (with their weights), in descending weight order, until either the client server list is full or the list of non-zero weight eligible servers is exhausted, whichever occurs first. In the latter case, identification of successive candidate servers (with their weights) are then written into this list in descending weight order, in order to fill the list if possible.

However, if the eligible servers are only those with zero weights--i.e., those eligible servers with relatively little capacity, then each of these eligible servers are successively assigned a common weight of one and selected in seriatim until the identifications of all these servers (including their weights) have been written into the client server list. An improvement may be to rotate the server entries among the different systems. Owing to their apparent inability to handle more than a small number, if any, of additional work requests, hence none of these servers is now particularly favored for new work. Thereafter, if any candidate servers are to be selected, each of these servers are successively assigned a weight of one and selected, again in seriatim, until all the candidate servers are selected or the client server list becomes full, whichever occurs first. In view of a lack of capacity information, none of these candidate servers is particularly favored as well.

As a result of our inventive work request assignment method, a list of servers and their corresponding server weights is identified to the client application which, in turn, will directly route a portion of the total work requests thereat to each of these servers in proportion to its weight. The client may request a refresh of this balancing assessment on a periodic basis.

With respect to session placement, our inventive method only places sessions when all relevant systems collectively have non-zero capacity information, i.e. are goal-oriented. In the absence of such systems, we will revert back to session count balancing. If all systems have capacity information, the eligible servers therein that are meeting their goals are selected first, followed by those not meeting their goals.

In particular, our inventive method first selects system(s) with sufficient available capacity at a lowest business importance level; if multiple systems result, then the one system that provides the largest amount of service per session is then selected. Competing servers on the selected system(s) are then evaluated based on their corresponding session count data to yield a single server. As a result of our inventive session placement method, the identification of that single server is returned to the client application which, in turn, will directly establish a session therewith.

Furthermore, our inventive technique provides the feature, as pertaining to work request assignment, of alternate routing and distribution of these requests. Not only can the individual work requests be routed from a client application directly, through a network, to corresponding sysplex servers, but also this routing can alternatively be accomplished within the sysplex itself. Sysplex based routing, via disadvantageously consuming additional overhead, advantageously frees the client application from any need to distribute and route work requests to individual servers and hence simplifies its programming.

Another feature of our invention, as it pertains to session placement, involves use of increasingly refined decision criteria to select among competing servers in order to account for transient server conditions. Rather than making sharp distinctions in choosing among competing servers for session placement, increasingly fine decision criteria can be used to give preference, that would not otherwise occur, to a system (and its servers) which, over a relatively short period of time, can be converted, through resource re-allocation, from one not meeting its goals to one that is. Such a system may experience only a temporary loss in capacity caused by factors unrelated to insufficient CPU access, such as, e.g., inadequate memory size. The MVS OS continually re-assesses and, if required, re-allocates system resources at relatively short intervals, such as every ten seconds. Hence, a system not currently meeting its goals through receiving the needed resources can be converted into one, even with a new session established thereat, that will meet its goals. Thus, by spreading a new session to an additional server, sysplex performance is enhanced; hence further effectuating sysplex policy.

A further feature of our invention is that if a relatively large number of sessions have recently been placed at any given system within a relatively small time period, then any work flowing from a new session subsequently placed at that system may well experience some degree of latency. This arises from a latent demand at that system to process work subsequently requested by then existing, i.e., previously established, active and/or pending sessions thereat. Unfortunately, the amount of such latent work at any one system generally can not be accurately estimated a priori. Moreover, owing to latency, the capacity statistics in a system capacity utilization table will not update immediately upon placement of each new session, but rather will require some time, to accurately reflect the actual processing capacity utilized by that session. Consequently, our invention compensates for this latent demand.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

FIG. 1 depicts illustrative sysplex 100 with a typical associated computing environment;

FIG. 2 depicts the correct alignment of the drawing sheets for FIGS. 2A and 2B;

FIGS. 2A and 2B collectively depict a simplified high-level diagram for illustrative message flow that results from use of our present invention in sysplex 100, with, e.g., System E serving as a routing node therein;

FIG. 3 depicts System Capacity Utilization (Importance Level Service Summary--ILSS) table 300 as used by our present invention;

FIG. 4 depicts Sysplex Router Registered User (SSRU) table 400 also used by our present invention;

FIG. 5 depicts a high level flowchart of Output Server List Determination routine 500 which embodies our present invention for assigning and balancing new work requests throughout the sysplex;

FIG. 6 depicts the correct alignment of the drawing sheets for FIGS. 6A-6C;

FIGS. 6A-6C collectively depict a high level flowchart of Eligible and Candidate Systems and Servers Determination routine 600 which is executed by routine 500, the latter shown in FIG. 5;

FIG. 7 depicts the correct alignment of the drawing sheets for FIGS. 7A-7C;

FIGS. 7A-7C collectively depict a high level flowchart of Eligible Server List and Weight Determination routine 700 which is also executed by routine 500, the latter shown in FIG. 5;

FIG. 8 depicts the correct alignment of the drawing sheets for FIGS. 8A and 8B;

FIGS. 8A and 8B collectively depict a high level flowchart of Selection of Candidate Servers and Weights Determination routine 800 which is also executed by routine 500, the latter shown in FIG. 5;

FIG. 9 depicts the correct alignment of the drawing sheets for FIGS. 9A-9C;

FIGS. 9A-9C collectively depict a high level flowchart of Server Assignment routine 900 which is also executed by routine 500, the latter shown in FIG. 5;

FIG. 10A depicts Generic Resource Real Instance (GRRI) table 1000 also used by our present invention;

FIG. 10B depicts Generic Resource Selected Systems (GRSS) table 1050 also used by our present invention;

FIG. 11 depicts a high level flowchart of Session Placement Determination routine 1100 which embodies our present invention for assigning and balancing new session placements throughout the sysplex;

FIG. 12 depicts the correct alignment of the drawing sheets for FIGS. 12A and 12B;

FIGS. 12A and 12B collectively depict a high level flowchart of System Set Ascertaining routine 1200 which is executed by routine 1100, the latter shown in FIG. 11;

FIG. 13 depicts the correct alignment of the drawing sheets for FIGS. 13A-13C;

FIGS. 13A-13C collectively depict a high level flowchart of System Determination routine 1300 which is also executed by routine 1100, the latter shown in FIG. 11;

FIG. 14 depicts the correct alignment of the drawing sheets for FIGS. 14A and 14B; and

FIGS. 14A and 14B collectively depict a high level flowchart of Server Selection routine 1400 which is also executed by routine 1100, the latter shown in FIG. 11.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to various figures.

DETAILED DESCRIPTION

After considering the following description, those skilled in the art will clearly realize that the teachings of our present invention for dynamically assigning new work requests and placing new sessions across multiple servers can be used in computer installations that have any one of a wide variety of different multi-server and/or multi-processing architectures. Illustratively, these architectures can range from a relatively simple computer installation which utilizes a single physical processor that simultaneously executes several applications in separate operating system (OS) images, each image essentially implementing a separate application server, to a large sysplex that employs multiple computer processing complexes (CPCs), i.e., multiple physical processors, in which each CPC usually concurrently executes multiple applications against multiple OS images, the latter typified by a well known "PR/SM" MVS environment. (PR/SM is a trademark of the International Business Machines Corporation.) In such a multi-CPC environment, again each OS image implements a separate server. Through use of our present invention, work requests and session placements are dynamically assigned across a plurality of, generally all, the servers consistent with the business importance of the underlying work and available sysplex resources and overall business goals. With this in mind and to simplify the ensuing description, we will describe our invention in an illustrative context of use in a sysplex, such as the ES/9000 Series computer, that has separate systems all capable of accessing shared data; the sysplex and all its components are currently manufactured by the International Business Machines (IBM) Corporation of Armonk, New York (which is also the present assignee hereof).

A. Overview

FIG. 1 depicts typical sysplex 100 and a typical associated networked computing environment. As shown, remote client computers 10, having client computers 10.sub.1, . . . , 10.sub.n, are connected through connections 20, network 30, and connections 40, to sysplex 100, and specifically through the network to any of systems 110 residing within the sysplex. Illustratively, client computers 10.sub.1, . . . , 10.sub.n execute respective client applications 15.sub.1, . . . , 15.sub.n ; for simplicity, one such application, e.g., a DB2 database application, is shown as executing at each client computer, though, in actuality, each client computer can simultaneously execute different client applications and/or conduct multiple sessions of the same application. Those skilled in the art realize that a "session" is an example of a connection between two end-points, e.g. a client and a server. Hence, the term "session" will be defined and used in that generic context hereinafter. To simplify the drawing, the sysplex is shown as containing merely five systems: 110.sub.A, 110.sub.B, 110.sub.C, 110.sub.D and 110.sub.E (also designated as SYSTEM A, SYSTEM B, SYSTEM C, SYSTEM D and SYSTEM E, respectively). Since, from the standpoint of the present invention, it is immaterial as to which specific CPC implements any given system or how many systems reside on that CPC and, for that matter how many CPCs are utilized in sysplex 100, the CPC level has simply been omitted from the figure and will not be discussed further. All that is required, for the present invention, is that all the systems communicate, as discussed below, their current capacity utilization data, their own identification and identification of their resident and available server(s) with each other.

Each system implements a separate corresponding and unique application processing environment. Each of these environments utilizes a separate copy of an OS, such as the MVS OS, which forms a so-called OS "image", along with an instance(s) of corresponding application program(s) and a dedicated storage area (typically a logical partition--"LPAR"); the OS and LPAR are not specifically shown in the figure. Each application instance that executes on any such system constitutes a separate application server (henceforth referred to as simply a "server" or "real instance") to service a portion of the total workload presented to the overall application (also referred to as the "generic resource") on the sysplex. A system, based on its processing capacity and that required by the corresponding applications, can implement multiple corresponding servers. In that regard, sysplex 100 illustratively implements twelve separate servers 115; namely, servers 115.sub.1, 115.sub.2, and 115.sub.3
(also denoted as SERVER 1, SERVER 2, SERVER 3, respectively) on system 110.sub.A ; servers 115.sub.4 and 115.sub.5 (also denoted as SERVER 4 and SERVER 5, respectively) on system 110.sub.B ; servers 115.sub.6, 115.sub.7, 115.sub.8 and 115.sub.9 (also denoted as SERVER 6, SERVER 7, SERVER 8 and SERVER 9, respectively) on system 110.sub.C ; servers 115.sub.10 and 115.sub.11 (SERVER 10 and SERVER 11, respectively) on system 10.sub.D; and server 115.sub.12 (also denoted as SERVER 12) on system 110.sub.E. One must bear in mind that a "system" as used herein is an intangible construct distinct from the actual physical CPC, i.e., the "machine", and specifically the physical processor therein, on which that system executes. Inasmuch as the invention addresses allocation and balancing of work requests and sessions among systems and servers thereon--regardless of where they reside in the sysplex, the "machine" level is irrelevant and will not be discussed any further. Furthermore, each of the systems
110 is capable of directly communicating with each other as symbolized by paths 120.

Shared data device 130 provides common data accessible among the systems. To the extent relevant here, device 130 stores policy information in dataset 135 that is commonly accessible by all the systems. The shared data device communicates with each of the systems 110 as symbolized by paths 125. The shared data device may be implemented by illustratively a coupling facility or a direct access storage device (DASD).

Each incoming work request, as well as session request, from any of client computers 10, specifically from client applications 15 running thereon, to the sysplex is accompanied by one or more associated attributes. The request attributes can span a wide variety and are typified by, e.g., user identification (USERID), accounting information, job name or transaction name. For any given request, policy information in dataset 135, using embedded pre-defined rules, is used to map the attributes for that request into a service class. Each different service class has a set of business goals associated therewith. These goals are varied and can constitute, e.g., required response time and required execution velocity--the latter signifying how fast a given piece of work is actually running vis-a-vis how fast that piece of work could run if nothing else was running at the time. Each of these goals has a numeric importance level associated with it which signifies the business importance attached to achieving that particular goal and hence the service class. These levels range from one to five, with one representing the highest importance level, five representing the lowest. Where insufficient sysplex capacity exists to immediately dispatch all work requests then assigned to any given system in the sysplex, those pending requests having higher importance levels prevail and are accorded higher dispatch priority for execution over those requests having lower importance levels. The present invention does not address either the actual prioritizing and dispatch of requests that have already been assigned to a given server or the policy based mapping of the request attributes into proper service classes and importance levels. Rather, as will be discussed in quite some detail below, the present invention is directed to a technique that, based upon business importance of existing requests (active work) and available sysplex resources, actually assigns these requests among the available systems and servers therein for subsequent execution in a manner that satisfies overall business goals of the sysplex, as embodied in the policy. Generally, some of the systems will be running under the goal based policy, while others may not. As will be seen below, our inventive process accommodates both types of systems.

As noted previously, network 30 provides a communication path from each of client computers 10 to any of systems 110. To facilitate this connection, each of the systems contains a network inter-connect facility, i.e. router 140. In use, incoming session and work requests initially flow from the network to one given system, a so-called "routing node", here shown as illustratively system 110.sub.E, to the sysplex. This particular node, through our inventive process, then provides information as to where subsequent requests should flow and communicates this information back to the requesting client computer, particularly the client application thereon, which, in turn, submits its work requests to system(s) and server(s) thereon. To accentuate the routing node function of system 110.sub.E, the communication path in paths 40 to that node is shown as a solid line, while every other such path in paths 40 from network 30 to each of the other four systems is shown as a dashed line.

B. Illustrative Message Flow

FIGS. 2A and 2B collectively depict a simplified high-level diagram of illustrative message flow that results from use of our present invention in sysplex 100 with system 110.sub.E serving as the routing node; the correct alignment of the drawing sheets for these figures is shown in FIG. 2.

As depicted in FIGS. 2A and 2B, whenever illustrative client application (e.g., DB2) 15.sub.1 executing within client computer 10.sub.1 needs to either submit a work request to the sysplex or establish a session therewith, the client first establishes a connection to the sysplex. The network, transparent to the client application, extends this connection to the routing node, i.e., system 110.sub.E, within the sysplex and particularly to router 140; the entire connection being symbolized by solid line 230. As such, and as symbolized by dashed line 233, the work or session request is routed to router 140. This router, resident on system 110.sub.E, passes this request, as symbolized by line 224, to MVS OS 210 that executes on this system.

MVS OS 210, through particularly workload manager 216, assigns the work or session request, in accordance with the teachings of our present invention, to an available server(s), based upon current capacity information, consistent with the business importance of the request so as to satisfy overall business goals of the sysplex--as embodied in the policy. In so doing, the workload manager contains and executes our inventive work load and session assignment process, collectively depicted here as block 218, as discussed below as routines 500-900 and 1100-1400 and shown with FIGS. 5-9 and 11-14, respectively. Through our present invention, the work request and session placement that previously occurred on a basis of balancing session counts among servers, is modified to permit the workload manager to assign work and session requests in our inventive manner.

To the extent relevant, the MVS OS, which is replicated on each of the systems--though only OS 210 is shown, contains a workload manager; a Sysplex Router Registered User (SSRU) table--of which table 400 is illustrative; Generic Resource Real Instance (GRRI) tables--of which table 215 is illustrative; GRSS table 1050; and System Utilization tables--of which tables 220 are illustrative. The SSRU table, as discussed below in conjunction with FIG. 4, maintains a list of systems and their associated servers, and other associated information therefor, for the entire sysplex. The same exact table, with the same information, is maintained in each of the different systems. As new systems are brought on-line, that system undergoes a registration process, which, appropriately updates the SSRU table in each of the other systems to account for the new system and its servers. The GRRI table, also discussed below in conjunction with FIG. 10A, maintains a list of specific application servers (which support an overall application, i.e., a "generic resource", with each separate instance of an application resident on that server being a so-called "real instance") available on a given system. Each system maintains its own GRRI table and supplies, as a new server is brought on-line or taken off-line, every other system with all updates to that GRRI table. As such, exact copies of all the current GRRI tables are maintained on every system. Though not specifically illustrated, GRRI tables 215 contain five separate GRRI tables, one for each of systems 110.sub.A-E. The GRSS.sub.-- Selections table 1050, which is also discussed below in conjunction with FIG. 10B, indicates the number of times, measured over, e.g., a ten second interval, that a session has been assigned, i.e., bound, by our inventive process, to a given system. This same table is also maintained and updated independently on every system to which our inventive process can assign a session. System utilization (Importance Level Service Summary--ILSS) tables 220 are formed of five tables (specifically 220.sub.A, 220.sub.B, 220.sub.C, 220.sub.D and 220.sub.E), again one, as discussed below in conjunction with FIG. 3, for each system (systems 110.sub.A-E, respectively), that reports the current utilization, in terms of service unit sums and percentages of total capacity, for that particular system. However, only each of those systems that run in so-called "goal" mode, i.e., under a "policy", maintains an ILSS table for itself. Here, for purposes of illustration, all five systems are assumed to be running in this mode. System capacity is measured over illustratively three separate implementionally specific measurement periods, e.g.: 60 seconds, 120
second and 180 seconds. Alternatively, these periods could be formed of one or two intervals and possibly of different durations. Again, each system reports its current capacity information over these three measurement periods to every other system, thus allowing each of the five systems to maintain a complete current copy of ILSS tables 220 for all the sysplex systems. However, regardless of whether each system is running in goal mode or not, and hence maintaining its own ILSS table or not, that system maintains a corresponding ILSS table for every other system in the sysplex that is running in goal mode. Thus, if a sysplex has 32 systems all running in goal mode, the tables 220 (for each of those systems) will contain 32 separate ILSS tables, one for each of these goal mode systems.

Once the work or session request has been assigned, then, as symbolized by line 228, the workload manager passes the assignment information back to router 140. For work requests, the assignment constitutes a list of available servers and the percentage, in terms of a proportionate weight, of then total future work requested by the client application, e.g., client application 15.sub.1, that is to be routed to each of these servers. For a session placement request, the assignment information constitutes identification of a single server at which a new session requested by the client application is to be established. Once router 140 receives this assignment information, it routes, as symbolized by dashed line 237, this information back through network 30 to requesting client application 15.sub.1. In response to this information, the client application, for a work request, then sends, as symbolized by dot-dashed lines 240, work requests to each of the servers, such as servers
115.sub.1-12, specified in the list and in a proportionate amount, of the total work to be placed, as specified by the corresponding weight for each of these servers. For a session request, the client application will simply send a session establishment request to just the single server identified to the application--rather than to one and often more servers that are to process a work request. Routing of all the session establishment requests to their corresponding identified servers, but for all twelve servers 115, is also symbolized by dot-dashed lines 240.

C. ILSS and SSRU Tables

FIG. 3 depicts System Capacity Utilization (ILSS) table 300. As discussed above, each system running in goal mode maintains its own ILSS table and, through its OS, communicates updates to that table to each of the other systems so as to maintain an exact current duplicate of that table at each of the other systems. Inasmuch as all the ILSS tables are identical in form, FIG. 3 shows one such table.

As shown, this table contains real-time measurement data, as measured by the workload manager, for capacity utilization, in terms of measured service unit sums, over each of three time intervals: 60, 120 and 180 seconds with corresponding measurement and percentage data in columns 310 and 320, 330 and 340; and 350 and 360. Measurement data is provided for eight importance levels (noted as 0-7), with numerically larger levels denoting increasingly less important work. Importance level 0, being reserved for system overhead tasks undertaken by the MVS OS, constitutes the most important work. No work requests can be assigned at this level to any system. Levels 1-5 are customer specifiable importance levels at which work can be dispatched according to the importance levels of the corresponding service classes associated with the underlying assigned work requests. Level 6 is a discretionary level and signifies work at a lower importance level than any of the customer specified levels. If any excess capacity remains for a system after all assigned work at all higher importance levels has been dispatched for execution on that system, that capacity is indicated in row 7 for "unused (available)" capacity. Work at this level is accorded the lowest dispatch importance level. Each entry in the service unit sum columns for any importance level is the sum, in measured service units, of service units consumed over the corresponding time period (e.g., 60, 120 or 180 seconds) by work at that importance level summed with service units consumed over the identical period by work at all lower importance levels (both discretionary and unused). As noted, this data is continually measured and reported by the workload manager. The percentage column indicates the percentage, in terms of total service units, of total capacity that is being consumed at any one and all lower importance levels during the corresponding period, truncated to a whole percentage. The data shown in table 300 is merely illustrative and, of course, will vary, sometimes widely, in real-time.

FIG. 4 depicts Sysplex Router Registered User (SSRU) table 400. This table, as noted, maintains a list of systems and their associated servers for the entire sysplex. This table contains columns 410, 420, 430 and 440. A separate entry exists for each set of the same application servers which resides on a different system. This table specifies, in column 410, a name of the set of servers (e.g., "ATM" or "NYC") resident on each system--this name can be a location name, if desired; in column
420, a system name (e.g., "SYS A" or "SYS B"); a logical unit (LU) identification for each system (e.g., "T5732A" or "T9723M"); and in column 440, a network identification for that system (e.g., "BANKING" or "ADMIN"). The server set name must be unique for each and every different set of servers. Though not specifically shown by the illustrative data in table 400, different sets of servers can reside on either a common system or different corresponding systems. The logical unit combined with the network identification, which are both specified by a customer, is unique for each and every system. As illustrated in table 400, a server set can have multiple instances. As each system is brought on-line, that system undergoes a registration process conducted by its MVS OS, which, appropriately updates its own SSRU table and sends messages to all the all other systems to update each of their own SSRU tables in order to account for the new system and all its resident servers. Accordingly, through registration, all the SSRU tables maintained on all the active servers are identical. Conversely, should a system be taken off-line, then, as a result of, e.g., non-responded status inquiries, the MVS OS on each of the other systems updates its own SSRU table to de-register, i.e., delete, the off-line system therefrom. While individual servers are not aware that any other server exists, through use of table 400 on each system, each MVS OS has knowledge of all the servers, by their set name, and the systems on which that set of servers resides. Through use of our invention, one of the sets of servers specified in table 400 is selected by the client application to receive work assignments with our invention then specifying the specific ratio of work requests sent to each of these servers in accordance with a corresponding dynamically determined weight. Since the only data of interest, for purposes of our present invention, is server name column 410, we will not discuss columns 420-440 any further.

D. Work Request Assignment

Our inventive work request assignment method categorizes systems and their resident servers into two classes: eligible and candidate, and assigns work requests for a client application first to various eligible systems and eligible servers thereon based on their current capacity to accept new work in a manner that meets business goals inherent in the policy; followed, if additional servers are requested by that application, to candidate systems and candidate servers thereon. Basically, as will be discussed in detail below, eligible systems are those goal-oriented systems running under a policy and for which current capacity information is known; candidate systems are those for which no current capacity information is known.

In essence, as we specifically teach, in response to a routing selection request from a client application, a list of appropriate systems is first fabricated. This list is populated first by selected eligible systems and then, if space remains in the list, by selected candidate systems. Those eligible systems selected for inclusion in the list are those then exhibiting a pre-defined minimum level of capacity utilization at a lowest business importance level. Weights are assigned to each of the eligible systems based on the actual capacity utilized at these lowest levels, over a pre-defined time interval, illustratively three minutes, with respect to total capacity utilized at that level across all eligible systems. A server weight for each of the eligible servers (that are part of a collection of common servers which supports the client application) on each eligible system is then calculated by dividing the weight for that eligible system by the number of active application servers thereon. In the event that the number of such servers exceeds the system weight thereby otherwise resulting in a fractional server weight, then the system weight is assigned to one of these servers on that eligible system; zero to all the others residing thereon. Thereafter, if candidate systems are to be selected, then weights are assigned to each candidate system and active candidate server thereon. If only candidate systems are to be selected, then the weight of each candidate system is set to one. Alternatively, if eligible servers are to be selected as well, then the weight of each candidate system is set as a minimum of the average and a median of all the system weights for all the eligible systems. In a similar fashion as with the eligible servers, the weight of each candidate server is calculated by dividing the candidate system weight by the number of candidate servers thereon. Here too any individual candidate server weight can not be fractional. Hence, in the event that the number of candidate servers exceeds the weight for the corresponding candidate system thereby resulting in a fractional server weight, then the system weight is assigned to one of these servers on that candidate system; zero to all the others residing thereon.

If non-zero weight eligible servers exist, then an output client server list is populated with identification of these servers (with their weights), in descending weight order, until either the client server list is full or the list of non-zero weight eligible servers is exhausted, whichever occurs first. In the latter case, identification of successive candidate servers (with their weights) are then written into this list in descending weight order, in order to fill the list if possible.

However, if the eligible servers are only those with zero weights--i.e., those eligible servers with relatively little capacity, then each of these eligible servers are successively assigned a common weight of one and selected in seriatim until the identifications of all these servers (including their weights) have been written into the client server list. An improvement here may be to rotate servers around different systems before the same system is selected again for a different server. In any event, owing to their apparent inability to handle more than a small number, if any, of additional work requests, hence none of these servers is now particularly favored for new work. Thereafter, if any candidate servers are to be selected, each of these servers are successively assigned a weight of one and selected, again in seriatim, until all the candidate servers are selected or the client server list becomes full, whichever occurs first. In view of a lack of capacity information, none of these candidate servers is particularly favored as well.

As a result of our inventive work request assignment method, a list of servers and their corresponding server weights is identified to the client application which, in turn, will directly route a portion of the total work requests thereat to each of these servers in proportion to its weight.

With the above overview in mind, FIG. 5 depicts a high level flowchart of Output Server List Determination routine 500 which embodies our present invention for assigning and balancing new work requests throughout the sysplex. As noted above, routine 500, as well as subservient called routines 600-900, all execute as part of the Workload Manager in the routing node--which for purposes of illustration is system 110.sub.E shown in FIGS. 1 and 2A-2B.

Routine 500, as shown in FIG. 5, is entered upon receipt of a routing selection request originating from a client application. Upon entry into this routine, execution proceeds to block 510 which executes routine 600 to determine two sets of servers: a set of "eligible" servers, and a set of "candidate" servers from all the sets of servers then registered in the SSRU tables. These sets of servers are also specified in terms of the corresponding systems on which each one of these sets reside. Eligible servers are defined as those which are running under a policy, i.e., goal oriented servers, and for which importance-based capacity (ILSS) information is available. Candidate servers include those for which capacity information is not available, e.g., servers that either are not running under a goal-oriented policy, or are goal-oriented but for which capacity information is not currently available. Servers that do not run under a goal-oriented policy instead utilize some type of resource controls and hence a different customer-defined metric, typically unrelated to underlying business importance, for dispatching assigned work, e.g., such as illustratively maximizing use of available processing cycles, dispatch priority time and storage targets. As will be seen, our inventive process can assign work requests across both types of servers.

Once routine 600 has executed to select the eligible and candidate servers and systems, execution proceeds to block 520 which, when entered, executes routine 700. Routine 700 specifies the maximum number of servers that can be selected from either of two server lists (eligible and candidate) and subsequently provided to the client application, and determines the weights for each of the eligible servers. Thereafter, execution proceeds to decision block 530 which determines whether any of the systems specified by routine 600 is running without goals. If no such non-goal oriented system then exists, then execution merely proceeds, via NO path 533, to block 550. Alternatively, if such a non-goal oriented system exists, then decision block
530 routes execution, via YES path 537, to block 540. This latter block executes routine 800 to set the weights of the "candidate" servers, i.e., the servers on the group of candidate systems specified by block 510. Candidate systems are those for which no capacity information is currently known. Two types of candidate systems exist. To easily distinguish between the two types of candidate systems and their servers: i.e., those systems which run under a policy but which lack current capacity information from those systems that are non-goal oriented and clearly provide no capacity information, we will refer to the latter type of candidate systems simply as "black box" systems and their resident servers as "black box" servers--hence, from policy and capacity perspectives, neither a policy nor capacity information exists for "black box" systems and servers. Routine 800 handles all the candidate servers including the "black box" systems and their "black box" servers. Once routine 800
fully executes, execution exits from block 540 and passes to block 550.

Block 550, when executed, invokes routine 900, to assign servers, either the eligible or candidate servers depending on the results of routines 700 and 800, and their corresponding weights to an output server list. The size of this list is specified by the particular client application that is requesting work. Once this list is formulated, execution proceeds to block 560 which sends this list back through the network to the client application. Once this occurs, execution exits from routine 500. The client application then sends a proportionate share of its current work through the network directly to the servers specified in the output list with the proportion given by the weight for each of these servers.

FIGS. 6A-6C collectively depict a high level flowchart of Eligible and Candidate Systems and Servers Determination routine 600; the correct alignment of the drawing sheets for these figures is shown in FIG. 6. As noted above, routine 600
determines a set of "eligible" and a set of "candidate" servers, and corresponding systems, to which work is to be assigned.

Upon entry into routine 600, execution first proceeds to block 605 which initializes various variables. In that regard, a list of eligible systems, i.e., ELIGIBLE.sub.-- SYSTEMS.sub.-- LIST, and a list of eligible servers, i.e., ELIGIBLE.sub.-- SERVER.sub.-- LIST, are both set to empty. Similarly, a list of candidate systems, i.e., CANDIDATE.sub.-- SYSTEMS.sub.-- LIST, and a list of candidate servers, i.e., CANDIDATE.sub.-- SERVER.sub.-- LIST, are also both set to empty. A list of remaining systems, i.e., REMAINING.sub.-- SYSTEMS.sub.-- LIST, is filled with all the systems, by name. A variable for target importance level, i.e., TARGET.sub.-- IMPORTANCE level, is set to zero (the highest importance level). Lastly, a variable for total service units, i.e., TOTAL.sub.-- SERVICE.sub.-- UNITS, is also set to zero. Once this initialization has completed, execution proceeds to decision block 610. This decision block determines, by ascertaining whether the REMAINING.sub.-- SYSTEMS list is empty, whether all the systems have been processed. If all the systems registered in the SRRU table have been processed, then execution simply exits, via YES path 612, from routine 600 and then returns to routine 500.

Alternatively, if any system remains to be processed, i.e., the list of REMAINING.sub.-- SYSTEMS, i.e., variable REMAINING.sub.-- SYSTEMS.sub.-- LIST, is not empty, then decision block 610 routes execution, via NO path 614, to block 615. This latter block, when executed, sets variable SYSTEM to designate the next successive system, i.e., the "current" system, to be processed in REMAINING.sub.-- SYSTEMS.sub.-- LIST. Thereafter, block 615 removes the system now designated by variable SYSTEM from REMAINING.sub.-- SYSTEMS.sub.-- LIST. Once this occurs, execution proceeds to decision block 620 which tests whether the current system is active. If this system is not active, execution loops back, via NO path 622 and path 670, to block 610 to again determine whether all the systems have been processed and so on. Alternatively, if the current system is active, then decision block 620 routes execution, via its YES path 624, to decision block 625. This decision block determines whether the current system has an suitable server, i.e., whether a functioning application server resides on that system. If the designated system does not have a suitable server, then that system is no longer considered. Hence, then decision block 625 routes execution, via its NO path 626 and path 670, back to decision block 610 to again determine whether all the systems have been processed and so on. Alternatively, if the designated system is home to a suitable server, then execution proceeds, via YES path
628 emanating from decision block 625, to decision block 630. This latter block determines whether capacity information (an associated ILSS table) exists for this particular system. If such capacity information does not exist, then decision block 630
routes execution, via its NO path 632, to block 665. All such goal-oriented systems for which capacity information is missing as well as the "black box" systems, which are non-goal oriented and hence provide no such information, are all collectively classified by routine 600 as candidate systems, with their resident servers being classified as candidate servers. Hence, block 665, when executed, merely adds the current system, whether it is a "black box" system or not, to the list of candidate systems, i.e., CANDIDATE.sub.-- SYSTEMS.sub.-- LIST, and adds all the servers resident thereon, by server set name (stored in SYSTEM.sub.-- SERVER.sub.-- LIST and originally accessible through the SRRU table), to the list of candidate servers, i.e., CANDIDATE.sub.-- SERVER.sub.-- LIST. Should capacity information become available later, then, through subsequent execution of routine 600 at that time, specifically decision block 630, this particular system and its set of resident servers will not then be categorized as a candidate system and candidate servers, respectively. Alternatively, if capacity information is currently available for the current system, then decision block 630 routes execution, via its YES path 634, to decision block 635.

Decision block 635 ascertains, based on current system capacity as shown in the ILSS table for the current system, whether that particular system has had at least 5% available capacity at the target importance level for the last three minutes, i.e., whether ILSS.sub.-- %.sub.180 .gtoreq.5 at TARGET.sub.-- IMPORTANCE.sub.-- LEVEL? If such capacity is not available, then that system is viewed as having insufficient capacity and thus is not a better choice than any other system previously chosen, through the current execution of routine 600, that had such capacity available. Consequently, for that current system that does not have such capacity, that system is effectively ignored. Specifically, decision block 635 routes execution, via its NO path 636 and path 670, back to decision block 610 to again determine whether all the systems have been processed and so on. The target importance level is initially set to the highest importance level, i.e., zero, and, as will be shortly seen, is then lowered accordingly. At the highest importance level, 5% capacity is always available. In the event the current system has at least 5% available capacity at the target importance level, then decision block 635 routes execution, via its YES path 638, to block 640. This latter block sets a variable CURRENT.sub.-- LEVEL equal to the highest numerical importance level (lowest level in terms of actual business importance) for the current system at which available capacity exists that equals or exceeds 5% For example for the ILSS.sub.-- %.sub.180 data in table 300 shown in FIG. 3, this lowest level occurs at numerical level 5.

Returning to FIGS. 6A-6C, once the CURRENT.sub.-- LEVEL variable is set, then execution proceeds from block 640 to decision block 645. This latter block determines whether the value of CURRENT.sub.-- LEVEL is greater numerically than the value of TARGET.sub.-- IMPORTANCE.sub.-- LEVEL, i.e., whether the current system has available capacity at a lower importance level than a previous system choice, in essence whether a "better" server has now been found. If the current system is no better in terms of importance level at which the 5% available capacity exists, i.e., CURRENT.sub.-- LEVEL is not numerically greater than TARGET.sub.-- IMPORTANCE.sub.-- LEVEL, then decision block 645 routes execution, via its NO path 648, to block 655. This latter block, when executed, merely adds the current system to the list of eligible systems, i.e., ELIGIBLE.sub.-- SYSTEMS.sub.-- LIST, and adds the set of servers thereon, by name, to the list of eligible servers, i.e., to ELIGIBLE.sub.-- SERVER.sub.-- LIST. Execution then proceeds to block 660. Alternatively, if the current system is indeed "better" than a previous system choice, i.e., it has sufficient available capacity at a lower business importance level (greater numerically), then decision block 645 routes execution, via its YES path 64