Multiprocessor system - Patent 4228496


United States Patent		4228496
Katzman , ; et al.		October 14, 1980

	*Title*

Multiprocessor system

	*Abstract*

A multiprocessor system the kind in which two or more separate processor modules are interconnected for parallel processing includes two redundant interprocessor buses dedicated exclusively to interprocessor communication. Any processor module may send information to any other processor module by either bus. The buses are shared in use by the processor modules on a time-sharing basis. Use of each bus is controlled by a special bus controller. The multiprocessor system includes an input/output system having multi-port device controllers and input/output buses connecting each device controller for access by the input/output channels of at least two different processor modules. Each device controller includes logic which insures that only one port is selected for access at a time. The multiprocessor system includes a distributed power supply system which insures nonstop operation of the remainder of the multiprocessor system in the event of a failure of a power supply for a part of the system. The distributed power supply system includes a separate power supply for each processor module and two separate power supplies for each device controller. Either one of the two power supplies provides the entire power for the device controller in the event the other power supply fails. The distributed power supply system permits any processor module or device controller to be powered down so that on-line maintenance can be performed in a power-off condition while the rest of the multiprocessor system is on-line and functional. The multiprocessor system includes a memory system in which the memory of each processor module is divided into four logical address areas--user data, system data, user code and system code. The memory system includes a map which translates logical addresses to physical addresses and which coacts with the multiprocessor system to bring pages from secondary memory into primary main memory as required to implement a virtual memory system. The map also provides a protection function. It provides inherent protection among users in a multiprogramming environment, isolates programs from data and protects system programs from the actions of user programs. The map also provides a reference history information for each logical page as an aid to efficient memory management by the operating system. The multiprocessor system includes in the memory of each processor module an error detection and correction system which detects all single bit and double bit errors and which corrects all single bit errors in semiconductor memory storage.



Inventors:	Katzman; James A. (San Jose, CA), Bartlett; Joel F. (Palo Alto, CA), Bixler; Richard M. (Sunnyvale, CA), Davidow; William H. (Atherton, CA), Despotakis; John A. (Pleasanton, CA), Graziano; Peter J. (Los Altos, CA), Green; Michael D. (Los Altos, CA), Greig; David A. (Cupertino, CA), Hayashi; Steven J. (Cupertino, CA), Mackie; David R. (Ben Lomond, CA), McEvoy; Dennis L. (Scotts Valley, CA), Treybig; James G. (Sunnyvale, CA), Wierenga; Steven W. (Sunnyvale, CA)
Assignee:	Tandem Computers Incorporated (Cupertino, CA)
Appl. No.:	721043
Filed:	September 7, 1976

Current U.S. Class:			710/100 700/82
Field of Search:			364/2MSFile,9MSFile

	U.S. Patent Documents
3480914	November 1969	Schlaeppi
3820079	June 1974	Bergh et al.
3828321	August 1974	Wilber et al.
3886524	May 1975	Appelt
4001790	January 1977	Barlow
4015243	March 1977	Kurpanek et al.
4032899	June 1977	Jenny et al.
4034347	July 1977	Probert
4034794	October 1978	Matsumoto
4035777	July 1977	Moreton
4040028	August 1977	Pauker et al.
4041472	August 1977	Shah et al.
4130865	December 1978	Heart et al.




Primary Examiner:	Nusbaum; Mark E.
Attorney, Agent or Firm:	Feix; Donald C.


	*Claims*

We claim: 1. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer, said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connection that processor module to the interprocessor bus, and centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus, said bus controller means including, arbitration means for centrally arbitrating the priority of said data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, said processor module including: reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and writing means for writing data received from the interprocessor bus into a specied location in the association main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module. 2. The invention defined in claim 1 wherein the dedicated memory bus in each processor module connects the memory in a given processor module directly to the central processor unit in that processor module and is effective to exclude any direct connection to a central processing unit in any other processor module so that the multiprocessor system is not subject to contention by central processor units for a shared memory and is not subject to failure because of failure of a shared memory. 3. The invention defined in claim 1 wherein the bus controller means include processor select logic means and individual select lines, each select line extending from the processor select logic means to the interprocessor control means of one associated processor module, and wherein the processor select logic means generate select signals on the select lines to guarantee that one and only one processor module can send data on the interprocessor bus at a time. 4. The invention efined in claim 3 wherein the processor select logic means generate select signals over the individual select lines to enable transmissions from a processor module to the interprocessor bus solely under the control of the bus controller means. 5. The invention defined in claim 3 wherein the bus controller means also include bus control state logic means for identifying a sender-receiver pair of processor modules and for defining a time frame for a transfer of data over the bus between the sender-receiver pair. 6. The invention defined in claim 5 wherein the processor select logic means and interprocessor control means are operatively associated by the select signals. 7. The invention defined in claim 5 including qualifying means for enabling the receiver processor module to receive data from another processor module without execution of a receive data instruction by the receiver processor module. 8. The invention defined in claim 1 wherein the bus controller means include bus control state logic means wherein the qualifying means include the combination of first means in the processor select logic means for generating a select signal on the select line to the receiver processor module, second means in the bus control state logic means for producing a receive command signal to the receiver processor module for interrogating readiness of that receiver processor module to receive, and third means in the interprocessor control means of the receiver processor module for generating an affirmative or negative receive acknowledge signal to the bus control state logic means for indicating said readiness to receive. 9. The invention defined in claim 8 wherein the processor select logic means generate only one select signal at a time to prevent other processor modules from transmitting on the bus simultaneously with the established sender processor module. 10. The invention defined in claim 3 wherein each connection of the interprocessor bus to a processor module comprises a plurality of common data path lines which are shared by all modules of the multiprocessor system, a plurality of common bus protocol lines, one bus clock line and one select line. 11. The invention defined in claim 10 wherein the bus controller means and the interprocessor control means of a sender and a receiver processor module are operatively associated and include means to generate on the interprocessor bus in sequence a send request signal, from any processor module which is ready to send, to cause the bus controller to leave an idle condition and to begin a polling operation, select signal, from the bus controller to poll in sequence each processor module until a processor module which is ready to send is identified, send acknowledge signal, with the receiver processor module number on the data lines, from a processor module to indicate readiness to send in response to the select signal and also to identify the receiver processor module to the bus controller, select signal, with the receive command signal, and with the identified sender processor module number on the data lines, from the bus controller to interrogate the ready to receive condition of the identified receiver processor module and to identify the sender processor module to that receiver processor module, receive acknowledge signal, from the receiver processor module to indicate the ready to receive condition and in response to the select signal and the receive command signal from the bus controller means, select signal, with the send command signal, from the bus controller means in response to a positive receive acknowledge indication, to enable the sender processor module to sequentially transmit a plurality of data words on the data lines to the receiver processor module, after which the bus controller means returns to an idle condition. 12. The invention defined in claim 1 wherein each processor module includes a bus receive table means which contains a separate entry corresponding to each processor module and specifies both the buffer address where incoming data from a particular processor module is to be stored and the number of data words expected. 13. The invention defined in claim 1 wherein the memory address space of each processor is divided into four separate logical address areas comprising a user data area, to system data area, a user code area and a system code area to separate code from data so that code can be made non-modifiable and also to separate operating system programs and data from user programs and data so that user programs cannot destroy the operating system. 14. A multiprocesor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer, said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus, said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, said interprocessor control means in each processor module including, inqueue means for receiving a sequence of data words from the interprocessor bus in a time interval defined by the bus controller means and outqueue means for transmitting a sequence of data words to the interprocessor bus in the time interval defined by the bus controller means said processor module including: reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and writing means for writing data received from the interprocessor bus into a specied location in the associated main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module. 15. The invention defined in claim 14 wherein each processor module includes a central processing unit, the bus clock generator means include a bus clock and each central processing unit includes its own separate processor clock and including interlock means operatively associated with the inqueue means and the outqueue means for operating a part of the inqueue means and a part of the outqueue means in synchronism with the interprocessor bus at the bus clock rate and for operating another part of the inqueue means and another part of the outqueue means in synchronism with the central processing unit of each processor module at its own separate processor clock rate without loss of data. 16. The invention defined in claim 15 wherein the inqueue means and the outqueue means are constructed to provide transfer of information over the interprocessor bus at a rate independent of and substantially faster than the rate permitted by direct memory accesses into and out of the memory sections of the processor modules to insure adequate interprocessor bus availability when more than two processor modules are connected to the interprocessor bus. 17. The invention defined in claim 15 wherein the inqueue means include bus fill stage logic and the outqueue means include bus empty state logic which operate in synchronism with the bus clock and wherein the inqueue means include processor empty state logic and the outqueue means include procesor fill state logic which operate in synchronism with the central processing unit clock. 18. The invention defined in claim 17 wherein the interlock means include logic interlocks between the processor fill state logic and the bus empty state logic of the outqueue means and between the processor empty state logic and the bus fill stage logic of the inqueue means and wherein the logic interlocks qualify certain transitions of the logic from one state to another state and make correct operation independent of the relative clock rates. 19. The invention defined in claim 15 including a second interprocessor bus and a second bus controller means and wherein the interprocessor control means of each processor module include inqueue means for each interprocessor bus and outqueue means and wherein the interprocessor control means fill the outqueue means and empty and inqueue means in synchronism with the central processing unit clock and empty the outqueue means and fill the inqueue means in synchronism with the associated interprocessor bus clock. 20. The invention defined in claim 19 wherein each outqueue means include bus empty state logic, processor fill state logic, and outqueue counter, an outqueue buffer, a receive register and an outqueue pointer for connecting the outqueue buffer to one of the two interprocessor buses. 21. The invention defined in claim 15 wherein the inqueue means includes logic means for performing interprocessor bus control and data transfer concurrently with computation by the receiver processor module. 22. The invention defined in claim 14 wherein the bus controller means include processor select logic means for generating a signal for selecting a particular processor module as a receiver processor module and for generating a second signal for selecting another particular processor module as a sender processor module, bus control state logic means for responding to a signal from the receiver processor module by generating a send command signal for enabling the sender processor module to transmit information on the bus, and packet counter means for counting the bus clock cycles comprising a packet of information transmitted on the bus. 23. The invention defined in claim 14 wherein each inqueue means include bus fill state logic means for providing a ready to receive signal to the bus controller and for controlling the storing of words received from the interprocessor bus into inqueue buffer means, processor empty state logic means for providing an inqueue full signal to the central processor unit and for controlling the reading of words stored in inqueue buffer means, inqueue buffer means for storing words received from the interprocessor bus, and inqueue counter means for counting words stored into and read from the inqueue buffer means. 24. The invention defined in claim 14 wherein each central processing unit includes a microprocessor and a control memory and wherein the control memory controls the microprocessor to cause the microprocessor to interact with the interprocessor control means to effect the inqueue emptying and the outqueue filling and wherein the bus controller means control the actual transmission of the data on the interprocessor bus including outqueue emptying and inqueue filling. 25. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, a plurality of interprocessor bus means each separate and distinct from an input/output system and from a memory bus and each interconnecting the processor modules for direct processor module to processor module signaling and data transfer and wherein each interprocessor bus means is separate and distinct from the other interprocessor bus means so that failure of a particular interprocessor bus means does not prevent signaling and data transfer over the other interprocessor bus means, each said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connecting that processor module to each interprocessor bus, and a separate bus controller means operatively associated with a particular interprocessor bus and with each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus, each of said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus. 26. The invention defined in claim 25 wherein the interprocessor control means and bus controller means are operatively associated to permit each processor module to send data over any one of the interprocessor buses to any one of the processor modules and to permit each processor module to receive data simultaneously over a plurality of the interprocessor buses 27. The invention defined in claim 26 including operating means for operating all of the interprocessor buses and bus controller means simultaneously and independently of each other to provide multiple paths of interconnection between the separate processor modules. 28. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer, said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus, said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, said interprocessor means in each processor module including, inqueue means for receiving a sequence of data words from the interprocessor bus in a time interval defined by the bus controller means and outqueue means for transmitting a sequence of data words to the interprocessor bus in a time interval defined by the bus controller means, each central processing unit including a microprocessor and an associated control memory, said control memory including, send controller means for dividing a block of data into individual packets each comprising a predetermined number of data words and for interacting with the interprocessor control means to fill the outqueue means with each packet transmitted, and bus receive controller means for interacting with the interprocessor control means to empty the inqueue means of each packet received and to reassemble received packets into the original data block, and wherein the bus controller means includes packet counter means operatively associated with the bus clock generator means for defining a time interval for transmission of each packet over the interprocessor bus to permit the usage of the interprocessor bus to be multiplexed on a packet by packet basis. 29. The invention defined in claim 28 including a plurality of interprocessor buses operatively associated with each processor module and a separate bus controller means operatively associated with each interprocessor bus for determining the priority of each data packet transferred between any two processor modules over that interprocessor bus and wherein the interprocessor control means in each processor module include inqueue means operatively associated with each interprocessor bus connected to that processor module. 30. The invention defined in claim 29 wherein each processor module memory includes receiver buffer means for accumulating packets received from each related processor module and also includes bus receive table means for directing incoming data from an interprocessor bus to a specified location in the related receiver buffer means in the memory of the processor module. 31. The invention defined in claim 30 wherein the bus receive table means in the receiver processor module provide a bus receive table entry which contains the current address where the incoming data is to be stored and the remaining number of words expected from the sender processor module. 32. The invention defined in claim 30 wherein the bus receive controller means are operatively associated with the inqueue means and the bus receive table means for storing the received data in the memory of a receiver processor module and for updating the count and address words in the bus receive table without interrupting the program currently executing in the receiving processor module. 33. The invention defined in claim 32 wherein the bus receive controller means update the bus receive table means after the receipt of each packet by the receiver processor module and provide a program interrupt when the bus receive table means indicate an entire data block has been received. 34. The invention defined in claim 32 wherein the bus receive controller means also provide a bus receive completion interrupt to the receiver processor module in response to the detection of an error by the bus receive controller means. 35. The invention defined in claim 29 including receiver control means in each processor module for controlling the ability of that processor module to receive packets over each interprocessor bus. 36. The invention defined in claim 35 including a bus receive table in each processor module for directing incoming data to a specified location in memory and wherein the receiver control means include mask register means for providing an interlock to prevent the receipt of additional data over an interprocessor bus until after certain critical operations such as software updating of the bus receive table have been completed. 37. The invention defined in claim 36 wherein the receiver control means include a completion interrupt flag set by the bus receive controller means. 38. The invention defined in claim 37 wherein the receiver control means include an interrupt mask register in each central processing unit corresponding to each interprocessor bus and having a mask which permits microinterrupts for that bus when the mask bit is on. 39. The invention defined in claim 29 wherein the central processing unit control memory in each processor module includes a send controller which causes the microprocessor in the central processing unit to interact with the outqueue means of the interprocessor control means to read a data block from memory, to divide the data block into packets, to calculate and to append a packet checksum on the data words in each packet, and to fill the packet into the outqueue for transmission one packet at a time over the interprocessor bus. 40. The invention defined in claim 39 wherein each central processing unit control memory includes a bus receive controller means in the receiving processor module for taking a received data packet from an inqueue means, storing the data into a memory buffer, and verifying the packet checksum. 41. The invention defined in claim 40 including bus receive table means in the receiver processor module operatively associated with the bus receive controller means for reassembling the packets into blocks. 42. The invention defined in claim 41 wherein the bus receive table means and the bus receive controller means cause the microprocessor to perform interleaved assembly of data blocks from packets from a plurality of sender processor modules and error checking transparently to the process executing in the receiving processor module. 43. The invention defined in claim 29 including error check means for insuring that data on each interprocessor bus is transmitted and received correctly. 44. The invention defined in claim 43 wherein the error check means include the addition of a checkword containing a checksum of the data words as the last word in each packet for verifying correct packet transfer. 45. The invention defined in claim 44 wherein the control memory includes a sending processor module controller means for generating the checksum and a receiving processor module controller means for comparing the checksum to verify the integrity of the transmission. 46. The invention defined in claim 45 wherein the error check means include an additional check for the data block to insure that a packet is not lost. 47. The invention defined in claim 29 wherein a data block sent over an interprocessor bus in one or more packets is always preceded by a control packet and is always followed by a trailer packet. 48. The invention defined in claim 47 wherein the control packet identifies the message and indicates how much data is to be received in the message. 49. The invention defined in claim 48 wherein the trailer packet means indicate whether or not there is an error during the data transmission and also provides a means for recognizing when too little or too much data has been transmitted. 50. The invention defined in claim 29 including power failure warning signal means for supplying a power failure warning signal to the inqueue means to cause the inquene means to load, at most, one further packet of information after the receipt of a power failure warning signal. 51. The invention defined in claim 28 wherein the control memory causes the microprocessor to transfer an entire block of data in response to a single send instruction. 52. The invention defined in claim 51 wherein the bus controller means include bus control state logic means for identifying a sender-receiver pair of processor modules and for defining the time frame for a packet transfer over the interprocessor bus between the sender-receiver pair. 53. The invention defined in claim 52 wherein the inqueue means include microinterrupt means for supplying a microinterrupt to the microprocessor in the receiver processor module after receipt of each packet and wherein the microprocessor supplies a program interrupt to the central processing unit of the receiver processor module after an entire data block has been received in the receiver processor module. 54. The invention defined in claim 52 wherein the interprocessor control means of the sender and receiver processor modules include means for logically disconnecting those processor modules from the interprocessor bus at the end of each data packet transfer to permit the bus control state logic means to establish a different sender-receiver pair of processor modules and to define a time frame for making a packet transfer between the other pair of sender-receiver processor modules so that a plurality of data block transfer between different sender-receiver pairs of processor modules can be interleaved on a packet by packet basis. 55. The invention defined in claim 54 including polling means in the bus controller means for polling at least one processor module to identify a processor module which is ready to send data on the interprocessor bus. 56. The invention defined in claim 55 wherein the polling means include sender counter means for sequentially polling the processor modules in a circular sequence starting after the last identified sender processor module to thereby insure that all processor modules have an opportunity to send data over the interprocessor bus. 57. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer, said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus, said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, input/output channel means in each processor module for connecting that processor module to a device controller, and a device controller operatively associated with the input/output channel means for controlling the transfer of data between the processor module and peripheral devices, said processor module including: reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and writing means for writing data received from the interprocessor bus into a specied location in the associated main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module. 58. The invention defined in claim 57 including a plurality of peripheral devices, a plurality of ports in each device controller, and a plurality of input/output buses connecting each device controller for access by a plurality of different processor modules. 59. The invention defined in claim 58 including a second interprocessor bus and a second bus controller means operatively associated with each interprocessor control means to provide, in combination with the multiple input/output buses, multiple separate paths for connecting each device controller with each processor module so that in the event of a failure of one path any processor module can still communicate with any device controller over another path. 60. The invention defined in claim 58 including interface common logic means operatively associated with each device controller for insuring that only one port of the device controller is logically connected to the device controller at a time. 61. The invention defined in claim 58 including stress responsive buffer means in each device controller for permitting data to be transmitted over each input/output bus between a port and an input/output channel means in a series of bursts. 62. The invention defined in claim 61 including jumpering means for varying the stress at which a buffer reconnect request is made to the input/output channel means. 63. The invention defined in claim 58 including input/output control table means having an entry for every device connected to the input/output channel means and wherein each entry contains a buffer address in memory and a count word for the number of bytes to be transferred in an input/output data transfer. 64. The invention defined in claim 63 wherein the input/output control table entries are located in the memory to protect that memory from a failing device controller or input/output bus. 65. The invention defined in claim 64 including another copy of the input/output control table located in each processor module that is operatively associated with the same device controller so that the failure of one processor module does not cause the loss of the device controller to the system. 66. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, and an input/output channel, interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer, said interprocessor bus means including, a common shared interprocessor bus, interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus, said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, a plurality of peripheral devices, a plurality of device controllers for controlling the transfer of data between the processor modules and the peripheral devices, multiple ports in each device controller, multiple associated input/output buses connecting each device controller for access by input/output channels of different processor modules, multiple separate power supplies operatively associated with each device controller for powering each device controller, and connecting means for powering a device controller from the remaining associated power supplies in the event of a failure or turn off of another power supply for that device controller whereby the power supply operatively associated with a processor module can be powered down while the rest of the multiprocessor system is on-line and functional. 67. The invention defined in claim 66 wherein the connecting means include diode switching means for providing logic power from the associated power supplies when the associated power supplies are operative and for providing logic power from the remaining power supplies in the event one power supply fails or is turned off and for providing a changeover in the event of failure or turn off which is smooth and without interruption or pulsation in the logic power supplied. 68. The invention defined in claim 66 including power-on circuit means in each processor module and device controller for detecting when the logic power supply to the processor module or device controller is below specifications for correct logic operation and for resetting the processor module or device controller to prevent all activity at its interfaces and to hold it in a state that is controlled when its logic power is affected by loss of line power, turn off, power supply failure, or power distribution failure. 69. The invention defined in claim 68 wherein the power-on circuit means also release the processor module or device controller and return it to operation after the power is restored and logic power is supplied at the proper specification to it. 70. The invention defined in claim 69 including transceiver means in the processor modules and device controllers which coact with the power-on circuit means to control the output of the transceiver means at the processor module and device controller interfaces as power comes up or goes down rather than leaving the output of the transceiver means undefined when power drops below the level required for correct operation. 71. The invention defined in claim 70 including an enable line connected to the transceivers at the device controller interfaces and wherein the power-on circuit coacts with the enable line to keep the transceivers in the off state until the power has risen to the level where the off state will be maintained by correct functioning of the logic elements in the device controller. 72. The invention defined in claim 66 wherein the interprocessor control means are constructed to enable additional processor modules to be added to the system without the need to modify either the existing hardware or software and wherein the input/output channel means are constructed to enable additional device controllers and peripheral devices to be added to the system whereby the system can be expanded both in processing capacity with additional processor modules and in input/output capacity with additional input/output channels, device controllers, and peripheral devices. 73. A processor module for a multiprocessor system of the kind in which separate processor modules communicate for signaling and data transfer over an interprocessor bus having a bus clock provided by an associated bus controller, said processor module comprising, a central processing unit having a processor clock which is independent of the bus clock, a memory containing instructions and data for that module, and interprocessor control means for connecting the processor module to the interprocessor bus, said interprocessor control means including buffer means for storing data transferred between the interprocessor bus and the processor memory, and buffer control means for controlling filling and emptying of the buffer means, said buffer control means having first logic means for operating in synchronism with the bus clock and second logic means for operating in synchronism with the processor clock, and first interlock means associated with the first logic means for receiving the state of the second logic means and for responding to enable certain state changes of said first logic means, and second interlock means associated with the second logic means for receiving the state of the first logic means and for responding to enable certain state changes of said second logic means, so that the interprocessor bus operates in synchronism with the bus clock and the central processing unit and memory of the processor module operate in synchronism with the processor clock without loss or duplication of the data being transferred and without loss of information about the state of the transfer sequence. 74. A method of interconnecting a plurality of processor modules by a multimodule communication path for insuring that one processor module ready to send can establish a sender-receiver path with another one of the processor modules ready to receive and can transmit data on a time synchronized basis without reliance on handshake acknowledgement between the pair of processor modules for each data word transmitted, said method comprising, connecting all of the processor modules with a multimodule communication path, controlling the multimodule communication path by a bus controller, operating the bus as a synchronous bus with a portion of each processor module synchronized with a bus clock provided by the bus controller, transmitting data in the form of multiword packets, polling the processor modules on a demand basis to identify a processor module which is ready to send a packet on the bus and to identify an intended receiver processor module, establishing a sender-receiver path by interrogating the readiness of the receiver processor module to receive and then allocating a bus time for the sending of a packet which time corresponds exactly to the length of the packet itself, and then releasing the bus and returning the bus controller to its idle state, or if further demanded again to its polling state. 75. The invention defined in claim 74 including specifying the intended receiver processor module and detecting the readiness of the receiver processor module to receive before sending the packet. 76. The invention defined in claim 75 wherein each processor module has an individual select line extending to that processor module from the bus controller and including producing select signals over the individual select lines to set up data transfers over the bus to establish the sender-receiver pair of processor modules and to define the time frame for the transfer of information over the bus between the sender-receiver and to exclude other processor modules from transmitting on the bus simultaneously with the selected sender processor module. 77. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising, a plurality of separate processor modules, interprocessor bus means including an interprocessor bus connecting each processor module for direct processor module to processor module signaling and data transfer and wherein the interprocessor bus means are separate and distinct from an input/output system and from a memory system bus, each processor module having a central processing unit, a memory for that module, and interprocessor control means for connecting the processor module to the interprocessor bus for signaling and data transfer with another processor module, bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers between any two processor modules over the interprocessor bus and for controlling transmissions over the interprocessor bus, each processor module including input/output channel means for transferring data between the processor module and one or more device controllers, a plurality of peripheral devices, a plurality of device controllers for controlling the transfer of data between the processor modules and the peripheral devices, a plurality of ports in each device controller, a plurality of input/output buses connecting each device controller for access by a plurality of different processor modules and wherein the input/output channel means, input/output buses and device controllers are operatively connected to provide, directly or in combination with the interprocessor bus means and a processor module, a path for data transfer between each processor module and any device controller. 78. The invention defined in claim 77 including a second interprocessor bus interconnecting each processor module and providing an alternate path for direct processor module to processor module signaling and data transfer. 79. The invention defined in claim 78 including distributed power supply means for insuring continuous operation of the remainder of the multiprocessor system in the event of a failure of a single power supply to a part of the multiprocessor system, said distributed power supply means including a separate power supply for each processor module and connecting means for connecting a plurality of the separate power supplies to each device controller and effective to supply power to each device controller from the remaining separate power supplies in the event of the failure of one of the power supplies connected to any device controller whereby the power supply operatively associated with a processor module can be powered down so that on-line maintenance can be performed in a power-off condition on that processor module while the rest of the multiprocessor system is on-line and functional. 80. A multiprocessor system constructed for continued system operation in the event of a failure of any single component in the system and comprising, a plurality of separate processor modules, each module including a central processing unit and a local memory, interprocessor bus means including a plurality of interprocessor buses for signaling and data transfer between separate processor modules, at least two interprocessor buses being connected to each processor module, each interprocessor bus being separate and distinct from an input/output system and from a memory bus, a plurality of device controllers each adapted for connection to at least one peripheral device, and a plurality of input/output buses each connected between a related processor module and one or more device controllers, each device controller being accessible to at least two processor modules by means of associated input/output buses each of said processor modules including: reading means for reading data from the associated local memory of that processor module for transmission to an interprocessor bus, and writing means for writing data received from an interprocessor bus into a specified location in the associated local memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module.


	*Description*

BACKGROUND OF THE INVENTION This invention relates to a multiprocessor computer system in which interconnected processor modules provide multiprocessing (parallel processing in separate processor modules) and multiprogramming (interleaved processing in one processor module). This invention relates particularly to a system which can support high transaction rates to large on-line data bases and in which no single component failure can stop or contaminate the operation of the system. There are many applications which require on-line processing of large volumes of data at high transaction rates. For example, such processing is required in retail applications for automated point of sale, inventory and credit transactions and in financial institutions for automated funds transfer and credit transactions. In computing applications of this kind it is important, and often critical, that the data processing not be interrupted. A failure of an on-line computer system can shut down a portion of the related business and can cause considerable loss of data and money. Thus, an on-line system of this kind must provide not only sufficient computing power to permit multiple computations to be done simultaneously, but it must also provide a mode of operation which permits data processing to be continued without interruption in the event some component of the system fails. The system should operate either in a fail-safe mode (in which no loss of throughput occurs as a result of failure) or in a fail-soft mode (in which some slowdown occurs but full processing capabilities are maintained) in the event of a failure. Furthermore, the system should also operate in a way such that a failure of a single component cannot contaminate the operation of the system. The system should provide fault-tolerant computing. For fault-tolerant computing all errors and failures in the system should either be corrected automatically, or if the failure or error cannot be corrected automatically, it should be detected, or if it cannot be detected, it should be contained and should not be permitted to contaminate the rest of the system. Since a single processor module can fail, it is obvious that a system which will operate without interruption in an on-line application must have more than one processor module. Systems which have more than one processor module can therefore meet one of the necessary conditions for noninterruptible operation. However, the use of more than one processor module in a system does not by itself provide all the sufficient conditions for maintaining the required processing capabilities in the event of component failure, as will become more apparent from the description to follow. Computing systems for on-line, high volume, transaction oriented, computing applications which must operate without interruption therefore require multiprocessors as a starting point. But the use of multiprocessors does not guarantee that all of the sufficient conditions will be met, and fulfilling the additional sufficient conditions for on-line systems of this kind has presented a number of problems in the prior art. The prior art approach to uninterrupted data processing has proceeded generally along two lines--either adapting two or more large, monolithic, general purpose computers for joint operation or interconnecting a plurality of minicomputers to provide multiprocessing capabilities. In the first case, adapting two large monolithic general purpose computers for joint operation, one conventional prior art approach has been to have the two computers share a common memory. Now in this type of multiprocessing system a failure in the shared memory can stop the entire system. Shared memory also presents a number of other problems including sequencing accesses to the common memory. This system, while meeting some of the necessary conditions for uninterruptible processing, does not meet all of the sufficient conditions. Furthermore, multiprocessing systems using large general purpose computers are quite expensive because each computer is constructed as a monolithic unit in which all components (including the packaging, the cooling system, etc.) must be duplicated each time another processor is added to the system even though many of the duplicated components are not required. The other prior art approach of using a plurality of minicomputers has (in common with the approach of using large general purpose computers) suffered from the drawback of having to adapt a communications link between computers that were never originally constructed to provide such a link. The required links were, as a result, usually made through the input/output channel. Connections through the input/output channel are necessarily slower than internal transfer within the processor itself, and such interprocessor links have therefore provided relatively slow interprocessor communication. Furthermore, the interprocessor connections required special adapter cards that added substantially to the cost of the overall system and that introduced the possibility of single component failures which could stop the system. Adding dual interprocessor links and adapter cards to avoid problems of critical single components failures increased the overall system cost even more substantially. Providing dual links and adapter cards between all processors generally became very cumbersome and quite complex from the standpoint of operation. Another problem of the prior art arose out of the way in which connections were made to peripheral devices. If a number of peripheral devices are connected to a single input/output bus of one processor in a multiprocessor system and that processor fails, then the peripheral devices will be unavailable to the system even though the failed processor is linked through an interprocessor connection to another processor or processors in the system. To avoid this problem, the prior art has provided an input/output bus switch for interconnecting input/output busses for continued access to peripheral devices when a processor associated with the peripheral devices on a particular input/output bus fails. The bus switches have been expensive and also have presented the possibility of single component failure which could down a substantial part of the overall system. Providing software for the prior art multiprocessor systems has also been a major problem. Operating systems software for such multiprocessing systems has tended to be nonexistent. Where software had been developed for such multiprocessor systems, it quite often was restricted to a small number of processors and was not adapted for the inclusion of additional processors. In many cases it was necessary either to modify the operating system or to put some of the operating system functions into the user's own program--an expensive, time-consuming operation. The prior art lacked a satisfactory standard operating system for linking processors. It also did not provide an operating system for automatically accommodating additional processors in a multiprocessing system constructed to accommodate the modular addition of processors as increased computering power was required. A primary object of the present invention is to construct a multiprocessor system for on-line, transaction-oriented applications which overcomes the problems of the prior art. A basic objective of the present invention is to insure that no single failure can stop the system or significantly affect system operation. In this regard, the system of the present invention is constructed so that there is no single component that attaches to everything in the system, either mechanically or electrically. It is a closely related objective of the present invention to guarantee that every error that happens can be either corrected, detected or prevented from contaminating the system. It is another important objective of the present invention to provide a system architecture and basic mode of operation which free the user from the need to get involved with the system hardware and the protocol of interprocessor communication. In the present invention every major component is modularized so that any major component can be removed or replaced without stopping the system. In addition, the system can be expanded in place (either horizontally by the addition of standard processor modules or in most cases vertically by the addition of peripheral devices) without system interruption or modification to hardware or software. SUMMARY OF THE INVENTION The multiprocessor system of the present invention comprises multiple, independent processor modules and data paths. In one specific embodiment of the present invention 16 separate processor modules are interconnected by an interprocessor bus for multiprocessing and multiprogramming. In this specific embodiment each processor module supports up to 32 device controllers, and each device controller can control up to eight peripheral devices. Multiple, independent communication paths and ports are provided between all major components of the system to insure that it is always possible to communicate between processor modules and between processor modules and peripheral devices over at least two paths and also to insure that a single failure will not stop system operation. These multiple communication paths include multiple interprocessor busses interconnecting each of the processor modules, multiports in each device controller, and input/output busses connecting each device controller for access by at least two different processor modules. Each processor module is a standard module and includes as part of the module a central processing unit, a main memory, an interprocessor control and an input/output channel. Each processor module has a pipelined microprocessor operated by microinstructions included as a basic instruction set in each processor module. The basic instruction set in each processor module recognizes the fact that there is an interprocessor communications link; and when an additional processor module is added to the system, the operating system (a copy of which resides in each processor module) is informed that a new resource is available for operation within the existing operating system without the need to modify either the system hardware or software. To increase performance and to maintain very high transaction rates each processor module includes a second microprocessor which is dedicated to input/output operations. A dual port access to the main memory by both the central processing unit and the input/output channel permits direct memory access for the input/output transfers to also increase performance. Each processor module is physically constructed to fit on a minimum number of large printed circuit boards. Using only a few boards for each processor module conserves space for packaging and minimizes the length of the interprocessor bus required to interconnect all of the processor modules. A relatively short interprocessor bus minimizes the deterioration of the signals on the interprocessor bus and permits high speed of communication over the interprocessor bus. Each interprocessor bus is a high speed, synchronous bus to minimize overhead in interprocessor communications and to enable the system to achieve high throughput rates. A separate bus controller monitors all transmissions over the bus. The bus controller includes processor select logic for determining the priority of data transfer between any two processor modules over the interprocessor bus. The bus controller also includes bus control state logic for establishing a sender-receiver pair of processor modules and a time frame for a transfer of information over the bus between the sender-receiver pair. Each bus controller includes a bus clock, and each central processing unit of each processor module has its own separate clock. There is no master clock system subject to a single component failure which could stop the entire multiprocessor system. Each processor module includes, in the interprocessor control of the processor module, a certain amount of circuitry on the printed circuit boards which is dedicated to communications over the interprocessor buses. Each interprocessor control also includes fast buffers (inqueue buffers and an outqueue buffer) which can be emptied and filled by the central processing unit without interfering with the interprocessor bus. This makes it possible to sustain a higher data rate on the interprocessor bus than could be sustained by any single pair of processors. Several data transfers between pairs of processor modules can be interleaved on an apparent simultaneous basis. Because the interprocessor bus operates asynchronously with each particular central processing unit, each inqueue and outqueue buffer is clocked either by the processor module or by the bus controller, but not by both simultaneously. Each inqueue buffer and outqueue buffer therefore has associated with it in the interprocessor control some logic that operates in synchronism with the bus clock and other logic that operates in synchronism with the central processing unit clock. Logic interlocks qualify certain transitions of the logic from one state to another state to prevent loss of data in transfers between the asynchronous interprocessor buses and processor module. The logic is also arranged so that in the event a processor module is powering down, there will be no transient effect on the interprocessor buses because the processor module is losing control. The powering down of the processor module on an interprocessor bus will therefore not disrupt any other interprocessor bus activity. The bus controller and interprocessor control of each processor module coact to perform all interprocessor bus management in parallel with processing by the central processing units so that there is no waste of processing power. This bus management is performed with low protocol overhead in that it takes very few interprocessor bus cycles to establish a bus transfer--what processor bus module is sending and what processor module is receiving--relative to the amount of information actually transmitted. The processor select logic of the bus controller includes an individual select line which extends from the processor select logic to each processor module. The select lines are used in three ways in the protocol of establishing a sender-receiver pair of processor modules and a time frame for transfer of information over the interprocessor bus between the sender-receiver pair. The select lines are used (1) in polling to determine which particular processor module wants to send, (2) in receiving to inquire of a receiver processor module whether the particular processor module wants to receive, and (3) in combination with a send command to let the sender processor module know the time frame for sending. The receiver processor module is qualified to receive incoming data unsolicited by the receiver processor module and without a software instruction. Blocks of data between a sender-receiver pair of processor modules are transmitted over the interprocessor bus in packets. At the end of each packet transfer the interprocessor control of a receiver processor module logically disconnects from the interprocessor bus to permit the bus control state logic to establish another sequence of a different sender-receiver pair of processor modules and a time frame for making a packet transfer between the other pair of sender-receiver processor modules. Thus, as noted above, several data block transfers between different sender-receiver pairs of processor modules can therefore be interleaved on the interprocessor bus on an apparently simultaneous basis because of the faster clock rate of the interprocessor bus as compared to the slower memory speed of the processor modules. Each processor module memory includes a separate buffer for each combination of a processor module and an interprocessor bus. Each memory also includes a bus receive table for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module. Each bus receive table provides a bus receive table entry which contains the address where the incoming data is to be stored and the number of words expected from the sender processor module. The bus receive table entry is updated by firmware in the processor module after the receipt of each packet and is effective with the firmware either to provide a program interrupt when the entire data block has been successfully received or to provide an interrupt to the software program currently executing in the processor module in response to the detection of an error in the course of the transmission of the data over the interprocessor bus. Producing a program interrupt only at the completion of the data block transfer enables the transfer of data to be made transparent to the software currently executing in the processor module. The interrupt in response to the detection of an error provides an integrity check on the transmission of data. The input/output subsystem of the multiprocessor system of the present invention is constructed to insure that no single processor module failure can impair system operation. In addition, the input/output subsystem is constructed to handle very high transaction rates, to maximize throughput, and to minimize interference with programs running in the processor modules. As noted above, each processor module includes a microprocessor which is dedicated to input/output operations. The input/output system is an interrupt driven system and provides a program interrupt only upon completion of the data transfer. This relieves the central processing unit from being dedicated to the device while it is transferring data. Each input/output channel is block multiplexed to handle several block transfers of data from several device controllers on an apparent simultaneous basis. This is accomplished by interleaving variable length bursts of data in transfers between the input/output channel and stress responsive buffers in the device controllers. As noted above, each device controller has multiports, and a separate input/output bus is connected to each port so that each device controller is connected for access by at least two different processor modules. The ports of each device controller are constructed so that each port is logically and physically independent of each other port. No component part of one port is also a component of another port so that no single component failure in one port can affect the operation of another port. Each device controller includes logic which insures that only one port is selected for access at a time so that transmitting erroneous data to one port can never contaminate another port. The input/output system of the present invention interfaces the peripheral devices in a failsoft manner. There are multiple paths to each particular device in case of a failure on one path. And a failure of the device or a failure of a processor module along one path does not affect the operation of a processor module on another path to the device. The input/output system of the present invention is also constructed so that any type of device can be put on the system, and the input/output system will still make maximum usage of the input/output channel bandwidth. The device controllers are buffered such that all transfers between the device controllers and the input/output channel occur at the maximum channel rate. The device controller may transfer between itself and a peripheral device in bytes, but the device controller must pack and unpack data to transfer words between itself and the input/output channel. Because the buffers are located in the device controllers rather than in the input/output channel, the present invention limits the buffering to only the buffering required by a particular system configuration. The present invention does not require a separate buffer for each peripheral device in order to prevent overruns, as would be required if the buffers were located in the input/output channel rather than in the device controllers as had often been the practice in the prior art. As noted above, each buffer is a stress responsive buffer and this provides two advantages. First of all, each buffer can be constructed to have an overall depth which is related to the type and number of devices to be serviced. Each device controller can therefore have a buffer size which is related to the kind of devices to be controlled. Secondly, the stress responsive buffer construction and mode of operation of the present invention allows the buffers to cooperate without communicating with each other. This in turn permits optimum efficient use of the bandwidth of the input/output channel. The stress placed on a particular buffer is determined by the degree of the full or empty condition of the buffer in combination with the direction of the transfer with respect to the processor module. Stress increases as the peripheral device accesses the buffer, and stress decreases as the input/output channel means access the buffer. Each buffer has a depth which is the sum of a threshold depth and a holdoff depth. The threshold depth is related to the time required to service higher priority device controllers, and the holdoff depth is related to the time required to service lower priority device controllers connected to the same input/output channel. The stress responsive buffer includes control logic for keeping track of the stress placed on the buffer. The control logic is effective to make reconnect requests to the input/output channel as the stress passes through a threshold depth of the buffer. Each buffer having a reconnect request pending is individually connected to the input/output channel in accordance with a polling scheme which resolves priority among all the device controllers having a reconnect request pending. When the device controller is connected to the input/output channel, the data is transferred between the buffer and the input/output channel in a burst at or near memory speed. Thus, because the buffers transmit data to and from the peripheral devices at the relatively slow device speed and can transmit the data to and from the processor modules at or near memory speed in burst transfers, and in response to buffer stress, the burst transfers can be time division multiplexed so that individual bursts from several device controllers can be interleaved to optimize efficient use of the bandwidth of the input/output channel and also to permit several block transfers from different device controllers to be made on an apparent simultaneous basis. Comprehensive error checks and provision for error containment are provided for all data transfers over the data paths of the multiprocessor system. The error checks include check summing and parity checks on the data paths and error detection and correction in the main memory system. The error checks also include time out limitations in the input/output channel. Error containment is provided in the input/output system by an input/output control table having a two-word entry for each peripheral device to define a buffer area in the memory for the particular device controller and device. Each two-word entry describes the buffer location in main memory and the remaining byte count length to be transferred at any particular time for a particular data transfer to a device. The input/output control table is located in each processor instead of in the device controllers to contain the results of any failure in the countword or address word to the single processor module in which the countword or address word is physically located. Each of the processor modules that is connected for access to common device controllers and related devices contains its own copy of the input/output control table. The failure of a table entry in one processor module does not affect the other processor module because the other processor module has its own correct copy of the table entry. The multiprocessor system of the present invention includes a power supply system which distributes separate power supplies to the processor modules and device controllers in a way to insure non-stop operation of the remainder of the multiprocessor system in the event of failure of a power supply for part of the multiprocessor system. Any processor module or device controller can be powered down so that on-line maintenance can be performed in a powered-off condition while the rest of the multiprocessor system is on-line and functional. The power supply system includes a separate power supply for each processor module and two separate power supplies for each device controller. The two separate power supplies are operatively associated with the device controller by a switch which permits one power supply to supply all of the power for the device controller in the event of a failure of the other power supply. The power supply system of the present invention also produces a power failure warning signal which is effective to save the state of the logic in a processor module in the event of a failure of a power supply associated with that processor module. When power is restored, the processor module is returned to operation in a state that is known and without the loss of data. The memory of the multiprocessor system of the present invention is divided into four logical areas--user data, system data, user code and system code. This division of memory into four separate logical address areas separates code from data so that code can be made nonmodifiable and also separates operating system programs from user programs so that users cannot inadvertently destroy the operating system. The multiprocessor system of the present invention includes a memory map which performs a number of functions. One function of the map is to provide a virtual memory system in which all code and data are inherently relocatable so that the user need not be concerned with the actual physical location of either system or user programs or the amount of physical memory attached to the system. The map translates logical addresses to physical addresses for pages in main memory and provides page fault interrupts for pages not in main memory. The operating system brings pages from secondary memory (i.e., memory stored in peripheral devices) into the primary main memory in the processor module as required to implement a virtual memory system in which the physical page addresses are invisible to users and in which logical pages need not reside in contiguous physical pages and need not be in physical main memory but may be in secondary memory. The map also provides a protection function and a memory management function. The map provides a separate map for each separate logical area of memory. This provides protection by separating code from data and also by separating the user programs from the system programs, as pointed above. It also provides protection among users in a multiprogramming environment because the map which is in effect for particular user points only to the physical memory pages of that user's program. This prevents one user from writing into a program page of another user's program. This feature of a user map therefore protects, without the need for protection registers, one user from destroying another user's program. The map in conjunction with the operating system performs a map memory management function to reduce operating overhead in the management of the memory system by (1) making pages available from secondary memory, (2) keeping track of frequency of use of physical pages in primary memory, (3) reducing virtual memory page input/output transfers, and (4) reducing interrupts to the operating system. The way that the map accomplishes these functions provides an efficient virtual memory system. The number of pages available in physical main memory is limited. Physical pages must therefore sometimes be brought into physical main memory from secondary memory. One important aspect of efficient memory management is to keep track of what pages in physical main memory are being used frequently enough so as to need to be retained in physical main memory. Another important aspect is to know whether any particular pages in physical main memory can be written over (overlaid) without having to be first swapped out to secondary storage. The map includes history bits as a part of the map entry for each page. These history bits (which are physically in the map entry) give a histogram of usage of the given physical page over a period of time. And, in the present invention, the history bits are periodically updated by hardware without the need for program intervention. Each map entry also includes a "dirty bit" for indicating whether a particular page has been written into since it was last brought in from secondary storage. The map therefore includes in the map itself information which permits the memory manager to determine whether a particular page in physical main memory is a good candidate for being overlaid (when it is necessary to bring a page in from secondary storage and no empty page or code page in physical main memory is available for an overlay) and to determine also, if an overlay is required, whether or not it is necessary to swap the overlaid page out to secondary storage before the page can be overlaid. Since copies of all non-dirty pages are kept in secondary storage, no swap is required if the dirty bit is not on. The map is contained in a part of the memory which is separate from the main memory. Each map is constructed to provide significantly faster access than the access to physical main memory so that the map can be rewritten in the time that a physical memory access is being accomplished. The rewriting of the map therefore does not increase memory cycle time. As noted above, the memory includes dual port access for the central processing unit and the input/output channel. The input/output channel can therefore access the memory directly, without having to go through the central processing unit, for data transfers to and from a device controller. Central processing unit accesses to memory and input/output channel accesses to memory can therefore be interleaved in time. All data transfers to and from memory by the input/output channel are made by way of the system data map. The system data map adds additional bits in the course of translating the logical addresses to physical addresses. This permits a larger number of words of physical memory to be accessed by using a shorter logical address to access a larger physical space than the word width itself would normally allow. The present invention also provides a syndrome decoding method for detecting and correcting errors in semiconductor memory modules. The storage area of the semiconductor memory module comprises words of 22 bits. Each word has a 16 bit data field and a six bit check field. Each memory module includes an error detector for simultaneously correcting all single bit and detecting all double bit errors and detecting many of the errors of 3 bits or more anywhere in the 22 bit word. The error correction includes a check bit generator, a check bit comparator, and a syndrome decoder. The check bit generator provides a code in which each check bit is a linear combination of eight data bits and in which each data bit is a component of exactly three check bits. The check bit comparator provides six output syndrome bits. The input of each of the output syndrome bits is eight data bits and one check bit. The syndrome decoder interprets the value of the six output syndrome bits and identifies the presence or absence of errors and the type of errors, if any, in the 22 bit word. A data bit complementer is also provided for inverting a single data bit error detected by the syndrome decoder and thus correcting the error. The semiconductor memory system is therefore tolerant of single bit failures and can be operated with single bit failures until such time as it is convenient to repair the memory. Multiprocessor system apparatus and methods which incorporate the structure and techniques described above and which are effective to function as described above constitute further, specific objects of this invention. Other and further objects of the present invention will be apparent from the following description and claims and are illustrated in the accompanying drawings which, by way of illustration, show preferred embodiments of the present invention and the principles thereof and what are now considered to be the best modes contemplated for applying these principles. Other embodiments of the invention embodying the same or equivalent principles may be used and structural changes may be made as desired by those skilled in the art without departing from the present invention and the purview of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an isometric, block diagram view of a multiprocessor system constructed in accordance with one embodiment of the present invention. FIG. 1 shows several processor modules 33 connected by two interprocessor buses 35 (an X bus and a Y bus) with each bus controlled by a bus controller 37. FIG. 1 also shows several dual-port device controllers 41 with each device controller connected to the input/output (I/O) buses 39 of two processor modules. FIG. 2 is a block diagram view showing details of the connections of the X bus controller and the Y bus controller to the individual processor modules. FIG. 2 shows, in diagrammatic form, the connections between each bus controller and the interprocessor control 55 of an individual processor module. FIG. 3 is a detailed diagrammatic view of the logic of one of the bus controllers 37 shown in FIG. 2. FIG. 4 is a detailed diagrammatic view of the logic for the shared output buffer and control 67 in the interprocessor control 55 of a processor module as illustrated in FIG. 2. FIG. 5 is a view like FIG. 4 but showing the logic for an inqueue buffer and control 65 of the interprocessor control 55 for a processor module. FIG. 6 is a state diagram of the logic 81 for a bus controller 37 and illustrates how the logic responds to the protocol lines going into the bus controller and generates the protocol lines going out of the bus controller to the processor modules. FIG. 7 is a state diagram like FIG. 6 but showing the logic 73 and 75 for the shared outqueue buffer and control 67 of FIG. 4. FIG. 8 is a state diagram like FIGS. 6 and 7 but showing the logic 93 and 101 for the inqueue buffer and control 65 of FIG. 5. FIG. 9 is a diagrammatic view showing the time sequence for the transmission of a given packet between a sender processor module and a receiver processor module. FIG. 10 is a logic diagram of the bus empty state logic section 75 and the processor fill state logic section 73 of the outqueue buffer and the control 67 shown in FIG. 4. FIG. 11 is a listing of logic equations for the logic diagram shown in FIG. 10. FIG. 12 is a block diagram of the input/output (I/O) system of the multiprocessor system shown in FIG. 1. FIG. 13 is a block diagram of the input/output (I/O) channel 109 of a processor module. FIG. 13 shows the major components of the I/O channel and the data path relating those component parts. FIG. 14 is a detailed view showing the individual lines in the I/O bus 39 of FIG. 1. FIG. 15 is an I/O channel protocol diagram showing the state changes of the T bus 153 for an execute input/output (EIO) caused by the microprogram 115 in the CPU 105. The sequence illustrated is initiated by the CPU 105 and is transmitted through the I/O channel 109 of the processor module 33 and on the T bus 153 to a device controller 41 as shown in FIG. 1. FIG. 16 is an I/O channel protocol diagram showing the state changes of the T bus 153 for a reconnect and data transfer sequence initiated by the I/O channel microprogram 121 in response to a request signal from a device controller 41. FIG. 17 is an I/O channel protocol diagram showing the state changes of the T bus 153 for an interrogate I/O (IIO) instruction or an interrogate high priority I/O (HIIO) instruction initiated by the CPU microprogram 115. The sequence illustrated is transmitted over the T bus 153 to a device controller 41. FIG. 18 is a table identifying the functions referred to by the mnemonics in FIGS. 15 through 17. FIG. 19 is a block diagram showing the general structure of the ports 43 and a device controller 41 as illustrated in FIG. 1. FIG. 20 is a block diagram of a port 43 shown in FIG. 19. This FIG. 20 shows primarily the data paths within a port 43. FIG. 21 is a block diagram showing the data path details of the interface common logic 181 of the device controller 41 shown in FIG. 19. FIG. 22 is a block diagram showing the component parts of a data buffer 189 in the control part of a device controller 41 as illustrated in FIG. 19. FIG. 23 is a graph illustrating the operation of the data buffer 189 illustrated in FIG. 22 and FIG. 19. FIG. 24 is a timing diagram illustrating the relationship of SERVICE OUT (SVO) from the channel 109 to the loading of data into the port data register 213 (FIG. 21) and illustrates how the parity check is started before data is loaded into the register and is continued until after the data has been fully loaded into the register. FIG. 25 is a schematic view showing details of the power on circuit (PON) shown in FIGS. 19 and 21. FIG. 26 is a logic diagram of the buffer control logic 243 of the data buffer 189 (shown in FIG. 22) of a device controller 41. FIG. 26 shows how the buffer control logic 243 controls the handshakes on the data bus and controls the input and output pointers. FIG. 27 is a listing of the logic equations for the select register 173 shown in FIG. 20. These logic equations are implemented by the port control logic 191 shown in FIG. 20. FIG. 28 is a timing diagram showing the operation of the two line handshake between the I/O channel 109 and the ports 43. FIG. 29 is a logic diagram showing the logic for the general case of the handshake shown in FIG. 28. The logic shown in FIG. 29 is part of the T bus machine 143 of the input/output channel 109 shown in FIG. 13. FIG. 30 is a block diagram of a power distribution system. FIG. 30 shows how a plurality of independent and separate power supplies 303 are distributed and associated with the dual port device controllers 41 for insuring that each device controller has both a primary and an alternate power supply. FIG. 31 is an enlarged, detailed view of the switching arrangement for switching between a primary power supply and an alternate supply for a device controller. The switching structure shown in FIG. 31 permits both automatic switching in the event of a failure of the primary power supply and manual switching in three different modes--off, auto and alternate. FIG. 32 is a block diagram showing details of one of the separate and independent power supplies 303 illustrated in FIG. 30. FIG. 33 is a block diagram view showing details of the vertical buses and the horizontal buses for supplying power from separate power supplies 303 shown in FIG. 30 to the individual device controllers 41. The particular bus arrangement shown in FIG. 33 permits easy selection of any two of the individual power supplies as the primary and the alternate power supply for a particular device controller. FIG. 34 is a block diagram of the memory system and shows details of the memory 107 of a processor module 33 shown in FIG. 1. FIG. 35 is a block diagram showing details of the map section 407 of the memory 107 shown in FIG. 34. FIG. 36 is a block diagram showing the organization of logical memory into four logical address areas and four separate map sections corresponding to the four logical address areas. FIG. 36 also shows details of the bits and fields in a single map entry of a map section. FIG. 37 is a block diagram showing details of one of the memory modules 403 illustrated in FIG. 34. The memory module 403 shown in FIG. 37 is a semiconductor memory module. FIG. 38 is a diagram of a check bit generator used in the semiconductor memory module 403 shown in FIG. 37. FIG. 38 also lists logic equations for two of the eight bit parity trees used in the check bit register. FIG. 39 is a diagram of a check bit comparator used in the semiconductor memory module 403 shown in FIG. 37. FIG. 39 includes the logic equation for nine bit parity tree for syndrome bit zero. FIG. 40 is a diagram of a syndrome decoder used in the semiconductor memory module 403 shown in FIG. 37. FIG. 37 also lists the logic equations for the operation of the logic section 511 of the syndrome decoder. FIG. 41 is a logic diagram of a bit complementer used in the semiconductor memory module 403 shown in FIG. 37. FIG. 42 shows the various states of a two processor system running an application program which is required to be running continuously. The diagrams illustrate the two processors successively failing and being repaired and the application program changing its mode of operation accordingly. DESCRIPTION OF THE PREFERRED EMBODIMENTS THE MULTIPROCESSOR SYSTEM FIG. 1 is an isometric diagrammatic view of a part of a multiprocessor system constructed in accordance with one embodiment of the present invention. In FIG. 1 the multiprocessor system is indicated generally by the reference numeral 31. The multiprocessor system 31 includes individual processor modules 33. Each processor module 33 comprises a central processing unit 105, a memory 107, an input/output channel 109 and an interprocessor control 55. The individual processor modules are interconnected by interprocessor buses 35 for interprocessor communications. In a specific embodiment of the multiprocessor system 31, up to sixteen processor modules 33 are interconnected by two interprocessor buses 35 (indicated as the X bus and the Y bus in FIG. 1). Each interprocessor bus has a bus controller 37 associated with that bus. The bus controllers 37, interprocessor buses 35 and interprocessor controls 55 (FIG. 1), together with associated microprocessors 113, microprograms 115 and bus receive tables 150 (FIG. 2) provide an interprocessor bus system. The construction and operation of this interprocessor bus system are illustrated in FIGS. 2-11 and 42 and are described in more detail below under the subtitle The Interprocessor Bus System. The multiprocessor system 31 has an input/output (I/O) system for transferring data between the processor modules 33 and peripheral devices, such as the discs 45, terminals 47, magnetic tape drives 49, card readers 51, and line printers 53 shown in FIG. 1. The I/O system includes one I/O bus 39 associated with each I/O channel 109 of a processor module and one or more multi-port device controllers 41 may be connected to each I/O bus 39. In the specific embodiment illustrated, each device controller 41 has two ports 43 for connection to two different processor modules 33 so that each device controller is connected for access by two processor modules. The I/O system includes a microprocessor 119 and a microprogram 121 in the I/O channel 109 (See FIG. 12.) which are dedicated to input/output transfers. As also diagrammatically illustrated in FIG. 12, the microprocessor 113 and microprogram 115 of the central processing unit 105 and an input/output control table 140 in the main memory 107 of each processor module 33 are operatively associated with the I/O channel 109. The construction and operation of these and other components of the I/O system are illustrated in FIGS. 12-29 and are described in detail below under the subtitle The Input/Output System and Dual Port Device Controller. The multiprocessor system includes a power distribution system 301 which distributes power from separate power supplies to the processor modules 33 and to the device controllers 41 in a way that permits on-line maintenance and also provides redundancy of power on each device controller. As illustrated in FIG. 30, the power distribution system includes separate and independent power supplies 303. A separate power supply 303 is provided for each processor module 33, and a bus 305 supplies the power from the power supply 303 to the central processing unit 105 and memory 105 of a related processor module 33. As also illustrated in FIG. 30, each device controller 41 is connected for supply of power from two separate power supplies 303 through an automatic switch 311. If one power supply 303 for a particular device controller 41 fails, that device controller is supplied with power from the other power supply 303; and the changeover is accomplished smoothly and without any interruption or pulsation in the power supplied to the device controller. The power distribution system coacts with the dual port system of the device controller to provide nonstop operation and access to the peripheral devices in the event of a failure of either a single port 43 or a single power supply 303. The multiprocessor system includes a power on (PON) circuit 182 (the details of which are shown in FIG. 25) in several components of the system to establish that the power to that particular component is within certain acceptable limits. For example, the PON circuit 182 is located in each CPU 105, in each device controller 41, and in each bus controller 37. The purpose of the PON circuit is to present a signal establishing the level of power applied to that particular component; and if the power is not within certain predetermined acceptable limits, then the signal output is used to directly disable the appropriate bus signal of the component in which the PON is located. The power-on circuit functions in four states--power off; power going from off to on; power on; and power going from on to off. The power-on circuit initializes all of the logic states of the system as the power is brought up; and in the present invention, the power-on circuit provides an additional and very important function of providing for a fail-safe system with on line maintenance. To do this, the power-on circuit in the present invention is used in a unique way to control the interface circuits which drive all of the intercommunication buses in the system. The construction and operation of the power distribution system are illustrated in FIGS. 30-33 and are described in detail below under the subtitle Power Distribution System. The multiprocessor system includes a memory system in which the physical memory is divided into four logical address areas--user data, system data, user code and system code (See FIG. 36.). The memory system includes a map 407 and control logic 401 (See FIG. 34.) for translating all logical addresses to physical addresses and for indicating pages absent from primary storage bit present in secondary storage as required to implement a virtual memory system in which the physical page addresses are invisible to users. The memory system incorporates a dual port access to the memory by the central processing unit 105 and the I/O channel 109. The I/O channel 109 can therefore access the memory 107 directly (without having to go through the central processing unit 105) for data transfers to and from a device controller 41. The construction and operation of the memory system are illustrated in FIGS. 34-41 and are described in detail below under the subtitle Memory System. An error detection system is incorporated in the memory system for correcting all single bit and detecting all double bit errors when semiconductor memory is used in the memory system. This error detection system utilizes a 16 bit data field and a 6 bit check field as shown in FIG. 37 and includes a data bit complementer 487 as also shown in FIG. 37 for correcting single bit errors. FIGS. 37 through 41 and the related disclosure illustrate and describe details of the error detection system. Before going into the detailed description of the systems and components noted generally above, it should be noted that certain terminology will have the following meanings as used in this application. The term "software" will refer to an operating system or a user program instructions; the term "firmware" will refer to a microprogram in read only memory; and the term "hardware" will refer to actual electronic logic and data storage. The operating system is a master control program executing in each processor module which has primary control of the allocation of all system resources accessible to that processor module. The operating system provides a scheduling function and determines what process has use of that processor module. The operating system also allocates the use of primary memory (memory management), and it operates the file system for secondary memory management. The operating system also manages the message system. This provides a facility for information transfer over the interprocessor bus. The operating system arrangement parallels the modular arrangement of the multiprocessor system components described above, in that there are no "global" components. At the lowest level of the software system, two fundamental entities are implemented--processes and messages. A process is the fundamental entity of control within a system. Each process consists of a private data space and register values, and a possibly shared code set. A process may also access a common data space. A number of processes coexist in a processor module 33. The processes may be user written programs, or the processes may have dedicated functions, such as, for example, control of an I/O device or the creation and deletion of other processes. A process may request services from another process, and this other process may be located in the same processor module 33 as a process making the request, or the other process may be located in some other processor module 33. The processes work in an asynchronous manner, and the processes therefore need a method of communication that will allow a request for services to be queued without "races" (a condition in which the outcome depends upon the sequence of which process started first)--thus the need for "messages" (an orderly system of interprocessor module communication described in more detail below). Also, all interprocessor module communication should appear the same to the processes, regardless of whether the processes are in the same or in different processor modules. As will become more clear from the description to follow, the software structure parallels the hardware; and different processes can be considered equivalent to certain components of the hardware in arrangement and function. For example, just as the I/O channel 109 communicates over the I/O bus 39 to the device controller 41, a user process can make a request (using the message system) to the process associated with that device controller 41; and then the device process returns status back similar to the way the device controller 41 returns information back to the I/O channel 109 over the I/O bus 39. The other fundamental entity of the software system, the message, consists of a request for service as well as any required data. When the request is completed, any required values will be returned to the requesting process. When a message is to be sent between processes in two different processor modules 33, the interprocessor buses 35 are used. However, as noted above, all communication between processes appears the same to the processes, regardless of whether they are in the same or in different processor modules 33. This software organization provides a number of benefits. This method of structuring the software also provides for significantly more reliable software. By being able to compartmentalize the software structure, smaller module sizes can be obtained, and the interfaces between modules are well defined. The system is also more maintainable because of the compartmentalization of function. The well defined modules and the well defined interfaces in the software system also provide advantages in being able to make it easily expandible--as in the case of adding additional processor modules 33 or device controllers 41 to the multiprocessor system. Furthermore, there is a benefit to the user of the multiprocessor system and software system in that the user, writing his program, need not be aware of either the actual machine configuration or the physical location of other processes. Just as the hardware provides multiple functionally equivalent modules with redundant interconnects, so does the software. For example, messages going between processes in different processor modules 33 may use either interprocessor bus 35. Also, device controllers 41 may be operated by processes in either of the processor modules 33 connected to the device controller 41. The multiprocessor hardware system and software system described above enable the user to develop a fault tolerant application system by virtue of its replicated modules with redundant interconnects. THE INTERPROCESSOR BUS SYSTEM As pointed out above, the individual processor modules 33 are interconnected by two interprocessor buses 35 (an X bus and a Y bus) with each bus controlled by a related bus controller 37. Each interprocessor bus 35, in combination with its bus controller 37 and a related interprocessor control 55 in each processor module 33, provides a multi-module communication path from any one processor module to any other processor module in the system. The use of two buses assures that two independent paths exist between all processor modules in the system. Therefore, a failure in one path (one bus) does not prevent communication between the processor modules. The bus controller 37 for each interprocessor bus 35 is a controller which is, in a preferred form of the invention, separate and distinct from the processor modules 33. Each interprocessor bus 35 is a synchronous bus with the time synchronization provided by a bus clock generator in the bus controllers 37. The interprocessor control portions 55 of all of the modules associated with the bus make state changes in synchronism with that bus clock during transfers over the bus. As will be described in more detail below, the CPU 105 operates on a different clock from the interprocessor bus clock. During the filling of an outqueue or the emptying of an inqueue in the interprocessor control 55 by the CPU, the operation takes place at the CPU clock rate. However, transmission of packets over the interprocessor bus always takes place at the bus clock rate. It is an important feature of the present invention that the information transmitted over the interprocessor bus is transferred at high transmission rates without any required correspondence to the clock rates of the various CPUs 105. The information transfer rate over the interprocessor bus is also substantially faster than would be permitted by direct memory accesses into and out of the memory sections 107 at memory speed. This ensures that there is adequate bus bandwidth even when a large number of processor modules is connected in a multiprocessor system. A benefit of using separate clocks for each CPU 105 is that a master system clock is not required, and this eliminates a potential source of single component failure which could stop the entire system. The interprocessor control 55 incorporates logic interlocks which make it possible to operate the interprocessor buses 35 at one clock rate and each CPU 105 at its own independent clock rate without loss of data. The information transmitted over the bus is transmitted in multiword packets. In a preferred form of the present invention each packet is a sixteen word packet in which fifteen of the words are data words and one word is a check word. The control logic within the bus controller 37 and the interprocessor controls 55 of the individual modules 33 follows a detailed protocol. The protocol provides for establishing a sender-receiver pair and a time frame for the data packet transfer. At the end of the time frame for the transmission of the data packet, the bus controller 37 is released for another such sequence. The specific manner in which these functions are carried out will become more apparent after a description of the structural features of FIGS. 3-9 below. X bus 35 is identical in structure to the Y bus 35, so the structure of only one bus will be described in detail. As illustrated in FIG. 2, each bus 35 comprises sixteen individual bus data lines 57, five individual bus protocol lines 59, and one clock line 61, and one select line 63 for each processor module 33. As also illustrated in FIG. 2, the interprocessor control 55 of each processor module 33 includes two inqueue sections 65 (shown as an X inqueue section and a Y inqueue section in FIG. 2) and a shared outqueue section 67. With the specific reference to FIG. 4, the shared outqueue section 67 includes an outqueue buffer 69 which performs a storage function. In a preferred form the buffer 69 has sixteen words of sixteen bits each. The buffer 69 is loaded by the CPU and holds the data until the packet transmission time, at which time the data is gated out to the bus, a will be described in more detail below. The outqueue section 67 also includes a receive register 71, which in a preferred form of the invention is a four bit register. This register is loaded by the CPU with the number of the processor module to which the data will be sent. The control part of the outqueue section 67 includes a processor fill state logic section 73 which operates in synchronism with the CPU clock, a bus empty state logic section 75 which operates in synchronism with the X or Y bus clock, and an outqueue counter 77. During filling of the outqueue buffer 69 by the CPU, the outqueue counter 77 scans the buffer 69 to direct the data input into each of the sixteen words of the buffer; and, as the sixteenth word is stored into the outqueue buffer 69, the outqueue counter 77 terminates the fill state.