REGISTER  |  SIGN IN
United States Patent7168042
Braun , ; et al.January 23, 2007
Title
Force effects for object types in a graphical user interface
Abstract
Force effects for a graphical user interface of a computer are provided based on user preference information, which indicates the types of graphical objects in the graphical user interface which are to have force effects associated with them, and at least one particular force effect assigned to each of those types of graphical objects. An output of a force sensation by a haptic feedback device to the user is based on those assigned force effects and occurs when a displayed cursor controlled by a user interacts with a graphical object having one of the types. An architecture for a host computer allowing multi-tasking application programs to interface with a feedback device is also disclosed.
Inventors:Braun; Adam C. (Sunnyvale, CA), Beamer; Jonathan L.  (Menlo Park, CA), Rosenberg; Louis B.  (Pleasanton, CA), Chang; Dean C.  (Mountain View, CA)
Assignee:Immersion Corporation (San Jose, CA)
Appl. No.:09/974,197
Filed:October 9, 2001
PCT Pub Date:January 28, 2007
Current U.S. Class:715/701 715/702 345/156 
Current International Class:G06F 3/00 (20060101)
Field of Search:345/700-704,156,157,161,163,520,522 341/20,27 364/709.01
U.S. Patent Documents
20010002126May 2001Rosenberg et al.
2972140February 1961Hirsch
3157853November 1964Hirsch
3220121November 1965Cutler
3497668February 1970Hirsch
3517446June 1970Corlyon et al.
3623064November 1971Kagan
3902687September 1975Hightower
3903614September 1975Diamond et al.
3911416October 1975Feder
3919691November 1975Noll
4127752November 1978Lowthorp
4148014April 1979Burson
4160508July 1979Frosch et al.
4236325December 1980Hall et al.
4262549April 1981Schwellenbach
4333070June 1982Barnes
4464117August 1984Foerst
4484191November 1984Vavra
4513235April 1985Acklam et al.
4546347October 1985Kirsch
4560983December 1985Williams
4581491April 1986Boothroyd
4599070July 1986Hladky et al.
4708656November 1987de Vries et al.
4713007December 1987Alban
4734685March 1988Watanabe
4782327November 1988Kley et al.
4794384December 1988Jackson
4794392December 1988Selinko
4799055January 1989Nestler et al.
4823634April 1989Culver
4839838June 1989LaBiche et al.
4868549September 1989Affinito et al.
4885565December 1989Embach
4891764January 1990McIntosh
4930770June 1990Baker
4934694June 1990McIntosh
4935728June 1990Kley
4949119August 1990Moncrief et al.
4961038October 1990MacMinn
5019761May 1991Kraft
5022384June 1991Freels et al.
5022407June 1991Horch et al.
5035242July 1991Franklin et al.
5038089August 1991Szakaly
5044956September 1991Behensky et al.
5065145November 1991Purcell
5078152January 1992Bond et al.
5095303March 1992Clark et al.
5103404April 1992McIntosh
5107262April 1992Cadoz et al.
5116180May 1992Fung et al.
5146566September 1992Hollis, Jr. et al.
5165897November 1992Johnson
5175459December 1992Danial et al.
5181181January 1993Glynn
5185561February 1993Good et al.
5186629February 1993Rohen
5186695February 1993Mangseth et al.
5193963March 1993McAffee et al.
5195179March 1993Tokunaga
5197003March 1993Moncrief et al.
5212473May 1993Louis
5223776June 1993Radke et al.
5235868August 1993Culver
5240417August 1993Smithson et al.
5264768November 1993Gregory et al.
5271290December 1993Fischer
5275174January 1994Cook
5275565January 1994Moncrief
5280276January 1994Kwok
5283970February 1994Aigner
5296846March 1994Ledley
5296871March 1994Paley
5298890March 1994Kanamaru et al.
5299810April 1994Pierce et al.
5309140May 1994Everett, Jr. et al.
5334027August 1994Wherlock
5341459August 1994Backes
5354162October 1994Burdea et al.
5355148October 1994Anderson
5374942December 1994Gilligan et al.
5374972December 1994Nakane et al.
5396266March 1995Brimhall
5398044March 1995Hill
5405152April 1995Katanics et al.
5414337May 1995Schuler
5436622July 1995Gutman et al.
5437607August 1995Taylor
5457479October 1995Cheng
5466213November 1995Hogan et al.
5471571November 1995Smith et al.
5477237December 1995Parks
5491477February 1996Clark et al.
5513100April 1996Parker et al.
5524195June 1996Clanton, III et al.
5526480June 1996Gibson
5530455June 1996Gillick et al.
5547382August 1996Yamasaki et al.
5550562August 1996Aoki et al.
5565887October 1996McCambridge et al.
5565888October 1996Selker
5570111October 1996Barrett et al.
5575761November 1996Hajianpour
5576727November 1996Rosenberg et al.
5589828December 1996Armstrong
5589854December 1996Tsai
5596347January 1997Robertson et al.
5623582April 1997Rosenberg
5625576April 1997Massie et al.
5628686May 1997Svancarek et al.
5629594May 1997Jacobus et al.
5642469June 1997Hannaford et al.
5643087July 1997Marcus et al.
5666138September 1997Culver
5690582November 1997Ulrich et al.
5691747November 1997Amano
5691898November 1997Rosenberg et al.
5694013December 1997Stewart et al.
5709219January 1998Chen et al.
5714978February 1998Yamanaka et al.
5721566February 1998Rosenberg et al.
5731804March 1998Rosenberg
5734373March 1998Rosenberg et al.
5736978April 1998Hasser et al.
5742278April 1998Chen et al.
5745715April 1998Pickover et al.
5755577May 1998Gillio
5757358May 1998Osga
5760764June 1998Martinelli
5766016June 1998Sinclair et al.
5767839June 1998Rosenberg
5769640June 1998Jacobus et al.
5771037June 1998Jackson
5781172July 1998Engel et al.
5784052July 1998Keyson
5785630July 1998Bobick et al.
5790108August 1998Salcudean et al.
5802353September 1998Avila et al.
5805140September 1998Rosenberg et al.
5805165September 1998Thorne, III et al.
5808601September 1998Leah et al.
5818423October 1998Pugliese et al.
5825308October 1998Rosenberg
5828197October 1998Martin et al.
5844392December 1998Peurach et al.
5877748March 1999Redlich
5884029March 1999Brush, II et al.
5889672March 1999Schuler et al.
5896139April 1999Strauss
5914716June 1999Rubin et al.
5956484September 1999Rosenberg et al.
5959613September 1999Rosenberg et al.
5973670October 1999Barber et al.
5973689October 1999Gallery
5977977November 1999Kajiya et al.
5990860November 1999Takeuchi
5990869November 1999Kubica et al.
5990875November 1999Bi et al.
5999168December 1999Rosenberg et al.
6005551December 1999Osborne et al.
6020876February 2000Rosenberg et al.
6024576February 2000Bevirt et al.
6028593February 2000Rosenberg et al.
6078308June 2000Rosenberg et al.
6084587July 2000Tarr et al.
6088017July 2000Tremblay et al.
6088019July 2000Rosenberg
6100874August 2000Schena et al.
6104158August 2000Jacobus et al.
6111562August 2000Downs et al.
6111577August 2000Zilles et al.
6147674November 2000Rosenberg et al.
6157367December 2000Van Der Haar et al.
6160489December 2000Perry et al.
6160907December 2000Robotham et al.
6166723December 2000Schena et al.
6169540January 2001Rosenberg et al.
6181318January 2001Lim
6215495April 2001Grantham et al.
6219032April 2001Rosenberg et al.
6219033April 2001Rosenberg et al.
6219034April 2001Elbing et al.
6252579June 2001Rosenberg et al.
6266053July 2001French et al.
6285351September 2001Chang et al.
6288705September 2001Rosenberg et al.
6292170September 2001Chang et al.
6292174September 2001Mallett et al.
6300936October 2001Braun et al.
6317116November 2001Rosenberg et al.
6326964December 2001Snyder et al.
6343349January 2002Braun et al.
6374255April 2002Peurach et al.
6422941July 2002Thorner et al.
6424356July 2002Chang et al.
6433771August 2002Yocum et al.
Foreign Patent Documents
0 349 086Jan., 1990EP
0265011Apr., 1988EP
0607580Jul., 1994EP
0626634Nov., 1994EP
H2-185278Jul., 1990JP
H4-8381Jan., 1992JP
H5-192449Aug., 1993JP
H7-24147Jan., 1995JP
WO 9200559Jan., 1992WO
WO 9520788Aug., 1995WO
WO 9532459Nov., 1995WO
WO 9628777Sep., 1996WO
WO 9721160Jun., 1997WO
WO 9731333Aug., 1997WO
Other References
Baigrie, "Electric Control Loading--A Low Cost, High Performance Alternative," Proceedings of Interservice/Industry Training Systems Conference, pp. 247-254, Nov. 6-8, 1990. cited by other .
Iwata, "Pen-based Haptic Virtual Environment," 0-7803-1363-1/93 IEEE, pp. 287-292, 1993. cited by other .
Russo, "The Design and Implementation of a Three Degree of Freedom Force Output Joystick," MIT Libraries Archives pp. 1-131, May 1990, archived Aug. 14, 1990. cited by other .
Brooks et al., "Hand Controllers for Teleoperation--A State-of-the-Art Technology Survey and Evaluation," JPL Publication 85-11, NASA-CR-175890, pp. 1-84, Mar. 1, 1985. cited by other .
Jones et al., "A perceptual analysis of stiffness," ISSN 0014-4819 Springer International (Springer-Verlag); Experimental Brain Research, vol. 79, No. 1, pp. 150-156, 1990. cited by other .
Burdea et al., "Distributed Virtual Force Feedback, Lecture Notes for Workshop on Force Display in Virtual Environments and its Application to Robotic Teleoperation," 1993 IEEE International Conference on Robotics and Automation, pp. 25-44, May 2, 1993. cited by other .
Snow et al., "Model-X Force-Reflecting-Hand-Controller," NT Control No. NPO-17851; JPL Case No. 7348, pp. 1-4 with 45 pages of attachments, Jun. 15, 1989. cited by other .
Ouh-Young, "Force Display in Molecular Docking," Doctoral Dissertation, University of North Carolina at Chapel Hill, UMI Order No. 9034744, p. 1-369, 1990. cited by other .
Tadros, "Control System Design for a Three Degree of Freedom Virtual Environment Simulator Using Motor/Brake Pair Actuators," MIT Archive, pp. 1-88, Feb. 1990, archived Aug. 13, 1990. cited by other .
Caldwell et al., "Enhanced Tactile Feedback (Tele-Taction) Using a Multi-Functional Sensory System," 1050-4729/93, pp. 955-960, 1993. cited by other .
Adelstein et al., "Design and Implementation of a Force Reflecting Manipulandum for Manual Control research," DSC-vol. 42, Advances in Robotics, pp. 1-12, 1992. cited by other .
Gotow et al., "Controlled Impedance Test Apparatus for Studying Human Interpretation of Kinesthetic Feedback," WA11-11:00, pp. 332-337. cited by other .
Stanley et al., "Computer Simulation of Interacting Dynamic Mechanical Systems Using Distributed Memory Parallel Processors," DSC-vol. 42, Advances in Robotics, pp. 55-61, ASME 1992. cited by other .
Russo, "Controlling Dissipative Magnetic Particle Brakes in Force Reflective Devices," DSC-vol. 42, Advances in Robotics, pp. 63-70, ASME 1992. cited by other .
Kontarinis et al., "Display of High-Frequency Tactile Information to Teleoperators," Telemanipulator Technology and Space Telerobotics, Won S. Kim, Editor, Proc. SPIE vol. 2057, pp. 40-50, Sep. 7-9, 1993. cited by other .
Patrick et al., "Design and Testing of A Non-reactive, Fingertip, Tactile Display for Interaction with Remote Environments," Cooperative Intelligent Robotics in Space, Rui J. deFigueiredo et al, Editor, Proc. SPIE vol. 1387, pp. 215-222, 1990. cited by other .
Adelstein, "A Virtual Environment System For The Study of Human Arm Tremor," Ph.D. Dissertation, Dept. of Mechanical Engineering, MIT, Jun. 1989, archived Mar. 13, 1990. cited by other .
Bejczy, "Sensors, Controls, and Man-Machine Interface for Advanced Teleoperation," Science, vol. 208, No. 4450, pp. 1327-1335, 1980. cited by other .
Bejczy et al., "Generalization of Bilateral Force-Reflecting Control of Manipulators," Proceedings Of Fourth CISM-IFToMM, Sep. 8-12, 1981. cited by other .
McAffee et al., "Teleoperator Subsystem/Telerobot Demonstrator: Force Reflecting Hand Controller Equipment Manual," JPL 1988, JPL D-5172. cited by other .
Minsky, "Computational Haptics: The Sandpaper System for Synthesizing Texture for a Force-Feedback Display," Ph.D. Dissertation, MIT, Jun. 1995, archived Jul. 6, 1995. cited by other .
Jacobsen et al., "High Performance, Dextrous Telerobotic Manipulator With Force Reflection," Intervention/ROV '91 Conference & Exposition, Hollywood, Florida, May 21-23, 1991. cited by other .
Shimoga, "Finger Force and Touch Feedback Issues in Dextrous Telemanipulation," Proceedings of Fourth Annual Conference on Intelligent Robotic Systems for Space Exploration, Rensselaer Polytechnic Institute, Sep. 30-Oct. 1, 1992. cited by other .
IBM Technical Disclosure Bulletin, "Mouse Ball-Actuating Device With Force and Tactile Feedback," vol. 32, No. 9B, Feb. 1990. cited by other .
Terry et al., "Tactile Feedback In A Computer Mouse," Proceedings of Fourteenth Annual Northeast Bioengineering Conference, University of New Hampshire, Mar. 10-11, 1988. cited by other .
Howe, "A Force-Reflecting Teleoperated Hand System for the Study of Tactile Sensing in Precision Manipulation," Proceedings of the 1992 IEEE International Conference on Robotics and Automation, Nice, France, May 1992. cited by other .
Eberhardt et al., "OMAR--A Haptic display for speech perception by deaf and deaf-blind individuals," IEEE Virtual Reality Annual International Symposium, Seattle, WA, Sep. 18-22, 1993. cited by other .
Rabinowitz et al., "Multidimensional tactile displays: Identification of vibratory intensity, frequency, and contractor area," Journal of The Acoustical Society of America, vol. 82, No. 4, Oct. 1987. cited by other .
Bejczy et al., "Kinesthetic Coupling Between Operator and Remote Manipulator," International Computer Technology Conference, The American Society of Mechanical Engineers, San Francisco, CA, Aug. 12-15, 1989. cit- ed by other .
Bejczy et al., "A Laboratory Breadboard System For Dual-Arm Teleoperation," SOAR '89 Workshop, JSC, Houston, TX, Jul. 25-27, 1989. cited by other .
Ouhyoung et al., "A Low-Cost Force Feedback Joystick and Its Use in PC Video Games," IEEE Transactions on Consumer Electronics, vol. 41, No. 3, Aug. 1995. cited by other .
Marcus, "Touch Feedback in Surgery," Proceedings of Virtual Reality and Medicine The Cutting Edge, Sep. 8-11, 1994. cited by other .
Bejczy, et al., "Universal Computer Control System (UCCS) For Space Telerobots," CH2413-3/87/0000/0318501.00 1987 IEEE, 1987. cited by other .
Hirota et al. "Development of Surface Display" 1993 Univ. of Tokyo, IEEE, 1993, pp. 256-262. cited by other .
Minsky et al. "Creating an Illusion of Feel: Control Issues in Force Display" Univ. of North Carolina at Chapel Hill, 1989, pp. 1-14. cited by other .
Buttolo et al.,"Pen-Based Force Display for Precision Manipulation in Virtual Environments," Univ. of Washington, IEEE, 0-8186-7084-3, 1995, pp. 217-224. cited by other .
Kilpatrick, P.J., "The Use of a Kinesthetic Supplement in an Interaction Graphics System" Univ. of North Carolina, 1976, pp. 1-161. cited by other .
Rosenberg et al., "Perceptual Decomposition of Virtual Haptic Surfaces" Proc. IEEE Symp. on Research Frontiers in Virtual Reality, 1993. cited by other .
Rosenberg, L., "Virtual haptic overlays performance in telepresence tasks" Stanford Univ., Dept. of Engineering, 1994. cited by other .
Rosenberg, L., "The Use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation" Wright-Patterson AFB AL/CF-TR-1994-0139, 1993, pp. 1-42. cited by other .
Su, S. A. et al. "The Virtual Panel Architecture: a 3D Gesture Framework" Univ. of Maryland, IEEE 4-7803-1363-1, 1993, pp. 387-393. cited by other .
Adachi et al. "Sensory Evaluation of Virtual Haptic Push-buttons" Technical Research Center, Suzuki Motor Corp., 1994. cited by other .
Iwata, H., "Pen-based Haptic Virtual Environment" Univ. of Tsukuba, IEEE 0-7803-1363-1, 1993, pp. 287-292. cited by other .
Kotoku, T., et al. "Environment Modeling for the Interactive Display (EMID) used in Telerobotics Systems," IEEE/RSJ Int'l Workshop on Intelligent Robots and Systems IROS '91, 91Tho375-6, pp. 999-1004. cited by other .
Iwata, H., "Artificial Reality with Force-Feedback: Development of Desktop Virtual Space with Compact Master Manipulator," Computer Graphics, vol. 24, No. 4, 1990, pp. 165-170. cited by other .
Brooks, F., et al. "Project Grope-Haptic Displays for Scientific Visualization," Computer Graphics, vol. 24, No. 4, 1990, pp. 177-185. cit- ed by other .
Yokokohji, Y., et al. "What you can see is What you can Feel--Development of a Visual/Haptic Interface to Virtual Environment," IEEE 0-8186-7295-1, 1996, pp. 46-54. cited by other .
Batter & Brooks "Grope 1--A Computer Display to the Sense of Feel," Proc. IFIP Congress 1971, pp. 759-763. cited by other .
Adelstein, B. et al., "Design and Implementation of a Force-Reflecting Manipulandum for Manual Control Research" NASA Ames Research Center, 1992, pp. 1-26. cited by other .
Adelstein, B. et al., "Design and Implementation of a Force Reflecting Manipulandum for Manual Control Research" NASA Ames Research Center, 1992, pp. 1-26. cited by other .
Russo, M., "The Design and Implementation of a Three Degree-of-Freedom Force Output Joystick," Dept. of Mech. Engineering, 1990, pp. 1-40. cited by other .
Schmult, B. et al., "Application Areas for a Force-Feedback Joystick" DSC--vol. 49, Advances in Robotics, Mechatronics, and Haptic Interfaces, ASME 1993, pp. 47-54. cited by other .
Atkinson, W., et al. "Computing with Feeling" Computer & Graphics, vol. 2, Pergamon Press, 1977, pp. 97-103. cited by other .
Winey III, Calvin McCoy, "Computer Simulated Visual & Tactile Feedback as an Aid to Manipulator and Vehicle Control", Dept. of Mech. Engineering, MIT 1991, pp. 1-79. cited by other .
Minsky, M., et al. "Feeling and Seeing: Issues in Force Display," ACM 089791-351-5, 1990, pp. 235-242, 270. cited by other .
B. Hannaford et al., "Force-Feeback Cursor Control," NASA Tech. Briefs, vol. 13, No. 11, Item #21, 1989, pp. 1-4. cited by other .
L. Rosenberg et al., "Commercially Viable Force Feedback Controller for Individuals w/ Neuromotor Disabilities," Wright Patterson AFB, 1996, pp. 1-33. cited by other .
Payette, Jule et al., "Evaluation of a Force Feedback (Haptic) Computer pointing Device in Zero Gravity," Dynamics Systems & Control Div. DSC--vol. 58, ASME 1996, pp. 547-553. cited by other .
Colgate, J. Edward et al., "Implementation of Stiff Virtual Walls in Force-Reflecting Interfaces," Northwestern University, 1993, pp. 1-7. cit- ed by other .
Louis Rosenberg, "Perceptual Design of a Virtual Rigid Surface Contact," Stanford University, Wright Patterson AFB, 1993, pp. 1-40. cited by other .
Louis Rosenberg et al., "The use of force feedback to enhance graphical user interfaces," Stereoscopic Displays and Virtual Reality Systems III, Proc. SPIE, 1996, pp. 243-248. cited by other .
Schmidt, Brian et al., "Application Areas for a Force-Feedback Joystick," DSC--vol. 49, Advances in Robotics, Mechatronics, and Haptic Interfaces, ASME 1993, pp. 47-54. cited by other .
Minsky, Margaret et al., "Feeling and Seeing: Issues in Force Display," University of North Carolina, ACM 1990, pp. 235-242, 270. cited by other .
Ramstein, Christophe, "Combining Haptic & Braille Technologies: Design Issues and Pilot Study," Assets, '96, ACM 0-89791-776-6, 1996, pp. 37-44. cited by other .
Munch, Stefan et al., "Intelligent Control for Haptic Displays," Eurographies '96, vol. 15, No. 3, Blackwell Publishers, 1996, pp. C218 to C226. cited by other .
Wiker, Stefan F. et al., "Development of Tactile Mice for Blind Access to Computers," Proc. of the Human Factors Society 35.sup.th Annual Meeting, 1991, pp. 707-712. cited by other .
Akamatsu, Motoyuki et al., "Multimodal Mouse: A Mouse-Type Device with Tactile and Force Display," Presence, vol. 3, No. 1, 1994, pp. 73-80. cit- ed by other .
Ramstein, Christophe et al., "The Pantograph: A Large Workspace Haptic Device for a Multimodal Human-Computer Interaction," Computer-Human Interaction, CHI '94, 1994, pp. 1-3. cited by other .
Kelley, A. J. et al., "On the Development of a Force-Feedback Mouse and it's Integration into a Graphical User Interface," 1994 Int'l Mechanical Engineering Congress and Exhibition, 1994, pp. 1-8. cited by other .
Kelley, A. J. et al., "Magic Mouse: Tactile and Kinesthetic Feedback in the Human-Computer Interface using and Electromagnetically Actuated Input/Output Device," University of British Columbia, 1993, pp. 1-27. cit- ed by other .
R. E. Ellis et al., "Design & Evaluation of A High-Performance Prototype Planar Haptic Interface," DSC--vol. 49, Advances in Robotics, Mechatronics, and Haptic Interfaces, ASME 1993, pp. 55-64. cited by other .
Rosenberg, Louis, "A Force Feedback Programming Primer," Immersion Corp., 1997, pp. 1-176. cited by other .
"Technical Manual Overhaul Instructions With Parts Breakdown, Coaxial Control Shaker Part No. C-25502," Safe Flight Instrument Corporation, Revised Jul. 15, 1980 (23 pages). cited by other .
Scannell, "Taking a Joystick Ride," Computer Currents, Boston Edition,, vol. 9, No. 11, Nov. 1994. cited by other .
Yamakita et al., "Tele-Virtual Reality of Dynamic Mechanical Model," Proceedings of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, Jul. 7-10, 1992. cited by other .
Noll, "Man-Machine Tactile," SID Journal, Jul./Aug. 1972 Issue. cited by other .
Rosenberg, "Virtual Fixtures: Perceptual Overlays Enhance Operator Performance in Telepresence Tasks," Ph.D. Dissertation, Stanford University, Jun. 1994. cited by other .
Patrick, "Design, Construction, and Testing of a Fingertip Tactile Display for Interaction with Virtual and Remote Environments," Master of Science Thesis, MIT, Aug. 1990, archived Nov. 8, 1990. cited by other .
Calder, "Design of A Force-Feedback Touch-Introducing Actuator For Teleoperator Robot Control," Bachelor of Science Thesis, MIT, May 1983, archived Jun. 23, 1983. cited by other .
Wiker, "Teletouch Display Development: Phase 1 Report," Technical Report 1230, Naval Ocean Systems Center, San Diego, Jul. 1988. cited by other .
Bliss, "Optical-to-Tactile Image Conversion for the Blind," IEEE Transactions on Man-Machine Systems, vol. MMS-11, No. 1, Mar. 1970. cited by other .
Johnson, "Shape-Memory Alloy Tactile Feedback Actuator," Armstrong Aerospace Medical Research Laboratory, AAMRL-TR-90-039, Aug., 1990. cited by other .
Kontarinis et al., "Tactile Display of Vibratory Information in Teleoperation and Virtual Environments," Presence, 4(4):387-402, Harvard Univ., 1995. cited by other .
Aukstakalnis et al., "Silicon Mirage: The Art and Science of Virtual Reality," ISBN O-938151-82-7, pp. 129-180, 1992. cited by other .
Eberhardt et al., "Inducing Dynamic Haptic Perception by The Hand: System Description and Some Results," DSC-vol. 55-1, Dynamic Systems and Control: vol. 1, ASME 1994. cited by other .
Gobel, et al., "Tactile Feedback Applied to Computer Mice," International Journal of Human-Computer Interaction, vol. 7, No. 1, pp. 1-24, 1995. cit- ed by other .
Pimentel et al., "Virtual Reality: through the new looking glass," 2.sup.nd Edition; McGraw-Hill, ISBN 0-07-050167-X, pp. 41:202, 1994. cite- d by other .
"Cyberman Technical Specification," Logitech Cyberman SWIFT Supplement to Logitech Mouse Technical Reference and Programming Guide, Apr. 5, 1994. cited by other .
Ouhyoung et al., "The Development of A Low-Cost Force Feedback Joystick and Its Use in the Virtual Reality Environment," Proceedings of the Third Pacific Conference on Computer Graphics and Applications, Pacific Graphics '95, Seoul, Korea, Aug. 21-24, 1995. cited by other .
Kaczmarek et al., "Tactile Displays," Virtual Environment Technologies, Chap. 9, pp. 349-414. cited by other .
Lake, "Cyberman from Logitech," at http://www.ibiblio.org/GameBytes/Issue21/greviews/cyberman.html, 1994. cited by other .
"Component Maintenance Manual With Illustrated Parts List, Coaxial Control Shaker Part No. C-25502," Safe Flight Instrument Corporation, Revised Jan. 28, 2002 (3 pages). cited by other .
Aliaga, "Virtual Objects in the Real World", Communications of the ACM, vol. 40, No. 3, Mar. 1997, pp. 49-54. cited by other.~
Primary Examiner: Tran; Henry N.
Attorney, Agent or Firm:Thelen Reid & Priest LLP Ritchie; David B.
Parent Case Text


This application is a continuation of U.S. application Ser. No. 08/970,953, filed Nov. 14, 1997, and entitled, "Force Feedback System Including Multi-Tasking Graphical Host Environment and Interface Device," now U.S. Pat. No. 6,300,936, which is incorporated herein by reference in its entirety.
Claims

What is claimed is:
1. A method, comprising: determining a force effect to be output at least partially based on user preference information, the user preference information including information associated with at least one type of graphical object from a plurality of types of graphical objects, the at least one type of graphical object being associated with a graphical user interface, the user preference information further including information associated with a least one force effect from a plurality of force effects, each force effect from the plurality of force effects being associated with a type of graphical object from the plurality of types of graphic objects; outputting a signal configured to cause the at least one force effect to be output when a cursor displayed in the graphical user interface interacts with a graphical object having a first type from the at least one type, the graphical object being associated with an application program from a plurality of application programs configured to run simultaneously, each application program from the plurality of application programs being configured to associate a force effect from the plurality of force effects with a graphical object from a plurality of graphical objects associated with that application program, each application program from the plurality of application programs providing information to a background application program about graphical objects associated with that application program.

2. The method of claim 1, wherein each application program from the plurality of application programs is configured to override a force effect associated with a graphical object associated with a background application program.

3. A system, comprising: means for determining a force effect to be output at least partially based on user preference information, the user preference information including information associated with at least one type of graphical object from a plurality of types of objects, the at least one type of graphical object being associated with a graphical user interface, the user preference information further including information associated with at least one force effect from a plurality of force effects, each force effect from the plurality of force effects being associated with a type of graphical object from the plurality of types of graphical objects; means for outputting a signal configured to cause the at least one force effect to be output when a cursor displayed in the graphical user interface interacts with a graphical object having a first type from the at least one type, the graphical object being associated with an application program from a plurality of application programs configured to run simultaneously, each application program from the plurality of application programs being configured to associate a force effect from the plurality of force effects with a graphical object from a plurality of graphical objects associated with that application program, each application program from the plurality of application programs being configured to interact with a background application program, the background application program being configured to associate a force effect from the plurality of force effects with at least one application program from the plurality of application programs.

4. The system of claim 3, wherein each application program from the plurality of application programs is configured to override a force effect associated with a graphical object associated with the background application program.

5. A processor-readable medium comprising code representing instructions to cause a processor to: determine a force effect to be output at least partially based on user preference information, the user preference information including information associated with at least one type of graphical object from a plurality of types of graphical objects, the at least one type of graphical object being associated with a graphical user interface, the user preference information further including information associated with at least one force effect from a plurality of force effects, each force effect from the plurality of force effects being associated with a type of graphical object from the plurality of types of graphical objects; output a signal configured to cause the at least one force effect to be output when a cursor displayed in the graphical user interface interacts with a graphical object having a first type from the at least one type, the graphical object being associated with an application program from a plurality of application programs, configured to run simultaneously, each application program from the plurality of application programs being configured to associate a force effect from the plurality of force effects with a graphical object from a plurality of graphical objects associated with that application program, each application program from the plurality of application programs being configured to interact with a background application program, the background application program being configured to associate a force effect from the plurality of force effects with an application program from the plurality of application programs.

6. A method, comprising: receiving a signal associated with a first application program, the first application program running during a first time period; receiving a signal associated with a second application program, the second application program running during a second time period that overlaps with the first time period in a multi-tasking environment; sending a first signal in response to receiving the signal from the first application program, the first signal configured to cause a force sensation associated with the signal associated with the first application program; and sending a second signal in response to receiving the signal from the second application program, the second signal configured to cause a force sensation associated with the signal associated with the second application program.

7. The method of claim 6, further comprising: storing in a first memory values associated with the signal from the first application program; and storing in a second memory values associated with the signal from the second application program.

8. The method of claim 6, wherein the signal from the first application program and the signal from the second application program include at least one command.

9. The system of 6, wherein the first signal is sent only when the first application program is indicated as active within the graphical user interface and the second signal is sent only when the second application program is indicated as active within the graphical user interface.

10. The method of claim 9, wherein the representation of the first application program becomes indicated as inactive when the representation of the second application program becomes indicated as active within the graphical user interface, and the representation of the second application program becomes indicated as inactive when the representation of the first application program becomes indicated as active within the graphical user interface.

11. The method of claim 10, further comprising: replacing commands from the first application program with commands from the second application program when the representation of the second application program becomes indicated as active within the graphical user interface and the representation of the first application program becomes indicated as inactive; and replacing commands from the second application program with commands from the first application program when the representation of the first application program becomes indicated as active within the graphical user interface and the representation of the second application program becomes indicated as inactive.

12. The method of claim 6, wherein the first signal is configured to be stored in a memory of a device and is configured to cause a first force sensation when the signal associated with the first application is received; and the second signal is configured to be stored in a memory of the device and is configured to cause a second force sensation when the signal associated with the second application is received.

13. The system of claim 6, wherein the first signal is configured to cause an enclosure sensation.

14. The system of claim 6, wherein the first signal is configured to cause an enclosure sensation, the enclosure sensation including different sensations for each of an inner portion and an outer portion.

15. A processor-readable medium comprising code representing instructions to cause a processor to: associate a context with a first application program from a plurality of application programs operating in a multi-tasking environment, the context defining at least one force effect corresponding to an event associated with the first application program; send a signal associated with the at least one force effect after detecting the corresponding event; and load the context into memory after the first application program becomes active.

16. The processor-readable medium of claim 15, further comprising code representing instructions to: associate a primary context with a master application program, the primary context defining at least one background force effect corresponding to an event associated the master application program; and send a signal associated with the at least one background force effect after detecting the corresponding event associated with the master application program.

17. The processor-readable medium of claim 15, further comprising code representing instructions to: associate a primary context with a master application program, the primary context defining at least one background force effect corresponding to an event associated with a background application program controlled by the master application program; and send a signal associated with the at least one background force effect after detecting the corresponding event associated with the master application program.

18. The processor-readable medium of claim 15, further comprising code representing instructions to: select the at least one force effect defined by the context via a graphical user interface.

19. The processor-readable medium of claim 15, further comprising code representing instructions to: uniquely associate the at least one force effect with the event associated with the first application program.

20. The processor-readable medium of claim 15, further comprising code representing instructions to: uniquely associate the at least one force effect with an object associated with the first application program.

21. The processor-readable medium of claim 15, further comprising code representing instructions to: call an object to send the signal associated with the at least one force effect.

22. The processor-readable medium of claim 15, further comprising code representing instructions to: call an object to send the signal associated with the at least one force effect, the object including at least one of an interface object, a device object, and an effect object.

23. An article of manufacture for use in a digital processing system, the article of manufacture comprising a digital processing system usable medium having readable program code embodied in the medium, the program code comprising: associate a context with a first application program from a plurality of application programs operating in a multi-tasking environment, the context defining at least one force effect corresponding to an event associated with the first application program; and send a signal associated with the at least one force effect after detecting the corresponding event, wherein the signal associated with the at least one force effect is sent only if it is determined that the first application program is active.

24. An apparatus, comprising: a device configured to output force sensations, the device being configured to communicate with a second processor; and a first processor configured to send the second processor a first signal and to receive a second signal, the first signal indicating relative position information of the apparatus if force feedback functionality is not enabled, the first signal indicating absolute position information of the apparatus if force feedback functionality is enabled.

25. The apparatus of claim 24, wherein the first processor is further configured to calculate the absolute position information at least partially based on the second signal.

26. The apparatus of claim 24, wherein the second signal is associated with ballistic parameters, the first processor being further configured to calculate the absolute position information at least partially based on the ballistic parameters.

27. The apparatus of claim 24, wherein the second signal is associated with the force feedback information, the first processor being further configured to cause the device to output force sensations.

28. A method, comprising: sensing a movement of a device; sending relative position information of the device to a processor if force feedback functionality associated with the device is not enabled; and sending absolute position information of the device to the processor if the force feedback functionality is enabled.

29. The method of claim 28, further comprising: sending an initial relative position parameter to the processor.

30. The method of claim 28, further comprising: receiving control information from the processor.

31. The method of claim 28, further comprising: receiving control information from the processor, the control information including an indication of whether the force feedback functionality is enabled.

32. The method of claim 28, further comprising: receiving control information from the processor, the control information including ballistic parameters.

33. The method of claim 28, further comprising: receiving control information from the processor, the control information including ballistic parameters; and calculating the absolute position information of the device based at least partially on the ballistic parameters.
Description
BACKGROUND OF THE INVENTION

The present invention relates generally to interface devices for allowing humans to interface with computer systems, and more particularly to computer interface devices that allow the user to provide input to computer systems and provide force feedback to the user.

Computer systems are used extensively to implement many applications, such as word processing, data management, simulations, games, and other tasks. A computer system typically displays a visual environment to a user on a display screen or other visual output device. Users can interact with the displayed environment to perform functions on the computer, play a game, experience a simulated environment, use a computer aided design (CAD) system, etc. One visual environment that is particularly common is a graphical user interface (GUI). GUI's present visual images which describe various graphical metaphors of a program or operating system implemented on the computer. Common operating systems using GUI's include the Windows.TM. operating system from Microsoft Corporation and the MacOS operating system from Apple Computer, Inc. The user typically moves a displayed, user-controlled graphical object, such as a cursor or pointer, across a computer screen and onto other displayed graphical objects or predefined screen regions, and then inputs a command to execute a given selection or operation. The objects or regions ("targets") can include, for example, icons, windows, pull-down menus, buttons, and scroll bars. Most GUI's are currently
2-dimensional as displayed on a computer screen; however, three dimensional (3-D) GUI's that present simulated 3-D environments on a 2-D screen can also be provided. Other programs or environments that may provide user-controlled graphical objects such as a cursor or a "view" controlled by the user include graphical "web pages" or other environments offered on the World Wide Web of the Internet, CAD programs, video games, virtual reality simulations, etc.

The user interaction with and manipulation of the computer environment is achieved using any of a variety of types of human-computer interface devices that are connected to the computer system controlling the displayed environment. In most systems, the computer updates the environment in response to the user's manipulation of a user-manipulatable physical object ("user object") that is included in the interface device, such as a mouse, joystick, etc. The computer provides feedback to the user utilizing the display screen. A computer mouse is a common user object used to interact with a GUI or other graphical environment. A mouse (and other mouse-type devices such as a track ball) is typically used as a position control device in which displacement of the mouse in a planar workspace (e.g. on a mouse pad) is directly correlated to displacement of the user-controlled graphical object, such as a cursor, displayed on the screen.

Force feedback interface devices allow a user to experience forces on the manipulated user object based on interactions and events within the displayed graphical environment. Typically, computer-controlled actuators are used to output forces on the user object in provided degrees of freedom to simulate various sensations, such as an obstruction force when moving the cursor into a wall, a damping force to resist motion of the cursor, and a spring force to bias the cursor to move back toward a starting position of the spring. Force feedback devices can be implemented in many forms, such as a joystick, mouse, steering wheel, etc.

When implementing force feedback sensations in a GUI of an operating system, several problems can arise. One problem is the use of force feedback when multiple application programs are simultaneously running in a multi-tasking environment on the host computer. Most operating systems allow such multi-tasking, for example, to allow a user to interact with one application while one or more applications are also running, receiving data, outputting data, or performing other tasks. For example, in the Windows.TM. operating system, one application is the "active" application that typically displays an active window in the GUI. The user can manipulate the functions of the active application using the cursor. Other inactive applications are also running and may have inactive windows displayed in the GUI. The user can switch to a different application by clicking the cursor in an inactive window, for example, which causes the new application to be the active application and the formerly active application to become inactive.

Each application run by an operating system may have its own set of force sensations that it needs to command to the force feedback device. Thus, one application may need to command spring, force, vibration, and texture force sensations, while a different application may need to command spring, damper, and jolt force sensations. The force feedback device typically cannot store all possible force sensations for each application running in the operating system, so there is a problem of which force sensations the force feedback device should store and implement at any one time. In addition, if two of the multi-tasking applications command conflicting force sensations, the force feedback device needs to choose one of the force sensations to output, and there currently is no system or method of doing so.

A different problem occurs when using a force feedback device with a GUI. Traditional mouse controllers used with GUI's are relative position reporting devices, i.e., they report only changes in position of the mouse to the host computer, which the host computer uses to calculate a new position for the cursor on the screen. Many force feedback devices, in contrast, are typically absolute position reporting devices which report an absolute position of the cursor, such as screen coordinates, to the host computer. This is because the force feedback device needs to know the cursor position to accurately determine when forces are to be applied and to accurately calculate the forces. However, it would be desirable in some instances to have a relative position reporting force feedback device, since the host computer is standardized to receive and interpret relative positions at the most basic level. Furthermore, such a relative device would permit the host computer to perform needed adjustments to cursor position, such as ballistics calculations which modify cursor position based on mouse velocity to provide enhanced control. If the host computer performs such adjustments, the force feedback device processors are relieved of computational burden. In addition, some types of interface devices such as trackballs are better suited to relative position reporting since an absolute, limited workspace is not easily defined for these devices.

Another problem occurs when force feedback is implemented with a GUI or other graphical environment and the graphical environment changes resolution or aspect ratio. For example, if a resolution of 640.times.480 is being displayed by the host computer on a screen, the force feedback device assumes that graphical objects in the GUI have a size proportional to screen dimensions and outputs forces accordingly. However, when the resolution is changed, the objects displayed on the screen change size in proportion to the screen dimensions. The force feedback device continues to check for conditions and generate forces as if the old resolution were active, resulting in forces that do not correlate with displayed interactions on the screen. The aspect ratio of a display screen can also change, e.g., when two screens are used to provide double the amount of displayed area in the GUI, the aspect ratio doubles in one dimension. Using prior art force feedback devices, forces can become distorted from such an aspect ratio change. For example, a circle object displayed on the screen may have forces at its borders that feel like an ellipse to the user of the interface device, since the aspect ratios of the screen and the mouse workspace are different.

SUMMARY OF THE INVENTION

The present invention is directed to a force feedback system and architecture which allow control of a force feedback device in a multi-tasking graphical host environment. In addition, force feedback device embodiments are disclosed which provide for relative position reporting and absolute position reporting to the host computer.

More specifically, a method of the present invention interfaces a multi-tasking graphical environment implemented on a host computer with a force feedback interface device coupled to the host computer, where multiple application programs may run in the multi-tasking environment. A context in created for association with each application program running in the multi-tasking graphical environment. Force effect commands are received from the application programs, where the force effect commands command the force feedback interface device to output a force effect specified by the command. The force effect commands are stored into the contexts. Each context is associated with one of the application programs running on the host computer, and each force effect command is stored in a context associated with the application program that sent the force effect command. The force effect commands in the context of a particular application program are sent to the force feedback device when that particular application program is active in the multi-tasking environment. Preferably, the force effect commands in contexts of inactive application programs are not sent to the force feedback device. Thus, only the active application program may command forces on the force feedback device.

When the application program becomes inactive and a new application program becomes active, new force effect commands are sent to the force feedback device to replace the force effect commands of the formerly active application. Preferably, a background application is provided which also provides force effects to the force feedback device and may output forces on the device even when not active. Events are also provided which allow a graphical action such as an interaction of the cursor in the graphical environment with another object to cause an event notification to be sent to the application program. Preferably, only the active and background application programs may receive events.

In another aspect of the present invention, a force feedback device, such as a force feedback mouse, provides relative positions to a host when force feedback is not enabled, and provides absolute positions to the host when force feedback is enabled. Sensor data is read and a position of a user manipulandum in a device frame is determined. A device delta position of the manipulandum includes the change in position of the manipulandum from a previous position and is reported to the host computer when the host computer is not enabled to receive an absolute screen position of the cursor from the force feedback device. When the host computer is enabled to receive the cursor screen position, the screen position is determined from the device delta position and is reported to the host computer to allow said host computer to display the cursor in the screen frame at the screen position. A scaled position or "ballistic position" of the manipulandum related to the screen position of the cursor is used in determining a force to be output by actuators of the force feedback device. Preferably, the scaled position is determined from the delta position and the screen position of said cursor is determined based on the scaled position. Ballistic parameters and screen size of the host display screen can be received from the host computer to determine the scaled position and screen position.

In another aspect of the present invention, a force feedback device, such as a force feedback mouse, provides only relative positions to a host computer. A local microprocessor on the force feedback device determines a delta position of the manipulandum that includes the change in position of the manipulandum from a previous position, and the delta position is reported to the host. A screen position of the cursor is modeled from the delta position and the modeled screen position is used in the determination of a force output by actuators of the device. An actual screen position of the cursor is received from the host computer and the modeled screen position is corrected based on the actual screen position. Preferably, the modeled screen position is corrected incrementally over time to smooth out the correction and reduce a perception of the error correction by the user. Preferably, both the host and the device adjust the screen position using a ballistics algorithm. The host also preferably sends ballistic parameters and screen size to the device to allow accurate modeling of the screen position.

In another aspect of the present invention, a force feedback device reads sensor data and determines a position of a user manipulandum in a workspace of the device. A position of the manipulandum is reported to the host computer so that the host can display the cursor in a graphical environment. Information is received from the host computer describing a current screen size of the display screen of the host computer, where the current screen size is included in a determination of the position reported to the host.

Also described herein are several force feedback sensations and structures, including enclosures, textures, and grids. For example, an enclosure is a rectangular or elliptical shape having walls that provide forces to an associated graphical object such as an icon or window. The enclosure includes walls, where each wall may be associated with a force. In one embodiment, a particular type of enclosure is provided to modify the forces of a different enclosure to prevent conflicts between the forces of overlapping or closely-positioned enclosures.

The present invention provides several embodiments of components in a force feedback system. The architecture on the host computer allows multi-tasking application programs to interface with the force feedback device without conflicts. The force feedback device provides both relative position reporting and absolute position reporting to allow great flexibility. A relative position reporting device allows maximum compatibility with existing software. Information such as ballistic parameters and screen size sent from the host to the force feedback device allow accurate mouse positions and cursor positions to be determined in the force feedback environment.

These and other advantages of the present invention will become apparent to those skilled in the art upon a reading of the following specification of the invention and a study of the several figures of the drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of one embodiment of a mouse interface system suitable for use with the present invention;

FIG. 2 is a perspective view of an embodiment of a mechanism suitable for the interface system of FIG. 1;

FIG. 3 is a block diagram of the system of FIG. 1 for controlling a force feedback interface device of the present invention;

FIG. 4 is a block diagram of an architecture for a host computer providing multiple application programs communicating with the force feedback device;

FIG. 5 is a diagrammatic illustration of a background application program control panel allowing a user to characterize background forces;

FIGS. 6a, 6b and 6c are diagrammatic illustrations of embodiments of an enclosure force effect;

FIG. 7 is a diagrammatic illustration of a texture force effect;

FIG. 8 is a diagrammatic illustration of a grid force effect;

FIG. 9 is a block diagram illustrating a preferred embodiment of implementing the force feedback device and system of the present invention;

FIG. 10 is a flow diagram illustrating a method of implementing position reporting for the embodiment of FIG. 9;

FIG. 11 is a flow diagram illustrating a method of determining cursor position and indexing in the embodiment of FIGS. 9 and 10;

FIG. 12 is a block diagram illustrating a second embodiment of implementing the force feedback device and system of the present invention;

FIG. 13 is a flow diagram illustrating a method of implementing position reporting for the embodiment of FIG. 12; and

FIGS. 14 and 15 are diagrammatic illustrations showing the error correction between cursor positions from host computer and force feedback device in the embodiment of FIG. 13.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a perspective view of a force feedback mouse interface system 10 of the present invention, capable of providing input from the user to a host computer based on the user's manipulation of the mouse and capable of providing force feedback to the user of the mouse system based on events occurring in an environment implemented by the host computer. Mouse system 10 includes an interface device 11 that includes a user manipulatable object or manipulandum 12 and an interface 14, and a host computer 18.

User object (or "manipulandum") 12 is an physical object that is preferably grasped or gripped and manipulated by a user. By "grasp," it is meant that users may releasably engage a portion of the object in some fashion, such as by hand, with their fingertips, etc. For example, images are displayed and/or modified on a display screen 20 of the computer system 18 in response to such manipulations. In the described embodiment, user object 12 is a mouse 12 that is shaped so that a user's fingers or hand may comfortably grasp the object and move it in the provided degrees of freedom in physical space. For example, a user can move mouse 12 to correspondingly move a computer generated graphical object, such as a cursor or other image, in a graphical environment provided by computer 18. The available degrees of freedom in which mouse 12 can be moved are determined from the interface 14, described below. In addition, mouse 12 preferably includes one or more buttons 15 to allow the user to provide additional commands to the computer system.

It will be appreciated that a great number of other types of user manipulable objects can be used with the method and apparatus of the present invention in place of or in addition to mouse 12. For example, such objects may include a sphere, a puck, a joystick, cubical- or other-shaped hand grips, a fingertip receptacle for receiving a finger or a stylus, a flat planar surface like a plastic card having a rubberized, contoured, and/or bumpy surface, a handheld remote control used for controlling web pages or other devices, or other objects. In one embodiment, a small fingertip joystick can be provided, where a small stick is moved in a small planar region with a user's fingertips. Spring forces can be provided by the actuators of the device 11 to bias the stick (or any type of joystick) toward the center of the planar region to simulate a spring return on the joystick. Since fingertips are used, output forces need not be as high a magnitude as in other embodiments. Also, mouse
12 can be provided with such a centering spring bias, e.g. when used like a joystick in gaming applications.

Interface 14 interfaces mechanical and electrical input and output between the mouse 12 and host computer 18 implementing the application program, such as a GUI, simulation or game environment. Interface 14 provides multiple degrees of freedom to mouse 12; in the preferred embodiment, two linear, planar degrees of freedom are provided to the mouse, as shown by arrows 22. In other embodiments, greater or fewer degrees of freedom can be provided, as well as rotary degrees of freedom. For many applications, mouse 12 need only be moved in a very small workspace area.

In a preferred embodiment, the user manipulates mouse 12 in a planar workspace, much like a traditional mouse, and the position of mouse 12 is translated into a form suitable for interpretation by position sensors of the interface 14. The sensors track the movement of the mouse 12 in planar space and provide suitable electronic signals to an electronic portion of interface 14. The interface 14 provides position information to host computer 18. In addition, host computer 18 and/or interface 14 provide force feedback signals to actuators coupled to interface 14, and the actuators generate forces on members of the mechanical portion of the interface 14 to provide forces on mouse 12 in provided or desired degrees of freedom. The user experiences the forces generated on the mouse 12 as realistic simulations of force sensations such as jolts, springs, textures, enclosures, circles, ellipses, grids, vibrations, "barrier" forces, and the like.

The electronic portion 26 of interface 14 may couple the mechanical portion 24 of the interface to the host computer 18. The electronic portion 26 is preferably included within the housing 21 of the interface 14 or, alternatively, the electronic portion may be included in host computer 18 or as a separate unit with its own housing. More particularly, interface 14 includes a local microprocessor distinct and separate from any microprocessors in the host computer 18 to control force feedback on mouse 12 independently of the host computer, as well as sensor and actuator interfaces that convert electrical signals to appropriate forms usable by the mechanical portion of interface 14 and host computer 18.

For example, a rigid surface is generated on computer screen 20 and a computer object (e.g., cursor) controlled by the user collides with the surface. In a preferred embodiment, high-level host commands can be used to provide the various forces associated with the rigid surface. The local control mode using a local microprocessor in interface 14 can be helpful in increasing the response time for forces applied to the user object, which is essential in creating realistic and accurate force feedback. For example, it is preferable that host computer 18 send a "spatial representation" to the local microprocessor, which is data describing the locations of some or all the graphical objects displayed in a GUI or other graphical environment which are associated with forces and the types/characteristics of these graphical objects. The microprocessor can store such a spatial representation in local memory, and thus will be able to determine interactions between the user object and graphical objects (such as the rigid surface) independently of the host computer. In addition, the microprocessor can be provided with the necessary instructions or data to check sensor readings, determine cursor and target positions, and determine output forces independently of host computer 18. The host could implement program functions (such as displaying images) when appropriate, and synchronization commands can be communicated between the microprocessor and host 18 to correlate the microprocessor and host processes. Such commands and related functionality is discussed in greater detail below. Alternatively, the computer 18 can directly send force feedback signals to the interface 14 to generate forces on mouse 12. A suitable embodiment of the electrical portion of interface 14 is described in detail with reference to FIG. 3.

The interface 14 can be coupled to the computer 18 by a bus 17, which communicates signals between interface 14 and computer 18 and also, in the preferred embodiment, provides power to the interface 14 (e.g. when bus 17 includes a USB interface). In other embodiments, signals can be sent between interface 14 and computer 18 by wireless transmission/reception. In preferred embodiments of the present invention, the interface 14 serves as an input/output (I/O) device for the computer 18. The interface 14 can also receive inputs from other input devices or controls that are associated with mouse system 10 and can relay those inputs to computer 18. For example, commands sent by the user activating a button on mouse 12 can be relayed to computer 18 by interface 14 to implement a command or cause the computer 18 to output a command to the interface 14.

Host computer 18 is preferably a personal computer or workstation, such as an IBM-PC compatible computer or Macintosh personal computer, or a SUN or Silicon Graphics workstation. For example, the computer 18 can operate under the Windows.TM. or MS-DOS operating system in conformance with an IBM PC AT standard. Alternatively, host computer system 18 can be one of a variety of home video game systems commonly connected to a television set, such as systems available from Nintendo, Sega, or Sony. In other embodiments, host computer system 18 can be a "set top box" which can be used, for example, to provide interactive television functions to users, or a "network-" or "internet-computer" which allows users to interact with a local or global network using standard connections and protocols such as used for the Internet and World Wide Web. Host computer preferably includes a host microprocessor, random access memory (RAM), read only memory (ROM), input/output (I/O) circuitry, and other components of computers well-known to those skilled in the art.

Host computer 18 preferably implements one or more application programs ("applications") with which a user is interacting via mouse 12 and other peripherals, if appropriate, and which can include force feedback functionality. For example, the host application programs can include a simulation, video game, Web page or browser that implements HTML or VRML instructions, word processor, drawing program, spreadsheet, scientific analysis program, or other application program that utilizes input of mouse 12 and outputs force feedback commands to the mouse 12. Typically, an operating systems such as Windows.TM., MS-DOS, MacOS, Unix, is also running on the host computer and preferably includes its own force feedback functionality. In one preferred embodiment, the operating system and application programs utilize a graphical user interface (GUI) to present options to a user, display data and images, and receive input from the user. In the preferred embodiment, multiple applications can run simultaneously in a multitasking environment of the host computer, as is detailed below. Herein, computer 18 may be referred as displaying "graphical objects" or "computer objects." These objects are not physical objects, but are logical software unit collections of data and/or procedures that may be displayed as images by computer 18 on display screen 20, as is well known to those skilled in the art. A displayed cursor or a simulated cockpit of an aircraft might be considered a graphical object. The host application program checks for input signals received from the electronics and sensors of interface 14, and outputs force values and/or commands to be converted into forces on mouse 12. Suitable software drivers which interface such simulation software with computer input/output (I/O) devices are available from Immersion Human Interface Corporation of San Jose, Calif.

Display device 20 can be included in host computer 18 and can be a standard display screen (LCD, CRT, etc.), 3-D goggles, or any other visual output device. Typically, the host application provides images to be displayed on display device 20
and/or other feedback, such as auditory signals. For example, display screen 20 can display images from a GUI. Images describing a moving, first person point of view can be displayed, as in a virtual reality game. Or, images describing a third-person perspective of objects, backgrounds, etc. can be displayed. Alternatively, images from a simulation, such as a medical simulation, can be displayed, e.g., images of tissue and a representation of a manipulated user object 12 moving through the tissue, etc.

There are two primary "control paradigms" of operation for mouse system 10: position control and rate control. Position control is the more typical control paradigm for mouse and similar controllers, and refers to a mapping of mouse 12 in which displacement of the mouse in physical space directly dictates displacement of a graphical object. Under a position control mapping, the computer object does not move unless the user object is in motion. Also, "ballistics" or other non-linear adjustments to cursor position can be used in which, for example, slow motions of the mouse have a different scaling factor for cursor movement than fast motions of the mouse, to allow more control of short cursor movements. Several different ways of implementing ballistics and other control adjustments in a force feedback device are described in U.S. Pat. No. 6,252,579, and these adjustments can be used in mouse system 10 if desired.

As shown in FIG. 1, the host computer may have its own "screen frame" 28 (or host frame) which is displayed on the display screen 20. In contrast, the mouse 12 has its own "device frame" (or local frame) 30 in which the mouse 12 is moved. In a position control paradigm, the position (or change in position) of a user-controlled graphical object, such as a cursor, in host frame 30 corresponds to a position (or change in position) of the mouse 12 in the local frame 28.

Rate control is also used as a control paradigm. This refers to a mapping in which the displacement of the mouse 12 is abstractly mapped to motion of a computer object under control. There is not a direct physical mapping between physical object (mouse) motion and computer object motion. Thus, most rate control paradigms are fundamentally different from position control in that the user object can be held steady at a given position but the controlled computer object is in motion at a commanded or given velocity, while the position control paradigm only allows the controlled computer object to be in motion if the user object is in motion.

The mouse interface system 10 is useful for both position control ("isotonic") tasks and rate control ("isometric") tasks. For example, as a traditional mouse, the position of mouse 12 in its local frame 30 workspace can be directly mapped to a position of a cursor in host frame 28 on display screen 20 in a position control paradigm. Alternatively, the displacement of mouse 12 in a particular direction against an opposing output force can command rate control tasks in an isometric mode. An implementation that provides both isotonic and isometric functionality for a force feedback controller and which is very suitable for the interface device of the present invention is described in U.S. Pat. No. 5,825,308, incorporated by reference herein.

Mouse 12 is preferably supported and suspended above a grounded pad 32 by the mechanical portion of interface 14, described below. Pad 32 or similar surface is supported by grounded surface 34. Pad 32 (or alternatively grounded surface 34) provides additional support for the mouse and relieve stress on the mechanical portion of interface 14. In addition, a wheel, roller, Teflon pad or other device can be used to support the mouse.

Mouse 12 can be used, for example, to control a computer-generated graphical object such as a cursor displayed in a graphical computer environment, such as a GUI. The user can move the mouse in 2D planar workspace to move the cursor to graphical objects in the GUI or perform other tasks. In other graphical environments, such as a virtual reality video game, a user can be controlling a computer player or vehicle in the virtual environment by manipulating the mouse 12. The computer system tracks the position of the mouse with sensors as the user moves it. The computer system may also provide force feedback commands to the mouse, for example, when the user moves the graphical object against a generated surface such as an edge of a window, a virtual wall, etc. It thus appears and feels to the user that the mouse and the graphical object are contacting real surfaces.

The mouse system 10 may also include an indexing function or "indexing mode" which allows the user to redefine the offset between the positions of the mouse 12 in the local frame and a user-controlled graphical object, such as a cursor, in the host frame displayed by host computer 18. In one implementation, the user may reposition the mouse 12 without providing any input to the host computer, thus allowing the user to redefine the offset between the objectis position and the cursoris position. Such indexing is achieved through an input device such as a button, switches, sensors, or other input devices. As long as the indexing device is activated, the mouse 12 is in indexing mode; when the button is released (or indexing mode otherwise exited), the position of the cursor is again controlled by the mouse 12. A hand weight switch can also be provided which inherently causes indexing when the user removes hand or finger weight from mouse 12. In one embodiment, the functionality of a safety switch and the indexing mode are integrated into one input device, as described in greater detail in U.S. Pat. Nos. 5,825,308 and 6,100,874.

A different way to allow indexing is to provide a combined position control and rate control device which allows different forms of control of the cursor depending on the position of the mouse in its workspace. This embodiment is described in greater detail below and in U.S. Pat. No. 6,252,579, which is hereby incorporated by reference herein.

FIG. 2 is a perspective view of a preferred embodiment of the mouse device 11 with the cover portion of housing 21 and the grounded pad 32 removed. Mechanical linkage 40 provides support for mouse 12 and couples the mouse to a grounded surface
34, such as a tabletop or other support. Linkage 40 is, in the described embodiment, a 5-member (or "5-bar") linkage. Fewer or greater numbers of members in the linkage can be provided in alternate embodiments.

Ground member 42 of the linkage 40 is a base for the support of the linkage and is coupled to or resting on a ground surface 34. The ground member 42 in FIG. 2 is shown as a plate or base that extends under mouse 12. The members of linkage 40
are rotatably coupled to one another through the use of rotatable pivots or bearing assemblies, all referred to as "bearings" herein. Base member 44 is rotatably coupled to ground member 42 by a grounded bearing 52 and can rotate about an axis A. Link member 46 is rotatably coupled to base member 44 by bearing 54 and can rotate about a floating axis B, and base member 48 is rotatably coupled to ground member 42 by bearing 52 and can rotate about axis A. Link member 50 is rotatably coupled to base member 48 by bearing 56 and can rotate about floating axis C, and link member 50 is also rotatably coupled to link member 46 by bearing 58 such that link member 50 and link member 46 may rotate relative to each other about floating axis D.

Linkage 40 is formed as a five-member closed-loop chain. Each member in the chain is rotatably coupled to two other members of the chain to provide mouse 12 with two degrees of freedom, i.e., mouse 12 can be moved within a planar x-y workspace. Mouse 12 is coupled to link members 46 and 50 by rotary bearing 58, and may rotate at least partially about axis D. A pad or other support can be provided under mouse 12 to help support the mouse.

Transducer system 41 is used to sense the position of mouse 12 in its workspace and to generate forces on the mouse 12. Transducer system 41 preferably includes sensors 62 and actuators 64. The sensors 62 collectively sense the movement of the mouse 12 in the provided degrees of freedom and send appropriate signals to the electronic portion of interface 14. Sensor 62a senses movement of link member 48 about axis A, and sensor 62b senses movement of base member 44 about axis A. These sensed positions about axis A allow the determination of the position of mouse 12 using known constants such as the lengths of the members of linkage 40 and using well-known coordinate transformations.

Sensors 62 are, in the described embodiment, grounded optical encoders that sense the intermittent blockage of an emitted beam. A grounded emitter/detector portion 71 includes an emitter that emits a beam which is detected by a grounded detector. A moving encoder disk portion or iarci 74 is provided at the end of members 44 and 48 which each block the beam for the respective sensor in predetermined spatial increments and allows a processor to determine the position of the arc 74 and thus the members 44 and 48 by counting the spatial increments. Also, a velocity of members 44 and 48 based on the speed of passing encoder marks can also be determined. In one embodiment, dedicated electronics such as a "haptic accelerator" may determine velocity and/or acceleration, as disclosed in U.S. Pat. No. 5,999,168, and hereby incorporated by reference herein. Sensors 62 are described in greater detail in U.S. Pat. Nos. 6,100,874 and 6,166,723, both incorporated by reference herein.

Transducer system 41 also preferably includes actuators 64 to transmit forces to mouse 12 in space, i.e., in two (or more) degrees of freedom of the user object. The bottom housing plate 65 of actuator 64a is rigidly coupled to ground member 42
(or grounded surface 34) and a moving portion of actuator 64a (preferably a coil) is integrated into the base member 44. The actuator 64a transmits rotational forces to base member 44 about axis A. The housing 65 of the grounded portion of actuator 64b is rigidly coupled to ground member 42 or ground surface 34 through the grounded housing of actuator 64b, and a moving portion (preferably a coil) of actuator 64b is integrated into base member 48. Actuator 64b transmits rotational forces to link member
48 about axis A. The combination of these rotational forces about axis A allows forces to be transmitted to mouse 12 in all directions in the planar workspace provided by linkage 40 through the rotational interaction of the members of linkage 40. The integration of the coils into the base members 44 and 48 is advantageous to the present invention and is discussed below. The operation of the electromagnetic actuators 64 is described in greater detail is U.S. Pat. Nos. 6,100,874 and 6,166,723. In other embodiments, other types of actuators can be used.

FIG. 3 is a block diagram illustrating the electronic portion of interface 14 and host computer 18 suitable for use with the present invention. Mouse interface system 10 includes a host computer 18, electronic interface 26, mechanical portion
24, and mouse or other user object 12. Electronic interface 26, mechanical portion 24, and mouse 12 can also collectively be considered the "force feedback interface device" 11 that is coupled to the host computer. A similar system is described in detail in U.S. Pat. No. 5,734,373, which is hereby incorporated by reference herein.

As explained with reference to FIG. 1, computer 18 is preferably a personal computer, workstation, video game console, or other computing or display device. Host computer system 18 commonly includes a host microprocessor 108, random access memory (RAM) 110, read-only memory (ROM) 112, input/output (I/O) electronics 114, a clock 116, a display device 20, and an audio output device 118. Host microprocessor 108 can include a variety of available microprocessors from Intel, AMD, Motorola, or other manufacturers. Microprocessor 108 can be single microprocessor chip, or can include multiple primary and/or co-processors. Microprocessor 108 preferably retrieves and stores instructions and other necessary data from RAM 110 and ROM 112 as is well known to those skilled in the art. In the described embodiment, host computer system 18 can receive sensor data or a sensor signal via a bus 120 from sensors of system 10 and other information. Microprocessor 108 can receive data from bus 120
using I/O electronics 114, and can use I/O electronics to control other peripheral devices. Host computer system 18 can also output commands to interface device 104 via bus 120 to cause force feedback for the interface system 10.

Clock 116 is a standard clock crystal or equivalent component used by host computer 18 to provide timing to electrical signals used by host microprocessor 108 and other components of the computer system 18. Display device 20 is described with reference to FIG. 1. Audio output device 118, such as speakers, can be coupled to host microprocessor 108 via amplifiers, filters, and other circuitry well known to those skilled in the art. Host processor 108 outputs signals to speakers 118 to provide sound output to the user when an "audio event" occurs during the implementation of the host application program. Other types of peripherals can also be coupled to host processor 108, such as storage devices (hard disk drive, CD ROM drive, floppy disk drive, etc.), printers, and other input and output devices.

Electronic interface 26 is coupled to host computer system 18 by a bi-directional bus 120. The bi-directional bus sends signals in either direction between host computer system 18 and the interface device 104. Bus 120 can be a serial interface bus providing data according to a serial communication protocol, a parallel bus using a parallel protocol, or other types of buses. An interface port of host computer system 18, such as an RS232 serial interface port, connects bus 120 to host computer system 18. In another embodiment, an additional bus can be included to communicate between host computer system 18 and interface device 11. One preferred serial interface bus used in the present invention is the Universal Serial Bus (USB). The USB standard provides a relatively high speed serial interface that can provide force feedback signals in the present invention with a high degree of realism. USB can also source power to drive actuators 64 and other devices of the present invention. In addition, the USB standard includes timing data that is encoded along with differential data. Alternatively, Firewire (also called 1392) can be used as bus 120; or, the bus can be between an interface card in the host computer 18, where the interface card holds components of device 11 such as microprocessor 130.

Electronic interface 26 includes a local microprocessor 130, local clock 132, local memory 134, sensor interface 136, and actuator interface 138. Interface 26 may also include additional electronic components for communicating via standard protocols on bus 120. In various embodiments, electronic interface 26 can be included in mechanical portion 24, in host computer 18, or in its own separate housing. Different components of interface 26 can be included in portion 24 or host computer 18
if desired.

Local microprocessor 130 preferably coupled to bus 120 and may be closely linked to mechanical portion 24 to allow quick communication with other components of the interface device. Processor 130 is considered "local" to interface device 11, where "local" herein refers to processor 130 being a separate microprocessor from any processors 108 in host computer 18. "Local" also preferably refers to processor 130 being dedicated to force feedback and sensor I/O of the interface system 10, and being closely coupled to sensors and actuators of the mechanical portion 24, such as within the housing of or in a housing coupled closely to portion 24. Microprocessor 130 can be provided with software instructions to wait for commands or requests from computer host 18, parse/decode the command or request, and handle/control input and output signals according to the command or request. In addition, processor 130 preferably operates independently of host computer 18 by reading sensor signals and calculating appropriate forces from those sensor signals, time signals, and force processes selected in accordance with a host command, and output appropriate control signals to the actuators. A suitable microprocessor for use as local microprocessor
130 includes the 8X930AX by Intel; or alternatively the MC68HC711E9 by Motorola or the PIC16C74 by Microchip, for example. Microprocessor 130 can include one microprocessor chip, or multiple processors and/or co-processor chips. In other embodiments, microprocessor 130 can include digital signal processor (DSP) functionality.

For example, in one host-controlled embodiment that utilizes microprocessor 130, host computer 18 can provide low-level force commands over bus 120, which microprocessor 130 directly transmits to the actuators. In a different local control embodiment, host computer system 18 provides high level supervisory commands to microprocessor 130 over bus 120, and microprocessor 130 manages low level force control loops to sensors and actuators in accordance with the high level commands and independently of the host computer 18. In the local control embodiment, the microprocessor 130 can process inputted sensor signals to determine appropriate output actuator signals by following the instructions of a "force process" that may be stored in local memory and includes calculation instructions, formulas, force magnitudes, or other data. The force process can command distinct force sensations, such as vibrations, textures, jolts, or even simulated interactions between displayed objects. An "enclosure" host command can also be provided, which causes the microprocessor to define a box-like enclosure in a graphical environment, where the enclosure has sides characterized by wall and texture forces, as described in U.S. Pat. No. 6,100,874. The host can send the local processor a spatial layout of objects in the graphical environment so that the microprocessor has a mapping of locations of graphical objects like enclosures and can determine interactions with the cursor locally. Force feedback used in graphical environments is described in greater detail in U.S. Pat. Nos. 6,219,032, 5,825,308, and 6,252,579, all of which are incorporated by reference herein.

Sensor signals used by microprocessor 130 are also reported to host computer system 18, which updates a host application program and outputs force control signals as appropriate. For example, if the user moves mouse 12, the computer system 18
receives position and/or other signals indicating this movement and can move a displayed cursor in response. These embodiments are described in greater detail in U.S. Pat. No. 5,734,373. In an alternate embodiment, no local microprocessor 130 is included in interface system 10, and host computer 18 directly controls and processes all signals to and from the interface 26 and mechanical portion 24.

A local clock 132 can be coupled to the microprocessor 130 to provide timing data, similar to system clock 116 of host computer 18; the timing data might be required, for example, to compute forces output by actuators 64 (e.g., forces dependent on calculated velocities or other time dependent factors). In alternate embodiments using the USB communication interface, timing data for microprocessor 130 can be retrieved from the USB interface. Local memory 134, such as RAM and/or ROM, is preferably coupled to microprocessor 130 in interface 26 to store instructions for microprocessor 130 and store temporary and other data. Microprocessor 130 may also store calibration parameters in a local memory 134 such as an EEPROM. As described above, link or member lengths or manufacturing variations and/or variations in coil winding or magnet strength can be stored. Memory 134 may be used to store the state of the force feedback device, including a reference position, current control mode or configuration, etc.

Sensor interface 136 may optionally be included in electronic interface 26 to convert sensor signals to signals that can be interpreted by the microprocessor 130 and/or host computer system 18. For example, sensor interface 136 can receive signals from a digital sensor such as an encoder and convert the signals into a digital binary number representing the position of a member of mechanical apparatus 14. An analog to digital converter (ADC) in sensor interface 136 can convert a received analog signal to a digital signal for microprocessor 130 and/or host computer 18. Such circuits, or equivalent circuits, are well known to those skilled in the art. Alternately, microprocessor 130 or host 18 can perform these interface functions. Other types of interface circuitry 136 can also be used, e.g., the electronic interface described in U.S. Pat. No. 5,576,727, which is hereby incorporated by reference herein.

Actuator interface 138 can be optionally connected between the actuators 64 and microprocessor 130. Interface 138 converts signals from microprocessor 130 into signals appropriate to drive the actuators. Interface 138 can include power amplifiers, switches, digital to analog controllers (DACs), and other components. Such interfaces are well known to those skilled in the art. In alternate embodiments, interface 138 circuitry can be provided within microprocessor 130 or in the actuators.

In the described embodiment, power is supplied to the actuators 64 and any other components (as required) by the USB. Since the electromagnetic actuators of the described embodiment have a limited physical range and need only output, for example, about 3 ounces of force to create realistic force sensations on the user, very little power is needed. For example, one way to draw additional power from the USB is to configure device 11 to appear as more than one peripheral to host computer
18; for example, each provided degree of freedom of mouse 12 can be configured as a different peripheral and receive its own allocation of power. Alternatively, power from the USB can be stored and regulated by device 11 and thus used when needed to drive actuators 64. For example, power can be stored over time and then immediately dissipated to provide a jolt force to the user object 12. A battery or a capacitor circuit, for example, can store energy and discharge or dissipate the energy when power is required by the system and/or when enough power has been stored. Alternatively, a power supply 140 can optionally be coupled to actuator interface 138 and/or actuators 64 to provide electrical power. The power storage embodiment described above, using a battery or capacitor circuit, can also be used in non-USB embodiments to allow a smaller power supply 140 to be used.

Mechanical portion 24 is coupled to electronic portion 26 and preferably includes sensors 62, actuators 64, and linkage 40. These components are described in detail above. Sensors 62 sense the position, motion, and/or other characteristics of mouse 12 along one or more degrees of freedom and provide signals to microprocessor 130 including information representative of those characteristics. Typically, a sensor 62 is provided for each degree of freedom along which mouse 12 can be moved, or, a single compound sensor can be used for multiple degrees of freedom. Example of sensors suitable for embodiments described herein are rotary optical encoders, as described above, linear optical encoders, analog sensors such as potentiometers, or non-contact sensors