REGISTER  |  SIGN IN
United States Patent6125385
Wies , ; et al.September 26, 2000
Title
Force feedback implementation in web pages
Abstract
Force feedback is provided to a user of a client computer receiving information such as a web page over a network such as the World Wide Web from a server machine. The client machine has a force feedback interface device through which the user experiences physical force feedback. Force feedback is correlated to web page objects by a force feedback program running on the client and based on input information from the interface device, web page objects, and the force feedback information. Generic force effects can be provided, which are applied uniformly at the client machine to all web page objects of a particular type as defined by user preferences at the client machine.
Inventors:Wies; Evan F. (Mountain View, CA), Chang; Dean C.  (Santa Clara, CA), Rosenberg; Louis B.  (San Jose, CA), Tan; Sian W.  (Mountain View, CA), Mallett; Jeffrey R.  (Boulder Creek, CA)
Assignee:Immersion Corporation (San Jose, CA)
Appl. No.:401563
Filed:September 22, 1999
Current U.S. Class:709/203 709/217 345/163 
Field of Search:703/203,217 345/145,156,161,160,163
U.S. Patent Documents
3919691November 1975Noll
4148014April 1979Burson
4216467August 1980Colston
4477043October 1984Repperger
4550221October 1985Mabusth
4550617November 1985Fraignier et al.
4769763September 1988Trieb et al.
4787051November 1988Olson
4800721January 1989Cemenska et al.
4811608March 1989Hilton
4823634April 1989Culver
4839838June 1989LaBiche et al.
4849692July 1989Blood
4868549September 1989Affinito et al.
4879556November 1989Duimel
4907970March 1990Meenen, Jr.
4907973March 1990Hon
4961138October 1990Gorniak
5007085April 1991Greanias et al.
5007300April 1991Siva
5044956September 1991Behensky et al.
5072361December 1991Davis et al.
5095303March 1992Clark et al.
5103404April 1992McIntosh
5116051May 1992Moncrief et al.
5139261August 1992Openiano
5142931September 1992Menahem
5143505September 1992Burdea et al.
5146566September 1992Hollis, Jr. et al.
5181181January 1993Glynn
5182557January 1993Lang
5184319February 1993Kramer
5185561February 1993Good et al.
5186629February 1993Rohen
5220260June 1993Schuler
5230623July 1993Guthrie et al.
5264768November 1993Gregory et al.
5275565January 1994Moncrief
5296846March 1994Ledley
5298890March 1994Kanamaru et al.
5341459August 1994Backes
5354162October 1994Burdea et al.
5386507January 1995Teig et al.
5389865February 1995Jacobus et al.
5396266March 1995Brimhall
5397323March 1995Taylor et al.
5402582April 1995Raab
5405152April 1995Katanics et al.
5414337May 1995Schuler
5428748June 1995Davidson et al.
5429140July 1995Burdea et al.
5436638July 1995Bolas et al.
5436640July 1995Reeves
5451924September 1995Massimino et al.
5459382October 1995Jacobus et al.
5471571November 1995Smith et al.
5513100April 1996Parker et al.
5526480June 1996Gibson
5565887October 1996McCambridge et al.
5565888October 1996Selker
5570111October 1996Barrett et al.
5576727November 1996Rosenberg et al.
5586257December 1996Perlman
5589854December 1996Tsai
5591924January 1997Hilton
5623582April 1997Rosenberg
5623642April 1997Katz et al.
5625576April 1997Massie et al.
5629594May 1997Jacobus et al.
5631861May 1997Kramer
5666138September 1997Culver
5666473September 1997Wallace
5691898November 1997Rosenberg et al.
5694013December 1997Stewart et al.
5709219January 1998Chen et al.
5714978February 1998Yamanaka et al.
5721566February 1998Rosenberg et al.
5734373March 1998Rosenberg et al.
5736978April 1998Hasser et al.
5742278April 1998Chen et al.
5745715April 1998Pickover et al.
5755577May 1998Gillio
5757358May 1998Osga
5760764June 1998Martinelli
5767839June 1998Rosenberg
5769640June 1998Jacobus et al.
5771037June 1998Jackson
5781172July 1998Engel et al.
5786818July 1998Brewer et al.
5790108August 1998Salcudean et al.
5791992August 1998Crump et al.
5795228August 1998Trumbull et al.
5802353September 1998Avila et al.
5805140September 1998Rosenberg et al.
5805165September 1998Thorne, III et al.
5825308October 1998Rosenberg
5844392December 1998Peurach et al.
5884029March 1999Brush, II et al.
5889506March 1999Lopresti et al.
5889670March 1999Schuler et al.
5956484September 1999Rosenberg et al.
5959613September 1999Rosenberg et al.
5990869November 1999Kubica et al.
6005551December 1999Osborne et al.
6020875February 2000Moore et al.
6020876February 2000Rosenberg et al.
6028593February 2000Rosenberg et al.
Foreign Patent Documents
0 626 634 A2Nov., 1994EP
WO 97/31333Aug., 1997WO
WO9502233Jan., 1995WO
WO9502801Jan., 1995WO
WO9510080Apr., 1995WO
WO9520787Aug., 1995WO
WO9520788Aug., 1995WO
WO9532459Nov., 1995WO
WO9622591Jul., 1996WO
WO97/12337Apr., 1997WO
WO9719440May., 1997WO
WO9721160Jun., 1997WO
Other References
Kelly, A.J. et al., "MagicMouse: Tactile and Kinesthetic Feedback in the Human-Computer Interface Using an Electromagnetically Actuated Input/Output Device," Dept. of Elec. Engineering, Univ. of British Columbia, 1993, pp. 1-27. .
Munch, Stefan, et al., "Intelligent Control for Haptic Displays," Institute for Real-Time Computer Systems & Robotics, University of Karlsrube Kaiserstr. 12, Germany, Eurographics '96, vol. 15, No. 3, pp. C-217-226. .
Ramstein, Christophe, et al., "The Pantograph: A Large Workspace Haptic Device for a Multimodal Human-Computer Interaction," Computer-Human Interaction, CHI '94. .
Ramstein, Christophe, "Combining Haptic and Braille Technologies: Design Issues and Pilot Study," Assets '96, 2nd Annual ACM Conf. On Assistive Technologies, Apr. 11-12, 1996, pp. 37-44. .
Rosenberg, Louis B., et al., "The Use of Force Feedback to Enhance Graphical User Interfaces," Stereoscopic Displays & Virtual Reality Systems III, Proc. SPIE 2653, 1996, pp. 243-248. .
Rosenberg, Louis, "A Force Feedback Programming Primer For PC Gaming Peripherals Supporting I-Force 2.0 and Direct-X 5.0," .COPYRGT.1997 Immersion Corporation. .
Payette, Julie, et al., "Evaluation of a Force Feedback (Haptic) Computer Pointing Device in Zero Gravity," DSC-vol. 58, Proceedings of the ASME Dynamics Systems and Control Division ASME 1996, pp. 547-553. .
Iwata, Hiroo, "Artificial Reality with Force-Feedback: Development of Desktop Virtual Space with Compact Master Manipulator," Computer Graphics, vol. 24, No. 4, Aug. 1990, pp. 165-170. .
Brooks, Frederick P., Jr., et al., "Project GROPE--Haptic Displays for Scientific Visualization," Computer Graphics, vol. 24, No. 4, Aug. 1990, pp. 177-185. .
Hannaford, B., et al., "Force-Feedback Cursor Control," JPL Invention Report NPO-17520/7034, NASA Tech Brief, vol. 13, No. 11, Item #21, Nov. 1989. .
Hirota, Koichi, et al., "Development of Surface Display," Department of Mechano-Informatics, Faculty of Engineering, University of Tokyo, IEEE 0-7803-1363-1, 1993, pp. 256-262. .
Akamatsu, Motoyuki, et al., "Multimodal Mouse: A Mouse-Type Device with Tactile and Force Display," Presence, vol. 3, No. 4, Winter 1994, pp. 73-80. .
Kelley, A.J., et al., "On the Development of a Force-Feedback Mouse and Its Integration Into a Graphical User Interface," Symp. on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 1994 Int'l Mechanical Engineering Congress and Exhibition, Chicago, IL, Nov. 1994. .
Perrochon, Louis, et al., "WAB: World Wide Web Access for Blind and Visually Impaired Computer Users," Institut fur Informationsysteme, ETH Zentrum, http://www.inf.ethz.ch/department/IS/ea/blinds/. .
Rosenberg, Louis B., "Virtual Fixtures as Tools to Enhance Operator Performance in Telepresence Environments," SPIE Telemanipulator Technology, 1993. .
Rosenberg, Louis B., "Virtual Haptic Overlays Enhance Performance in Telepresence Tasks," SPIE 1994. .
Rosenberg, Louis B., "The Use of Virtual Fixtures as Perceptual Overlays to Enhance Operator Performance in Remote Environments," Air Force Material Command, Sep. 1992, pp. 1-42. .
Rosenberg, Louis B., "Perceptual Design of a Virtual Rigid Surface Contact," Center for Design Research Stanford University, Air Force Material Command, Apr. 1993, pp. 1-41. .
Rosenberg, Louis B., The Use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation, Armstrong Laboratory, Mar. 1993, pp. 1-45. .
Rosenberg, Louis B. et al., "Perceptual Decomposition of Virtual Haptic Surfaces," IEEE, Oct. 1993. .
Colgate, J. Edward et al., "Implementation of Stiff Virtual Walls in Force-Reflecting Interfaces," 1993, pp. 1-9. .
Buttolo, Pietro et al., "Pen-Based Force Display for Precision Manipulation in Virtual Environments," IEEE Mar. 1995, pp. 1-8. .
Minsky, Margaret et al., "Feeling and Seeing: Issues in Force Display," ACM 1990, pp. 235-242. .
Su, S. Augustine et al., "The Virtual Panel Architecture: A 3D Gesture Framework," IEEE 1993, pp. 387-393. .
Adachi, Yoshitaka et al., "Sensory Evaluation of Virtual Haptic Push-Buttons," Technical Research Center, Suzuki Motor Corporation, Nov. 1994. .
Burdea, Grigore et al., "Distributed Virtual Force Feedback," IEEE, May 2, 1993, pp. 25-44. .
Ellis, R.E. et al., "Design and Evaluation of a High-Performance Prototype Planar Haptic Interface," ASME Dec. 3, 1993, DSC-vol. 49, pp. 55-64. .
Kotoku, Tetsuo et al., "Environment Modeling for the Interactive Display (EMID) Used in Telerobotic Systems," IEEE Nov. 3-5, 1991, pp. 99-1004. .
Adelstein Bernard D. et al., "A High Performance Two Degree-of-Freedom Kinesthetic Interface," Massachusetts Institute of Technology 1992, pp. 108-112. .
Jones, L.A., et al., "A Perceptual Analysis of Stiffness," Experimental Brain Research 1990, pp. 151-156. .
Bejczy, Antal K., "The Phantom Robot: Predictive Displays for Teleoperation with Time Delay," IEEE 1990, pp. 546-550. .
Iwata, Hiroo, "Pen-based Haptic Virtual Environment," Institute of Engineering Mechanics, University of Tsukuba, Japan, pp. 287-292. 1993. .
Adlestein, Bernard D. et al., "Design and Implementation of a Force Reflecting Manipulandum for Manual Control Research," 1992, pp. 1-24. .
Ouh-young, Ming et al., "Force Display Performs Better than Visual Display in a Simple 6-D Docking Task," IEEE 1989, pp. 1462-1466. .
Kim, Won S. et al., "Graphics Displays for Operator Aid in Telemanipulation," IEEE 1991, pp. 1059-1067. .
Hannaford, Blake et al., "Performance Evaluation of a Six-Axis Generalized Force-Reflecting Teleoperator," IEEE May/Jun. 1991, vol. 21, No. 3, pp. 620-633.. .
Kim, Won S. et al., A Teleoperation Training Simulator with Visual and Kinesthetic Force Virtual Reality. No date. .
Iwata, Hiroo et al, Volume Haptization, IEEE 1993, pp. 16-18. .
Batter, James J. et al., "Grope-1: A Computer Display to the Sense of Feel," pp. TA-4-188-TA-4-192. 1971. .
Yamakita, M. et al., Tele-Virtual Reality of Dynamic Mechanical Model, IEEE Jul. 7-10, 1992, pp. 1103-1110. .
Fischer, Patrick et al., "Specification and Design of Input Devices for Teleoperation," 1990. .
Fisher, S.S. et al., "Virtual Environment Display System," ACM Interactive 3D Graphics, Oct. 1986. .
Wiker, Steven F. et al., "Development of Tactile Mice for Blind Access to Computers: Importance of Stimulation Locus, Object Size, and Vibrotactile Display Resolution," Proceedings of the Human Factors Society 35th Annual Meeting 1991, pp. 708-712. .
"The Personal Digitizer.TM.," Immersion Human Interface Corporation 1994. .
Meyer, Kenneth et al., "A Survey of Position Trackers," The Massachusetts Institute of Technology 1992, Presence, vol. 1, No. 2. .
Smith, Geoffrey, "Call It Palpable Progress," Business Week, Oct. 9, 1995, p. 93, 96. .
Krueger, Myron W., Artificial Reality 1988, pp. 54-75. .
Gotow, J.K., et al., "Perception of Mechanical Properties at the Man-Machine Interface," IEEE 1987, pp. 688-689. .
"Proceedings of the IFIP Congress 65," International Federation for Information Processing, Information Processing 1965, vol. 3, New York, May 24-29, 1965, pp. 506. .
Atkinston, William D. et al, "Computing with Feeling," Comput. & Graphics, vol. 2, No. 2-E, pp. 97-103. 1976. .
Noll, A. Michael, "Man-Machine Tactile Communication Dissertation," Polytechnic Institute of Brooklyn, Jun. 1971, pp. 1-88. .
Ouh-Young, Ming, "Force Display in Molecular Docking," Chapel Hill 1990, pp. 1-85. .
Ouh-young, Ming et al., "Using a Manipulator for Force Display in Molecular Docking," IEEE 1988, pp. 1824-1829. .
Tan, Hong Z et al., "Manual Resolution of Compliance When Work and Force Cues are Minimized," ASME 1993, DSC-vol. 49, pp. 99-104. .
Kotoku, Tetsuo, "A Predictive Display with Force Feedback and its Application to Remote Manipulation System with Transmission Time Display," IEEE 1992, Jul. 7-10, 1992, pp. 239-246. .
Howe, Robert D., "Task Performance with a Dextrous Teleoperated Hand System," Proceedings of SPIE, Nov. 1992, vol. 1833, pp. 1-9.
Primary Examiner: Geckil; Mehmet B.
Attorney, Agent or Firm:Riegel; James R.
Government Interests:

This invention was made with Government support under Contract Number
F41624-96-C-6029, awarded by the Department of Defense. The Government has
certain rights in this invention.
Parent Case Text


CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of copending prior U.S. application Ser. No. 09/244,622, filed on Feb. 2, 1999, which claims priority to 1) U.S. application Ser. No. 08/691,852, now U.S. Pat. No. 5,956,484, filed Aug. 1, 1996, which is a continuation-in-part of U.S. patent application Ser. No. 08/571,606, filed Dec. 13, 1995; 2) copending U.S. application Ser. No. 08/970,953, filed Nov. 14, 1997; 3) U.S. Provisional Application No. 60/073,518, filed Feb. 3, 1998; the disclosures of which are all incorporated herein by reference.
Claims

What is claimed is:
1. A method for providing force feedback over a network comprising:
establishing a connection between a server machine and a client machine over a network, said client machine including a visual display and an interface device providing computer-controlled physical force feedback to a user of said interface device;
receiving web page information from said server machine over said network, said web page information including screen display information representing a visual layout of a web page and force feedback information related to providing a feel sensation correlated with said visual layout;
displaying on said visual display of said client machine said web page based upon said screen display information;
receiving input information from said human computer interface device for positioning a displayed cursor with respect to said visual layout of said web page; and
providing a force feedback signal that is based upon said input information and based upon said web page information received over said network wherein said force feedback information includes a call to a force feedback program running on said client machine that provides said force feedback signal, wherein said force feedback program running on said client machine is an ActiveX control, said ActiveX control being a force-only ActiveX control able to output force feedback signals for different force effects to said interface device, said force feedback signal being received by said interface device, wherein said interface device outputs computer-controlled physical force feedback to said user correlated with said visual layout of said web page on said visual display, said force feedback being based upon said force feedback signal.

2. A method as recited in claim 1 wherein said force-only ActiveX control is referenced by script instructions included in said force feedback information.

3. A method as recited in claim 2 wherein said script instructions are provided in JavaScript.

4. A method as recited in claim 2 wherein said force feedback information in said web page includes said script instructions and force effect parameters, wherein said script instructions provide said force effect parameters to said force feedback program running on said client machine, wherein said force feedback program provides said force feedback signal to said interface device based on said force effect parameters.

5. A method as recited in claim 2 wherein said script instructions determine when to provide said force feedback signal to said interface device.

6. A method for providing force effects for a web page, the method comprising:
receiving web page information from a server machine over a network, said web page information including screen display information representing a plurality of web page objects to be displayed in a web page;
determining which of said web page objects are force web page objects to be associated with at least one force effect, wherein said force web page objects are web page objects having a predefined type; and
assigning a generic force effect to each of said force web page objects, said generic force effects being defined by effect information derived from a client machine, wherein said generic force effects cause a force signal to be output by said client machine when a user-controlled cursor interacts with one of said force web page objects, wherein a force control program running on said client machine detects whether said user-controlled cursor is contacting one of said force web page objects having said predefined type said force control program being integrated in a web browser program, said cursor and said force web page objects being
displayed on said web page by said web browser running on said client machine, wherein said force signal is output to an actuator of a force feedback interface device coupled to said client machine to cause a force sensation to a user of said force feedback interface device, and wherein said force signal is based on said effect information.

7. A method for providing force feedback over a network comprising:
establishing a connection between a server machine and a client machine over a network, said client machine including a visual display and an interface device providing computer-controlled physical force feedback to a user of said interface device;
receiving web page information from said server machine over said network, said web page information including screen display information representing a visual layout of a web page and force feedback information related to providing a feel sensation correlated with said visual layout;
displaying on said visual display of said client machine said web page based upon said screen display information;
receiving input information from said human computer interface device for positioning a displayed cursor with respect to said visual layout of said web page; and
providing a force feedback signal that is based upon said input information and based upon said web page information received over said network, wherein said force feedback information includes a call to a force feedback program running on said client machine that provides said force feedback signal, wherein said force feedback program running on said client machine is an ActiveX control, said ActiveX control implementing a particular force effect at a specified area in said web page, said force feedback signal being received by said interface device, wherein said interface device outputs computer-controlled physical force feedback to said user correlated with said visual layout of said web page on said visual display, said force feedback being based upon said force feedback signal.

8. A method for providing force feedback over a network comprising:
establishing a connection between a server machine and a client machine over a network, said client machine including a visual display and an interface device providing computer-controlled physical force feedback to a user of said interface device;
receiving web page information from said server machine over said network, said web page information including screen display information representing a visual layout of a web page and force feedback information related to providing a feel sensation correlated with said visual layout;
displaying on said visual display of said client machine said web page based upon said screen display information;
receiving input information from said human computer interface device for positioning a displayed cursor with respect to said visual layout of said web page; and providing a force feedback signal that is based upon said input information and based upon said web page information received over said network, wherein said force feedback information includes a call to a force feedback program running on said client machine that provides said force feedback signal, wherein said force feedback program running on said client machine is a Java applet, said force feedback signal being received by said interface device, wherein said interface device outputs computer-controlled physical force feedback to said user correlated with said visual layout of said web page on said visual display, said force feedback being based upon said force feedback signal.
Description
BACKGROUND OF THE INVENTION

This application contains Microfiche Appendix A consisting of (1) slide and 36 Frames.

This invention relates generally to human/computer interfaces, and more particularly to human/computer interfaces with force feedback that can operate over a network.

The Internet has, of late, become extremely popular. While the use of the Internet has been prevalent for many years now, its use has been limited by the arcane and difficult commands required to access the various computer on the network. To address this problem, a protocol known as the "World Wide Web" or "WWW" was developed to provide an easier and user-friendlier interface for the Internet. With the World Wide Web, an entity having a domain name creates a "web page" or "page" which can provide information and, to a limited degree, some interactivity.

A computer user can "browse", i.e. navigate around, the WWW by utilizing a suitable web browser and a network gateway (e.g., an Internet Service Provider (ISP)). Currently, popular web browsers, include Netscape.RTM. Navigator.RTM. made by Netscape Corporation of Mountain View, Calif., and Internet Explorer made by Microsoft.RTM. Corporation of Redmond, Wash. A web browser allows a user to specify or search for a web page on the WWW, and then retrieves and displays web pages on the user's computer screen.

The Internet is based upon a transmission protocol known as "Transmission Control Protocol/Internet Protocol" (or "TCP/IP" for short), which sends "packets" of data between a host machine, e.g. a server computer on the Internet, and a client machine, e.g. a user's personal computer connected to the Internet The WWW is an Internet interface protocol which is supported by the same TCP/IP transmission protocol. Intranets are private networks based upon Internet standards since they adhere to Internet standards, can often use the same web browser software and web server software as are used on the Internet.

A web page typically includes static images, animated images (e.g. video), and/or text. The images and text are specified in a "HyperText Mark-up Language" ("HTML") file that is sent from the web server to the client machine. This HTML file is parsed by the web browser in order to display the text and images on the display of the client machine. Other standardized languages or protocols are also being developed for use with the Internet and the World Wide Web. For example, the Virtual Reality Modeling Language (VRML) is used to provide visual virtual 3-D environments and allow one or many users to navigate through and interact as "avatars" in such an environment using a web browser or other software on a client computer system.

Furthermore, additional functionality may be provided in web pages with information downloaded over the WWW in the form of scripts or programs. Scripting, typically in the form of VBScript or JavaScript, allows a series of instructions to be performed on the client computer once the instructions have been downloaded in a web page. Programs can be provided in such standard languages as Visual Basic, C++, or currently in the form of Java "applets" or ActiveX.RTM. controls. Java includes a platform-independent interpreter running on the client machine that executes downloaded applets within web pages or other programs, e.g., to display animated images, retrieve data over the WWW, output feedback to

the user, or perform operating system tasks. ActiveX controls similarly execute on the client computer once he program instructions are resident on the client. ActiveX controls are programmable acts that can be embedded into Web pages and may be written in any (platform-specific) language. Java and ActiveX controls can add functionality to a Web page that would normally be difficult, or even impossible, using HTML or scripting languages. ActiveX controls can also be controlled with a scripting language. Alternatively, functionality can be added to a web page through the use of "plug-ins", which are application programs running in conjunction with certain web browsers to parse plug-in-specific code in the web page which the browser cannot understand.

Other WWW-related functionality includes Dynamic HTML. Dynamic HTML is a set of features currently incorporated in browsers such a, Microsoft Internet Explorer that enable authors to dynamically change the rendering and content of an HTML document. Using Dynamic HTML, a content developer or programmer can access the attributes of a document's contents or objects (such as an object's position on the page and type). In addition, event messages are generated when a user interacts with the web page content (such as when a user clicks on a graphical button image). The features of Dynamic HTML can be elicited through the use of VBScript or JavaScript scripts embedded in a Web page or programmatically through Visual Basic or C++.

The Internet and the WWW also permit sound data to be transmitted over the Internet. For example, references to sound files can be embedded in HTML pages and can be played by the web browser. Data "packets" coded in TCP/IP format can also be sent from one client machine to another over the Internet to transmit sound data. This last-mentioned technique forms the basis for Internet telephony.

While the transmission of visual images (both static and dynamic), text, and sound over the Internet is well-known, the transmission of other types of sensory data has not been well explored. In particular, the transmission of data over the Internet pertaining to the sense of touch and/or force has not been established "Force feedback" allows a user to experience or "feel" tactile sensations as provided through computational information. Using computer-controlled actuators and sensors on a force feedback device, a variety of realistic sensations can be modeled and experienced by the user. This useful and highly immersive sensory modality for interacting with the Internet has hereto been unavailable.

In addition, there needs to be tools to allow web page authors to quickly and easily add forces to web page content or adjust/modify existing forces as desired. Such tools should preferably allow an author or user to intuitively include forces in web pages without needing a knowledge of force feedback instruction or programming constructs. In addition, the author should be provided with an intuitive way to design or adjust the forces. Thus, a tool is needed for assisting the user, programmer or developer in intuitively and easily creating web pages and in setting force feedback characteristics to provide desired force sensations in web pages.

SUMMARY OF THE INVENTION

The present invention is related to the transmission/reception of information pertaining to force feedback to provide feel in transmitted information such as web pages. The force feedback provided by the methods and apparatus of the present invention enhance the sensory experience of the user to provide a richer, more interesting, and more enjoyable interaction with information received over networks.

In one aspect of the present invention for providing force feedback over a network, information is received by a client machine over a network from a server machine. The client machine (typically a personal computer) has a visual display and a force feedback interface device. Force feedback interface devices can include force feedback mice, joysticks, finger cradles, trackballs, steering wheels, and yokes. The force feedback interface device includes sensors and actuators and preferably a local microprocessor for communicating with the client machine and controlling the actuators of the interface device.

Several embodiments are disclosed in which forces are implemented based on the information received over the network. The disclosed embodiments emphasize the World Wide Web or the Internet as the network provided between client and server computers, where web page information received by the client is in HTML or other standard format and a web browser running on the client machine displays the web page. One embodiment provides authored force effects, which are effects particular to a web page that are customized for each web page object and are downloaded with the web page. The received web page information includes screen display information representing a visual layout of a web page and force feedback information related to providing a feel sensation. Input information from the interface device is used to position a cursor with respect to the visual layout of the web page. A force feedback signal is provided based upon the input information and the web page information, and the received force feedback information includes a call to a force feedback program running on the client machine that provides the force feedback signal. The force feedback signal is received by the interface device, which outputs computer-controlled physical force feedback to the user correlated with the visual layout of the web page. The force feedback program running on the client can be ActiveX control, a Java applet, a plug-in, etc. In one preferred embodiment, the ActiveX control is called by script instructions, e.g. in JavaScript, included in the received force feedback information. The script instructions can provide downloaded force effect parameters to the force feedback program to output a desired force effect.

In a different embodiment, generic force effects are implemented. Generic effects are applied uniformly to all web page objects of a particular type. A web page need not include force information to allow generic force effects to be applied to the web page content. Thus, a received web page includes screen display information representing web page objects. The client determines which web page objects are force web page objects to be associated with at least one generic force effect, when the force web page objects are of a predefined type. A generic force effect is assign to each type of web page object as defined by effect information derived from the client machine. Generic force effects are output when a user-controlled cursor interacts wit a force web page object. A force control program running on the client machine can detect whether the cursor is contacting one of the force web page objects and output the appropriate force feedback signals to the interface device. In another embodiment, a Downloaded web page can be preprocessed before being displayed to add force feedback information to the web page. The client can perform the preprocessing, or a proxy server can preprocess the web page before sending the web page to the client. In other embodiments, both generic effects and authored effects can be provided for a particular web page. If both types of force effects are used, either type of force effect can be overridden by the other depending on user preferences.

In another aspect of the present invention, a web page authoring interface is provided that includes the ability to add force sensations to a web page. The web page authoring interface is displayed on a display device of a computer and displays a web page including web page objects. Input from a user is received to the authoring interface and selects a web page object and a force effect to be associated with the selected web page object. A web page is output by the interface, including the web page objects and including force information to allow the force effect to be implemented when the web page is displayed by a client machine after being received from a server machine over a network. The authoring tool can include a force design interface for creating or modifying the force effect, or a separate force design inter ace can accessed. The force effects added to web page objects can also be experienced directly by the author during the design process. In one embodiment, the author is also able to spatially designate an area of a web page object to be associated with a selected force effect, e.g. a graphical mark or outline can designate sub-areas of an image or text to be associated with forces. In addition, the user can preferably associate sound data with a web page object, such that when said force effect is output, a sound is output to the user synchronized with the force effect. Finally, a predefined graphical identifier can be inserted into the web page to indicate to a user that the web page provides force effects, and which may also be a link that causes a linked force feedback resource web page to be downloaded.

The present invention provides several ways to implement force feedback over a network such as the World Wide Web. Embodiments included herein allow standard web pages without any force content to be assigned forces on a client machine including force feedback capability. In other embodiments, a web page author can include specific force effects in a web page to any desired level of customization. The described force feedback web page authoring tool allows force effects to be easily and quickly included in web pages to foster greater diversity and widespread use of feel in web pages.

These and other advantages of the present invention will become apparent upon reading the following detailed descriptions and studying the various figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial representation of the Internet, a web server machine, and two client machines;

FIG. 2 is a block-diagram of a client machine used in the present invention;

FIG. 3 is a block-diagram of a force feedback system in accordance with the present invention;

FIG. 4a is a perspective view of a preferred human/computer interface ("force feedback device") of the present invention;

FIG. 4b is a perspective view of the mechanism of the force feedback device of FIG. 4a;

FIG. 5 is a block diagram of a wide area network (WAN) based upon Internet TCP/IP protocol and supporting World Wide Web (WWW) HTML protocols in accordance with the present invention;

FIG. 6 is an illustration of a web browser and an image displayed on a visual display of a client computer as generated from a downloaded HTML web page file;

FIG. 7 is a flow-diagram illustrating a first embodiment of an embodiment using Dynamic HTML to provide force effects in web pages;

FIG. 8 is a diagrammatic illustration of the coordinate frames provided in Dynamic HTML;

FIG. 9 is a diagrammatic illustration of coordinates used in a transformation to obtain screen coordinates of an object;

FIG. 10 is a flow-diagram of illustrating a second embodiment of an embodiment using Dynamic HTML to provide force effects in web pages;

FIGS. 11a and 11b are diagrammatic illustrations of the two embodiments shown in FIGS. 7 and 10;

FIGS. 12, 13a, 13b, 14, and 15 are illustrations of displayed web pages in which web page objects are associated with authored force effects using an ActiveX control;

FIG. 16 is a diagrammatic illustration showing a proxy server for providing preprocessing to web pages to add force effects;

FIGS. 17a and 17b are illustrations of embodiments of web page authoring applications of the present invention that includes force feedback editing functionality;

FIG. 18 is a block diagram illustrating the web page authoring interface of the present invention and web pages produced by the authoring interface;

FIGS. 19a, 19b, and 19c are illustrations showing force effect editing interface examples which can be used in conjunction the authoring application of FIGS. 17a-17b;

FIGS. 20-22 illustrate a web page editor displaying an image in a web page and providing an outline function to add force effects to portions of the image;

FIGS. 23 and 24 illustrate the web page editor of FIGS. 20-22 adding force effects to a different image;

FIG. 25 is a diagrammatic illustration showing features of a first embodiment of the force feedback web page editor of the present invention; and

FIG. 26 is a diagrammatic illustration showing feature of a second embodiment of the force feedback web page editor of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In FIG. 1, a network system 10 includes a wide area network (WAN) such as the Internet 12, and a number of computers or "machines" coupled to the Internet 12. For example, a first client machine 14, a second client machine 16, and a web server machine 18, are coupled to the Internet 12.

As noted previously, both the Internet 12 and Intranets operate using the same TCP/IP protocols. This allows Intranets to use similar or the same server machine software and client machine software as are used in Internet 12 applications. The Internet 12 includes a number of nodes 20 that are interconnected by data transmission media 22. These nodes are typically routers, switches, and other intelligent data transmission apparatus which route "packets" of TCP/IP information to the desired destination. In some instances, the nodes 20 comprise an Internet service provider (ISP) 20a which allows a client machine to access the "backbone" of the Internet. Alternatively, client machines and web servers can be coupled directly into the backbone of the Internet.

As noted previously, the present invention is directed to the implementation of force feedback over a network, such as the Internet 12. To provide a user of a client machine with the experience of force feedback, force feedback human/computer interfaces (hereafter "force feedback devices") 24 and 26 can be provided as part of the client machines 14 and 16, respectively. The client machines 14 and 16 are typically provided with computer video monitors 28 and 30 (which is one example of a "visual display"), respectively, which can display images I1 and I2, respectively. Preferably, forces developed by force feedback devices 24 and 26 are correlated with the images I1 and 12 of the client machines 14 and 16, respectively.

The machines 14-18 are considered, in the language of the Internet, to be "resources," and each has its own unique Uniform Resource Locator or "URL." In one embodiment of the present invention, a client machine, such as client machine 14 or 16, sends a request for a "web page" residing on, for example, web server machine 18. This is accomplished by the client machine sending a connection request and a URL which specifies the address of the web page to the web server machine 18. The web server machine 18 then sends a web page 32 in HTML format back to the requesting client machine where it is "cached" in the memory (typically the RAM, hard disk, or a combination of the two) of the client machine. In this embodiment of the invention, the image on the video display of the client machine is generated from the HTML web page file cached on the client machine, and force feedback is provided to a user through the force feedback device as he manipulates a user manipulable object of the force feedback device. The embodiments of the present invention can also be used with other collections of data or documents besides web pages.

In another aspect of the present invention, a first client machine, such as client machine 14, and a second client machine, such as client machine 16, directly communicate force feedback commands to each other in standard TCP/IP protocol over the Internet 12. More particularly, client machine 14 can send force feedback and other information to the URL of the client machine 16, and the client machine 16 can send force feedback and other information in standard TCP/IP pickets to the URL of the client machine 14. In this way, users of client machine 14 and client machine 16 can interact physically over the Internet 12. Of course, a server machine 18 can likewise directly communicate force feedback commands to a client machine 12 or 14, or all three machines can interact.

In FIG. 2, a "personal" computer 34 architecture that can be used for client machine 14 or client machine 16 is shown in block diagram form. It should be noted that a variety of machine architectures can be used to access the Internet 12, such as a PC compatible computer under the Windows or MS-DOS operating system, Macintosh personal computer, SUN or Silicon Graphics workstation, home video game systems (such as systems available from Nintendo, Sega, or Sony), "set top box", "network access computers", portable/handheld computers of all types, or any general-purpose computer. The particular architecture shown for the computer 34 is a typical personal or "PC" computer architecture. Web server machines can also have similar architectures, but are often more powerful "workstations" that, for example, operate under some variant of the UNIX.RTM. operating system. The Internet service providers 20a are likewise often UNIX-based computers or powerful personal computers running Windows NT.RTM.. The nodes 20 are most commonly routers built by Cisco Systems of San Jose, Calif. Client machine 14 or 16 can also take other forms, such as a television including or connected to a microprocessor for Internet access. Force feedback devices used with such client machines can be appropriate for the particular embodiment, e.g., a TV remote control used for internet browsing on the abovementioned television can include force feedback functionality.

The personal computer system 34 includes a microprocessor 36 clocked by a system clock CLK and which is coupled to a high speed or memory bus 38 and to a lower speed or I/O bus 40. The system RAM 42 and ROM 44 are typically coupled to the high speed memory bus, while various peripherals, such as the video display, hard disk drive, Internet interface (often either a modem or an Ethernet connection), and force feedback device, are typically coupled to the slower I/O bus. The microprocessor executes programs stored in the various memories or other computer-readable medium of the computer 34 (RAM, ROM, hard disk, signal propagated by a carrier wave, etc.) to control, for example, the image display on the video display and the forces provided by the force feedback device. The manufacture and use of computers, such as personal computer 34, are well-known to those skilled in the art.

In FIG. 3, a client machine 46 in accordance with the present invention includes a personal computer system 48 and a force feedback human/computer interface or "force feedback device" 50. A user 52 can receive visual information 54 and auditory information 56 from the personal computer 48 and can manipulate the force feedback device 50 as indicated at 58a and 58b to provide input, e.g., to command a cursor location on a visual display or other provide other control information. In addition, the user 52 can receive force feedback 60 from the force feedback device 50 to represent physical "feel" or force sensations.

The personal computer system 48 includes the microprocessor 36, the system clock 62, a video monitor 64 (which is one type of "visual display"), and an audio device 66. The system clock 62, as explained previously, provides a system clock signal CLK to the microprocessor 36 and to other components of the personal computer system 48. The display device 64 and the audio output device 66 are typically coupled to the I/O bus 40 (not shown in this figure).

In this preferred embodiment, the force feedback device 50 preferably includes a local microprocessor 68, a local clock 70, optional local memory 71 for the local microprocessor 68, a sensor interface 72, sensors 74, a user manipulatable object
76, "other" input interface 78, an actuator interface 80, a safety switch 82, and actuators 84 which provide a force F to the object 76, and an optional power supply 86 to provide power for the actuator interface 80 and actuator 84.

The microprocessor 36 of the personal compute system 48 is coupled for communication with the local microprocessor 68 of the force feedback device 50. This communication coupling can be through a serial port coupling 88 to the personal computer system. A preferred communication coupling the Universal Serial Bus (USB) of a personal computer, although an RS-232 serial bus, or other communication coupling (such as Firewire), a parallel bus, an Ethernet bus, or other types of interfaces or communication links can also be used.

In use, the user 52 of the client machine 46 grasps the object 76 of the force feedback device 50 and manipulates (i.e. exerts a force to move or attempt to move) the object to cause a "pointer" icon (cursor) to move in the image displayed by the display device 64. This pointer icon typically takes the form of a small arrow, a pointing hand, or the like. The sensor 75 senses the movement of the object 76 and communicates the movement to the local microprocessor 68 through the sensor interface
72. The local microprocessor 68 then communicates through communication coupling 88 to the microprocessor 36 to cause the microprocessor 36 to create a corresponding movement of the pointer icon on the image displayed upon the visual display 64. In some embodiments, the sensors 74 can communicate directly to microprocessor 36 without the use of local microprocessor 68. The user can also create other input, such as a "button click," through the other input 78 which are communicates to the microprocessor 36 by the local microprocessor 68 or directly, e.g., using a game port.

If the pointer icon on the display device 64 is at a position (or time) that correlates to a desired force feedback to the user 52, the microprocessor 36 sends a force feedback command to the local microprocessor 68 over the serial port connection 88. The local microprocessor 68 parses this force feedback command and sends signals to the actuator interface 80 which causes the actuator 84 to create force, F on object 76, which are experienced by the user 52 as indicated at 60. The safety switch 82, sometimes referred to as a "deadman switch", blocks the signal from the actuator interface 80 if, for example, the user 52 is no longer grasping the object 76. In this way, the user 52 can interact with the client machine 46 in a visual, auditory, and tactile fashion.

The hardware architecture described above is also described in co-pending U.S. Pat. No. 5,739,811, filed Nov. 28, 1995, the disclosure of which is incorporated herein by reference. The high level command protocol between the computer and the force feedback device is also described in U.S. Pat. No. 5,734,373, filed Dec. 1, 1995, the disclosure of which is incorporated herein by reference. Force feedback as implemented in a graphical user interface is described in U.S. patent application Ser. No. 08/571,606, filed Dec. 13, 1995, and incorporated herein by reference.

FIG. 4a is a perspective view of a force feedback mouse interface system 90 of the present invention, capable of providing input from the user to a host computer based on the user's manipulation of the mouse and capable of providing force feedback to the user of the mouse system in accordance with the present invention. Mouse system 90 includes interface device 50 that includes a user manipulatable object or manipulandum 36 and an interface 100, and host (client) computer 48.

User manipulatable object (or "manipulandum") 36, in the described embodiment, is a mouse that is shaped so that a user's fingers or hand may comfortably grasp the object and move it in the provided degrees of freedom in physical space. For example, a user can move mouse 36 to correspondingly move a computer generated graphical object, such as a cursor or other image, in a graphical environment provided by computer 48. The available degrees of freedom in which mouse 36 can be moved are determined from the interface 100, described below. In addition, mouse 36 preferably includes one or more buttons 102 to allow the user to provide additional commands to the computer system.

It will be appreciated that a great number of other types of user manipulable objects can be used with the method and apparatus of the present invention in place of or in addition to mouse 36. For example, such objects may include a sphere, as trackball, a puck, a joystick, a knob, a wheel, a dial, cubical-, cylindrical-, or other-shaped hand grips, a fingertip receptacle for receiving a finger or a stylus, a flat planar surface like a plastic card having a rubberized, contoured, and/or bumpy surface, a handheld remote control used for controlling web pages or other devices, pool cue, or other physical objects.

Interface 100 is provided in housing 101 and interfaces mechanical and electrical input and output between the mouse 36 and host computer 48 implementing the application program, such as a GUI, simulation or game environment. Interface 100
provides one or more degrees of freedom to mouse 36; in the preferred embodiment, two linear, planar degrees of freedom are provided to the mouse, as shown by arrows 104. In other embodiments, greater or fewer degrees of freedom can be provided, as well as rotary degrees of freedom. For many applications, mouse 36 need only be moved in a very small workspace area.

The interface 100 provides position information to computer 48 via bus 112, such as a Universal Serial Bus (USB). In addition, computer 48 and/or interface 100 provide force feedback signals to actuators coupled to interface 10, and the actuators generate forces on members of the mechanical portion of the interface 100 to provide forces on mouse 36 in provided or desired degrees of freedom. The user experiences the forces generated on the mouse 36 as realistic simulations of force sensations such as jolts, springs, textures, enclosures, circles, ellipses, grids, vibrations, "barrier" forces, and the like. The electronic portion of interface 100 may couple the mechanical portion of the interface to the host computer 48. Interface
100 preferably includes a local microprocessor 68 as described above. Mouse 36 is preferably supported and suspended above a grounded pad 110 by the mechanical portion of interface 100.

Computer 48 is preferably a personal or other computer, workstation, or game console as described above. Computer 48 preferably implements one or more application programs ("applications") with which a user is interacting via mouse 36 and other peripherals, if appropriate, and which can include force feedback functionality. For example, one of the application programs is preferably a Web browser that implements HTML or VRML instructions. Herein, computer 48 may be referred as displaying "graphical objects" or "objects." These objects are not physical objects, but are logical software unit collections of data and/or procedures that may be displayed as images by computer 48 on display screen 64, as is well known to those skilled in the art. Display device 64 can be included in host computer 48 and can be a standard display screen (LCD, CRT, etc.), 3-D goggles, or any other visual output device.

As shown in FIG. 4a, the host computer may have its own "screen frame" 118 (or host frame) which is displayed on the display screen 64. In contrast, the mouse 36 has its own "device frame" (or local frame) 120 in which the mouse 36 is moved. In a position control paradigm, the position (or change in position) of a user-controlled graphical object, such as a cursor, in host frame 118 corresponds to a position (or change in position) of the mouse 36 in the local frame 120.

FIG. 4b is a perspective view of a preferred embodiment of the mechanical portion 108 of mouse device 50. Mechanical linkage 130 provides support for mouse 36 and couples the mouse to a grounded surface 124, such as a tabletop or other support. Linkage 130 is, in the described embodiment, a 5-member (or "5-bar") linkage. Fewer or greater numbers of members in the linkage can be provided in alternate embodiments.

Ground member 132 of the linkage 130 is a base for the support of the linkage and is coupled to or resting on a ground surface 124. The members of linkage 130 are rotatably coupled to one another through the use of rotatable pivots or bearing assemblies, all referred to as "bearings" herein. Base member 134 is rotatably coupled to ground member 132 by a grounded bearing 142 and can rotate about an axis A. Link member 136 is rotatably coupled to base member 134 by bearing 144 and can rotate about a floating axis B, and base member 138 is rotatably coupled to ground member 132 by bearing 142 and can rotate about axis A. Link member 140 is rotatably coupled to base member 138 by bearing 146 and can rotate about floating axis C, and link member 140 is also rotatably coupled to link member 136 by bearing 148 such that link member 140 and link member 136 may rotate relative to each other about floating axis D. Mouse 36 is coupled to link members 136 and 140 by rotary bearing 148 and may rotate at least partially about axis D.

Transducer system 150 is used to sense the position of mouse 36 in its workspace and to generate forces on the mouse 36. Transducer system 150 preferably includes sensors 152 and actuators 154. The sensors 152 collectively sense the movement of the mouse 36 in the provided degrees of freedom and send appropriate signals to the electronic portion of interface 100. Sensor 152a senses movement of link member 138 about axis A, and sensor 152b senses movement of base member 134 about axis A. These sensed positions about axis A allow the determination of the position of mouse 36 using known constants such as the lengths of the members of linkage 130 and using well-known coordinate transformations. Sensors 152 are, in the described embodiment, grounded optical encoders that sense the intermittent blockage of an emitted beam. A grounded emitter/detector portion 156 includes an emitter that emits a beam which is detected by a grounded detector. A moving encoder disk portion or "arc" 158 is provided at the end of members 134 and 138 which each block the beam for the respective sensor in predetermined spatial increments and allows a processor to determine the position (and velocity) of the arc 158 and thus the members 134 and 138 by counting the spatial increments.

Transducer system 150 also preferably includes actutors 154 to transmit forces to mouse 36 in space, i.e., in two (or more) degrees of freedom of the user object. The bottom housing plate 157 of actuator 154a is rigidly coupled to ground member
132 (or grounded surface 124) which includes, e.g. a magnet, and a moving portion of actuator 154a (e.g. a wire coil) is integrated into the base member 134. The actuator 154a transmits rotational forces to base member 134 about axis A. The housing 157
of the grounded portion of actuator 154b is coupled to ground member 132 or ground surface 124 through the grounded housing of actuator 154b, and a moving portion (e.g. a coil) of actuator 154b is integrated into base member 138. Actuator 154b transmits rotational forces to link member 138 about axis A. The combination of these rotational forces about axis A allows forces to be transmitted to mouse 36 in all directions in the planar workspace provided by linkage 130 through the rotational interaction of the members of linkage 130. The operation of the electromagnetic actuators 154 is described in greater detail in co-pending applications Ser. No. 08/881,691 and aforementioned 08/965,720. In other embodiments, other types of actuators, such as electrical DC motors, can be used. A different embodiment of a force feedback device can include flexure members to allow movement in provided degrees of freedom.

In FIG. 5, a conceptual representation of the network system 10 with force feedback includes a server machine 18, a client machine 14 provided with a force feedback device 24, and one or more additional client machines 16, each of which may be provided with additional force feedback devices 26. As noted in this figure, the server machine is a computer or "processor" running, for example, the TCP/IP server software and is which is connected to the Internet. The client machine 14 includes a computer or "processor" running Internet browser software and force feedback driver software. The processor of the client machine is connected to the Internet and to the force feedback device 24. The force feedback device 24 has sensors and actuators so that it can track movement of the user manipulatable object, monitor for button presses and/or other ancillary input devices, and provide output force feedback sensations. The force feedback device 24 sends object tracking information to the client machine, and receives force feedback commands from the client machine 14. The "additional client", such as client machine 16, also includes computers or "processors" running Internet browser software and force feedback driver software. The processors of these additional clients are also connected to the Internet and are connected to force feedback devices associated with that client.

As noted in FIG. 5, a client machine 14 can send data request to the server machine 18 and, in return, receive a web page including HTML instructions and/or other instructions for implementing a web page, e.g. Dynamic HTML instructions, JavaScript instructions, Java or ActiveX code, an embedded plug-in reference (such as the "IFF" extension described below), etc. For

example, if an embedded IFF reference is used, the server must also have a modified configuration file which let it know that IFF is a valid MIME type. This modified file would be a SRM.CONF or other .CONF file, as will be appreciated by those skilled in the art. The client machine 14 then sends force feedback commands to the force feedback device 24 and receives tracking and button data from the force feedback device 24. Client machine 16 can likewise send a data request to the server machine 18 and receive an HTML file and/or other web page data or other document or collection of data. The client machine 16 can then interact with the force feedback device 26 by sending force feedback commands to the device 26 and by receiving tracking and button data from the force feedback device 26.

In another embodiment, a force-related application or control program running on the client machine 16 can check if any peripherals including force feedback functionality are connected to the client machine. For example, after downloading a web page having force effects, the client machine can first check whether force feedback mouse is connected. If no mouse is connected, the client can check whether another force feedback peripheral, such as a force feedback joystick, is connected. If so, the client can enable the joystick to control the position of the cursor within the web page so that the user is able to experience the force effects associated with web page objects. Other force feedback peripherals can also be enabled for cursor control in a web page. The force-related application can perform the necessary adjustments to receive input from an absolute input device or a relative input device, as necessary, or from a "cursor control device" or a "gaming device", etc.

In addition to communicating with the server machine the client machines can communicate directly with each other over the Internet using an Internet communication protocol. For example, client machine 14 can communicate with client machine 16
through a TCP/IP connection. This is accomplished making the URL of the client machine 16 known to the client machine 14, and vice versa. In this fashion, direct communication between client machines can be accomplished without involving the server machine 18. These connections can send force feedback information and other information to the other client machine. For example, a process on the client machine 16 can send force feedback information over a TCP/IP Internet connection to the client machine 14, which will then generate a force feedback command to the force feedback device 24. When the user reacts to the force feedback at force feedback device 24, this information can be sent from client machine 14 to client machine 16 to provide force feedback to the user on force feedback device 26.

As is well known to those skilled in the art, a client machine normally connects to another computer, such as a server or other client, over the Web by sending a connection request to the "host" of the desired URL. The host, in this example, is a server machine 18 and the desired URL is the URL of the desired wet page residing on the server machine 18. Alternatively, the desired web page can reside on another server or resource and be retrieved by server machine 18. In response to the connection request of step 150, the server machine 18 sends the HTML file representing the web page over the Internet to be received by the client machine. The HTML file includes a number of "components" which are typically commands, command fragments, instructions, and data which permit the display of the web page and other functionality. HTML components are parsed and interpreted (processed) as they are received, e.g., even before the entire HTML file is received at the client machine. Alternatively, the entire HTML file can be received before the processing begins.

A HTML file typically includes a number of "components" which are parsed and interpreted as previously described. For example, an HTML file begins with a <HTML>command or "tag" to indicate the start of the HTML file, and a <BODY> tag to indicate that the body of the HTML file is beginning. Then, an arbitrary number of HTML commands are provided to, for example, display images of the web page on the video display of the client machine. The body of the HTML file is terminated by the </BODY> command, and the end of the HTML file is indicated with the </HTML> command, i.e. this command is the "eof" command of the HTML file.

Force Effects in Web pages

There are many types of touch interaction that can be achieved in Web pages. The force sensation resulting from this interaction is know herein as a force effect or an "effect." Two main classes of tactile interaction with Web content described herein are "generic effects" and "authored effects," as described in greater detail below. These effects are applied to web page objects in the most common implementation. Herein, "web page objects" or "web objects" are any graphical objects (or regions) appearing on a web page, such as images, text, buttons, or other standardized or custom objects. In some cases, a web page object may not actually appear on the web page when displayed, but may have an effect on how the web page is visually displayed or how force effects are output on the force feedback interface device.

Generic effects and authored effects are preferably composed from a basic set of stock force effects. The stock effects include vector forces vibrations, springs, textures, and others, as described in U.S. Pat. No. 5,825,308 and patent applications Ser. Nos. 08/571,606, 08/747,841, now U.S. Pat. No. 5,959,613, Ser. Nos. 08/846,011 and 08/879,296, all incorporated by reference herein. Effects of differing complexity can be provided as stock effects; for example, a primitive effect such as a simple vector force to be output in a specified direction at a specified magnitude can be provided, or a more complex effect that includes multiple primitive effects can be provided. One particularly significant, more complex effect is the enclosure. An enclosure is a set of forces that occur only when the cursor is in or near a geometrically bounded ("enclosed") area of the screen. Enclosures can be associated with forces at their borders to attract the cursor to the inside of the bounded area, keep the cursor outside the bounded area, or attract the cursor to the border surrounding the bounded area. The enclosure may take a variety of shapes, for example rectangular or elliptical, and may also be associated with one or more other force effects when the cursor or pointer is positioned inside the enclosure. Examples of force effects that can be provided and programmed are specified in the FEELit Application Programming Interface (API) from Immersion Corporation of San Jose, Calif., detailed in patent application Ser. No. 08/970,953, filed Nov. 14, 1997, Docket no. IMM1P035, entitled, "Force Feedback System Including Multi-Tasking Graphical Host Environment and Interface Device" and incorporated by reference herein.

For example, a hyperlink ("link") is an input-sensitive area displayed on a web page which, when selected by the user, causes a "link action" in the web browser. Such link actions can include retrieving a different web page over the Web whose address is specified by the link and/or displaying that web page, causing a different section of the web page currently in memory to be displayed, retrieving data for an animation or other modification of the current web page, sending data to a server over the web, or other action either locally and/or involving data transfer over the WWW. Typically, one or more web objects on a web page are defined as hyperlinks, such as text, an image, an animated image, an icon, or other object; such an object is referred to as a "link object" herein. The user typically selects a link by moving the mouse pointer over the link object and providing a command gesture such as clicking a mouse button. Alternatively, links might automatically be selected when the mouse pointer is moved over the link object, i.e., the interaction of pointer and link object is the command gesture in such a case.

An enclosure can be defined as the visual boundaries of a hyperlink object (enclosures can also be defined as a portion of an object or as a portion of the background to the web page). The forces of the enclosure can be used to "snap" the pointer to that link object, i.e., assist the pointer to move onto or "acquire" that link object. To the user, this might feel as if the link has a magnetic attraction. A text hyperlink, for example, is no longer purely discerned by the user by the text's appearance (usually color and/or underlining), but is also discerned by its feel. Generally, hyperlinks are some of the more interesting features of a Web page and are intended to stand out from the other web page content to inform the user that other content can be accessed from the displayed web page, e.g. hyperlinks are the areas on the web page where the location of the mouse pointer can cause a link event, while other areas on the web page may have no effect based on input from a mouse. The snap enclosure effect physically grabs the user's attention at a link using forces until the user pushes the user manipulatable object (such as a mouse) hard away from the link object. Instead of interacting with unresponsive text, the enclosures constrain the user to the most important content on the web page that is responsive to input from the mouse (e.g. a command gesture such as a button click).

In addition, some objects on the web page may be defined as a hyperlink, but there is no visual evidence to the user that these objects are links and can cause a link action such as calling tip a different web page. For instance, many graphical images are also hyperlinks, but this fact is not discernible visually; previously, the user had to move the pointer over the image, which would cause the appearance of the mouse pointer to change if the image were a link object (or cause a change in other displayed information). By providing enclosure effects on hyperlinks, tactile cues are offered instead of or in addition to visual cues; these tactile cues are consistent with the user interface so the user easily understands their meaning. The user can more quickly determine which objects on a web page are links by using the tactile sense compared with the visual sense. Adding feel to web pages increases both the user's efficiency and the quality of their experience. Some studies have shown an 88% improvement in mouse targeting performance with over 1000 test subjects.

Other objects on a web page besides hyperlinks may also be associated with forces such as the enclosure to assist acquiring the object. For example, standard GUI graphical objects such as text boxes, input buttons, radio buttons, checkboxes, icons, are readily adaptable to enclosures. In addition, the user can be constrained by enclosures to the input fields or the areas associated with frames. For example, an input field where the user must enter data such as a name, a credit card number, or click a drop down menu are prime areas which can be associated with enclosure "snap to" forces. Similarly, frames having particular objects of interest can be associated with enclosure effects. "Frames" are those areas of a web page which can be accessed, manipulated, scrolled, etc. independently of other displayed areas (other frames) of the web page. For example, a frame on the left side of a web page often is used to display the main subject areas of the web page while the larger, main frame on the right displays the content of a selected one of those subject areas.

Finally, an extra dimension of user experience can be added by invoking a "pop" or jolt force effect when the user clicks on a buttons, checkboxes, or other graphical object. A pop is a time based sensation as opposed to, e.g., an enclosure implemented as a snap sensation, which is a spatially based sensation.

FIG. 6 illustrates a web browser 200 such as Netscape Navigator in which displayed web page objects are associated with forces. For example, these force effects can be authored force effects using any of the authored effect embodiments described below. Control area 210 of the browser includes well known controls such as navigation buttons 212 and 214 and URL address field 216. The display area 230 has been provided with a number of web page objects to illustrate some of the concepts of the present invention. The force feedback device controls the position of a pointer icon (cursor) 240 which can be caused to interact with the various web page objects.

As an example, when the force feedback device is manipulated by the user to cause the pointer icon 240 to move within a "texture" region 242, force feedback commands can be created for the force feedback device to provide a desired "texture" to the force feedback device. For example, the texture can feel "rough" to the user by causing the force feedback device to place forces on the user manipulatable object that emulate a rough or bumpy surface. In a region 244, a viscosity or "liquid" force feedback can be provided. With this form of force feedback as the pointer icon is moved through field 244, a viscous "drag" force is emulated on the user manipulatable object as if the user object were moved through a thick liquid. In a region 246, inertial forces can be felt. Therefore, a pointer icon being moved through an "inertia" region would require relatively little or no force to move in a straight line, but would require greater forces to accelerate in a new direction or to be stopped. The inertial force sensations can be applied to the user manipulatable object and felt by the user.

In a "keep out" region 248, the pointer image is prevented from entering the region. This is accomplished by creating a repulsive force on the user manipulatable object using a force feedback command to the force feedback device which prevents or inhibits the user from moving the user manipulatable object in a direction of the region 248 when the pointer icon 240 contacts the periphery of the region 248. In contrast, a "snap-in" region 250 will pull a pointer icon 240 to a center 252 whenever the pointer icon engages the periphery of the snap-in region 250 and apply a corresponding attractive force on the user manipulatable object. A "spring" region 254 emulates a spring function such that a pointer icon moving into the spring region "compresses" a spring, which exerts a spring force on the user manipulatable object which opposes the movement of the pointer icon. A region 256 is a "Force To Left" region where the pointer icon within the region 256 is forced to the left side of the region and the user manipulatable object is forced in a corresponding direction as if influenced by some invisible magnetic force or gravitational force. A region 258 illustrates that regions can be of any size or shape and that within a region different force effects can be developed. In this example, within region 258 there is a texture core 260 surrounded by a vibration ring 262. Therefore, as the pointer icon 240 moves into the region 258, the user first experiences vibration from the ring 262, and then experiences a texture as the pointer icon moves within the core 260.

The exemplary force feedback web page of FIG. 6 is also provided with several force feedback buttons. In a first button 264, the placement of the pointer icon 240 over the button and the pressing of a mouse button (i.e., a switch) on the mouse
36 to create a "button click", "button down", or simply a "button event" input, will then cause a "buzz" command to be sent to the force feedback device. The buzz command would, for example, cause a vibration force on the user manipulatable object. Similarly, the selection of the "jolt" button 266 will cause a jolting force e.g., a short-duration pulse of force) to be provided at the force feedback device, and the pressing of the "detent" button 268 will cause a "detent" to be created for the force feedback device. By "detent" it is meant that the user manipulatable object will be controlled by the force feedback actuators such that it feels as if a mechanical-type detent exists at the position that the user manipulatable object was in when the detent button 268 was activated.

Generic Effects

Generic effects are force effects that are applied uniformly to all objects in a received document having a particular type. For example, in a received web page, web page objects of standard types include hyperlinks, images, text, text entry fields, tables, headings, image maps, marquees, buttons, check boxes, radio buttons, drop-down menus, or other standard web page objects. Different generic effects can also be applied to different characteristics of an object; for example, one effect is associated with bold text, while a different effect is associated with text having the color blue. Generic effects are achieved by defining a mapping between object types and feel sensations, where feel sensations are uniformly associated with particular type of web page objects and/or particular types of interactions (or "events") with objects. "Interactions" or "events" may include clicking a mouse button when the pointer is positioned over the object, moving the pointer onto a link from

above the link (e.g. as differing from moving the pointer onto the link from below the link), or any other cursor/object interaction defined by the developer or user.

In a preferences file or other storage of the client machine, particular force effects are each mapped to a particular type of object provided in a web page (and/or GUI). Using generic effects, the same force effect is applied to all graphical objects of a particular type. For example, all hyperlinks on a page can be assigned with an enclosure having the same force characteristics (e.g., forces of the same magnitude), although the area enclosed by each enclosure on the page will vary with the size of the web page object with which it is associated.

A major advantage of using generic effects is that the force effects are implemented exclusively on the client side of the network. Thus, the author/content developer of a web page does not need to specify any forces or force characteristics of the objects in the web page code itself. When the web page is received by the client computer, the force-enabling code implemented by the client associates the forces from the preferences file (or default forces) to each graphical object and outputs the forces as appropriate. This allows standard web pages that have no force effects in their code to be implemented as a force feedback web page on the client end, which allows a force feedback interface device to be used immediately without requiring special web pages tailored for force output. In other embodiments, a web page can include force feedback information for authored effects (described below), and generic effects can also be applied to web page objects not having any authored effects associated with them, or to override particular authored effects as desired by the user of the client.

For example, the force magnitudes or directions associated with an enclosure are defined on the client computer. This enclosure is assigned to objects of a particular type, e.g. link objects. A different enclosure with different associated forces can be assigned to another type of object, such as a (non-link) image or an animation. The preferred embodiment allows the user to adjust the effects to suit their preferences. For example, one way to allow a user to assign forces is analogous to the approach applied to the color of a hyperlink object, where the web browser automatically assigns the hyperlink a color based on the user's preference but web authors can override those default colors as desired, e.g. to make the hyperlinks' color match the web page's color scheme. The user preferably is able to adjust the characteristics of generic effects using a central control panel, e.g., a control dialog box associated with their force feedback GUI preferences, or a dialog box associated with the web browser preferences.

In a different embodiment, different sets of generic effects can be provided which are associated with different web pages or different group of web pages. For example, the user can specify a first set of generic effects in which snap enclosures are to be associated with link objects on the downloaded web page. This first set is to be associated with web pages of a particular type or origin, e.g. all web pages received from a particular URL or domain name will have objects assigned to the first set of effects on the client computer. Likewise, a second set of generic effects can be characterized by the user, e.g. assigns vibration forces to link objects. The second set of generic effects is assigned to web pages received from a different surface or having a different characteristic. Alternatively, different generic effect sets can be provided by different web site providers, developers, companies. Thus, for example, Immersion Corporation can provide a generic effect set that the user can download and which will automatically be assigned to web pages that are downloaded from specified servers or locations on the WWW. (The specified locations can be designated in the generic effect set file, or otherwise). The web browser (or a plug-in or script) can check for these specified locations when a web page is downloaded.

Authored Effects

In contrast, authored effects are effects particular to a certain web page that are specified or modified by a content developer or other author and which can provide a more rich, custom experience than generic effects can offer. For example, one or more particular graphical objects on a web page can be assigned a different, customized force effect by an author (the "author" can be the creator/developer of the web page or the user on the client machine that receives the web page). Web page objects having the same type can be assigned different force effects. Authored effects can be used, for example, to intentionally correlate a particular, unique force with an object of a web page. Typically, at least part of the instructions governing the type and/or characteristics of an authored effect are included in the code of the web page that is downloaded by the client. Thus, standard non-force web pages do not include authored force effects (unless there is preprocessing done to the web page before it is displayed by the client, as described in greater detail below).

Any standard force effect can be an authored effect, e.g. those feel sensations supported by the FEELit API from Immersion Corp. Alternatively, authored effects can be partially or completely customized to be specific to a desired interaction or display. One example of an authored effect is gravity toward a particular object. A gravity force is an attractive force that biases or actively attracts the mouse in a direction that causes the mouse pointer to move toward a defined point of the desired object. For example, an author can use gravity to attract the mouse pointer towards advertisement image objects on a web page. Many web sites are sponsored or supported by advertisers; such sites usually include web pages that display graphical hyperlinks to a sponsor's web site. Although the advertisement visuals (animations, etc.) can attract the attention of a site's visitor, an authored effect can physically attract the mouse toward the advertisement, forcing the user to acknowledge the advertisement's existence. However, not all the graphical images on the same web page may be desired to have a gravity effect, so that generic effects may not be suitable in this situation. In other examples, authored effects can enable users to feel the texture of clothing at a commercial Web site by providing a texture effect associated with a graphical clothing image, or to feel the forces of a falling ball at a physics education web site by providing a ball image that causes the desired forces when the pointer is moved in contact with the ball image. Authored effects allow an author to provide a "theme" of feel sensations in accordance with a theme or subject of a particular web page; for example, a vibration force effect can be associated with the links of a web site that gives information about earthquake.

Authored effects can be "dynamic," i.e., the behavior of the effect can change depending on the author's intent and a current state or other state of the Web page. In the gravitational advertisement example, the gravity can be turned off once the mouse cursor has reached the advertisement. Thus, the user's experience of the site will not be compromised after the cursor has been guided to the advertisement. The dynamic nature of these effects can be achieved either through scripting (e.g. using such languages as VBScript or JavaScript) or programming (e.g., using such programming constructs as Java applets and ActiveX controls). The ability to incorporate compelling authored effects in Web sites is limited only by the developer's creativity.

Although authored effects can allow much more compelling forces to be provided in web pages because they can be carefully correlated with images, animations, and sounds. They have the disadvantages of typically requiring specific force effect instructions in the web page that are downloaded by the client and sometimes require specific force-enabled resources on the client end.

A different embodiment uses a form of authored effects to specify generic effects. For example, if an author does not want to include authored effects in a web page but wishes to specify one or more generic force effects for object types on the web page, the author can include generic effect designations. For instance, a number of standardized generic effects sets (as described above) can be provided on client machines, and each web page downloaded can specify in an HTML tag (or otherwise) which generic effect set (or a particular generic effect within a set) to use for the downloaded page. Thus, the author has, in effect, authored the force effects by referring in the web page to generic effects resident on the client. Or, the tag in the web page code can refer to an official or customized generic effect set that the user is presumed to have previously installed on the client machine (and, if that customized effect set is not installed on the client, a default generic effect set can be used). The author can simply specify that there is a force enclosure on an object, but allow client defaults or preferences to assign forces and/or force characteristics to the enclosure (e.g. to the walls and/or interior of the enclosure).

A user can experience both authored effects and generic effects in a particular downloaded web page. For example, the user can set generic effects to types of objects displayed in the web page, as defined by the user's preferences stored on the client. There can also be authored effects in the web page that are specified in information downloaded with the web page. Any authored effect that conflicts with a generic effect can be set, by default, to override the conflicting generic effect. This allows special authored effects to still be felt by the user, yet also allows any other web page objects not assigned any authored effects to have the user's preferred generic effects assigned to them. As is analogous to the color of hyperlinks, the user in a client machine can also preferably specify whether to override any authored forces of a downloaded web page with preferred generic effects; or the user can specify to use the authored effects or the generic effects specified by the web page author (which may fit the theme of the web page better than the user's choice of generic effects, for example).

In yet another embodiment of the present invention, either authored effects or generic effects can be implemented by continuous reception of force feedback data over a network, e.g. similar to "streaming" video and audio. For example, parameters to specify and characterize forces can be included in streaming video and audio data to provide tactile sensations to the user of the client machine receiving the streaming data. Tags or references to force effects already resident on the receiving client machine can be included in the streaming data. Alternatively, all or most of the data defining the force effects and the data required to implement the force effects can be included in the streaming data.

Embodiments for Implementing Force Feedback over Networks

The following subsections detail several embodiments of software that allow a user to feel web pages. These embodiments retrieve web page data, track user interaction with the web page, and output the appropriate force effects to the tactile display.

Proprietary Browser

An efficient embodiment to enable feel in web pages is to implement the force effect functionality directly in the web browser, such as Netscape Communicator by Netscape Communications or Internet Explorer by Microsoft Corp. A web browser parses Hypertext Markup Language (HTML) files retrieved over the Internet and visually displays the content of the web page. In the present embodiment, a browser may also track a user-controlled cursor's interactions with the displayed content and apply force effects accordingly. Since the web browser has intimate knowledge of the web content, it can perform these tracking operations more easily than many other methods can do so. This embodiment is most feasible for entities that have already written a Web browser (such as Microsoft and Netscape). In addition, publicly available source code of a web browser (e.g., Netscape Communicator) can be modified by a different party to incorporate force feedback functionality directly into the browser and re-released to the public.

This embodiment can apply generic effects to web page objects, which is typically used for standard web pages. However, HTML tags (commands) can also be created which can define authored effects in a downloaded web page and which the web browser can interpret and implement, similar to the function of the plug-in software described below in the plug-in embodiment. The use of authored effects can alternatively be achieved in this embodiment by providing Java applets or ActiveX controls in the web page, as described in greater detail below.

Dynamic HTML

Dynamic HTML is a set of features currently in browsers such as Microsoft Internet Explorer 4.0 that enable authors to dynamically change the rendering and content of an HTML document. Using Dynamic HTML, a content developer or programmer can access the attributes of the graphical objects of an HTML document (such as the object's position on the web page and the object's HTML tag or type). In addition, event messages are generated when a user selects or interacts with the Web objects (such as when a user clicks a button while the mouse pointer is over the object). The power of Dynamic HTML can be elicited, for example, through the use of VBScript or JavaScript scripts embedded in a web page or programmatically, e.g. in a web browser, through Visual Basic or C++, as is well known to those skilled in the art.

Dynamic HTML can be used to enable the feeling of generic effects in web pages. FIG. 7 presents a flow diagram of the present invention in accordance with a first Dynamic HTML embodiment in which a separate application program runs in parallel with a web browser on a client machine. A user starts a separate application program on a client machine in step 284. In a step 286, the application launches a web browser on the client machine, and creates/maintains a connection to the web browser. In step 290, this application then "hides" in the background of the operating system. After a web page in HTML has been received and displayed by the web browser, the application checks for an event notification in step 292. The application receives an event notification from the browser whenever a user moves the mouse (and/or when another predefined event occurs). When the application receives this notification, it can ascertain in step 294 whether the mouse pointer is touching any relevant object (e.g., a web page object having forces associated with it) on the web page through the use of Dynamic HTML functions. If no event notification is received in step 292, the process continues to check for a notification until the user quits the web browser or the separate application in step 300, at which point the connection to the web browser is destroyed in step 302.

One example of determining if a relevant object is being touched is provided in the pseudocode below. Dynamic HTML is hierarchical in nature, where characteristics of an object may be provided in parent objects to the object. Since Dynamic HTML is hierarchical, the element passed immediately to a requesting application or function may not be of interest, but something in that element's hierarchy may be of interest. For example, only bold text can be associated with a force effect. The application thus must check whether the current object that is "touched" by the pointer is bold text. The below function checks this hierarchy. Given a Dynamic HTML object, called "elem", this function will determine whether the object is something to which a force-effect is to be applied (i.e., it is "relevant"). If a text object is passed to this function will search the hierarchy of the text object looking for a "B" tagname that indicates the text is bold. It will return null if the object is not relevant, otherwise it will return the relevant tagname.

______________________________________ // This is pseudo-code for the TouchingRelevantObject function. TAG TouchingRelevantObject( HTMLElement elem ) // Is this element valid? // This will happen if we've reached the end of the recursion. if ( elem IS null ) return false; if ( elem.tagName IS.sub.-- IN A.sub.-- SET.sub.-- OF.sub.-- TOUCHABLE.su b.-- TAGS ) return elem.tagName; else

return TouchingRelevantObject( elem.parentElement ); } ______________________________________

If the mouse pointer happens to be touching a relevant object, i.e. an object associated with one or more force effects (such as a hyperlink), the application calculates the screen coordinates of the object in step 296 using the dynamic HTML functions. The application then generates the appropriate force effect in step 298, such as an enclosure, on the user manipulatable object of the interface device. The force effect can be generated by using high level commands sent to the microprocessor of the interface device 50, or can be generated by direct low level commands from the host application.

One difficulty of this approach lies in the determination of screen coordinates of the relevant object. The difference between the coordinate frame of the mouse and the coordinate frame of the Dynamic HTML object requires that one of the coordinate frames be transformed to be operable with the other. The mouse pointer position and effects such as enclosures are described in screen coordinates. Dynamic HTML objects such as hyperlinks are described in coordinates relative to other web page objects; to derive the absolute screen coordinates of a Dynamic HTML object, several coordinate frame transformations may have to be applied based on the returned Dynamic HTML object. An example of the hierarchy of objects and the transformations required to achieve screen frame coordinates are illustrated in FIG. 8, which shows coordinate frames in Dynamic HTML. The frames may include a screen frame (absolute frame) 310, a window frame 312, a web browser client area frame 314 (which is also HTML frame #1), an HTML frame #2 316, and a hyperlink frame 318. A particular frame is specified by its leftmost and topmost coordinates, relative to the coordinate frame containing the particular frame. The hierarchy leading from the hyperlink frame to the screen frame is the reverse order of the frames as listed above, starting with the hyperlink frame and traversing to the screen frame.

As currently implemented, it is not possible to compute the absolute screen coordinates of a Dynamic HTML object by traversing its transformation hierarchy because the farthest traversal is to the client window frame; there are no mechanisms available for transforming this client window frame to the screen frame. Also, traversing this hierarchy can be computationally expensive, relative to the degree of responsiveness that is required in a force-feedback system. If a user moves over a hyperlink object, the enclosure should snap immediately; any delays severely degrade the user experience.

The preferred way to resolve this problem is simple, elegant, and fast. Dynamic HTML returns an "event object" when the application is notified of a mouse interaction; this event object allows the program to query the mouse position in both relative and screen coordinates. These values can be used to ascertain a Dynamic HTML object's transformation between relative and screen coordinates. For example, the relevant "event" object fields that return the desired values are shown below:

offsetX--event X coordinate relative to the srcElement's container's frame

offsetY--event Y coordinate relative to the srcElement's container's frame

screenX--event X coordinate relative to physical screen size

screenY--event Y coordinate relative to physical screen size

srcElement--the object that cause the event

The relevant DHTML object fields are provided below:

offsetleft--X coordinate of object's left edge, relative to its container's frame

offsettop--Y coordinate of object's top edge, relative to its container's frame

These values indicate the coordinates of the web page object (also known as an "element" of the web page) with reference to the object's "container", which is the frame that encloses the object (such as the web browser). Once these values are obtained, the absolute screen coordinates of the object can be determined as follows:

objectScreenX=event.screen.X-(event.offsetX-vent.scrElement.offsetLeft)

objectScreenY=event.screen.Y-(event offsetY-vent.scrElement.offsetTop)

These values are illustrated in FIG. 9.

Since generic force effects are often used with normal HTML code having no force effects included therein, the Dynamic HTML is preferably only implemented by the browser or other program providing the generic effects. For example, after the HTML document is received, it is processed into Dynamic HTML by the browser or separate application on the client so that web object properties can be obtained and generic forces applied. Alternatively, the web page can be provided as Dynamic HTML instructions and downloaded in that form.

A second implementation of integrating generic force effects with Dynamic HTML can be used in which the web browser itself has the ability to check user interactions and apply force effects. As shown in a method 320 of FIG. 10, the user again starts an application program on the client machine in step 322. In step 324, this application opens a browser window in the foreground that hosts navigation buttons and a browser "engine", e.g., an MSHTML object. MSHTML it rendering engine used by Microsoft's Internet Explorer and is also available for all application developers to use. The MSHTML object performs the parsing and rendering of HTML content. To allow users to feel the displayed objects of a web page, the application program then applies the methodology described above in FIG. 7 with the MSHTML object to directly determine (by calculating screen coordinates) when the mouse pointer is positioned in relation to a relevant web object (in step 326) such that screen coordinates are calculated in step 328 and forces are to be output to cause the force feedback interface device to output the appropriate forces in step 330. The process quits via step 332 when the pointer no longer touches a relevant object.

The two implementations described above are summarized diagrammatically in FIGS. 11a and 11b. In a first implementation of FIG. 11a, a proprietary application includes a connection to Internet Explorer, Netscape Navigator, or other available, standard web browser application. The browser 340 display web page content 342. The proprietary application 344 also runs simultaneously (e.g. multitasks) which is used to control forces for the web page content. The browser 340 provides event notifications to the application 344 indicating, for example, when the mouse pointer touches a relevant object. Upon event notification, the application 344 verifies if the element is touchable; if so, the application queries the browser 340 for element information such as the element frame in screen coordinates. The proprietary application 344 determines and outputs commands for generic force effects to the API (such as Feelit API) for the mouse or other manipulandum, which are output as force sensations correlated with the web page content displayed by the web browser 340.

In a second implementation of FIG. 11b, a proprietary application 348 hosts MSHTML as described above, and does not connect to a standard, available web browser. This implementation uses the same notification/query mechanism used in the first implementation.

The advantage of the first implementation is that a standard, well known web browser is used as the interface to the web page, which is familiar to users. Also, users can easily take advantage of web browser upgrades and improvements as provided by the browser company. Unfortunately, the connection between web browser and the feel-enabling application program can be tenuous. The connection can be prone to many bugs and may be slow, and may not have the responsiveness required by a force-feedback application. On the other hand, in the second implementation, there is an extremely tight coupling between the browser engine (e.g. MSHTML object) and the feel-enabling application because they are in the same process space. Thus, event notification in this implementation is robust and fast. The drawback is that users must use an interface that is not as full-featured as a proprietary web browser nor as familiar. They are identical in terms of the display of web content, but a proprietary web browser typically has many more features and functionality for manipulating, editing, navigating among web pages, such as "Favorites" and "History" options to provide easier browsing.

The