Three-dimensional electronic interaction

3D electronic interaction filmed through the lens of a Project North Star augmented reality headset

3D electronic interaction in 1974

Three-dimensional electronic interaction (3DEI) is the use of an electronic product that has a physical interface that requires three dimensions to use. 3D electronic interaction is a richer form of interaction with external working memory in the form of visual representations than 2D interfaces. 3D electronic interaction includes augmented reality (AR) and virtual reality (VR).

3D electronic interaction is theorized to be a next-generation mode of using mainstream computing devices, improving in all aspects over the standard 2D interface devices that have been used starting in about the 1980s. 3D electronic interaction improves upon 2D electronic displays that have been in use for about 100 years.

3D electronic interaction systems are under development and have been developed starting in the 1960s. Industrial progress in 3D electronic interaction systems had a period of expansion starting in the 1990s with the consumer phenomenon of the first wave of virtual reality.

Uses of 3D electronic interaction include 3D visualization and planning, 3D computer-aided design, and annotation of 3D models. 3D electronic interaction can be used for casual communication and fun.

3D electronic interaction requires a display and a 3D input device. The display can be a 3D-tracked monoscopic 2D display, where the rendered viewpoint changes according to the user's tracked position. Many forms of 3D interaction use a 3D-tracked head-mounted display and a 3D-tracked handheld controller. In 3DEI, the position of a handheld 3D input device can be used to control the position of a viewport instead of the headset position controlling the position of the viewport, even if displaying on a head-mounted display.

3D interaction enables doing tasks in a virtual 3D space faster than with a 2D system to a high degree. 3D vision increases the information transfer from the real world to the human brain to a high degree. 3D electronic interaction allows for the leverage of actions of a user with a computer system to increase.

Main areas of interest are 3D direct interaction and isomorphic interaction.

For the purpose of 3DEI, if 3D tracking is available, a monoscopic HMD with 3D tracking beats a binocular fixed-focus display that is two-view with 3D tracking. In case 3D tracking is not available, a binocular fixed-focus two-view display with no 3D tracking beats a zograscope with no 3D tracking. 6DOF tracking was invented in the 1960s and 1970s, whereas binocular stereoscopes have had a run of popularity starting in the 1800s, which displaced the popular focus on zograscopes.

Features

3D interface more allows a user to interact with a three dimensional (3D) environment or data stored in the computer in a natural and direct manner.

In order for a spatial computing system to be successful, it needs to be self hosting, just like existing developments in computing, such as Doug Engelbart's work and personal computing developments in the 1970s.

In a 3D electronic interaction system, the most important factors in order are speed of display update, the computerized system itself, and head-and-hand 3D tracking. Quality of the graphics such as having color, filled polygons, or sharpness is not as important as those three factors. Focal variety is also not as important.

Motion sickness due to latency primarily comes from the visual periphery. Vertical position tracking is not needed in AR/VR as a requirement. Computers in general act as a medium for expression. 3D computing offers an upgraded medium.

Mapping to the human brain

Modalities of 3D interactive environments

2D and 3D electronic media in general act as a substrate for human thought. Electronic media, whether 2D or 3D, provide a huge amount of external working memory. Compared to 2D electronic interaction, 3D electronic interaction (3DEI) provides better saturation of the brain's neural coding capacity.

3D representations provide a richer medium for thought that engage the brain in ways that 2D representations cannot. The human brain prefers to represent as much information as possible using the fewest active neurons to save energy. In 2D, interfaces are an unnatural abstraction, so the brain has to burn a massive amount of metabolic energy in the prefrontal cortex just to hold abstract, disconnected concepts in working memory. The parts of the human brain that control muscles, muscle memory, and the sense of where the body is in space (proprioception) and the visual system evolved to work together to process the world in 3D.

Without 3DEI, 3D data can be used and analyzed by computers, but there is no direct 3D interface, making all access from the user indirect and slow, using 2D input and output devices, like 2D monitors and computer mice.

3D spatial computing is an architectural necessity for widening the data conduit between machine and mind.

Parietal cortex coding saturation

3D GUIs aim to saturate the posterior parietal cortex (PPC), which is beyond basic retinotopic mapping, compared to the effect of 2D GUIs, which in general only saturate the primary visual cortex.

Brain processing

2D interfaces rely on the ventral stream (the "what" pathway), focusing on object recognition and symbolic decoding.

3D interfaces engage the dorsal stream (the "where/how" pathway) to a high degree, which involves the posterior parietal cortex (PPC).

Cognitive load is reduced if the parietal lobe processes visual and other phenomenon in physically local places with in it. For example, processing the hand's movement directly at the 3D location of a visual stimulus, like when picking up a real object. This is how the human brain evolved.

Human psychology

3D personal electronic devices exploit the lack of a known bandwidth ceiling for internet communications, in light of the known latency ceiling.

3D human-computer interaction is theorized to improve the communication bandwidth between the human cortex and a digital tertiary layer. A large part of the focus on 3D information technology is communicating using three dimensions (3D) using the visual cortex, to be able to use more of the human brain for complex cognition tasks.

One of the highest-amount-of-information-processing parts of the brain is the visual cortex.

According to Louis Rosenberg, the ideal way to interaction with digital information is in 3D.

3D human-computer interaction can force parietal dominance as a default cognitive mode, which underlies much of what is called genius-level reasoning. It forces the prefrontal cortex to solve problems using spatial invariants.

Hardware

3D electronic interaction requires three main parts: A display, a 3D control device, and electronic memory. The display and memory can be thought of as two types of electronic media.

In an HMD system, 3D tracking enables 3D control device input and alignment of digital and physical content by using head tracking.

Displays

• 3D displays, including multifocal displays in a VR or AR configuration

• Dynamic focus/varifocal displays: To give a natural 3D view of a computer-generated scene, with all non-pictorial depth cues like in the real world. This is crucial for long-term comfort and realism.

3D control

• Isomorphic interaction, enabled through 6DOF controls.
• Motion Controllers: Devices like VR controllers that detect movement in three dimensions using positional tracking, either optical or otherwise.
• Gesture Recognition: Cameras and sensors (e.g., Microsoft Kinect, Leap Motion) that capture body movements and hand gestures.
• Haptic Feedback: Systems that provide tactile feedback to the user, enhancing the sense of touch in a virtual environment.

People are not very good at drawing lines in space without the support of a writing surface. A more intuitive method is "sculpting".

History

The first VR system that was used with a 3D controller

3D electronic interaction research began in the 1960s. The first 3D computer input device was the Lincoln Wand in 1966.

Use of two-dimensional computer input/output devices, like 2D screens and light pens, became commonplace between 1960 and 1974. As of 1974, the use of perspective graphics displays started happening, which show a 3D scene on a 2D monitor.

At the beginning of real-time 3DEI research in the 1960s, it seemed obvious that three-dimensional computer input devices might be interesting and useful, but there had been little to no corresponding development of these devices. At the time, the only 3D input device available as a commercial product was a three-dimensional version of a Science Accessories tablet in the early 1960s.

Early research was done by A. Michael Noll.

• Ivan Sutherland
• Leap Motion

Software

• Molecular visualization
• Computer-aided design
• Military planning and communication
• 3D mathematical visualization software
• Real-time Collaborative 3D Workspaces: Multiple users, locally or remotely, can simultaneously interact with the same 3D information in a shared volumetric space. Engineers can intuitively collaborate on a car design, walking around it virtually, making changes, and feeling the impact of those changes in real-time.

Interaction techniques

Interaction techniques include manipulation of 3D objects: Techniques for selecting, rotating, scaling, and otherwise interacting with virtual objects in a three-dimensional space.

One method is world-in-miniature.

References

• Moss, Ceci (2008-12-02). "Spatial Operating Environment "g-speak"". https://rhizome.org/editorial/2008/dec/02/spatial-operating-environment-quotg-speakquot/.
• Bowman, Doug A. (2023-08-12). "3D User Interfaces". https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/3d-user-interfaces.