Spatial computing

Microscribe, an example of a 3D input device, being used

Spatial computing in 1974

3D human-computer interaction (3D HCI), also known as spatial computing, refers to methods and technologies that allow users to interact with computers in three-dimensional space.

3D HCI includes augmented reality and virtual reality. In spatial computing, the user's tasks are performed directly in 3D spatial contexts. A 3D computer system can transmit more information to the brain in a form it evolved to process efficiently.

The goal of 3D human-computer interaction is to saturate the neural coding capacity of the human brain. 3D environments engage complex neural networks in the parietal and visual cortexes in ways that 2D interfaces cannot. 3D environments are high-entropy sources that match the dimensionality of human neural structures, like place cells and grid cells in the hippocampal formation.

Full-duplex 3D HCI requires a 3D input device and a 3D display. The 3D display can be flat focus 3D, lightfield, or holographic.

It can be summed up as a situation where a person can move something around in 3D, and a computer knows where it is, and the computer can move something virtual around in 3D, and the person knows where it is.

Main areas of interest are 3D direct interaction and isomorphic interaction. 3D HCI allows for the leverage of actions of a user with a computer system to increase.

History

The first VR system with a 3D controller

The first 3D computer input device was the Lincoln Wand in 1963.

Between 1960 and 1974, use of two-dimensional computer input/output devices, like 2D screens and light pens became commonplace.

As of 1974, the use of perspective graphics displays started happening, which show a 3D scene on a 2D monitor. To researchers, it seemed obvious at first that corresponding three-dimensional computer input devices might be interesting and useful, but there had been no corresponding development of these devices. At the time, the only commercially available 3D input device was a three-dimensional version of the Science Accessories acoustic tablet.

• Ivan Sutherland
• Leap Motion
• Looking Glass Factory
• CREAL

Mapping to the human brain

Computer technology in general acts as a substrate for human thought. It provides near-infinite working memory.

The parts of the human brain that control muscles, muscle memory, and the sense of where the body is in space (proprioception) and the visual system evolved to work together to process the world in 3D, but the way computers have been used has been 2D graphical user interface elements since the 1980s.

This sort of control doesn't take advantage of how the human brain has evolved.

3D spatial computing is an architectural necessity for widening the data conduit between machine and mind.

History

The Macintosh GUI was explicitly an attempt to saturate more of the user’s perceptual and cognitive bandwidth than text-based, symbolic interfaces. Its goal was to escape the low-capacity bottleneck of command lines.

2D GUIs largely saturate the primary visual cortex through retinotopic mappings, but do not saturate the parietal lobe.

3D GUIs aim to saturate the posterior parietal cortex (PPC), which is beyond basic retinotopic mapping.

Brain processing

2D interfaces rely heavily on the ventral stream (the "what" pathway), focusing on object recognition and symbolic decoding.

3D interfaces engage the dorsal stream (the "where/how" pathway). By using 3D GUIs, we activate the Posterior Parietal Cortex (PPC).

Cognitive load is reduced if the parietal lobe processes visual and other phenomenon in physically local places with in it. For example, processing the hand's movement directly at the 3D location of a visual stimulus, like when picking up a real object. This is how the human brain evolved.

Displays

• 3D displays, including biscopic displays and holographic displays
• VR headsets

Input devices (control peripherals)

• Isomorphic interaction, enabled through 6DOF controls.
• Motion Controllers: Devices like VR controllers that detect movement in three dimensions using positional tracking, either optical or otherwise.
• Gesture Recognition: Cameras and sensors (e.g., Microsoft Kinect, Leap Motion) that capture body movements and hand gestures.
• Haptic Feedback: Systems that provide tactile feedback to the user, enhancing the sense of touch in a virtual environment.

People are not very good at drawing lines in space without the support of a writing surface. A more intuitive method is "sculpting".

Interaction techniques

• Manipulation of 3D objects: Techniques for selecting, rotating, scaling, and otherwise interacting with virtual objects in a three-dimensional space.

Application areas

This is a list of uses of 3D computer interface that consists of 3D control peripherals and 3D displays.

• Molecular visualization
• CAD
• Military planning and communication
• 3D mathematical visualization software

3D information technology

3D information technology is information technology that operates using three-dimensional space. Unlike traditional 2D (two-dimensional) information technology which only considers a 2D plane, 3D information technology adds a dimension of depth.

3D personal electronic devices exploit the lack of a known bandwidth ceiling for internet communications, in light of the known latency ceiling.

An example is a 3D display. Another example is a 3D controller.

Properly done 3D human-computer interaction is theorized to improve the communication bandwidth between the human cortex and a digital tertiary layer. A large part of the focus on 3D information technology is communicating using three dimensions (3D) using the visual cortex, to be able to use more of the human brain for complex cognition tasks.

One of the highest-amount-of-information-processing parts of the brain is the visual cortex.

According to Louis Rosenberg, the ideal way to interaction with digital information is in 3D.

A standard 2D computer serves as external working memory to the prefrontal and visual cortexes, but does not serve as rich working memory for the posterior parietal area.

3D human-computer interaction can force parietal dominance as a default cognitive mode, which underlies much of what we call genius-level reasoning. It forces the prefrontal cortex to solve problems using spatial invariants.

Comparison with 2D

• A 2D list of files is better than a messy 3D pile of icons, but a well-designed 3D environment (like a physical library) uses human spatial memory—which is significantly more powerful than the ability to memorize a flat list of text.

• Steve Jobs' contribution to information technology was his push at Apple to get the technology to portray a 2D spatial map and basic memory functions through parietal processing. This was through the graphical user interface (GUI). 2D information technology relies on Rote/List Memory and 2D parietal processing. 3D perspective computer graphics no a 2D screen utilize the parietal lobe to a small extent, but are generally limited by a keyboard and mouse, which are 2D.

Hardware

• Multifocal displays

• Dynamic focus/varifocal displays: To give a natural 3D view of a computer-generated scene, with all focal cues like in the real world. This is crucial for long-term comfort and realism.

• Advanced spatial tracking (sub-millimeter precision): For both the user and physical objects in the environment, enabling precise interaction and alignment of digital and physical content.

Software

• Real-time Collaborative 3D Workspaces: Multiple users, locally or remotely, can simultaneously interact with the same 3D information in a shared volumetric space. Engineers can intuitively collaborate on a car design, walking around it virtually, making changes, and feeling the impact of those changes in real-time.