Spatial computing

Types of spatial computing

Spatial computing in 1974

Spatial computing is the use of a computer system that has a three-dimensional interface, which more directly and naturally allows a user to interact with a three dimensional (3D) environment or data stored in the computer. Spatial computing includes augmented reality, virtual reality, and other modes of operation.

3D representations provide a richer medium for thought that engage the brain in ways that 2D representations cannot. Computers and the internet can process and exchange 3D data amongst themselves just as easily as two-dimensional (2D) data. Spatial computing is a development of the human interface that exploits this fact. Prior to spatial computing, 3D data could be used and analyzed by computers, but was always indirectly accessed using 2D input and output devices, like 2D monitors and computer mice, which made it harder to use as a human than 2D data.

Spatial computing that is full-duplex requires a 3D input device and a 3D display. The 3D display can be fixed-focus 3D, lightfield, or holographic.

Main areas of interest are 3D direct interaction and isomorphic interaction. 3D HCI allows for the leverage of actions of a user with a computer system to increase.

Features

In order for a spatial computing system to be successful, it needs to be self hosting, which follows the lineage of Doug Engelbart's work and Apple Computer in the 1970s.

Mapping to the human brain

Computer technology in general acts as a substrate for human thought. It provides near-infinite working memory.

Compared to physical 3D structures, spatial computing gives near-infinite 3D working memory. Compared to 2D computing, spatial computing provides better saturation of the brain's neural coding capacity.

The brain prefers to represent as much information as possible using the fewest active neurons to save energy. In 2D, interfaces are an unnatural abstraction, so the brain has to burn a massive amount of metabolic energy in the prefrontal cortex just to hold abstract, disconnected concepts in working memory. The parts of the human brain that control muscles, muscle memory, and the sense of where the body is in space (proprioception) and the visual system evolved to work together to process the world in 3D.

3D spatial computing is an architectural necessity for widening the data conduit between machine and mind.

Parietal cortex coding saturation

3D GUIs aim to saturate the posterior parietal cortex (PPC), which is beyond basic retinotopic mapping, compared to the effect of 2D GUIs, which largely only saturate the primary visual cortex.

Brain processing

2D interfaces rely heavily on the ventral stream (the "what" pathway), focusing on object recognition and symbolic decoding.

3D interfaces engage the dorsal stream (the "where/how" pathway). By using 3D GUIs, we activate the Posterior Parietal Cortex (PPC).

Cognitive load is reduced if the parietal lobe processes visual and other phenomenon in physically local places with in it. For example, processing the hand's movement directly at the 3D location of a visual stimulus, like when picking up a real object. This is how the human brain evolved.

Human psychology

3D personal electronic devices exploit the lack of a known bandwidth ceiling for internet communications, in light of the known latency ceiling.

Properly done 3D human-computer interaction is theorized to improve the communication bandwidth between the human cortex and a digital tertiary layer. A large part of the focus on 3D information technology is communicating using three dimensions (3D) using the visual cortex, to be able to use more of the human brain for complex cognition tasks.

One of the highest-amount-of-information-processing parts of the brain is the visual cortex.

According to Louis Rosenberg, the ideal way to interaction with digital information is in 3D.

3D human-computer interaction can force parietal dominance as a default cognitive mode, which underlies much of what is called genius-level reasoning. It forces the prefrontal cortex to solve problems using spatial invariants.

History

The first VR system with a 3D controller

Spatial computing research began in the 1960s. The first 3D computer input device was the Lincoln Wand in 1966.

Use of two-dimensional computer input/output devices, like 2D screens and light pens, became commonplace between 1960 and 1974. As of 1974, the use of perspective graphics displays started happening, which show a 3D scene on a 2D monitor.

To researchers, it seemed obvious intially that three-dimensional computer input devices might be interesting and useful, but there had been little to no corresponding development of these devices. At the time, the only commercially available 3D input device was a three-dimensional version of a Science Accessories tablet.

Early research was done by A. Michael Noll.

• Ivan Sutherland
• Leap Motion

Hardware

• Multifocal displays

• Dynamic focus/varifocal displays: To give a natural 3D view of a computer-generated scene, with all focal cues like in the real world. This is crucial for long-term comfort and realism.

• Advanced spatial tracking (sub-millimeter precision): For both the user and physical objects in the environment, enabling precise interaction and alignment of digital and physical content.

Displays

• 3D displays, including multifocal displays in a VR or AR configuration

Input devices (control peripherals)

• Isomorphic interaction, enabled through 6DOF controls.
• Motion Controllers: Devices like VR controllers that detect movement in three dimensions using positional tracking, either optical or otherwise.
• Gesture Recognition: Cameras and sensors (e.g., Microsoft Kinect, Leap Motion) that capture body movements and hand gestures.
• Haptic Feedback: Systems that provide tactile feedback to the user, enhancing the sense of touch in a virtual environment.

People are not very good at drawing lines in space without the support of a writing surface. A more intuitive method is "sculpting".

Software

This is a list of uses of 3D computer interface that consists of 3D control peripherals and 3D displays.

• Molecular visualization
• CAD
• Military planning and communication
• 3D mathematical visualization software
• Real-time Collaborative 3D Workspaces: Multiple users, locally or remotely, can simultaneously interact with the same 3D information in a shared volumetric space. Engineers can intuitively collaborate on a car design, walking around it virtually, making changes, and feeling the impact of those changes in real-time.

Interaction techniques

• Manipulation of 3D objects: Techniques for selecting, rotating, scaling, and otherwise interacting with virtual objects in a three-dimensional space.
• World-in-miniature

References

• Moss, Ceci (2008-12-02). "Spatial Operating Environment "g-speak"". https://rhizome.org/editorial/2008/dec/02/spatial-operating-environment-quotg-speakquot/.
• Bowman, Doug A. (2023-08-12). "3D User Interfaces". https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/3d-user-interfaces.