Design of three-dimensional spatial packaging of interconnected systems with physical interactions (SPI2) plays a vital role in the functionality, operation, energy usage, and life cycle of a broad range of engineered systems, from chips to ships. SPI2 design problems are highly nonlinear, involving tightly constrained component placement, governed by coupled physical phenomena (thermal, hydraulic, electromagnetic, or other behaviors), and involve energy and material transfer through intricate geometric interconnects. While many aspects of engineering system design have advanced rapidly in the last few decades, including breakthroughs in computational support, SPI2 design has largely resisted automation and in practice requires the bottleneck of at least some human-executed placement and routing design. This article explores a vision of a holistic SPI2 design approach needed to develop next-generation automated design methods that dramatically reduce the time needed to design SPI2 systems and to increase the complexity of designable systems. We review several technical domains related to holistic SPI2 design, discuss existing knowledge gaps and practical challenges, examine exciting opportunities at the intersection of multiple domains to enable comprehensive exploration of SPI2 design spaces, and present one viable two-stage SPI2 design automation framework. Holistic SPI2 design opens up a new direction of high industrial and societal relevance for the design research community.