Virtual and augmented reality

In recent years, continuous technological change is influencing and modifying almost all sectors. The industrial one, it can be said, is one of the first to be affected by these changes, through the adoption of new techniques and working methodologies aimed at making all kinds of activities more efficient. This paper focused on researching and implementing enabling technologies of Industry 4.0 in the field of oil and gas industrial plant design and management by presenting and describing two real applications. Augmented Reality and Digital Twin technologies have been implemented, using the software PTC Creo Parametric and Vuforia Studio, with the aim of realising experiences that show the potential of the technology itself and the software that realises its use.

Keywords: augmented reality , digital twin , industry 4.0 , smart factory

Phantoms play a crucial role in planning and simulating surgical procedures, evaluating new medical devices, and training purposes. Current simulators, such as human cadavers, animal models, and virtual reality systems, are quite expensive and not easily accessible. This research aims to design and develop an affordable 3D replica of a child's head for paediatric neurosurgery. The process began with a Magnetic Resonance Imaging (MRI) scan of a child's head. Afterward, a phantom was fabricated using nylon 3D printing and incorporating agar to replicate the skull and soft tissue. Moreover, Vaseline balls simulating tumors and a silicone skin were added to the phantom, enabling realistic MRI scans for surgical navigation. The functionality of a novel paediatric neurosurgery robot was then evaluated using the phantom as a realistic representation of a child's head. The cost-effectiveness of this approach makes it an ideal and reproducible method for surgical simulation and training objectives.

Keywords: 3D printing , Agar , MRI-visible phantom head , Neurosurgery

Retrievals of small and medium scale are one of the fundamental and most numerous parts of the entire museal heritage set. Their exposition represents an issue, looking for supporting structures that could be non-invasive, aesthetically pleasant, customized, and, last but not least, safe and stiff enough to avoid any damage to the artifacts. Through the usage of several engineering methods, starting from the retrieval and the exhibition requirements, optimized supporting structure can be developed. The procedure here reported and applied starts from the Reverse Engineering acquisition in order to obtain the 3D model of each retrieval, also comprehending color and texture information. After that, exhibition and preservation requirements, given by heritage experts, are translated into engineering requirements, with the usage of an immersive Virtual Reality environment. With these inputs, the developed Generative Design Method (GDM), using CAD, CAE and optimization tools synergically, can optimize the configuration of supporting interfaces with the artifact and then can generate a lightweight conceptual design through a generative design process. In this way, we can take into account all the given requirements, obtaining a lightweight support, topologically optimized, highly customized, non-invasive, possibly pleasant, able to resist in safety, and additively manufacturable. In this paper, the procedure is applied on different retrievals from Museo Civico dell'Agrocimino of Soriano nel Cimino, Italy, with different scales and materials, proving the robustness of the method.

Keywords: Cultural Heritage , Generative Design , Parametric Optimisation , Reverse Engineering , Virtual Reality

Since the beginning, virtual reality (VR) has been envisioned as a technology that engages multiple senses. Examples include Morton Heiliga's Sensorama simulator and Ivan E. Sutherlanda's notion of the Ultimate Display described in the 1960s. In more than 60 years, technological development and scientific knowledge have predominantly focused virtual and extended reality (VR and XR) applications on the sense of sight and hearing. Despite their importance in everyday interaction, the other senses remain on the margins of VR and XR applications. The potential to include additional senses has been acknowledged and extensively explored in the literature. Those who deal with this topic encounter challenges in understanding the functioning mechanism of the human sensorial and perceptive components and creating interfaces capable of communicating through these senses. Last but not least, it is important to design case studies capable of exploiting the potential of this technology and encouraging investors to believe in its development and end users to adopt it once it is ready. These challenges drive the research activities included in this special issue.

Conducting thoracoscopic surgery necessitates a high level of skill and specialized training. In addition to practicing on actual patients, surgeons utilize various safer training methods. A widely used training apparatus is the laparoscopic box, which contains multiple compartments fitted with training modules to replicate diverse surgical scenarios accurately. Unfortunately, the cost of these devices can be prohibitively high, making it challenging for individual students to acquire their own. Virtual Reality has emerged as a promising solution, especially due to the decreasing costs of VR equipment, which makes it more accessible. This study details our progress in enhancing training for lung surgeries by leveraging cost-effective VR technologies with head-mounted displays. In particular, simulations of thoracoscopic surgery scenarios within a VR environment have been developed, aiming to create training exercises that serve as an initial step before advancing to laparoscopic simulation boxes and, ultimately, actual surgical procedures on patients. For reproducing the behaviour of the soft lung tissues, we used a recent version of the Extended Position-Based Dynamics (XPBD) model, which solves some of the known problems encountered in the traditional PBD system, thereby enhancing the realism of the simulation.

Keywords: Position-based Dynamics , Training , Virtual Reality

Markerless systems and inertial sensors, such as inertial measurement units, offer significant advantages over traditional marker-based methods by eliminating the need for laboratory conditions or markers placement. Virtual reality is an innovative technology used in rehabilitation, recently investigated for obtaining kinematic motion data. This research aims to validate a markerless system using a MediaPipe-based algorithm to reconstruct joint kinematics during exercises with the Nirvana virtual reality rehabilitation system. In the first phase, the markerless system was evaluated against the Xsens inertial measurement unit system, analyzing knee, hip, elbow and shoulder movements of 7 healthy participants at the Politecnico di Milano. The MediaPipe algorithm showed high performance in tracking lower limb movements but was less accurate for upper limbs, as indicated by RMSE and %RMSE values. However, it accurately replicated the time course of all angles based on ICC and Spearman's coefficient, suggesting virtual reality's feasibility in rehabilitation and movement analysis. In the second phase, a pilot study was carried out at the Fondazione TOG on 2 participants with left hemiparesis to assess the efficacy of the system in a pathological context. The results provide a basis for evaluating the potential clinical applicability of the system.

Keywords: Inertial Measurement Units , Markerless , MediaPipe algorithm , Rehabilitation , Virtual reality

Inclusive sport participation for students with disabilities, particularly those within the ASPOC association [1] (Aged 18-24) demographic, remains a challenge in many educational and recreational settings. This paper proposes a novel approach utilizing Virtual Reality (VR) and mixed reality technologies to enhance the inclusion and engagement of ASPOC students in sporting activities [2]. Focusing on kayak and rowing simulations, this study explores the potential of immersive experiences to promote physical activity, skill development, and social inclusion among ASPOC students. By integrating VR and mixed reality technologies personalized adaptive experiences can be tailored to meet individual needs, preferences, and abilities. The paper discusses the design considerations, implementation strategies, and potential impacts of this innovative approach on fostering inclusive sport environments for ASPOC students.

Keywords: ASPOC students , inclusion , kayak , kayak simulation , mixed reality , rehabilitation , rowing simulation , Sport , students with disabilities , unity , virtual reality

This paper presents an innovative initiative aimed at enhancing cognitive abilities, fostering social inclusion, and promoting labor inclusion for young students with disabilities. Through the collaborative efforts of the ASPOC association, bachelor, and master degree students from Politecnico di Milano, Lecco Campus, we integrate Mixed Reality (MR) technology via the Meta Quest 3 device with hands-on training and assembly procedures using the Infento Kit. This holistic approach not only empowers young individuals with disabilities but also provides valuable experiential learning opportunities for university volunteers. The paper discusses the development of training and assembly procedures, the role of volunteer engagement, and the potential impact on cognitive, social, and vocational outcomes.

Keywords: ASPOC Association , Cognitive Enhancement , Disabilities , Infento Kit , Labor Inclusion , Meta Quest Device , Mixed Reality , Social Inclusion , Volunteer Engagement

Hydroponic agriculture and vertical farming hold significant promise for sustainable food production, but accessing training programs for these techniques can be challenging, especially for young students with disabilities. This paper presents an innovative approach to address this issue by developing an augmented reality (AR) application tailored for hydroponic agriculture and vertical farming training. Collaborating with the Association for Cognitive Enhancement (ASPOC), the AR platform aims to provide inclusive and accessible learning experiences for students with disabilities, empowering them to acquire practical skills in agricultural practices. By leveraging AR technology, the system enhances engagement, interactivity, and real-time feedback, thereby fostering a more inclusive educational environment.

Keywords: Augmented Reality , Autism , Down Syndrome , Hydroponic Agriculture , Inclusive Education , Neurodevelopmental Disorder , Special Education , Special Educational Needs

Virtual reality (VR) enters educational processes today as a tool capable of promoting immersive learning experiences and facilitating the engagement and participation of all students, including those with Special Educational Needs (SEN). This paper aims to present a case study concerning the use of VR to improve manual skills and promote immersive and enjoyable cognitive training for students with disabilities. A VR cooking food preparation app is introduced to illustrate how VR serves as a tool for skill training on one hand and as a genuine digital learning environment on the other. In addition to presenting the technical features of the app, the International Classification of Functioning (ICF) approach by the World Health Organization (WHO) will be discussed. This approach is useful for identifying the best strategies to promote learning for students with cognitive disabilities who participated in this case study. Furthermore, the perspective of universal learning design (UDL), also known as Universal Design for Learning (UDL) by CAST, will be explored to guide teachers and trainers in designing digital and innovative learning activities that can accommodate all students.

Keywords: Autism , Down Syndrome , Inclusive Education , Neurodevelopmental Disorder , Special Education , Special Educational Needs , Training , Virtual Reality

Training individuals with special needs for cleaning activities presents unique challenges due to diverse learning styles and accessibility requirements. Augmented Reality (AR) technology offers a promising solution by providing interactive, immersive, and customizable training experiences. This paper presents the development and evaluation of an AR application tailored for training cleaning activities specifically designed to accommodate the needs of individuals with special needs. The application utilizes AR to provide personalized guidance, adapt training content to individual capabilities, and enhance engagement through interactive simulations. We discuss the design considerations, implementation details, and user feedback gathered through a pilot study involving individuals with various special needs. Results indicate that the AR application significantly improves the accessibility and effectiveness of training for cleaning activities, thereby empowering individuals with special needs to develop essential life skills and achieve greater independence.

Keywords: Augmented Reality , Cleaning Activities , Immersive Experience , Instructional Guides , Training

Extended Reality (ER) stands out as one of the major technology trends currently, with expectations of even further growth in the near future. The primary objective of this paper is to develop an ER application for the Sharebot One and Sharebot Next Generation 3D Printers, specifically designed to assist students with special needs. The application follows an inclusive design perspective based on the International Classification of Functioning, Disability, and Health (ICF) by the World Health Organization (WHO), enabling students to work and interact with the machine safely. To achieve this goal, three configurations have been developed and tested: a desktop version running on a PC, an Android tablet-based version, and a solution optimized for the Microsoft HoloLens device.

Keywords: Autism , Down Syndrome , Extended Reality , Inclusive Education , Neurodevelopmental Disorder , Special Education , Special Educational Needs , Training

Uban Air Mobility is a new concept of air transportation system that include platforms with different specifications, mainly vertical take-off and landing vehicles, aiming to transport goods and people within urban settings by alternatively managing traffic in congested ground mobility. To ensure the operability of Urban air Mobility, some challenges and objectives in terms of public acceptance and effective design of operational scenarios still need to be achieved before its full integration and deployment. Extended Reality technologies can contribute to the successful realization of these goals as they offer immersive human-machine interfaces that can improve the implementation of such innovative transportation systems. This paper explores Extended Reality-based scenarios and their potential applications for Urban Air Mobility within the framework of the Reality Virtuality Continuum and end user of Urban Air Mobility are elicited. The most promising applications of the proposed approach are discussed, focusing on the main feature that can be enhanced though the adoption of Extended Reality-based technologies.

Keywords: Sustainable Mobility , U-space services , UAM , UTM , XR

Mixed Reality (MR) environments hold immense potential for inducing a sense of embodiment, where users feel like their bodies are present within the virtual space. This subjective experience has been traditionally assessed using subjective reports and behavioral measures. However, neurophysiological approaches offer unique advantages in objectively characterizing embodiment. This review article explores the current state of the art in utilizing neurophysiological techniques, particularly Electroencephalography (EEG), Photoplethysmography (PPG), and Electrodermal activity (EDA), to investigate the neural and autonomic correlates of embodiment in MR for out-of-the-lab applications. More specifically, it was investigated how EEG, with its high temporal resolution, PPG, and EDA, can capture transient brain activity associated with specific aspects of embodiment, such as visuomotor synchrony, visual feedback of a virtual body, and manipulations of virtual body parts. The potential of such neurophysiological signals to differentiate between subjective experiences of embodiment was discussed, with a particular regard to identify the neural and autonomic markers of early embodiment formation during MR exposure in real settings. Finally, the strengths and limitations of the neurophysiological approach in the context of MR embodiment research were discussed, in order to achieve a more comprehensive understanding of this multifaceted phenomenon.

Keywords: augmented reality , EEG , mixed reality , neurophysiology , virtual reality

Companies face the pressing challenge of bridging the skills gap caused by relentless technological progress and high worker turnover rates, necessitating continuous investment in upskilling and reskilling initiatives. Virtual reality (VR) is already used in training programs. However, despite its advantages, the authoring time for creating and customizing virtual environments often makes using this technology not completely convenient. This article proposes an architecture that aims to facilitate the integration of artificial intelligence (AI) assistance into VR training environments to improve user engagement and reduce authoring effort. The proposed architecture was tested in a study that compared a virtual training session with and without a digital assistant powered by AI. Results indicate comparable levels of usability and perceived workload between the two conditions, with higher performance satisfaction in the assisted condition.

The future is bright and challenging: we envision a very complex combination of mixed reality (MR), artificial intelligence, and spatial computing as a smart integration of interconnected systems—seamlessly bridging both physical and virtual spaces. The upcoming generation of everyday wearable, embedded, and ubiquitous technology allows the ideation and validation of a wide range of next-generation context-aware, or intelligent MR user experiences. Realization of this vision is promising, especially given recent interdisciplinary advancements in key areas and growing interest in applying novel approaches in real-world application domains. This Special Issue aimed to attract ambitious contributions addressing this intelligent vision for MR leveraging, new computational interfaces, spatial computing, rich and diverse data sources, smart devices' extensive computation power, artificial intelligence, and sensing capabilities, as well as ubiquitous connectivity, computation resources, and storage offered by the existing communication and network infrastructure. We sought contributions that addressed new methods, technologies, approaches, applications, case studies, and industrial showcases, paving the way for novel insights, breakthrough designs, discussions, and developments in this rapidly evolving domain.

TOTTA outlines the spatial position and rotation guidance of a real/virtual tool (TO) towards a real/virtual target (TA), which is a key task in Mixed reality applications. The task error can have critical consequences regarding safety, performance, and quality, such as surgical implantology or industrial maintenance scenarios. The TOTTA problem lacks a dedicated study and it is scattered in different domains with isolated designs. This work contributes to a systematic review of the TOTTA visual widgets, studying 70 unique designs from 24 papers. TOTTA is commonly guided by the visual overlap -an intuitive, pre-attentive "collimation"feedback- of simple shaped widgets: Box, 3D Axes, 3D Model, 2D Crosshair, Globe, Tetrahedron, Line, Plane. Our research discovers that TO and TA are often represented with the same shape. They are distinguished by topological elements (e.g. edges/vertices/faces), colors, transparency levels, and added. shapes, widget quantity, and size. Meanwhile some designs provide continuous "during manipulation feedback"relative to the distance between TO and TA by text, dynamic color, sonification, and amplified graphical visualization. Some approaches trigger discrete "TA reached feedback"such as color alteration, added sound, TA shape change, and added text. We found the lack of golden standards, including in testing procedures, as current ones are limited to partial sets with different and incomparable setups (different target configurations, avatar, background, etc.). We also found a bias in participants: right-handed, young male, non-color impaired.

Keywords: 3D positioning , 3D user interface , tool to target manipulation , Virtual environments , widgets

—Mixed Reality (MR) is proven in the literature to support precise spatial dental drill positioning by superimposing 3D widgets. Despite this, the related knowledge about widget's visual design and interactive user feedback is still limited. Therefore, this study is contributed to by co-designed MR drill tool positioning widgets with two expert dentists and three MR experts. The results of co-design are two static widgets (SWs): a simple entry point, a target axis, and two dynamic widgets (DWs), variants of dynamic error visualization with and without a target axis (DWTA and DWEP). We evaluated the co-designed widgets in a virtual reality simulation supported by a realistic setup with a tracked phantom patient, a virtual magnifying loupe, and a dentist's foot pedal. The user study involved 35 dentists with various backgrounds and years of experience. The findings demonstrated significant results; DWs outperform SWs in positional and rotational precision, especially with younger generations and subjects with gaming experiences. The user preference remains for DWs (19) instead of SWs (16). However, findings indicated that the precision positively correlates with the time trade-off. The post-experience questionnaire (NASA-TLX) showed that DWs increase mental and physical demand, effort, and frustration more than SWs. Comparisons between DWEP and DWTA show that the DW's complexity level influences time, physical and mental demands. The DWs are extensible to diverse medical and industrial scenarios that demand precision.

Keywords: co-design , dentistry , Dynamic widgets , mixed reality , precise tool positioning , usability testing

The incorporation of immersive technologies such as virtual reality into medical settings, particularly in the context of preoperative planning, has the potential to enhance the accuracy and efficiency of surgical procedures. This preliminary study aims to develop an application specifically focused on the surgical intervention of tibial osteotomy. More precisely, this research integrates virtual reality to enhance preoperative planning, affording surgeons direct interaction with patient anatomical structures within a virtual environment, thereby facilitating precise decision-making. The manipulation of virtual three-dimensional reconstructions derived from medical imaging modalities such as computed tomography and magnetic resonance imaging is achievable through the integration of advanced modelling software and a virtual environment mimicking the operation room. This enables user navigation, object interaction, and outcome preservation. The study provides a detailed account of the design and implementation of a virtual reality application to simulate the surgical planning phase. This enables surgeons to visualise and execute the procedure intuitively. Despite promising progress toward virtual planning, further enhancements are necessary to fully supplant conventional methods. This includes the emulation of soft tissue interactions and the refinement of graft design processes within the virtual environment. Notwithstanding these challenges, this investigation demonstrates the potential of virtual reality to revolutionise preoperative planning paradigms, heralding a new era of immersive and efficacious surgical practices.

Keywords: Paediatric Orthopaedics , Surgical Planning , Virtual Reality

This work investigates the application of Augmented Reality (AR) in assisting industrial companies in facilitating the adoption of vertical-axis wind turbines for domestic use. Despite its growth, high upfront costs limit broader wind turbine adoption, especially in urban areas. AR is an emerging technology that can enhance the spread of renewable energy. In the wind energy sector, it is frequently integrated with data monitoring to support the decision-making process. In the literature, many studies address this topic, focusing primarily on improving performance in industry rather than evaluating application usability and user experience. Therefore, our study intends to explore an AR application's usability and user experience integrated with data monitoring to create customer service in the wind energy sector. We developed an application targeted to potential customers planning to adopt wind power to produce electricity within their homes. The aim is to encourage the spread of this renewable energy in an urban setting by facilitating the choice of the most suitable wind turbine based on the local weather conditions and the needs of the individual. To achieve this goal, specifically, we paid significant attention to data visualization, including the estimated power output of the chosen turbine, to better engage the user through customizable graphs rather than traditional textual information. We evaluated the usability and user experience of the implemented application through a user study involving 10 potential buyers of a wind turbine to be installed in their property. The results reveal that the usability is 'excellent' with a positive user experience evaluation, especially in attractiveness and novelty. Thus, our proposal turns out to be promising in creating customer service in the wind energy sector and other branches of renewable energy, such as solar-powered systems.

Keywords: augmented reality , data monitoring , data visualization , user acceptance , wind turbine