Virtual and augmented reality

Point cloud 3D models are becoming more and more popular thanks to the spreading of scanning systems employed in many fields, like autonomous vehicles and robotics. When used for rendering purposes, point clouds are usually displayed with their original color acquired at scan time, without considering the lighting condition of the scene where the model is placed. This leads to a lack of realism in many contexts, especially in the case of animated point clouds employed in eXtended Reality applications where it would be desirable to have the model reacting to incoming light and integrating with the surrounding environment. This paper proposes the application of Physically Based Rendering (PBR), a rendering technique widely used in Real-Time Computer Graphics applications, to animated point cloud models for reproducing specular reflections, and achieving a photo-realistic and physically accurate look under any lighting condition. Firstly, we consider the extension of commonly used animated point cloud formats, to include normal vectors, and PBR parameters (such as roughness and Metalness), as well as the encoding of the animated environment maps required by the technique. Then, an animated point cloud model is rendered with a shader implementing the proposed PBR method. Finally, the proposed PBR pipeline is compared to traditional renderings of the same point cloud obtained with commonly used shaders, under different lighting conditions in different environments. It will be shown how, using the proposed PBR method, the point cloud better integrates visually with its surroundings. Moreover, it will be shown that using this rendering technique it is possible to render different materials, by exploiting the features of PBR and the encoding of the surrounding environment.

Keywords: Cloud computing , Lighting , Rendering (computer graphics) , Signal encoding

Industry 4.0 will bring not only transformation to the manufacturing technologies but also to the profile of the workforce. Education system should be revised to prepare the future graduates embracing the knowledge of the ongoing revolution. Initiatives on the modification of curriculum and tools to deliver the concepts of Industry 4.0 must be taken. This paper examines limitations of current engineering tools and proposes a virtual reality (VR)-based learning platform to support the teaching and learning activity for Industry 4.0, focusing on the design of Human–robot Collaboration (HRC). The development process began with the identification of the general Intended Learning Outcomes (ILOs) which were extracted from the review of the HRC design issues, followed by the evaluation of the current teaching tool. From these results, the new requirements of the tools to achieve ILOs are redefined. The implementation of the system is demonstrated to show the feasibility of the proposed learning platform. This workflow can serve as initial development of another learning platform for the other innovative concepts that form the building blocks of Industry 4.0.

Keywords: Human–robot collaboration , Industry 4.0 , Learning factory , Virtual reality

Virtual Reality (VR) is one of the major technology trends right now and will increase much more in the future. This technology promotes a new way of interaction, communication, and productivity. This paper aims to allow students with special needs to work and interact with Google Earth using VR. The theoretical framework used, the International Classification of Functioning, Disability, and Health (ICF) perspective and checklists, allows us to identify the best way to propose an interactive and funny learning process for students with Special Educational Needs (SEN). By means of ICF, the definition of the proper technological tools and of the main steps in which the learning process can be divided is possible. This approach, indeed, leads to the translation of cognitive tasks that can result too complex for students with cognitive fragilities into concrete experiences more easily to do for all, promoting inclusion and equity in our modern digital society. In the present work, the inclusive VR-based technology is tested on a group of 10 students with SEN in Google Earth environment, and the final outcomes in terms of system usability are presented.

Keywords: Autism , Down Syndrome , Extended Reality , Inclusive Education , Neurodevelopmental Disorder , Special Education , Special Educational Needs , Training

In the recent times the concept of Co-creation, involving stakeholders, customers, and partners in the design, development, and delivery of products and services, has gained importance in various industries, including aviation. This PhD research paper presents the concept of cocreation in the aviation sector, focusing on its key elements and the potential of extended reality (XR) technologies as enablers. The paper highlights collaboration, interaction, and user experience as the essential elements of co-creation and discusses existing co-creation platforms and tools. Traditional co-creation platforms have limitations, prompting the proposal to leverage XR technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) as future co-creative platforms. These immersive technologies offer the potential to enhance collaboration, visualization, and engagement in a virtual environment. The study explores how XR tools can aid digital co-creation methods and whether XR systems can induce effective co-creation sessions. In this context, the research develops a multi-user real-time XR environment that fosters collaborative and interactive co-creation aiming to enhance engagement, communication, and idea generation. This paper includes use-case scenarios for adapting such environments in view of investigating the potential of XR tools to improve the quality and novelty of ideas, reduce communication challenges, and decrease time and costs in the aviation industry's co-creation process.

Keywords: Aviation Industry , Co-creation , Digital Methods , Extended Reality

This work analyzes the impact of subjective factors on users' immersive experience, while using glove-based handling devices as input system. A user-study has been conducted on a Virtual Training framework, evaluating usability and mental effort perceived with such devices.

Keywords: Frustration , Gloves , Handling devices , Usability , User study , Virtual Training

The creation and management of content are among the main open issues for the spread of Augmented Reality. In Augmented Reality interfaces for procedural tasks, a key authoring strategy is chunking instructions and using optimized visual cues, i.e., tailored to the specific information to convey. Nevertheless, research works rarely present rationales behind their choice. This work aims to provide design guidelines for the localization of in-view and not occluded components, which is recurrent information in technical documentation. Previous studies revealed that the most suited visual cues to convey this information are auxiliary models, i.e., abstract shapes that highlight the space region where the component is located. Among them, 3D arrows are widely used, but they may produce ambiguity of information. Furthermore, from the literature, it is unclear how to design auxiliary model shapes and if they are affected by the component shapes. To fill this gap, we conducted two user studies. In the first study, we collected the preference of 45 users regarding the shape, color, and animation of auxiliary models for the localization of various component shapes. According to the results of this study, we defined guidelines for designing optimized auxiliary models based on the component shapes. In the second user study, we validated these guidelines by evaluating the performance (localization time and recognition accuracy) and user experience of 24 users. The results of this study allowed us to confirm that designing auxiliary models following our guidelines leads to a higher recognition accuracy and user experience than using 3D arrows.

Keywords: Augmented reality , Authoring , Auxiliary model , Localization

Mixed Reality (MR) technology has gained popularity in the manufacturing industry over the last few decades. In the maintenance field, it helps workers in document analysis and decision-making during operations. While numerous studies have demonstrated the empirical benefits of MR, few have examined its applicability in a real industrial context. Among them, many focus on user performance. The novelty of our work is to evaluate mainly the acceptance of MR technology by experienced technicians using smart glasses for maintenance tasks in a real industrial context. We conducted a user study involving 7 workers who were asked to follow the instructions provided by an MR Technical Documentation (MRTD) developed considering authoring guidelines already validated in the literature but only in a laboratory context. The industrial operators performed both navigation tasks within the menu of the MRTD and a disassembly task, following the instructions displayed on the smart glass. The evaluation in the real working scenario showed that MRTD is highly appreciated by workers even if they have no previous experience with smart glasses. These results allow us to confirm the goodness of the design guidelines followed for the authoring of the MRTD.

Keywords: Industrial Operator Support , Mixed Reality , Smart Glasses , Technical Documentation , User Acceptance

The biophilic design of virtual workplaces consists of introducing additional natural elements in the virtual environment (VE) with respect to those needed for the main task. It has received increasing attention in recent years, and an increasing number of studies are showing that exposure to biophilic elements in an immersive VE produces positive effects on human well-being. However, in the literature, there are no guidelines about the setting of the different variables of biophilic design in a VE. In this work, we investigated one of these variables, i.e., the effect of the animations of biophilic elements, formulating the following research question: how do animations of biophilic virtual elements affect user attention while performing a working task? We carried out an experiment in which users performed two levels of difficulty of the n-Back cognitive test in three VEs: two identical biophilic VE, one with all static elements and another one in which some of these elements were animated, and an additional empty VE used as a baseline. Performance measurements, eye-tracking measurements, and subjective measurements were collected from 24 users. We found that introducing animations of biophilic elements does not affect user performance in the cognitive task. Furthermore, we found that users' gaze remains fixed on the main task for more than 94% of the time. However, the use of animations does not bring added value to users in terms of user experience, whereas it causes an increase in perceived distraction and mental effort.

Keywords: Animations , Biophilic Design , Positive Computing , Virtual Reality

Immersive audio rendering techniques allow for generating a 3D scenario where the listener can perceive the sound from all directions. An important aspect of these approaches is the subjective perception of the listener and how these types of systems are perceived from the emotional point of view and how they can influence the listener's mood. In this context, a deep investigation of immersive sound perception considering subjective perception in terms of flowing emotion is performed. Starting from a 4-channels immersive audio system and an emotion-aware system based on the analysis of the user's facial expressions, several experiments have been performed to investigate a correlation between immersive perception and the listener's emotions.

Keywords: emotion recognition , immersive audio , subjective perception

The use of eXtended Reality (XR) technologies, including augmented reality (AR), virtual reality (VR), and mixed reality (MR), has become increasingly popular in museums to enhance the visitor experience. However, the impact of XR technologies on Learning Performance in the context of archeological museums needs to be better understood. This study aims to investigate the relationships between Usability, Presence and Learning Performance by developing XR experiences showcasing archeological artefacts and conducting user testing to evaluate their effectiveness. A laboratory test is conducted to compare a VR application with a mobile AR one, presenting the digital models of five archeological findings. Descriptive statistics are used to compare the two case studies, providing valuable insights into the impact of XR technologies on the visitor experience from a learning perspective. The study confirms that Usability has a more significant effect on learning than Presence and can help designers and museum managers better understand the factors contributing to a successful XR experience. The findings suggest that while Presence is an important factor in improving visitors' experience, Usability should be the priority when designing XR experiences for museums.

Keywords: Cultural heritage , Technological Benchmarking , XR Technologies

In the last decade, museums and exhibitions have benefited from the advances in Virtual Reality technologies to create complementary virtual elements to the traditional visit. The aim is to make the collections more engaging, interactive, comprehensible and accessible. Also, the studies regarding users' and visitors' engagement suggest that the real affective state cannot be fully assessed with self-assessment techniques and that other physiological techniques, such as EEG, should be adopted to gain a more unbiased and mature understanding of their feelings. With the aim of contributing to bridging this knowledge gap, this work proposes to adopt literature EEG-based indicators (valence, arousal, engagement) to analyze the affective state of 95 visitors interacting physically or virtually (in a VR environment) with five handicraft objects belonging to the permanent collection of the Museo dell'Artigianato Valdostano di Tradizione, which is a traditional craftsmanship museum in the Valle d'Aosta region. Extreme Gradient Boosting (XGBoost) was adopted to classify the obtained engagement measures, which were labeled according to questionnaire replies. EEG analysis played a fundamental role in understanding the cognitive and emotional processes underlying immersive experiences, highlighting the potential of VR technologies in enhancing participants' cognitive engagement. The results indicate that EEG-based indicators have common trends with self-assessment, suggesting that their use as ‘the ground truth of emotion' is a viable option.

Keywords: craftsmanship , cultural heritage , EEG , user engagement , Virtual Reality , XGBoost

In recent decades industrial development has led to increasingly sophisticated machinery and systems, which require complex maintenance routines. Consequently, maintenance operators may not have the sufficient skills to perform recovery procedures properly and quickly, so that the need of assistance from the manufacturer's after-sales service or companies specialized in maintenance services. Such actions usually lead to very long recovery times, high maintenance costs, and a temporary drop in production. In this scenario, we should consider that Industry 4.0 is making available innovative technologies, such as Augmented Reality (AR), suitable for improving the skills and competencies of operators without burdening their cognitive load, and consequently wellbeing. However, technologies must be selected, designed, and used according to the users' needs to be effective and useful. The paper presents a user experience (UX)-driven methodology for designing user-centric AR applications for complex maintenance procedures. The methodology was applied to a real industrial case concerning the management of CNC machines in a plant producing tractors components, where a smartphone-based AR application was designed and tested with users. The satisfactory results highlighted the potential benefits of AR in industry and specifically in maintenance.

Keywords: Augmented Reality , Industry 4.0 , Maintenance , Operator 4.0 , User experience design

Nowadays, logistics is increasingly becoming a crucial process in the modern factories, and numerous companies are paying growing attention to the continuous improvement of quality and efficiency of logistic processes. Although Industry 4.0 concept introduced new digital technologies and led to high degree of automation of industrial and logistic systems, many tasks still rely on human work, especially in the warehouse. As a consequence, the consideration of the operators' needs is definitely a key topic to guarantee high quality work performance, according to Industry 5.0 paradigm. In this context, the quality of the training procedures is fundamental to speed up the knowledge transfer and empower operators through the use of digital technologies. This paper presents the development of a novel training methodology based on Augmented Reality (AR), promoting a learning-by-doing approach to enhance both operators' satisfaction and process efficiency and flexibility. A dedicated AR application has been developed for a spare parts warehouse of a large enterprise operating in the agriculture and construction vehicles sector, according to a User-Centered Design (UCD) approach. Results from field testing showed how the AR application is positively rated by the operators, guaranteeing high user satisfaction, and limiting the training effort.

Keywords: Augmented Reality (AR) , Training , User-Centered Design (UCD) , warehouse Operation

With technological innovation and advancements, especially in autonomy, battery and digitization, the future of air transport and mobility is transiting towards a broader spectrum of Advanced Air Mobility (AAM) and Urban Air Mobility (UAM). UAM envisions safer, faster, and more sustainable air mobility for smarter cities and urban environments including passenger transport and goods delivery. Nevertheless, this concept is still considered extremely breakthrough and several technological and operational aspects are mostly undefined. In this context, a comprehensive approach to AAM/UAM may be to adapt cutting-edge technologies in developing sustainable framework and Human-Machine Interfaces (HMIs) in order to realize the challenges, benefits, and conditions of such transport system in advance for future safer, more reliable and globally approved operations. One of the technologies that can contribute to accelerate advancements through human centred simulating UAM processes and operations is XR (eXtended Reality). This paper presents the early steps of a multidisciplinary study performed under the framework of PNRR (Piano Nazionale di Ripresa e Resilienza) and MOST (Centro Nazionale Mobilità Sostenibile) project in analyzing the perspectives of XR based HMIs for UAM paradigm and potential AAM/UAM use case scenarios that can be simulated with XR in view of attaining efficient and effective future solutions. Furthermore, the work introduces the state-of-the-art overview on XR facilitated UAM applications and considers prospective potential use cases that can be developed through PNRR research study in demonstrating XR as an enabling technology in promising areas of the UAM framework. © 2023, Association of American Publishers.

Keywords: Advanced Air Mobility , Extended Reality , Human Machine Interfaces , Immersive Technologies , U-Space , Unmanned Aerial Systems , Urban Air Mobility

This work proposes an Augmented Reality (AR) application designed for HoloLens 2 which allows human operators, without particular experience or knowledge of robotics, to easily interact with collaborative robots. Building on the application presented in a previous work of the authors, the novel contributions are focused on a bi-directional interaction that manages the exchange of data from the robot to the human operator and, in the meantime, the flow of commands in the opposite direction. More in detail, the application includes the reading of the robot state, in terms of joint positions, velocities and torques, the visualization of the workspace and the generation and manipulation of the end-effector trajectory by directly moving a set of way-points displayed in the AR environment. Finally, the trajectory feasibility is verified and notified to the user by taking into account the workspace limits. A usability study of the AR platform has been conducted involving 45 participants with different ages and expertise in robot programming and Extended Reality (XR) platforms, comparing two programming methods: a classical kinesthetic teaching interface, provided by the Franka Emika Panda cobot, and the presented AR platform. Participants have reported the effectiveness of the proposed platform, experiencing less physical demand and higher intuitiveness and usability.

Keywords: augmented reality , head-mounted display , human-robot interaction , robotics

More than ever, precision surgery is making its way into modern surgery for functional organ preservation. This is possible mainly due to the increasing number of technologies available, including 3D models, virtual reality, augmented reality, and artificial intelligence. Intraoperative surgical navigation represents an interesting application of these technologies, allowing to understand in detail the surgical anatomy, planning a patient-tailored approach. Automatic superimposition comes into this context to optimally perform surgery as accurately as possible. Through a dedicated software (the first version) called iKidney, it is possible to superimpose the images using 3D models and live endoscopic images during partial nephrectomy, targeting the renal mass only. The patient is 31 years old with a 28 mm totally endophytic right-sided renal mass, with a PADUA score of 9. Thanks to the automatic superimposition and selective clamping, an enucleoresection of the renal mass alone was performed with no major postoperative complication (i.e., Clavien–Dindo < 2). iKidney-guided partial nephrectomy is safe, feasible, and yields excellent results in terms of organ preservation and functional outcomes. Further validation studies are needed to improve the prototype software, particularly to improve the rotational axes and avoid human help. Furthermore, it is important to reduce the costs associated with these technologies to increase its use in smaller hospitals.

Keywords: 3D models , artificial intelligence , augmented reality , kidney cancer , robotics

Objective: To evaluate the accuracy of our new three-dimensional (3D) automatic augmented reality (AAR) system guided by artificial intelligence in the identification of tumour's location at the level of the preserved neurovascular bundle (NVB) at the end of the extirpative phase of nerve-sparing robot-assisted radical prostatectomy. Methods: In this prospective study, we enrolled patients with prostate cancer (clinical stages cT1c–3, cN0, and cM0) with a positive index lesion at target biopsy, suspicious for capsular contact or extracapsular extension at preoperative multiparametric magnetic resonance imaging. Patients underwent robot-assisted radical prostatectomy at San Luigi Gonzaga Hospital (Orbassano, Turin, Italy), from December 2020 to December 2021. At the end of extirpative phase, thanks to our new AAR artificial intelligence driven system, the virtual prostate 3D model allowed to identify the tumour's location at the level of the preserved NVB and to perform a selective excisional biopsy, sparing the remaining portion of the bundle. Perioperative and postoperative data were evaluated, especially focusing on the positive surgical margin (PSM) rates, potency, continence recovery, and biochemical recurrence. Results: Thirty-four patients were enrolled. In 15 (44.1%) cases, the target lesion was in contact with the prostatic capsule at multiparametric magnetic resonance imaging (Wheeler grade L2) while in 19 (55.9%) cases extracapsular extension was detected (Wheeler grade L3). 3D AAR guided biopsies were negative in all pathological tumour stage 2 (pT2) patients while they revealed the presence of cancer in 14 cases in the pT3 cohort (14/16; 87.5%). PSM rates were 0% and 7.1% in the pathological stages pT2 and pT3 (<3 mm, Gleason score 3), respectively. Conclusion: With the proposed 3D AAR system, it is possible to correctly identify the lesion's location on the NVB in 87.5% of pT3 patients and perform a 3D-guided tailored nerve-sparing even in locally advanced diseases, without compromising the oncological safety in terms of PSM rates.

Keywords: Artificial intelligence , Augmented reality , Prostate cancer , Radical prostatectomy , Robotics

This article presents a software platform to design serious games for the rehabilitation of severe memory loss by means of virtual reality (VR). In particular, the focus is on retrograde amnesia, a condition affecting patient's quality of life usually after brain stroke. Currently, the standard rehabilitation process includes showing pictures of familiar environments to help memory recover, while the proposed rehabilitation solution aims at developing patient-specific serious games for memory loss starting from 3D scanning of familiar environments. The Occipital Structure sensor and the Skanect application have been used for the virtualization of the real objects and the environment. Instead of following the traditional approach to design a video game during which the game logic is specifically developed for a virtual scene and the software code is not meant to be recombined, a modular procedure has been designed using Unity to interface the virtual objects of each acquired environment without modifying the game logic. In addition, the developed solution makes available a set of software modules for patient's monitoring and data management to automatically generate medical reports, which can be easily connected to each new serious game. A test has been performed to assess the main features of the VR platform and its usability recruiting ten testers. Medical personnel evaluated positively the platform, and they highlighted the importance of objective data to improve the ecological validity of the cognitive rehabilitation for retrograde amnesia.

Keywords: Neurological disorders , Serious game , Virtual reality

This work proposes a novel virtual reality system which makes use of wearable sensors for testing and validation of cooperative workplaces from the ergonomic point of view. The main objective is to show, in real time, the ergonomic evaluation based on a muscular activity analysis within the immersive virtual environment. The system comprises the following key elements: a robotic simulator for modeling the robot and the working environment; virtual reality devices for human immersion and interaction within the simulated environment; five surface electromyographic sensors; and one uniaxial accelerometer for measuring the human ergonomic status. The methodology comprises the following steps: firstly, the virtual environment is constructed with an associated immersive tutorial for the worker; secondly, an ergonomic toolbox is developed for muscular analysis. This analysis involves multiple ergonomic outputs: root mean square for each muscle, a global electromyographic score, and a synthetic index. They are all visualized in the immersive environment during the execution of the task. To test this methodology, experimental trials are conducted on a real use case in a human–robot cooperative workplace typical of the automotive industry. The results showed that the methodology can effectively be applied in the analysis of human–robot interaction, to endow the workers with self–awareness with respect to their physical conditions.

Keywords: Cooperative workplace , Ergonomic analysis , Human– robot physical interaction , Virtual reality , Wearable sensors

As one of the most promising technologies for next-generation mobile platforms, Augmented Reality (AR) has the potential to radically change the way users interact with real environments enriched with various digital information. To achieve this potential, it is of fundamental importance to track and maintain accurate registration between real and computer-generated objects. Thus, it is crucially important to assess tracking capabilities. In this paper, we present a benchmark evaluation of the tracking performances of some of the most popular AR handheld devices, which can be regarded as a representative set of devices for sale in the global market. In particular, eight different next-gen devices including smartphones and tablets were considered. Experiments were conducted in a laboratory by adopting an external tracking system. The experimental methodology consisted of three main stages: calibration, data acquisition, and data evaluation. The results of the experimentation showed that the selected devices, in combination with the AR SDKs, have different tracking performances depending on the covered trajectory.

Keywords: Apple ARKit , benchmarking , Google ARCore , mobile augmented reality , simultaneous localization and mapping (SLAM) , tracking accuracy