Virtual and augmented reality

In the context of Industry 4.0, Operator 4.0 paradigm represents a key factor when dealing with the integration of new digital technologies into smart factories that are suited for workers with different skills, capabilities, and preferences. In this regard, to encourage the introduction of these new digital solutions and achieve high user acceptance, it is fundamental to consider human factors and put the worker at the center of the development process through the adoption of structured design strategies such as user-centered design (UCD) approaches. In this perspective, the paper proposes a novel Augmented Reality (AR) tool for supporting operators at the workplace, in real-time, while performing inspection activities on built products. The proposed tool has been developed according to a user-centered design approach by involving end-users in the various design and testing stages. Preliminary tests have been conducted with representative users on a real case study to assess the usability of the proposed solution. The outcomes are very encouraging and lead us towards further investigations for effective and valid implementation of this AR tool in an industrial scenario.

Keywords: Industrial augmented reality , Industry 4.0 , Operator 4.0 , Usability , User-centered design

Numerical simulations and Finite Element Analysis (FEA) have currently increased their applications in medical field for making preoperative plans to simulate the response of tissues and organs. Soft tissue simulations, such as colorectal simulations, can be adopted to understand the interaction between colon tissues and surrounding tissues, as well as the effects of instruments used in this kind of surgical procedures. This paper analyses through FEA the interaction between a surgical device and a colon tissue when it is fully clamped. Sensitivity analysis in the respect of the material mechanical behaviour, geometric approximation and the effect of thickness variation are investigated with the aim of setting up a virtual prototype of the surgical operation to aid mentoring and preliminary evaluation via haptic solutions. Through this investigation, the force feedback estimation that is necessary in many virtual-reality applications, may be estimated without discharging nonlinear effects that occur during clamping and that usually cannot be simulated efficiently to guarantee real-time solutions. Results are aligned with experimental data, confirming the reliability and right the set-up of FEA. Through them, the preliminary set-up of a haptic force feedback has been described and simulated through Simulink 3D animation, confirming the feasibility of the concept.

Keywords: FEA , Force feedback , Haptic device , Metamodeling , Surgical simulation , Virtual prototyping

In this work, we propose a Mixed Reality (MR) application to support laboratory lectures in STEM distance education. It was designed following a methodology extendable to diverse STEM laboratory lectures. We formulated this methodology considering the main issues found in the literature that limit MR's use in education. Thus, the main design features of the resulting MR application are students' and teachers' involvement, use of not distracting graphics, integration of traditional didactic material, and easy scalability to new learning activities. In this work, we present how we applied the design methodology and used the framework for the case study of an engineering course to support students in understanding drawings of complex machines without being physically in the laboratory. We finally evaluated the usability and cognitive load of the implemented MR application through two user studies, involving, respectively, 48 and 36 students. The results reveal that the usability of our application is “excellent” (mean SUS score 84.7), and it is not influenced by familiarity with Mixed Reality and distance education tools. Furthermore, the cognitive load is medium (mean NASA TLX score below 29) for all four learning tasks that students can accomplish through the MR application.

Keywords: augmented and virtual reality , distance education and online learning , improving classroom teaching , mixed reality , mobile learning

Although Virtual Reality Social Skills Training has proven its effectiveness in treating psychiatric disorders, this VR application field is still under-researched for two main reasons. The first one is the unavailability of low-cost VR technologies with sufficient computational capacity needed to render realistic Virtual Environments. The second one consists of the need for specialized VR application developers, usually far from the mental health research field. The recent diffusion of low-cost stereoscopic viewers and the introduction of easy and fast VR content authoring systems, such as Cinematic Virtual Reality (CVR), allow overcoming these limitations. CVR makes it possible to capture real scenes through 360 cameras, augment them with additional virtual objects, and finally immerse the user in these synthetic but highly immersive environments. We present the design and the features of the Entellect360 prototype -an innovative tool supporting the rehabilitation process of subjects affected by schizophrenia. It exploits CVR technology to create Virtual Environments aimed at the rehabilitation of psychiatric patients. The Entellect360 features allow for rehabilitation sessions and patient-performance data-collection even under conditions of social distancing. We also explain the experimental protocol and the validation procedure the prototype will undergo to assess its effectiveness.

Keywords: 360-degree virtual reality , Cognitive rehabilitation , Human-computer interaction , Mental health , Social skills training

Mixed Reality (MR) could help students in the understanding of complex concepts as well as increase their motivation in the learning process. In this work, our aim is to propose a MR application for the support of engineering students in the understanding of assembly drawings of complex machines. We presented the application of our design methodology for this case study. Then, based on the results of a user study with a sample of students, we tried to improve the usability and the user experience of the MR application, proposing an updated version. The usability of the revised application was in the range “good-excellent” (mean SUS score 77.0). We also presented the lessons learned in this case study, that can be a starting point for a renewal of consolidated didactic processes aiming at future application of MR in other STEM courses.

Keywords: Augmented and virtual reality , Distance education and online learning , Improving classroom teaching , Mixed Reality , Mobile learning

In the last years, interactive exhibitions based on digital technologies have become widely common, thanks to their flexibility and effectiveness in engaging visitors and creating memorable experiences. One of the topics in which digital technologies can be particularly effective is the communication of abstract concepts that are difficult for the human mind to imagine. An emblematic example is the astronomy discipline, which requires us to imagine and understand phenomena far away from our everyday life. In this paper, the authors present a research project, MARSS, in which digital technologies are used effectively to enhance the Users' Experience of the Museo Astronomico di Brera located in Milan. Specifically, the MARSS project aims at designing and developing a new digital journey inside the museum to allow different categories of visitors to enjoy the exhibition in an engaging and interactive way. The paper presents the design and development phases of the experience and its evaluation with users. The results of the evaluation indicate that the digital interactive experience is appreciated by users and is successful in translating the content of high scientific value into more engaging and easily understandable elements.

Keywords: augmented reality , cultural heritage , extended reality , interactive exhibitions , science museums , user experience

The issue of training operators in the use of machinery is topical in the industrial field and in many other contexts, such as university laboratories. Training is about learning how to use machinery properly and safely. Beyond the possibility of studying manuals to learn how to use a machine, operators typically learn through on-The-job training. Indeed, learning by doing is in general more effective, tasks done practically are remembered more easily, and the training is more motivating and less tiresome. On the other hand, this training method has several negative factors. In particular, safety may be a major issue in some training situations. An approach that may contribute overcoming negative factors is using Virtual Reality and digital simulation techniques for operators training. The research work presented in this paper concerns the development of a multisensory virtual reality application for training operators to properly use machinery and personal protective equipment (PPE). The context selected for the study is a university laboratory hosting manufacturing machinery. The application allows user to navigate the laboratory, to approach a machine and learn about how to operate it, and also to use proper PPE while operating a machine. Specifically, the paper describes the design and implementation of the application and presents the results of preliminary testing sessions.

The development of Virtual Reality in a wide range of field, including engineering related applications, has pushed towards the investigation of novel solutions that are able to take advantage of such new possibilities, while possibly trying to seamlessly integrate them within currently established workflows. Regarding conceptual sketching, which commonly represents one of the first activities taking place across Product Design development work-flows, there are examples of applications that allow to shift from the 2D layout of traditional drawing to a fully immersive 3D environment where the user is able to produce strokes in space by means of a set of natural gestures. Despite sounding extremely intuitive, this kind of approach also comes with potential issues: the lack of a supportive surface onto which the user can rely on to produce strokes with a high degree of precision while not feeling tired after prolonged sessions can be problematic. Based on these premises, a new hybrid approach is proposed: the user is still immersed in the Virtual Environment, but is able to make use of a traditional tablet device which lays on a physical desk in order to produce visible strokes in Virtual Reality, while having the possibility to simultaneously manipulate the position and the orientation of the scene thanks to a hand tracking device to break into the third dimen-sion. As designed, the application supports the generation of simple line strokes and few basic commands, but a thorough testing session is still needed to validate the solution and investigate on the necessary improvements.

Keywords: Conceptual design , Product design , Virtual reality

Design for Sustainability is a research area based on a multidisciplinary approach, which has become increasingly important in recent years. Among the several approaches to Design for Sustainability emerged in the past decades, great attention is paid to the “Design for sustainable behavior” approach, used to design products that can impact on users' behaviors, through embedded smart technologies, e.g., Internet of Things (IoT). In fact, IoT systems are able to "dialogue" with the users, supporting the identification of any misbehavior, and suggesting more sustainable ones. The authors identified the opportunity to design and develop AR interactive applications aiming to generate awareness about the impact of humans on Earth, make people reason about how they can directly contribute to limit the expansion of this problem and induce their behavioral change. The applications are meant to engage users in an active process of exploring and discovering informative contents and to foster them to elaborate a personal and critical vision and change their bad habits. Specifically, this paper presents two case studies about the design and development of Augmented Reality applications and IoT products to be used for supporting users towards more conscious food consumption in their daily life, in order to reduce food waste.

Keywords: Augmented reality , Design for behavioral change , Design for sustainable behavior , Food waste

The European Commission defined the new concept of Industry 5.0 meaning a more human-centric, resilient, and sustainable approach for the design of industrial systems and operations. A deep understanding of the work environment and organization is important to start analysing the working conditions and the resulting User eXperience (UX) of the operators. Also, the knowledge about users' needs and ergonomics is fundamental to optimize the workers' wellbeing, working conditions, and industrial results. In this context, the paper presents a strategy to effectively assess the UX of workers to promote human-centric vision of manufacturing sites, enhancing the overall sustainability of the modern factories. A set of non-invasive wearable devices is used to monitor human activities and collect physiological parameters, as well as questionnaires to gather subjective self-assessment. This set-up was applied to virtual reality (VR) simulation, replicating heavy duty work sequence tasks that took place in an oil and gas pipes manufacturing site. This approach allowed the identification of possible stressful conditions for the operator, from physical and mental perspectives, which may compromise the performance. This research was funded by the European Community's HORIZON 2020 programme under grant agreement No. 958303 (PENELOPE).

Keywords: Cognitive ergonomics , Human-centred design , Industry 5.0 , User experience , Virtual reality

The scientific literature highlights how Mixed Reality (MR) simulations allow obtaining several benefits in healthcare education. Simulation-based training, boosted by MR, offers an exciting and immersive learning experience that helps health professionals to acquire knowledge and skills, without exposing patients to unnecessary risks. High engagement, informational overload, and unfamiliarity with virtual elements could expose students to cognitive overload and acute stress. The implementation of effective simulation design strategies able to preserve the psychological safety of learners and the investigation of the impacts and effects of simulations are two open challenges to be faced. In this context, the present study proposes a method to design a medical simulation and evaluate its effectiveness, with the final aim to achieve the learning outcomes and do not compromise the students' psychological safety. The method has been applied in the design and development of an MR application to simulate the rachicentesis procedure for diagnostic purposes in adults. The MR application has been tested by involving twenty students of the 6th year of Medicine and Surgery of Università Politecnica delle Marche. Multiple measurement techniques such as self-report, physiological indices, and observer ratings of performance, cognitive and emotional states of learners have been implemented to improve the rigour of the study. Also, a user-experience analysis has been accomplished to discriminate between two different devices: Vox Gear Plus® and Microsoft Hololens®. To compare the results with a reference, students performed the simulation also without using the MR application. The use of MR resulted in increased stress measured by physiological parameters without a high increase in perceived workload. It satisfies the objective to enhance the realism of the simulation without generating cognitive overload, which favours productive learning. The user experience (UX) has found greater benefits in involvement, immersion, and realism; however, it has emphasized the technological limitations of devices such as obstruction, loss of depth (Vox Gear Plus), and narrow FOV (Microsoft Hololens).

Keywords: Augmented reality , Cognitive load , Medical education , Mixed reality , Simulation , Stress

Since custom-made 3D printed surgical guides for maxillofacial surgery are usually expensive, Augmented Reality (AR) can be efficiently employed to overcome the high costs. The proposed work aims to develop and test an AR application for different maxillofacial surgeries. The application consists in overlaying the cutting lines on the patient's mandible to guide the clinician during the procedure. It has been realized in Unity and preliminary tested with HoloLens 2 and a 3D printed mandible. Seven participants performed two consecutive trials. The mandible with the obtained surgical lines has been scanned after each test to digitally reconstruct the traced lines and compare them with the surgical lines previously designed. The results allowed the preliminary analysis of the developed AR system's accuracy and precision. Mean distances from the designed surgical guides showed good accuracy for the genioplasty (deviation error around 1.03 mm) and orthognathic surgery (deviation error around 1.27 mm), suggesting the applicability of HoloLens 2 for these kinds of surgery. On the contrary, the application was not suitable for the mandibular angle osteotomy (deviation error over 2.50 mm).

Keywords: Augmented reality , HoloLens 2 , Maxillofacial surgery , Surgical guides

In the industrial context, the wire harness represents a process with a high degree of manual work and significant customizability of final products. To reduce the workers' mental demand, the augmented reality (AR) can be an effective tool. The existing industrial applications mainly focus on manual assembly assistance, but few solutions have been developed for the wire harness. This paper tries to enhance such a process by supporting operators with an AR application that shows the work instructions and allows reducing their cognitive workload. The main goals are to propose a user-friendly and versatile tool and carry out a structured and complete evaluation of the user experience. Two experimental sessions were conducted in the laboratory by simulating the wire harness assembly and quality inspection. The use of the AR application is efficient and effective especially when instructions far from the workstation are considered as a reference.

Keywords: Augmented reality , Hololens 2 , Human-centered manufacturing , Industry 4.0 , Wire harness assembly

Human factors studies are becoming more and more crucial in the automotive sector due to the need to evaluate the driver.s reactions to the increasingly sophisticated driving-assistant technologies. Driving simulators allow performing this kind of study in a controlled and safe environment. However, the driving simulation.s Level of Detail (LOD) can affect the users. perception of driving scenarios and make an experimental campaign.s outcomes unreliable. This paper proposes a study investigating possible correlations between driver.s behaviors and emotions, and simulated driving scenarios. Four scenarios replicating the same real area were built with four LODs from LOD0 (only the road is drawn) to LOD3 (all buildings with real textures for facades and roofs are inserted together with items visible from the road). 32 participants drove in all the four scenarios on a fixed-base driving simulator; their performance relating to the vehicle control (i.e., speed, trajectory, brake and gas pedal use, and steering wheel), their physiological data (electrodermal activity, and eye movements), their subjective perceptions, opinions and emotional state were measured. The results showed that drivers. behavior changes in a very complex way. Geometrical features of the route and environmental elements constrain much more driving behavior than LOD does Emotions are not affected by LODs. Generally, different signals showed different correlations with the LOD level, suggesting that future studies should consider their measures while modeling the virtual scenario. It is hypothesized that scenario realism is more relevant during leisurely environmental interaction, whilst simulator fidelity is crucial in task-driven interactions.

Keywords: driver behavior , Driving simulator , environmental psychology , eye tracker , level of detail (LOD) , simulation reliability

The paper presents the human-centered interdisciplinary methodology “SafeMob - Safe Mobility Experiential and Environmental Assessment” developed by the authors. The method combines Objective Safety (OS) with Perceived Safety (PS) to evaluate the performance of mobility solutions. The interrelation of data from the person (physio/psycho) and the environment (road, vehicle, and the surrounding or interacting context - including flows, buildings, and weather) makes the methodology holistic and interdisciplinary. The final goal is to provide a ‘Decision Support System' for stakeholders in the mobility field, the automotive sector, and the urban planning area. The paper describes the overall theoretical approach and a specific case study application using a car simulator. Emotional reactions of users, driving through the same virtual scenario with different Level of Details, are assessed to gather information about the perceived safety of the environment.

Keywords: Drive simulator , Emotions , Level of Detail , Physiological measures , Virtual reality

Projector-based Spatial Augmented Reality (P-SAR) is a technology that allows to alter the external appearance of a physical object by means of an almost infinite variety of computer-generated contents. Thanks to the adoption of colored lights, a projected image acquires coherent spatial properties with respect to the data it represents; this has been demonstrated to facilitate the users' interpretation of complex information. The current paper presents the development of a P-SAR system aiming at supporting the real-time visualization and the inspection of engineering simulation results. Particular attention is paid on detailing the algorithms necessary for the generation of the color maps to be displayed on the prototype's surface. These are interpolated starting from a discrete array of output data coming from a generic simulation to resemble the configuration of sensors commonly adopted in real experimental setups. An illustrative case study applied to CFD analysis is finally discussed to show the applicability of such immersive environments in engineering fields that require to perform testing activities with equipment like wind tunnels.

Keywords: Engineering analysis , Simulation , Spatial augmented reality

Worldwide, stroke is the third cause of disability. The majority of people affected by this disease cannot perform activities of daily living. Bringing the therapy to the patients' home is complex, and in literature, there are still open challenges to face. Starting from therapists' and patients' needs, this paper describes a possible solution: HANDY, a rehabilitative active hand exoskeleton for post-stroke patients. With a desktop application, they perform three different types of exercises: passive, active and based on activities of daily living. They can also control the exoskeleton themselves in a serious-game approach with a leap motion controller. We evaluated our method with patients at the Villa Beretta rehabilitative center. Preliminary results from the session about comfort, usability and willingness to utilize the system are promising.

Keywords: Additive manufacturing , CAD modeling , Hand exoskeleton , Interactive applications , Stroke

Successful aircraft cabin design depends on how the different stakeholders are involved since the first phases of product development. To predict passenger satisfaction prior to the manufacturing phase, human response was investigated in a Virtual Reality (VR) environment simulating a cabin aircraft. Subjective assessments of virtual designs have been collected via questionnaires, while the underlying neural mechanisms have been captured through electroencephalographic (EEG) data. In particular, we focused on the modulation of EEG alpha rhythm as a valuable marker of the brain's internal state and investigated which changes in alpha power and connectivity can be related to a different visual comfort perception by comparing groups with higher and lower comfort rates. Results show that alpha-band power decreased in occipital regions during subjects' immersion in the virtual cabin compared with the relaxation state, reflecting attention to the environment. Moreover, alpha-band power was modulated by comfort perception: lower comfort was associated with a lower alpha power compared to higher comfort. Further, alpha-band Granger connectivity shows top-down mechanisms in higher comfort participants, modulating attention and restoring partial relaxation. Present results contribute to understanding the role of alpha rhythm in visual comfort perception and demonstrate that VR and EEG represent promising tools to quantify human–environment interactions.

In the last two decades a huge number of interactive and collaborative applications of Virtual Environments for designing products has been proposed. Such applications have been recommended as tools to implement Human Centered Design Approach in experiments where potential users are involved in participatory design sessions before going for production. In this kind of experiments, we observe that users are mainly involved in the validation of solutions previously elaborated by designers while in the last decade the most innovative approach in the creation of solutions seems to be moving from a user centered design to a co-design or co-creation process. Thus, it is essential to have a platform where the elements of co-creation can be fulfilled in building a successful project. The purpose of this paper is to outline the concept of co-creation and the significance of co-creation platforms alongside of proposing innovative tools for building the co-creative environments. The paper aims to layout a classification of the Extended Reality (XR) tools currently available and of their functionality as valuable means to actually embed co-design and co-creation concepts in Virtual Environments evaluating the advantages that this can bring to Industry through field studies. A review of innovative solutions like Virtual, Augmented and Mixed reality technologies is examined and drawn towards the requirements of the concept through a literature research. Additionally, a co-creative environment for designing aircraft cabin interiors is conceived and discussed a with company representative.

Keywords: Co-creation , Collaboration , Extended reality , Interaction platforms

The lifecycle of a product is getting shorter in today's market realities. Latest developments in the industry are heading towards achieving products that are easy to recycle, by developing further technological advances in raw materials ought to include input from End of Life (EOL) products so a reduction of natural harm could be achieved, hence reducing the overall production environmental footprint. Therefore, the approach taken as a design for environment, a key request nowadays in order to develop products that would ease the reverse manufacturing process leading to a more efficient element recycling for later use as spare parts or remanufacturing. The methodology proposed compares three probable disassembly sequences following a comparison of literature-found procedures between genetic algorithms and as a “state space search” problem, followed by a hybrid approach developed by the authors. Time and evaluation of these procedures reached to the best performing sequence. A subsequent augmented reality disassembly simulation was performed with the top-scored operation sequence with which the user is better able to familiarize himself with the assembly than a traditional paper manual, therefore enlightening the feasibility of the top performing sequence in the real world.

Keywords: closed loop , Disassembly , genetic algorithms , metaheuristic method , optimization , particle swarm optimizer