Virtual and augmented reality

In orthopedic surgery and maxillofacial there is a growing use of augmented reality (AR) as a technology to increase the visual perception of the surgeon in the operating room. The objective of this review is to analyze the state of the art in the use of AR for osteotomies, highlighting the advantages and the most-known open issues to be addressed in the future research. Scopus, Web of Science, Pubmed and IEEE Xplore databases have been explored with a keyword search, setting the time limits from January 2017 to January 2023, inclusive. Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines has been used in this review, focusing on anatomical districts, real-virtual environment interaction, advantaged and limitations of existing AR-based applications. 49 articles met the inclusion criteria and have been selected in the final analysis. For the sake of clarity, works have been grouped according to the anatomical district, but also the real-virtual environment interaction methodology was reported, as well as information regarding accuracy assessment. A Quality Function Deployment (QFD) has been used to assess the AR-based solutions with regards to the more traditional freehand (FH) and Patient Specific Template (PST) approaches. Finally, some suggestions to foster the AR-based solution adoption in osteotomies treatment have been drawn, considering the highlighted advantages and limitations of this technology. The AR resulted to meet the surgeons' needs more than other traditional approaches. Among the emerged advantages, AR can lead to a better surgical field accessibility, more flexible solutions and lower the management effort. Nonetheless, future research should address some well-known issues, among which the calibration time, the robustness of the tracking, and the HMDs discomfort.

Keywords: Augmented reality , Cutting guides , Maxillofacial surgery , Mixed reality , Orthopedic surgery , Osteotomies , Patient specific templates , Surgical guides

Virtual Reality (VR) technology has the potential to revolutionize engineering education by providing students with real-world simulation scenarios. This paper presents a user study to measure the effectiveness of VR applications in teaching technical representation and machine elements to first-year mechanical, energy, and aerospace engineering students. The VR application was designed to immerse the students in a virtual mechanical workshop where they could interact with a 3D model of a gearbox and access task instructions and technical drawings through an interactive whiteboard. A total of 448 students voluntarily participated in the one-hour lecture and were divided into groups of two to use the available 15 headsets. The study was assessed by analyzing the log files and their answers to a satisfaction and usability questionnaire. The results indicated that the VR application was an effective supplement to traditional instruction, as the students reported that it helped them better understand technical representation concepts and made the learning experience more enjoyable.

Keywords: Engineering education , Laboratory experience , User-centered evaluation , Virtual Reality

The 4.0 revolution is leading to increasingly automated, flexible, and intelligent manufacturing systems that require greater complexity to manage during maintenance and process control. In this context the optimization of the human-machine interaction plays a crucial important role in the design of modern industrial systems. Virtual Reality (VR) offers realistic simulation environments where users can be involved to replicate specific human tasks, detecting and solving problems before they occur. The paper proposes a human-centric digital design methodology that integrates VR technologies with human data analysis tools to support the design or redesign of complex industrial systems. Different wearable devices have been used to collect data about physical and mental user conditions to provide an early assessment of the operators' workload, while comparing different design solutions into the virtual space. An industrial use case related to the redesign of packaging automated machines was used to validate the proposed method and tools: a preliminary correlation between physiological parameters and machines interactions was found.

Keywords: digital simulation , ergonomics and human factors , Human data analysis , Human-centred design , Virtual Reality

The logistics industry involves various processes in the warehouse work environment on a daily basis, such as handling, storage, and packing. Therefore, workers are often engaged in manual activities such as pushing, pulling and lifting loads. These types of movements are repetitive and, together with adverse physical factors of the environment, they affect employees' health condition. The aim of this study is to test a proactive evaluation approach exploiting the potential of a professional virtual wall to simulate specific tasks performed in a warehouse to find the better solution in terms of working condition and productivity. The proposed framework includes the following steps: at first, it is required to design the virtual environment by means of 3D modelling tools; afterwards, simulation tests are performed by objectively assessing the physical working condition of the operator; finally, the valuable information are provided to improve the design of the workstation, based on the operator's ergonomics. The framework is modular and can be scaled to complex industrial environments.

Keywords: Ergonomics , Logistics , Virtual Reality , Virtual Wall

In the realm of smart manufacturing, Augmented Reality (AR) technology has gained increasing attention among researchers and manufacturers due to its practicality and adaptability. For this reason, it has been widely embraced in various industrial fields, especially for helping operators assemble products. Despite its widespread adoption, there is a debate in the research community about how effective AR is for improving user performance in assembly tasks, particularly when using handheld devices. These disparities can be attributed to differences in experimental approaches, such as the frequent use of qualitative methods, the inclusion of non-representative users, and the limited number of comprehensive case studies. In response to this, the paper delved into the benefits of AR applications, with a specific focus on measuring user performance and the cognitive workload perceived by users during assembly activities. To this end, an AR assembly guidance tool has been developed to assist users during assembly tasks, running on a mobile device, specifically a tablet, for freedom of movement and high portability. Experimentation involved the assembly of a comprehensive case study and a diverse user group, allowing the comparison representative users and experienced industrial operators. The results were promising, indicating that AR technology effectively enhances user performance during assembly-guided activities compared to conventional methods, particularly when users are unfamiliar with the task at hand. This study brings valuable insights by addressing previous research limitations and providing strong evidence of AR's positive impact on user performance in real-world assembly scenarios.

Keywords: AR Foundation , Assembly guidance , Augmented Reality , Hand-held devices , Industry 4.0 , User performance

In the context of Industry 4.0, the Operator 4.0 paradigm plays a crucial role in seamlessly integrating new digital technologies into smart factories. These technologies are designed to accommodate workers with varying skills, capabilities, and preferences. To promote the adoption of these innovative digital tools and ensure high user acceptance, it is essential to prioritize human factors and place workers at the heart of the development process. This can be achieved through structured design strategies, such as those employed in user-centered design approaches. Within this framework, this paper introduces an innovative Augmented Reality tool designed to facilitate and streamline real-time inspection activities performed by operators on assembled products at the workplace. The development of this tool was based on a user-centered design approach, which actively engaged end-users at different stages of design and testing phases. User studies were conducted using a real case study to evaluate the usability and user acceptance of the proposed solution. The results are encouraging, indicating a promising path forward for the effective and valid implementation of this AR tool in an industrial setting.

Keywords: Industrial augmented reality , Industry 4.0 , Operator 4.0 , Technology acceptance model , Usability , User-centered design

Over the past few years, the manufacturing industry has increasingly embraced Augmented Reality (AR) for inspecting real products, yet faces challenges in visualization modalities. In fact, AR content presentation significantly impacts user performance, especially when virtual object colors lack real-world context. Additionally, the lack of studies in this area compounds uncertainty about visualization effects on user performance in inspection tasks. This study introduces a novel AR recoloring technique to enhance user performance during industrial assembly inspection tasks. This technique automatically recolors virtual components based on their physical counterparts, improving distinctiveness. Experimental comparisons with AR experts and representative users, using objective and subjective metrics, demonstrate the proposed AR recoloring technique enhances task performance and reduces mental burden during inspection activities. This innovative approach outperforms established methods like CAD and random modes, showcasing its potential for advancing AR applications in manufacturing, particularly in the inspection of products.

Keywords: AR recoloring techniques , Assembly , Augmented Reality , Cognitive load , Color , Image color analysis , Industrial assembly inspection , Industry 4.0 , Inspection , Three-dimensional displays , Videos , Visualization , Visualization modalities

ugmented Reality (AR) is considered one of the key pillars of Industry 4.0 as it innovates the manufacturing process and promotes new ways of human-machine interaction. Among the main challenges influencing the implementation success of AR technology in industry, visualization modalities of AR content represent a crucial factor to be considered in relation to the application domain. In this regard, previous studies evaluating different AR-based content visualizations have demonstrated significant implications on worker performance and, in particular, specific factors such as colors applied to virtual information and uncontrolled environmental conditions, which can hinder the correct perception of AR content by users. Based on these considerations, the paper proposes a context-aware recoloring approach for AR-assisted maintenance applications for both indoor and outdoor environments. More specifically, the proposed approach is used to recolor virtual information on the basis of real work conditions in order to improve distinctiveness and visibility of virtual objects with respect to the real ones. By way of example, a use case is presented for exploring the benefits of the proposed solution in the field of maintenance activities. The outcomes provide the basis for further improvement and experimentation in a real industrial setting.

Keywords: Augmented Reality , Industrial Augmented Reality , Maintenance , Visualization modalities

Dental tool positioning is a challenging manual operation that requires precise 5DOF positioning of the drill -rotation around the tool is not influential- minimal error can result in grave consequences such as nerve and bone damage. Augmented Reality (AR) can help to assist tool positioning, but the widgets proposed in the literature are quasi-static, commonly using virtual 3D cylinders or lines to be visually collimated and eventually supported by color change. We draw our inspiration from the photographic viewfinder and propose a novel Virtual Stigmometer Widget (ViSti) to convey tool positioning error magnitude to the dentist. The widget slightly blurs the view and gradually focuses it while approaching the target (i.e., implant drilling position) till it is perfectly collimated. We conducted a within subjects experiment (N = 30) to compare our widget with the golden standard in 32 positioning tasks. NASA-TLX results demonstrated that our widget reduces frustration (−43%), and mental demand (−19%), and the user perceives a better performance (+8%). However, the expected tradeoff is effort (+34%), and physical (+33%) temporal demand (+4%). Our widget demonstrated to have a potential usage for dentistry and in further fields where 5DOF precision is crucial.

Keywords: 5DOF manipulation , Augmented Reality , Dental Implantology , Human Centered Design , Stigmometer

In an era characterized by the increasing complexity of products and the rapid turnover of the workforce across different companies, there is a growing need to invest significantly in quick and efficient training methods. Concurrently, the advancement of digitalization has rendered certain training practices anchored to paper-based materials obsolete. Numerous companies are directing their investments toward digital training, yet the optimal format to exploit the full advantages of digitalization remains unclear. This study undertakes a comparison of four distinct digital versions of the same training process with the aim of comprehending the tangible benefits. The findings indicate that to fully capitalize on the advantages of digital technology, a complete rethinking of training practices is necessary.

Keywords: virtual and augmented reality environments , virtual prototyping

Nowadays, there is a pressing need to incorporate sustainable practices and promote environmental awareness. The fashion industry, characterized by resource-intensive consumption, significantly contributes to global sustainability threats. The authors proposed the adoption of Virtual Reality (VR) to design and develop the Fashion Footprint application, an experience aimed at enhancing sustainability awareness and fostering behavioral change. The findings suggest that VR technology is valuable in promoting sustainability awareness and driving positive behavioral shifts in the fashion industry.

Keywords: behavioural design , sustainability , virtual reality (VR)

Using custom-made physical cutting guides in maxillofacial surgery presents several drawbacks, mainly related to design, time, and costs. Even if the literature provides several Augmented Reality (AR) solutions, they mostly display cutting planes and static assistance for repositioning. This work proposes a Mixed Reality (MR) application, deployed on Microsoft HoloLens2, to guide the surgeon in maxillofacial osteotomies and repositioning through intuitive and interactive digital content. A holographic panel allows selecting between the osteotomy and repositioning modes. With the first one, the cutting lines and the drilling points for the fixation plates are overlaid on the patient's skull. With the second one, three different kinds of feedback simultaneously guide the surgeon toward the correct final bone placement, step-by-step, according to the real-time fragment displacement. The MR app was tested by seven inexperienced subjects using a 3D-printed skull. Satisfactory results were obtained for maxillary osteotomy (i.e., less than 1.5 mm deviations) and relocation (i.e., the accuracy is about 0.7 mm and 0.6°).

Keywords: Augmented Reality , Healthcare Design , HoloLens 2 , Maxillofacial Surgery , Mixed Reality

The advancement of Augmented Reality (AR) technology has been remarkable, enabling the augmentation of user perception with timely information. This progress holds great promise in the field of interaction design. However, the mere advancement of technology is not enough to ensure widespread adoption. The user dimension has been somewhat overlooked in AR research due to a lack of attention to user motivations, needs, usability, and perceived value. The critical aspects of AR technology tend to be overshadowed by the technology itself. To ensure appropriate future assessments, it is necessary to thoroughly examine and categorize all the methods used for AR technology validation. By identifying and classifying these evaluation methods, researchers and practitioners will be better equipped to develop and validate new AR techniques and applications. Therefore, comprehensive and systematic evaluations are critical to the advancement and sustainability of AR technology. This paper presents a theoretical framework derived from a cluster analysis of the most efficient evaluation methods for AR extracted from 399 papers. Evaluation methods were clustered according to the application domains and the human–computer interaction aspects to be investigated. This framework should facilitate rapid development cycles prioritizing user requirements, ultimately leading to groundbreaking interaction methods accessible to a broader audience beyond research and development centers.

Keywords: augmented reality , evaluation framework , human–computer interaction , methods and tools , user evaluation , user testing

Surgical operations must be preformed by skilled and qualified personnel. Training is therefore of paramount importance to master enough skill to succeeds in the tasks without endangering the health of the patients. Combining different types of training activities, can help reaching the goal in a quicker and more effective way. In this scenario, Virtual Reality represents an interesting step, particularly thanks to the reduction in the cost of the equipment that have made their adoption affordable and their availability widespread. In this work, we present our advances in training lung operations with low-cost virtual reality simulation environments.

Keywords: Simulation , surgical operations , training , Virtual reality

The concept of an innovative human–machine interface and interaction modes based on virtual and augmented reality technologies for airport control towers has been developed with the aim of increasing the human performances and situational awareness of air traffic control operators. By presenting digital information through see-through head-mounted displays superimposed over the out-of-the-tower view, the proposed interface should stimulate controllers to operate in a head-up position and, therefore, reduce the number of switches between a head-up and a head-down position even in low visibility conditions. This paper introduces the developed interface and describes the exercises conducted to validate the technical solutions developed, focusing on the simulation platform and exploited technologies, to demonstrate how virtual and augmented reality, along with additional features such as adaptive human–machine interface, multimodal interaction and attention guidance, enable a more natural and effective interaction in the control tower. The results of the human-in-the-loop real-time validation exercises show that the prototype concept is feasible from both an operational and technical perspective, the solution proves to support the air traffic controllers in working in a head-up position more than head-down even with low-visibility operational scenarios, and to lower the time to react in critical or alerting situations with a positive impact on the human performances of the user. While showcasing promising results, this study also identifies certain limitations and opportunities for refinement, aimed at further optimising the efficacy and usability of the proposed interface.

Keywords: Air traffic control , Airport control tower , Augmented reality , Human machine interface , Multimodal interaction , Safety nets

Now a days new products are evaluated in the initial phases of product development in order to identify all design problems as early as possible before the prototype or manufacturing of the product. With advent growth in innovative technologies, simulation and testing of the human centered product designs are being performed using digital models & immersive to semi-immersive environments. There are numerous concepts and methods for this purpose, which have evidently proven to be effective solutions, such as Digital Human Modelling (DHM) or Human Model in the Loop simulations (HMITL), to facilitate ergonomic evaluations without interactions, and Virtual Reality (VR) technology to enable Human in the Loop (HIL) simulations in immersive environments along with interactions. These simulations are performed utilizing various software, which makes the data integration phase a complicated and time taking process. This paper elaborates on a new approach of adapting DHM analysis in a VR simulation for analyzing living space inside a 3D model using a single open-source platform, UNITY 3D. This process is expected to reduce the need for multiple programs during simulations, thereby reducing costs and time while providing uncomplicated visualizations. A use case scenario in view of assessing living space inside a novel aircraft lavatory design with baby table integration is also presented. This paper serves as a step towards inclusive design analysis and predicts its potential in various fields to study ergonomics and interaction of products at the early stage of product development.

Keywords: aircraft cabin design , Digital Human Modeling , Human Centered Design , product development , User Experience , Virtual Reality

In this work, we present a general control architecture for robotic systems dedicated to the remote handling of in-vessel components in fusion machines. This control architecture will be tested for the inspection and maintenance of the first wall components of the RFX-mod2 experiment, in the scope of the New Equipment For the Experimental Research and Technological Advancement for the Rfx Infrastructure (NEFERTARI) project. The architecture is split into low-level and high-level layers. The former is used for the low-level control of the robotics systems and to manage safety; the latter is used for the high-level planning and implements several sub-modules, such as a Virtual Reality (VR) environment for the visualisation of the robot's digital twin. Moreover, an Application Programming Interface (API) is intended to connect the two layers, and the communication between modules and layers is provided by the ROS2 framework. A typical usage of such a framework involves a human operator who is teleoperating the real robot, data from motor encoders are used as inputs for the dynamic model module. This module is used to compute the forward dynamics in real-time, providing an accurate simulation of the robot (digital twin) in the virtual environment.

Keywords: Control architecture , Digital twin , Remote handling , Robotics , ROS2

This paper introduces a unique, feature-based methodology for determining the best design features of a Virtual Reality Training System (VRTS). It offers a holistic approach by using a revised Technology Acceptance Questionnaire, which assesses design features from a user's perspective, rather than relying solely on objective measures of training effectiveness. We implemented the methodology in an experimental campaign, which involved simulating manual processes in two VR platforms with different features. Both novice and experienced VR users participated in the study, evaluating design features for perceived usefulness and ease of use. The findings provide insights into improving VRTS design based on user experience, highlighting the potential of our feature-based methodology.

Keywords: Design Features , Manual Layup , Technology Acceptance , User Experience , Virtual Reality , Virtual Training

This paper proposes a virtual reality-based dual-mode teleoperation architecture to assist human operators in remotely operating robotic manipulation systems in a safe and flexible way. The architecture, implemented via a finite state machine, enables the operator to switch between two operational modes: the Approach mode, where the operator indirectly controls the robotic system by specifying its target configuration via the immersive virtual reality (VR) interface, and the Telemanip mode, where the operator directly controls the robot end-effector motion via input devices. The two independent control modes have been tested along the task of reaching a glass on a table by a sample population of 18 participants. Two working groups have been considered to distinguish users with previous experience with VR technologies from the novices. The results of the user study presented in this work show the potential of the proposed architecture in terms of usability, both physical and mental workload, and user satisfaction. Finally, a statistical analysis showed no significant differences along these three metrics between the two considered groups demonstrating ease of use of the proposed architecture by both people with and with no previous experience in VR.

Keywords: control mode , digital twin , human-robot interface , man-machine systems , remote control , teleoperation , virtual reality

In response to the increasing number of traffic accidents, industrialized countries are investing in educational programs to raise awareness of safe driving. The most common type of intervention to reduce risky driving behaviors is the showing of traumatic experience videos related to the occurrence of traffic accidents caused by incorrect and high-risk driving habits. However, the effectiveness of this intervention depends on several factors, including how this content is experienced. Nowadays, thanks to Cinematic Virtual Reality (CVR) technologies, new modalities of video content fruition can be exploited. In this work, we implemented a CVR application to show 360° videos of traffic accidents and evaluate the effectiveness of CVR as an educational intervention tool to improve drivers' attitudes toward traffic safety. A sample of 86 users aged 18 to 56 years answered a questionnaire on Attitudes Toward Traffic Safety Scale (ATTSS) administered before and after the CVR experience. In addition, a user experience questionnaire (UEQ) was administered. The results analysis of ATTSS questionnaires shows that the CVR experience increases risk awareness by inducing greater attention to safety while driving. Finally, the results of the UEQ questionnaire reveal that users positively evaluate the experience of using the system from hedonistic, pragmatic, and usability perspectives.

Keywords: 360 video , Educational Intervention , Traffic Safety , Virtual Reality