Virtual and augmented reality

In this work, the design and the experimental characterization of a soft gripper which takes inspiration from the octopus tentacle is presented. The system is a single silicone-based pneumatic soft actuator which is able to perform bending motion thanks to its particular geometry and, due to the presence of an embedded suction cup, it can provide enhanced adhesion to the object surface of the goods to be grasped. Firstly, the design aspects as well as the fabrication process of the single-tentacle soft gripper are described. Then, its performances in terms of bending behavior, static deflection, shape variation under applied load and detachment force of the suction cup are evaluated. The experiments include grasping tests of spherical payloads with different weights. This gripper might have impact in the grasping and manipulation of delicate products like mozzarella in the agrifood field.

Keywords: soft grippers , soft robotics , tentacle gripper

CAD-to-XR is the workflow to generate interactive Photorealistic Virtual Prototypes (iPVPs) for Extended Reality (XR) apps from Computer-Aided Design (CAD) models. This process entails modelling, texturing, and XR programming. In the literature, no automatic CAD-to-XR frameworks simultaneously manage CAD simplification and texturing. There are no examples of their adoption for User-Centered Design (UCD). Moreover, such CAD-to-XR workflows do not seize the potentialities of generative algorithms to produce synthetic images (textures). The paper presents a framework for implementing the CAD-to-XR workflow. The solution consists of a module for texture generation based on Generative Adversarial Networks (GANs). The generated texture is then managed by another module (based on Shrinkwrap modelling) to develop the iPVP by simplifying the 3D model and UV mapping the generated texture. The geometric and material data is integrated into a graphic engine, which allows for programming an interactive experience with the iPVP in XR. The CAD-to-XR framework was validated on two components (rifle stock and forend) of a sporting rifle. The solution can automate the texturing process of different product versions in shorter times (compared to a manual procedure). After each product revision, it avoids tedious and manual activities required to generate a new iPVP. The image quality metrics highlight that images are generated in a “realistic” manner (the perceived quality of generated textures is highly comparable to real images). The quality of the iPVPs, generated through the proposed framework and visualised by users through a mixed reality head-mounted display, is equivalent to traditionally designed prototypes.

Keywords: Artificial Intelligence , CAD-to-XR , Extended Reality , Generative Adversarial Networks , User-Centered Design , Virtual Prototyping

This study investigates the use of electroencephalography (EEG) to characterize emotions and provides insights into the consistency between self-reported and machine learning outcomes. Thirty participants engaged in five virtual reality environments designed to elicit specific emotions, while their brain activity was recorded. The participants self-assessed their ground truth emotional state in terms of Arousal and Valence through a Self-Assessment Manikin. Gradient Boosted Decision Tree was adopted as a classification algorithm to test the EEG feasibility in the characterization of emotional states. Distinctive patterns of neural activation corresponding to different levels of Valence and Arousal emerged, and a noteworthy correspondence between the outcomes of the self-assessments and the classifier suggested that EEG-based affective indicators can be successfully applied in emotional characterization, shedding light on the possibility of using them as ground truth measurements. These findings provide compelling evidence for the validity of EEG as a tool for emotion characterization and its contribution to a better understanding of emotional activation.

Keywords: Affective elicitation , EEG , Emotion assessment , VR

[No abstract available]

Keywords: affective computing , affective elicitation , augmented/virtual reality , biometrics , emotion assessment , human-computer interaction , neuroscience , physiological data

Position-controlled servomechanisms are the core elements of flexible manufacturing plants, primarily utilized to actuate robotic systems and automated machines. To match specific torque and costs requirements, typical servomechanism arrangements comprise precision reducers, which introduce motion errors that heavily limit the final performance achievable. Such errors are complex to model and depend from speed, dynamic loading conditions and temperature. Accurate characterization is fundamental to develop digital twins and advanced control strategies aimed at their active prediction and compensation. To properly assess the servomechanisms behavior and elaborate high-fidelity virtual models, instrumented test rigs have been proposed which can replicate the time-varying working conditions encountered in real industrial environments. In this context, the present paper reports about a novel engineering method for developing an active loading apparatus, namely a programmable mechatronic device that can deliver custom loads in a highly dynamic manner. The proposed system, consisting of a secondary servomotor and related rotating vector reducer, is integrated and synchronized within an existing instrumented test rig and is controlled in torque mode via a programmable logic controller. The paper mainly focuses on the description of the implemented closed-loop control and on the related tuning and calibration processes, demonstrating that the proposed solutions avoid important measurement errors that could compromise the final effectiveness of the system. The study finally explores the potential benefits of introducing a filter to further enhance system performance. At last, to prove the importance of stabilizing the rig and demonstrate the influence of the control parameters on its measurements, a standard test aimed at assessing the reducer transmission error is conducted adopting different parameter settings.

Keywords: Loading system , Mechanical transmission , Servomechanisms , Stabilization , Test rig , Torque control

The design of modern production systems is requiring more and more integration of human-in-the-loop principles introduction of new working paradigms, where humans and machines cooperate and interact with each other. Such practices are reinforcing the need for human-centric smart manufacturing (HSM). Today, human-cyber-physical systems (HCPSs) allow to intrinsically meet the industrial complexity and bring insights in the development and implementation of HSM. In this context, the growing interest of academics and practitioners on the three key concepts of “sustainability, human-centric, and resiliency” related to the Industry 5.0 (I5.0) framework is justified. The aim of this study is to present a methodological approach for the design of adaptive human-cyber-physical systems to meet the I5.0 objectives. Starting from the analysis of the as-is user and process-related conditions, this innovative closed-loop framework guarantees a step-by-step tuning logic according to specified objectives. A use case concerning a quality control procedure for the Oil&Gas sector is proposed, where a preliminary HCPS prototype has been developed and tested. The solution provides a contextual support throughout all the value chain to the diverse activities carried out by design engineers up to end-users by offering a valuable operator's user experience (UX). The main results suggested that the conceived strategy overcomes traditional limitations of the technical drawings by integrating model-based engineering with human-centred design (HCD) principles. The systematic introduction of HCPS allows to redesign the human–machine interaction according to new paradigms of collaboration between the humans and the smart factory.

Keywords: Augmented reality , Human-cyber-physical system , Human-in-the-loop , Human-machine interaction , Industry 5.0

[No abstract available]

The manufacturing industry plays a crucial role in addressing sustainability challenges by integrating eco-friendly practices into product design. Employing a System Thinking methodology, this research comprehensively considers the interconnected components of a product, entities involved, and their interactions. The study focuses on developing a corporate sustainability tool for the entire supply chain, leveraging Digital Twin (DT), Virtual Reality (VR), and Augmented Reality (AR) to facilitate eco-design. Benchmarking existing monitoring systems and case studies in sustainable product design identifies gaps and opportunities, establishing guidelines for integrating digital technologies. Within the industrial context, the designer implements the system to ensure the development of eco-compatible products (Green Design), considering not only environmental impact, but also social change. The aim of this paper is to design a corporate sustainability tool based on a digital product model enriched with specific product and process information. This tool serves as an instrument for the company to evaluate environmental, social and governance data, make strategic sustainability decisions, and create company dashboards that facilitate the visualization and analysis of this data. The creation of a Digital Product Passport (DPP) will offer valuable information on the sustainability aspects of the product, benefiting end users who utilize the product. Conducted within a co-financed Ph.D. program with an Italian industrial machinery manufacturer, the research addresses challenges in academia-industry collaboration, enhancing the practical application of designed systems. The findings have significant implications for the manufacturing industry, contributing to sustainable product design practices and establishing a standardized framework for assessing and communicating product sustainability.

Keywords: Corporate sustainability digital tool , Green Design , Sustainable product design , System thinking , Transdisciplinary engineering

The advent of Industry 4.0 has led to the need to train highly skilled operators in complex and dynamic environments, through the convergence of physical and digital technologies. In this context, indoor navigation plays a crucial role in the training of Operators 4.0, allowing them to acquire spatial knowledge and specific skills to operate effectively in advanced industrial environments. However, the challenges associated with this technology, such as environmental complexity and obstacles, require an innovative approach to operator training. Mixed Reality (MR) represents a promising technological innovation for the development of these systems. Indeed, MR enables the co-represence of digital and real world, by content overlapping and integrated interaction between virtual and real assets, promoting efficiency, safety and user engagement. Within this scenario, the paper proposes a Holographic Navigation (HoloNav) methodology based on spatial and cloud computing for the development of an indoor navigation application according to a transdisciplinary approach. The application is deployed on the Microsoft HoloLens 2 as MR device, using Microsoft Azure Spatial Anchors (ASA) and Cloud Storage Account resources. This study aims to provide an applied methodology and guidelines for application development, and to analyze the broader social implications and potential for social change of such MR technology to support training processes. The proposed ASA-based HoloNav approach was applied through a case study, where a tour inside the spare parts warehouse was developed for new operators who need to start familiarizing themselves with all the different areas.

Keywords: Indoor navigation , Microsoft HoloLens 2 , Mixed reality , Transdisciplinary engineering , User experience

The use of Extended Reality (XR) technologies virtual and augmented reality (AR) is establishing itself more and more in design, construction and operation applications in industrial and research sectors that aim to employ digital technologies with a human-centered design approach. At ITER, a methodology that includes these technologies is applied in the design study of test blanket module (TBM) port cell (PC) components and replacement operation. The use of virtual prototyping at early design phases permits anticipating issues related to human operation and accessibility. Therefore, design optimization is performed before manufacturing the actual feasibility mock-ups. XR sessions are organized involving several design experts interacting with the various components 3-D models at the same time to facilitate the design decisions and to optimize the procedures. The main asset of this XR approach is to offer different means to support the decisions and onboard all participants. The simulations are prepared by building a virtual mock-up based on native CAD models available in databases for the different systems integrated into the Tokamak building. The complexity of the systems and the large amount of data to display require data preprocessing in order to ensure a reliable representation of the scenarios to be simulated. The reliability of the 3-D data and the consideration of the risks and the environmental conditions of the areas to be simulated are essential for making applicable and sound decisions. The aim of this article is to provide a method to realize reliable XR simulations following a documented workflow usable across the ITER project. The method is based on the realization of an XR-maintenance verification plan as an input for the XR simulation and of a maintainability verification document based on the results available from XR simulations. The experience and the results gained from the TBM PC design activities involving virtual reality (VR) simulations are also discussed.

Keywords: As low as reasonably achievable (ALARA) , early design validation , extended reality (XR) , maintenance , test blanket modules (TBMs) , virtual reality (VR)

In recent years, operator ergonomics has been playing an important role, driven by workplace regulations and worker safety in manufacturing settings characterized by physical demands and injury risks. Traditional ergonomic evaluation methods rely mainly on observation and manual measurements, which are subjective, reactive, and sometimes challenging to perform. Technologies, such as virtual reality devices and motion capture systems, offer opportunities to streamline data collection and enhance objectivity. Moreover, Industry 4.0 introduced digital twins, facilitating ergonomic evaluations in virtual environments. This paper presents a framework that integrates Virtual Reality technologies and motion capture analysis to support ergonomists in evaluating operators' working conditions in manufacturing settings. The operator wears a head mounted display to simulate working tasks in a virtual replica of the real plant. An inertial motion capture system is employed to automate data gathering and processing for an objective evaluation of the operator's postural condition. A Virtual Wall system is exploited by the ergonomist for the evaluation of the tasks using the motion captured data. A case study has been considered to test the functionality of the platform. A tester repeats a sequence of tasks both in real and virtual scenarios in order to compare the results. Ergonomics is evaluated through three standard indexes (i.e., RULA, REBA, OWAS), resulting in a strong correlation between data acquired interacting with physical and virtual environments.

Keywords: Digital Twin , Ergonomics , Semi-immersive Environment , Simulation

Despite the existence of numerous innovative solutions aimed at enhancing the objectivity and efficiency of physical ergonomics analysis, traditional methods are still widely used among ergonomists, relying on direct observation of workers and their task performances. This research addresses this gap by introducing AR4Ergo, an Augmented Reality platform specifically designed to assist ergonomists by computing real-time standard indexes and simulating musculoskeletal loads using biomechanical virtual models. Xsens inertial motion capture system is employed to monitor human movements, while the OpenSim Application Programming Interface has been integrated into AR4Ergo for the simulation of musculoskeletal loads. Microsoft Hololens v2 is used to visualize human and workplace digital twins with information relative to ergonomic indexes and biomechanical simulations during the observation of the tasks performed by workers. A test was conducted to evaluate the functionalities of the platform, with three testers taking turns performing the roles of both the ergonomists and the operators. The results demonstrate how real-time analysis with a visual interface can be performed to augment ergonomists' experience without modifying the basic observational procedure. Even if the feedback is positive, some improvements have been identified in order to allow the ergonomist to use AR4Ergo in daily practice for different industrial applications.

Keywords: Augmented Reality , Biomechanical Simulation , Ergonomics , Microsoft Hololens , Xsens

In an era characterized by the increasing complexity of products and the rapid turnover of the workforce across different companies, there is a growing need to invest significantly in quick and efficient training methods. Concurrently, the advancement of digitalization has rendered certain training practices anchored to paper-based materials obsolete. Numerous companies are directing their investments toward digital training, yet the optimal format to exploit the full advantages of digitalization remains unclear. This study undertakes a comparison of four distinct digital versions of the same training process with the aim of comprehending the tangible benefits.

Keywords: virtual and augmented reality environments , virtual prototyping

This paper presents a pioneering pilot study conducted at ARGOR-HERAEUS SA, focusing on integrating Augmented Reality (AR) HoloLens technology for emergency training and management in the scrubber System. The scrubber system, known for its relevance in terms of reducing pollutants and thus its potential dangers in case of malfunction, poses unique challenges for emergency preparedness and response. The study utilizes Unity3D to develop immersive training scenarios and emergency situations, enhancing trainee preparedness.

Keywords: ARGOR-HERAEUS SA , Augmented Reality , Emergency Management , Emergency Training , HoloLens , Pilot Study , Scrubber System , System Usability Scale (SUS) , Unity3D

The aim of “Precision Surgery” is to reduce the impact of surgeries on patients' global health. In this context, over the last years, the use of three-dimensional virtual models (3DVMs) of organs has allowed for intraoperative guidance, showing hidden anatomical targets, thus limiting healthy-tissue dissections and subsequent damage during an operation. In order to provide an automatic 3DVM overlapping in the surgical field, we developed and tested a new software, called “ikidney”, based on convolutional neural networks (CNNs). From January 2022 to April 2023, patients affected by organ-confined renal masses amenable to RAPN were enrolled. A bioengineer, a software developer, and a surgeon collaborated to create hyper-accurate 3D models for automatic 3D AR-guided RAPN, using CNNs. For each patient, demographic and clinical data were collected. A total of 13 patients were included in the present study. The average anchoring time was 11 (6–13) s. Unintended 3D-model automatic co-registration temporary failures happened in a static setting in one patient, while this happened in one patient in a dynamic setting. There was one failure; in this single case, an ultrasound drop-in probe was used to detect the neoplasm, and the surgery was performed under ultrasound guidance instead of AR guidance. No major intraoperative nor postoperative complications (i.e., Clavien Dindo > 2) were recorded. The employment of AI has unveiled several new scenarios in clinical practice, thanks to its ability to perform specific tasks autonomously. We employed CNNs for an automatic 3DVM overlapping during RAPN, thus improving the accuracy of the superimposition process.

Keywords: artificial intelligence , kidney cancer , nephron-sparing surgery , partial nephrectomy , renal cell carcinoma , robotic surgery , three-dimensional imaging

Objectives: The research's purpose is to develop a software that automatically integrates and overlay 3D virtual models of kidneys harboring renal masses into the Da Vinci robotic console, assisting surgeon during the intervention. Introduction: Precision medicine, especially in the field of minimally-invasive partial nephrectomy, aims to use 3D virtual models as a guidance for augmented reality robotic procedures. However, the co-registration process of the virtual images over the real operative field is performed manually. Methods: In this prospective study, two strategies for the automatic overlapping of the model over the real kidney were explored: the computer vision technology, leveraging the super-enhancement of the kidney allowed by the intraoperative injection of Indocyanine green for superimposition and the convolutional neural network technology, based on the processing of live images from the endoscope, after a training of the software on frames from prerecorded videos of the same surgery. The work-team, comprising a bioengineer, a software-developer and a surgeon, collaborated to create hyper-accuracy 3D models for automatic 3D-AR-guided RAPN. For each patient, demographic and clinical data were collected. Results: Two groups (group A for the first technology with 12 patients and group B for the second technology with 8 patients) were defined. They showed comparable preoperative and post-operative characteristics. Concerning the first technology the average co-registration time was 7 (3–11) seconds while in the case of the second technology 11 (6–13) seconds. No major intraoperative or postoperative complications were recorded. There were no differences in terms of functional outcomes between the groups at every time-point considered. Conclusion: The first technology allowed a successful anchoring of the 3D model to the kidney, despite minimal manual refinements. The second technology improved kidney automatic detection without relying on indocyanine injection, resulting in better organ boundaries identification during tests. Further studies are needed to confirm this preliminary evidence.

Keywords: artificial intelligence , kidney cancer , nephron-sparing surgery , renal cell carcinoma , robotic surgery , three-dimensional imaging

Robotic surgery, also known as robotic-assisted surgery (RAS), has rapidly evolved during the last decade. RAS systems are developed to assist surgeons to perform complex minimally invasive surgeries, and necessitate augmented interfaces for precise execution of these image-guided procedures. Extended Reality (XR) technologies, augmenting the real-world perception via integrating digital contents, show promise in enhancing RAS efficacy in various studies. Despite multiple reviews on technological and medical aspects, the crucial elements of human-robot interaction (HRI) and user experience (UX) remain underexplored. This review fills this gap by elucidating HRI dynamics within XR-aided RAS systems, emphasizing their impact on UX and overall surgical outcomes. By synthesizing existing literature, this systematic review study identifies challenges and opportunities, paving the way for improved XR-enhanced robotic surgery, ultimately enhancing patient care and surgical performance.

Keywords: extended reality , human-computer interaction , robotic-assisted surgery , surgical robots , user experience

In recent years, Augmented Reality (AR) has been effectively proposed as a tool to support workers in manual procedural tasks in industry, such as assembly and maintenance. It is very common for workers to deal with complete equipment in maintenance. Then, instructions may refer to components located in blind areas, i.e., visually occluded by part of the equipment. Displaying in-situ AR instructions in such blind areas requires handling occlusions; otherwise, side-by-side instructions can be exploited. In the literature, it is still unclear which solution better leverage workers' performance in maintenance tasks accomplished in blind areas. Thus, we designed three AR presentation modes to convey maintenance instructions in a real machine: 3D in-situ rendered with X-ray technique; 3D side-by-side on a CAD model replicating the blind area; 2D side-by-side on a virtual mirror. We conducted a user study comparing these three presentation modes with a 2D drawing extracted from the original maintenance documentation of the machine. The performance of 42 participants was evaluated in terms of completion time, accuracy, and cognitive load. The results revealed that both in-situ and 3D side-by-side presentation modes perform better than 2D drawing. Specifically, the in-situ presentation mode outperforms the 3D side-by-side mode in terms of completion time. The side-by-side virtual mirror does not improve performance with respect to 2D drawing, then it needs to be redesigned for effective use in AR maintenance interfaces.

Keywords: Augmented reality , Blind area , Industrial operator support , Maintenance , Occlusion , Work instructions

Augmented Reality (AR) is one of the leading technologies of the Industry 4.0 revolution, offering innovative interfaces to promote the diffusion of digital contents into industrial processes, thanks to flexible and robust solutions and cost-effective devices. In this context, this paper explores the adoption of AR in industrial logistics where several open issues still discourage its effective use in everyday scenarios. After a review of objectives, approaches and technics of AR integration in logistics operations, the paper presents a framework to identify goods in a warehouse, retrieve data relative to the package, display info to the user to drive operations. The approach aims at easing and speeding up the activity of the warehouseman to identify goods, check the relative information and to put each good on the correct shelf. A prototypal application was developed within the Unity platform and integrated with the company ERP system to manage data on the products and retrieve images of the identification labels. A real use case involving a primary company producing agricultural tractors is proposed to test usability of the prototype. Results showed that the developed application allows relevant benefits in terms of process effectiveness, error prevention, aiming at reducing the operator mental workload.

Keywords: Augmented Reality , Industry 4.0 , Logistics , Warehouse operations

Factory acceptance test is the inspection of equipment and components at the supplier's premises before delivery or final inspection. However, this control can represent a considerable cost for the customer, especially when the manufacturer's company is geographically far from the customer one and inspections must be frequent. In this paper, the authors present a framework to support the factory acceptance test based on augmented reality (AR) techniques and model-based definition aimed at dimensional checks that does not require the physical presence of the customer at the supplier's premises. The supplier must be previously equipped with an automatic measuring machine. Once the component under inspection is placed inside the machine, this reads the type and the position of the features to be measured along with the related specification limits directly from the annotated 3D model of the component. The results are automatically transmitted to the customer's site. Through a tablet, the supplier, guided by the customer, reads the results of the measures directly on the measured object through augmented or mixed reality techniques. Any out-of-specification dimension can be remeasured in real time with the customer's remote assistance using traditional measurement techniques. The proposed architecture, at an advanced stage of experimentation, is discussed with reference to an industrial case study proposed and using an entry level commercial 3D scanner.

Keywords: Augmented reality , Dimensional inspection , Factory acceptance test , Metrology