Virtual and augmented reality

This work is focused on the development of a “serious game”, i.e., a software with no playful purpose adopted to train and verify the skills of users to improve safety in the workplace. Particular attention has been given to the accidents that occur during typical operations done in factory sites, e.g., driving forklifts. Firstly, the different types of “serious games” already available on the market have been analyzed to identify the best way to carry on tests and simulations by means of virtual, mixed, and augmented reality. Afterward, once the best solution has been identified, the Unity development environment has been considered to define a standard that could also be used for future projects. So, the result is the development of a powerful editor of the scene, in which the user can program all the components adjusting the game to the tasks to be performed by the worker, the tools, the environment and the targets. So, an upgradeable “parametric path” has been created, which will be followed by the forklift, realizing a new game and test environment for the worker. Then, modular components have been conceived to contemplate the future developments of the project, including a graphic editor: these will lead to a software that can be easily modified according to the customer's requests.

Keywords: Occupational safety , Serious games , Virtual reality

Preface and Acknowledgements (Editorial)

Keywords: Editorial

This paper describes the design of a novel fabric–based antagonistic pneumatic actuator with multiple chambers that can be used for the development of soft continuum manipulators for collaborative tasks. The concept consists of three pneumatic chambers of fabric material capable of being actuated independently. By connecting multiple actuators of this kind, it is possible to obtain soft continuum manipulators capable of complex movements and able to change the stiffness of their elements. In this work we highlight the design and prototyping of the soft actuator and we present the preliminary experiments in terms of motion and stiffening capabilities.

Keywords: Fabric pneumatic artificial muscles , Soft actuators , Soft continuum manipulators , Soft robotics

The paper focuses on a preliminary study of an easy-to-customize capacitive soft sensor to measure forces that can enable soft robot features like sensitive skins or permits dexterous object manipulation thanks to the perception of the grasping force. The prototype has been realized overlapping five different layers choose among commercial and easy to find materials. The sensor is completely composed by customized or self-produced parts. The stack definition involves compatibility test to define the correct combination of layers and adhesives. An evaluation of the behavior has been performed applying weights in the range [20–5800] g finding a mean sensitivity of 0.143 pF/kg over an initial value C0 of 3.151 pF. The sensor prototype showed good performance in term of sensitivity and hysteresis in the defined application range. Dielectric viscoelastic phenomena and decreasing repeatability have been observed in the upper part of the measuring range. The sensor proposed shows promising characteristics encouraging future developments.

Keywords: Capacitive force sensor , Conductive inks , Customizable force sensor

This paper presents the use of surface electromyographic (sEMG) signals for the actuation of soft pneumatic artificial muscles. The idea behind this paper is finding a relationship between a natural muscle and an artificial muscle, do it through an analysis of the sEMG data. We start from the characterization of a specific soft pneumatic artificial muscle and we relate the root mean square value of the sEMG signal to the contraction of the actuator itself. This work might pave the way for the development of intuitive wearable interfaces for the actuation of soft robots.

Keywords: Human biosignal , Soft actuators , Surface electromyographic sensors , Wearable technologies

The evaluation of hip implantation success remains one of the most relevant problems in orthopaedics. There are several factors that can cause its failure, e.g.: aseptic loosening and dislocations of the prosthetic joint due to implant impingement. Following a total hip arthroplasty, it is fundamental that the orthopaedist can evaluate which may be the possible risk factors that would lead to dislocation, or in the worst cases, to implant failure. A procedure has been carried out with the aim of evaluating the Range of Movement (ROM) of the implanted prosthesis, to predict whether the inserted implant is correctly positioned or will be prone to dislocation or material wear due to the malposition of its components. Leveraging on a previous patented methodology that consists in the 3D reconstruction and movement simulation of the hip joint, this work aims to provide a more effective visualization of the simulation results through Mixed Reality (MR). The use of MR for the representation of hip kinematics and implant position can provide the orthopaedic surgeon with a deeper understanding of the orientation and position of implanted components, as well as the consequences of such placements while looking directly at the patient. To this end, an anchoring system based on a body-tracking recognition library was developed, so that both completely automatic and human-assisted options are available without additional markers or sensors. An Augmented Reality (AR) prototype has been developed in Unity 3D and used on HoloLens 2, integrating the implemented human-assisted anchoring system option.

Keywords: Computer-aided surgery , HoloLens 2 , Mixed reality , THA assessment , Total hip arthroplasty

In recent days product design and review cycles are considerably held to many constraints and requirements. One of the main requirements regards the usability of solutions and the impact that design choices have on final users. Therefore, it is becoming crucial to anticipate human factors concerns in the preliminary phases of product development process. Also, since human factors affect the users subconsciously and influence the decision making in a significant way, they need to be enabled to observe the users while interacting with the product in real-time. In this framework, the platform that the human factors expert and the users use plays a vital role in influencing the human factor assessments and evaluations. Latest developments in Extended Reality opened the way toward the possibility to conceive new simulation platforms that allow experts to deeply explore the products in real-time through collaborative and interactive environments before the physical mockup of the product. The purpose of this paper is to outline a Mixed Reality (MR) tool in the field of aircraft interior design to demonstrate it as a potential co-creative platform for involving human factor experts in the loop while the task is ongoing. An MR multi-user, co-located, collaborative and interactive environment of an aircraft galley is developed where the HF specialist and flight crew member can co-exist to visualize the real scale model of the galley and perform an operational task. Such co-creative tool is foreseen to execute usability tests during design review phase and reduce time, costs of product development cycle while meeting the user requirements.

Keywords: Co-creation , Design review , Human factors and ergonomics , Mixed reality

Collaborative robots (cobots) are designed to directly interact with human beings within a shared workspace. To minimize the risk of musculoskeletal disease for the workers, a physical ergonomic assessment of their interaction is needed. Virtual reality (VR) and motion capture (Mocap) systems can aid designers in building low-hazard collaborative environments. This work presents a framework based on VR and Mocap systems for the ergonomic evaluation of collaborative robotic workstations. Starting from the 3D models of the cobot and workstation components, a virtual environment is built in Unity and ROS is employed to manage the cobot behavior. The physical ergonomics is evaluated by means of RULA methodology, exploiting the body tracking capabilities of the device Kinect Azure, a low-cost markerless Mocap system. The framework has been tested by building a virtual environment for collaborative control of flanges with different diameters. The worker interacts with a six-axis Nyro One to move parts on the workstation. The ergonomic assessment is performed in real-time, and a report is generated for later uses and evaluations. The proposed framework fosters the design of collaborative robotics workstations based on an objective assessment of ergonomics. The results of this research work allow planning future development steps for the emulation of more complex workstations with cobots and the use of augmented reality to evaluate how to modify existing workstations to introduce a cobot.

Keywords: Collaborative robots , Ergonomics , Motion capture , Virtual reality

The development of cutting-edge technologies in the industrial sector has led to the demand for increasingly specific tools in the optimization of efficiency problems. The automotive sector is the one that makes the most use of these technologies, such as, among all, Virtual Reality (VR) and Augmented Reality (AR). A virtual reality tool is inserted as a guide tool for the user in drawing as quickly as possible the key information for optimizing the process. This research work fits into this context, the goal of which was the implementation of an immersive platform for carrying out accurate measurements within an entirely virtual automotive environment. A VR system of this type allows to check the dimensional and shape tolerances of car components, performing measurement, with high precision and in real time, in a custom-made virtual environment, in which it is possible to simulate the presence of a myriad of components and test their mutual interaction. A comparative test was carried out, obtained by varying the graphical and geometrical model in the VR settings, in order to evaluate the level of usability and the degree of efficiency of a tool for measuring distances between objects in a virtual reality environment, depending on the system parameters. The validation of a measurement instrument in VR is part of an increasingly current technological context in which the need to optimize the time-cost curve embraces the need for increasingly accurate results.

Keywords: Assembly training , Tolerance measurements , Virtual reality

Human-Robot Interaction (HRI) is an interdisciplinary research field aiming to study and develop platform in which robots (and in particular collaborative robots, also called cobots) can interact and collaborate with humans to execute dedicated tasks. Usually, cobots are intended as passive robotic devices for direct collaboration with a human operator within a shared workspace. They are designed to be used in open and uncontrolled environments; the robot shall be able to adapt its behaviors to the dynamic input of the surrounding environment. In this optic, Mixed Reality (MR) can play a crucial role supporting the flow of data between the actors (cobot and human) working in the shared environment, it can offer a simply, but remarkably advanced, communication interface between human and robot. Thanks to MR, tools to allow human operators, without particular experience or knowledge of robotics, to easily interact with the cobot can be developed. Our work is focused on development of a MR platform that integrates cutting-edge technologies, i.e. a Head Mounted Display (HMD), and a cobot in a shared environment. The experimental setup includes the Microsoft's Mixed Reality HMD HoloLens 2 and the Franka Emika Robot System.

Keywords: Cobot , Head Mounted Display , Human Robot Interaction , Mixed Reality , Robotics

Extended Reality (XR) technologies can be a valid tool for supporting operators with assembly instructions in real-time, directly superimposed on the product. A strategy to face dynamic effects, such as sudden environmental changes and objects in movement, still needs to be defined to implement effective XR applications in moving production lines. The physical environment must be tracked and recognized to determine the position and the orientation of digital content in space. This paper aims to address these open issues by proposing a Mixed Reality (MR) application to support workers in the wire harness process for the tractor's drivelines production, which requires the product to be in motion. For this aim, the Microsoft Azure Spatial Anchors with Microsoft HoloLens 2 were adopted. The designed and developed MR application allows the operator to display, step-by-step, consecutive work instructions provided through textual indications, pictures, videos, and animations. Through the Azure Spatial Anchors, the digital content can be anchored over the driveline parts. By knowing the speed of the production line and the duration of each wiring task, a time-controlled application has been developed, to assure the visualization of the instruction in correspondence of the relative driveline elements, in the right time. The application has been tested on-field with expert operators. Despite the presence of some drawbacks related both to the driveline motion and technology, the operators' feedback pointed out satisfactory and promising results since the application allowed them to reduce errors and forgetfulness.

Keywords: Azure Spatial Anchor , In-motion tracking , Industry 4.0 , Mixed Reality , Wire harness process

In this work, we present a technique to simplify the authoring of Augmented Reality Technical Documentation, allowing technical writers with limited knowledge in Augmented Reality (AR) to produce this new type of documentation in their companies. Contrarily to Traditional Technical Documentation, AR offers the opportunity to provide the exact amount of information needed through a careful design of the AR interface. However, in the literature, there are no established techniques to break work instructions down into elemental pieces of information and define how to convey this information through visual assets in AR. In this work, we proposed identifying the information contained in work instructions of technical documentation, extending the method of “Therbligs,” already used in the literature to describe assembly tasks. First, we defined six classes of information types: identity, location, order, way-to, notification, and orientation. Then, we showed how to use these information types to break down the work instructions of an assembly manual used as a case study. We found that the six information types were enough to analyze the complete manual. The second contribution of this work is the proposal of the most suitable visual asset and its properties for each information type. This goal was accomplished through a technical discussion in a focus group with ten experts in the design of AR technical documentation.

Keywords: Augmented reality , Authoring , Information presentation , Technical documentation , Work instructions

Augmented Reality (AR) has been proved to be effective in maintenance operations in the industrial field. In a concurrent engineering approach, the authoring of AR manuals, to convey instructions to operators exploiting AR, must be done during the design phase of the product lifecycle. A reliable solution is needed to speed up the AR manual development process when the product is not physically available. We compared three solutions for displaying a demo version of an AR manual when the real product is not available, opting to replace it with its CAD model. Based on the user study results, the main features for each demo version were collected. The Augmented Reality (AR) solution allows to show the product that is not physically available in a real scale with the drawback of needing a physical printed marker. The Desktop Virtual Reality (DVR) prototype overcomes the problem of managing different devices and real-life locations, but without a real scale and a natural interaction. The Augmented Desktop Virtual Reality (ADVR) prototype allows to distinguish more easily the virtual elements of the true AR from those simulating the real product, but with a less natural interaction due to the use of a secondary screen. As a case study, we chose a compressor that a local company is going to produce with its AR manual. Although users overall preferred the AR demo version, the company chose the ADVR solution due to a better perception of what would be the result of the true AR application.

Keywords: Augmented Reality , Prototyping , Task simulation , Work instructions

For the sake of being competitive in an ever-changing market, industrial companies need a redefinition of traditional design and integration of parts, equipment, and services such a redefinition allows effectively addressing the interaction between machines and operators, particularly in the area of complex production lines. In this context, enhancing ergonomics is crucial to reduce fatigue and stress of workers and increase work-place efficiency and comfort. Moreover, identifying ergonomic flaws in three-dimensional human-machine design problems (e.g., body posture, reach, visibility) at an early stage of the engineering process allows to prevent these issues at a low cost. Virtual reality (VR) is emerging as a powerful tool to improve the ergonomic assessment in the design of complex production lines. However, VR is not yet a well-consolidated practice for industrial companies, and the state-of-the-art applications are limited to simplified, isolated, and customized experiments. This work proposes the use of a virtual golden zone (VGZ) as a standard and efficient VR method for the ergonomic analysis and optimization of operator activities in manual manufacturing stations. The resulting effectiveness and benefits are highlighted through the application of the approach to a real industrial case study. Finally, the outcomes of a usability questionnaire, compiled by the professionals involved in the VR reviews, are presented to evaluate the usability of the VGZ methodology in the design process of complex production lines.

Keywords: Complex production lines , Ergonomics , Human computer interaction , Virtual reality

This work is part of a larger project aiming to develop a comprehensive Augmented Reality (AR) interface for recreative and professional nautical sailing navigation. Due to the complexity of the marine environment and the dynamism of the crew members on board, we propose to diversify the display of navigation information concerning the role, position and activity onboard. The interfaces are designed to be viewed by one or more crew members, solving the problem of everyone viewing the same data, and giving each one the information they need based on their role or position. The three novel proposals for sail-specific AR graphic interfaces are designed to be displayed on Head-Mounted Display HMD and each of them differs for the information stabilization: Screen-Stabilized, Body-Stabilized and Boat-Stabilized. Each interface is different from the others differentiated according to the type of navigation and the user who must use that information. These approaches have been prototyped and evaluated by a panel of experts in the field of sailing navigation and were able to showcase their potential for future evaluations in different scenarios.

Keywords: Augmented Reality , Interface , Nautical , Sailing , Spatiality , User-based

In the ever-changing scenario of technology evolution, designers need to develop new interactive systems that respond to users' needs. Augmented Reality (AR) could be a tool and an opportunity for designers to create novel interactive systems. AR has proven to be effective in several domains, showing potential for widespread deployment even in everyday life tasks such as the use of household appliances. Thus, leveraging the Human-Centred Design (HCD) approach, we integrate AR into the design and development process of an interactive system for household appliances. Based on our survey results with 463 participants, one of the appliances that could benefit most from AR is the kitchen machine. Starting from a case study, we develop a demo to prove the feasibility of designing interactive systems with the integration of AR technologies, following the HCD approach.

Keywords: Augmented Reality , Human-Centred Design , Industrial Design , Interactive System , User Interface

Conventionally, visitors of exhibitions cannot directly interact with artworks but remain mainly passive. This work presents a new way to discover paintings through an extended reality application with the aim of enhancing visitors' engagement. The workflow consists of digitally recreating a painting. All its constitutive elements are contoured and then removed by the artwork. These elements will be unveiled by the visitors with gesture interaction. The discovery of each element is followed by additional information on the subject. The application also proposes an alternative experience where the user paints the components of the artwork with gestures. The hands' position was tracked by the Leap Motion Controller, an optical sensor by Ultraleap. The process has been applied to one of the Tintoretto's pieces of art: “L'Annunciazione del Doge Grimani”. The extended reality application has been experienced by over twenty thousand visitors at the dedicated exhibition. Numerous positive comments received from both visitors and guides are encouraging.

Keywords: Cultural heritage , Exhibition , Extended reality , User interaction

This paper proposes a methodology to exploit virtual prototypes for the development of both VR and AR-based Training systems, with the same Software-Hardware architecture. This architecture provides a unique collaborative environment to enable the interaction between several users even if geographically distant, allowing to switch easily and rapidly between VR, AR and mixed technologies. The proposed Training Systems concern a simple but structured workcycle about the assembly of some Lego bricks. The selected development platform is IC.IDO: a commercial software with an industrial footprint, produced by ESI Group. Despite IC.IDO is originally designed for VR and not compatible for AR applications, it resulted fully compliant with the requisites of the proposed methodology. The paper firstly describes the path outlined from the conversion of a Training session from VR to AR compatible, within IC.IDO. Thanks to the adoption of a colour-filtering software as OBS Studio, the IC.IDO session easily becomes AR-compatible, using a unique hardware architecture: the HTC Vive Pro HMD and a VIVE tracker. In this work, also the Oculus Rift S has been employed for the VR applications to demonstrate the system's compatibility with other VR devices. For both VIVE and Oculus, the Virtual Desktop feature allowed to display the OBS Desktop window in the HMD. Furthermore, an innovative solution is proposed: thanks to IC.IDO's Cooperative Environment, trainer and trainee will cooperate within the same scene using simultaneously VR and AR. Finally, the role of the trainer is discussed, to outline a process of progressive independence for the trainee.

Keywords: Augmented reality , Cooperative environment , Virtual prototype , Virtual reality , Virtual training system

The current study aimed to propose a Deep Learning (DL) based framework to retrieve in real-time the position and the rotation of an object in need of maintenance from live video frames only. For testing the positioning performances, we focused on intervention on a generic Fused Deposition Modeling (FDM) 3D printer maintenance. Lastly, to demonstrate a possible Augmented Reality (AR) application that can be built on top of this, we discussed a specific case study using a Prusa i3 MKS FDM printer. This method was developed using a You Only Look Once (YOLOv3) network for object detection to locate the position of the FDM 3D printer and a subsequent Rotation Convolutional Neural Network (RotationCNN), trained on a dataset of artificial images, to predict the rotations' parameters for attaching the 3D model. To train YOLOv3 we used an augmented dataset of 1653 real images, while to train the RotationCNN we utilized a dataset of 99.220 synthetic images, showing the FDM 3D Printer with different orientations, and fine-tuned it using 235 real images tagged manually. The YOLOv3 network obtained an AP (Average Precision) of 100% with Intersection Over Unit parameter of 0.5, while the RotationCNN showed a mean Geodesic Distance of 0.250 (σ = 0.210) and a mean accuracy to detect the correct rotation r of 0.619 (σ = 0.130), considering as acceptable the range [r − 10, r + 10]. We then evaluate the CAD system performances with 10 non-expert users: the average speed improved from 9.61 (σ = 1.53) to 5.30 (σ = 1.30) and the average number of actions to complete the task from 12.60 (σ = 2.15) to 11.00 (σ = 0.89). This work is a further step through the adoption of DL and AR in the assistance domain. In future works, we will overcome the limitations of this approach and develop a complete mobile CAD system that could be extended to any object that presents a 3D counterpart model.

Keywords: Augmented reality , CAD assistance , Deep learning , Neural network

Airports are a fundamental node in the aviation system. The growth of this sector undergoes the evolution of the airports procedures and infrastructures. Air traffic control is considered to be one of the most important activity performed in an airport. This often involves numerous personnel, construction and maintenance costs etc. With the ongoing digitalisation process in various fields, the concept of remote and virtual control towers (RVT) has emerged owing to the innovation in this field. Technologies such as augmented reality (AR) have successfully paved their smooth way to bring in improvements to an RVT. The current work aims to enhance and improve the controller work in an RVT exploiting the adaption of virtual reality and AR systems. In particular, this concept is under investigation within the EU funded SESAR project: RETINA (Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision) and DTT (Digital Technologies for Tower). AR content generation by virtue of various display technologies are studied and, Optical Spatial See-through displays (O SSTDs) have been considered for its own advantages in developing this application. One of the necessary requirements of AR in this configuration is identified as the tracking of the operator's point of view (eye-tracking), to provide virtual content consistent with its real position. Thus, this paper elaborates the design of specific eye-tracking system using Microsoft Kinect V2 for the virtual control tower application. The need to have binocular vision to use AR content is assessed and the interface has been tested with few subjects to evaluate the precision of the measurements detected through the proposed solution.

Keywords: Air traffic control , Augmented reality , Eye-tracking , Microsoft kinect , Remote control tower , Spatial see-through displays