Abstract: Background and objective Three-dimensional (3D) augmented reality (AR) and artificial intelligence (AI) technologies have recently been introduced to enhance guidance during robot-assisted radical prostatectomy (RARP). By overlaying virtual and real-time images, this approach helps accurately localize hidden lesions during surgery, enabling the execution of tailored procedures. This study aimed to evaluate whether 3D-AI-AR guidance reduces positive surgical margins (PSMs) compared with standard tw0-dimensional (2D) magnetic resonance imaging (MRI)-based interventions. Methods In this prospective, multicenter randomized controlled trial (NCT06318559), 133 patients with extracapsular extension or bulging at preoperative MRI were enrolled and randomized (2:1) to either 2D MRI-guided ( n = 84) or 3D-AI-AR–guided RARP ( n = 49). All the patients underwent nerve-sparing RARP. Intraoperative selective biopsies were then performed at the level of the preserved neurovascular bundle (NVB): cognitive in the MRI group and AR guided in the 3D group. The primary outcomes included PSM rate. Prostate-specific antigen (PSA) levels, continence, and potency recovery were assessed during the 12 mo of follow-up. The use of postoperative radiotherapy was recorded. Biochemical recurrence (BCR) was defined as PSA >0.4 ng/ml. All the analyses were conducted with SAS Statistics Software v.9.4. Key findings and limitations Baseline and intraoperative characteristics were similar between the groups. While PSMs on prostate surface were comparable ( p = 0.8), 3D-guided excisional biopsies had a significantly higher positivity rate (52% vs 13%; p = 0.001), allowing an improved margin control. The 3D group had a lower overall PSM rate (22% vs 39%; p = 0.047), required less postoperative RT (18% vs 35%; p = 0.046), and showed higher continence at 12 mo (91% vs 71%; p = 0.03). Potency and BCR rates were similar. Conclusions and clinical implications The execution of a 3D-AI-AR–guided biopsy at the level of preserved NVBs during nerve-sparing RARP allows correct identification of the tumor with subsequent improvement of margin control. Longer follow-up is required to assess the functional and long-term oncological outcomes of this approach.

Keywords: Artificial intelligence | Augmented reality | Prostate cancer | Radical prostatectomy | Three-dimensional model

Abstract: This study aims to develop an intraoperative navigation system for in-vivo procedures leveraging optical flow techniques. Traditional methods for organ tracking often fail in intraoperative scenarios due to domain complexity and, in the case of machine learning methods, by lack of annotated data. To address this challenge, we present a novel organ tracking system based on optical flow and evaluate the performance of four state-of-the-art neural networks for optical flow to determine the most suitable one to use in our system. The proposed system combines semantic segmentation and optical flow estimation. Segmentation networks identify the organ, surgical tools, and background in each frame. Then, deep learning-based optical flow networks estimate its motion across frames. The resulting motion is used to compute the organ’s rotation and translation and update the 3D virtual model accordingly. The performances of the four neural networks for optical flow are tested on two video sequences of a robot-assisted partial nephrectomy, where the tracked organ of interest is the kidney. Among the tested networks, and for the concerns of this specific domain of use, RAFT and GMFlow achieved the most promising results in terms of IoU accuracy.

Keywords: 3D Organ Tracking | Deep Learning for Surgery | Partial Nephrectomy | Real-Time Surgical Assistance | Semantic Segmentation

Abstract: Novel View Synthesis techniques such as 3D Gaussian Splatting (3DGS) conventionally rely on Structure-from-Motion (SfM) to generate the sparse point cloud required for initialisation. SfM, however, fails systematically on textureless or flat surfaces, producing sparse geometry and visible artefacts. This paper presents a synthetic data generation pipeline that bypasses SfM entirely by decoupling geometric and photometric acquisition. A custom path tracing engine employs a toroidal sensor to capture omnidirectional data and produces a dense, ground-truth point cloud that serves as an optimal initialisation state for 3DGS. We investigate several surface sampling strategies and demonstrate that Colour-Based Importance Sampling outperforms uniform methods by concentrating samples on visually informative regions. Experimental results suggest that, within our synthetic setup, our pipeline can operate without Adaptive Density Control, achieves competitive reconstruction quality compared to standard SfM-based initialisations on the evaluated scenes, and matches COLMAP training times despite operating on a significantly denser primitive set.

Keywords: 3D Gaussian Splatting | Importance Sampling | Novel View Synthesis | Path Tracing | Synthetic Data Generation | Toroidal Sensor

Abstract: In today’s advanced era of medical science and techniques, acquiring comprehensive knowledge for conducting medical examinations has become increasingly challenging. While theoretical learning remains essential in the medical field, such as when performing ultrasound scans, the demand for practical learning opportunities prior to working with real patients has grown significantly. By leveraging on VR technology, we propose a solution that provide learners with an immersive and realistic environment where they can undergo training in ultrasound scans. This approach aims to simulate the experience of conducting actual scans, enabling learners to develop their skills effectively. In this paper, we describe our solution focusing on the technique we developed to transform any set of DICOM images into visually realistic ultrasound images to be used inside our real-time VR-based training simulation. The use of VR can overcome the financial obstacles associated with procuring expensive equipment for training purposes, while the flexibility of our approach enhances the learning experience by adapting to various scenarios and accommodating different skill levels. Learners can navigate through a range of simulated cases, reinforcing their knowledge and competence in ultrasound techniques.

Keywords: Practical Learning | Ultrasound Simulation | Virtual Reality

Abstract: Accurate simulation of respiratory dynamics is essential for advancing the diagnosis and treatment of pulmonary diseases. This review analyzes current methodologies for modelling lung mechanics during insufflation and exsufflation, focusing on airflow simulations in the tracheobronchial tree. 45 studies were selected through a structured screening process and evaluated based on modelling approaches, simulation techniques, boundary conditions, and clinical applicability. The review identifies three main strategies for 3D TB model generation: segmentation of DICOM images, CAD-based geometries, and hybrid methods. While DICOM segmentation ensures anatomical realism, it is limited in generational depth. Conversely, CAD and hybrid approaches extend model coverage but may compromise subject specificity. Simulation methods include Computational Fluid Dynamics, Fluid–Structure Interaction, biomechanical, structural and statistical models, MR-Linac workflows, and neural networks. Among these, CFD remains the most widely adopted due to its accessibility and maturity, whereas FSI and hybrid CFD–FSI models offer superior physiological fidelity. The review wants to highlight the importance of combining detailed anatomical modelling with dynamic simulation frameworks to improve clinical interventions, particularly in lung surgery. Future work should focus on integrating patient-specific imaging, advanced boundary conditions, and multiscale modelling to enable more precise and scalable respiratory simulations.

Keywords: Computational fluid dynamics (CFD) | Fluid–Structure Interaction (FSI) | Lung insufflation–exsufflation | Patient-specific modeling | Tracheobronchial tree

Abstract: The Bounding Volume Hierarchy (BVH) is a fundamental data structure in the ray tracing algorithms to accelerate the detection of ray-scene intersections. The Surface Area Heuristic (SAH), employed during the construction of the BVH, is based on the hypothesis that the ray distribution in the scene is uniform. In this paper, we show that the SAH hypothesis is not valid when importance sampling is used, and propose two novel heuristics. With the Projected Area Heuristic (PAH) we demonstrate how it is possible to estimate better the cost of a BVH. In particular, we replace the approximation of the probability an Axis Aligned Bounding Boc (AABB) is hit by a ray, from the ratio between the surface areas of the node and the root of the BVH to the ratio of their projected areas. The plane the AABBs are projected on and the kind of projection (either orthographic or perspective) are chosen based on the local ray distribution. With the Splitting Plane Facing Heuristic (SPFH) we show how we can build higher-quality BVHs by taking into account the knowledge of the ray distribution in a local region of the scene during the selection of the orientation of the plane used to split a node.

Keywords: acceleration structures | Bounding Volume Hierarchy | Ray-tracing

Abstract: Traditional surgical training methods often lack the immersive and interactive elements necessary for optimal skill acquisition. Virtual reality (VR) technology has emerged as a transformative tool in surgical education, offering realistic simulations that enhance technical proficiency, psychomotor skills, and cognitive planning. This paper presents a novel collaborative cross-platform VR framework that supports multi-user interaction, enabling trainees and instructors to engage in a shared virtual environment. Structured around a modular digital hospital, the system facilitates real-time training sessions, interactive knowledge sharing, and procedural simulations with integrated feedback mechanisms. A user study demonstrated a significant reduction in errors, improved precision, and faster task completion, underscoring the system's effectiveness in enhancing surgical training efficiency. Participants rated usability highly, highlighting the system's intuitive design, engagement, and potential for democratizing surgical education.

Keywords: Collaborative Learning | Digital Hospital | Medical Simulation | Multi-User VR | Surgical Training | Virtual Reality

Abstract: Soft tissue simulation is a complex and challenging research field with applications across various domains, including engineering, design, medicine, and biomechanics. By accurately modeling soft tissues, virtual and augmented reality applications—such as medical training simulations, image-guided surgical support systems, and injury analysis—can more effectively replicate the behavior of complex human body structures. While many models have been proposed to simulate soft bodies, a major challenge has always been the heavy computational workload required to model and reproduce elastic and mechanical interactions between structures in the human body, usually not optimal for real-time applications. The aim of this work is to present an innovative workflow to simulate soft tissues in real time, leveraging the computational power offered by graphic processing units (GPUs). The proposed workflow addresses the challenge of improving the real-time performance of soft bodies deformation based on Extended Position Based Dynamics (XPBD) by proposing an original two-fold integrated approach: firstly, we optimized the XPBD model for parallel execution by using graph coloring. This technique allowed us to run the simulation without the use of atomic operations, while retaining the faster convergence speed of the parallel Gauss-Seidel solver. Subsequently, we applied it on a soft body represented as a volumetric regular lattice of particles, instead of the well-known representation based on tetrahedral or hexagonal meshes, allowing us to apply the deformation results even on highly polygonal meshes. The effectiveness of the developed system is evaluated through a series of tests aimed at proving its accuracy and performance, with a focus on providing realistic visual representation alongside computational efficiency.

Keywords: computer graphics | graph coloring | lattice model | soft tissues simulation | XPBD

Abstract: Point cloud 3D models are gaining increasing popularity due to the proliferation of scanning systems in various fields, including autonomous vehicles and robotics. When employed for rendering purposes, point clouds are typically depicted with their original colors acquired during the acquisition, often without taking into account the lighting conditions of the scene in which the model is situated. This can result in a lack of realism in numerous contexts, especially when dealing with animated point clouds used in eXtended reality applications, where it is desirable for the model to respond to incoming light and seamlessly blend with the surrounding environment. This paper proposes the application of physically based rendering (PBR), a rendering technique widely used in real-time computer graphics applications, to animated point cloud models for reproducing specular reflections, and achieving a photo-realistic and physically accurate look under any lighting condition. To achieve this, we first explore the extension of commonly used animated point cloud formats to incorporate normal vectors and PBR parameters, like roughness and metalness. Additionally, the encoding of the animated environment maps necessary for the PBR technique is investigated. Then, an animated point cloud model is rendered with a shader implementing the proposed PBR method. Finally, we compare the outcomes of this PBR pipeline with traditional renderings of the same point cloud produced using commonly used shaders, taking into account different lighting conditions and environments. Through these comparisons, we demonstrate how the proposed PBR method enhances the visual integration of the point cloud with its surroundings. Furthermore, it will be shown that using this rendering technique, it is possible to render different materials, by exploiting the features of PBR and the encoding of the surrounding environment.

Keywords: extended reality | point cloud | real-time rendering

Abstract: The aim of “Precision Surgery” is to reduce the impact of surgeries on patients’ global health. In this context, over the last years, the use of three-dimensional virtual models (3DVMs) of organs has allowed for intraoperative guidance, showing hidden anatomical targets, thus limiting healthy-tissue dissections and subsequent damage during an operation. In order to provide an automatic 3DVM overlapping in the surgical field, we developed and tested a new software, called “ikidney”, based on convolutional neural networks (CNNs). From January 2022 to April 2023, patients affected by organ-confined renal masses amenable to RAPN were enrolled. A bioengineer, a software developer, and a surgeon collaborated to create hyper-accurate 3D models for automatic 3D AR-guided RAPN, using CNNs. For each patient, demographic and clinical data were collected. A total of 13 patients were included in the present study. The average anchoring time was 11 (6–13) s. Unintended 3D-model automatic co-registration temporary failures happened in a static setting in one patient, while this happened in one patient in a dynamic setting. There was one failure; in this single case, an ultrasound drop-in probe was used to detect the neoplasm, and the surgery was performed under ultrasound guidance instead of AR guidance. No major intraoperative nor postoperative complications (i.e., Clavien Dindo > 2) were recorded. The employment of AI has unveiled several new scenarios in clinical practice, thanks to its ability to perform specific tasks autonomously. We employed CNNs for an automatic 3DVM overlapping during RAPN, thus improving the accuracy of the superimposition process.

Keywords: artificial intelligence | kidney cancer | nephron-sparing surgery | partial nephrectomy | renal cell carcinoma | robotic surgery | three-dimensional imaging

Abstract: Surgical operations must be preformed by skilled and qualified personnel. Training is therefore of paramount importance to master enough skill to succeeds in the tasks without endangering the health of the patients. Combining different types of training activities, can help reaching the goal in a quicker and more effective way. In this scenario, Virtual Reality represents an interesting step, particularly thanks to the reduction in the cost of the equipment that have made their adoption affordable and their availability widespread. In this work, we present our advances in training lung operations with low-cost virtual reality simulation environments.

Abstract: This work discusses the development and application of a parametric CAD model of the human bronchial tree, for use in computational fluid dynamics simulations. The model, which represents the trachea, bronchi, and early airway bifurcations, is based on geometrical parameters derived from existing literature. It can be edited by easily varying parameters in an external spreadsheet, offering an efficient alternative to patient-specific models, which often require the use of time-consuming segmentation procedures. The developed model was utilized to run fluid dynamic simulations, including a scenario that represents respiratory system dysfunctions. These are typical of diseases such as acute respiratory distress syndrome, which can be also triggered by recently emerged COVID-19. The results of these simulations were critically analyzed: they turned out to be consistent with the stated objectives and methods, even in the context of the existing literature. The paper concludes by discussing the limitations and potential improvements of the research.

Keywords: ARDS | bronchial tree | CFD | parametric modelling

Abstract: Conducting thoracoscopic surgery necessitates a high level of skill and specialized training. In addition to practicing on actual patients, surgeons utilize various safer training methods. A widely used training apparatus is the laparoscopic box, which contains multiple compartments fitted with training modules to replicate diverse surgical scenarios accurately. Unfortunately, the cost of these devices can be prohibitively high, making it challenging for individual students to acquire their own. Virtual Reality has emerged as a promising solution, especially due to the decreasing costs of VR equipment, which makes it more accessible. This study details our progress in enhancing training for lung surgeries by leveraging cost-effective VR technologies with head-mounted displays. In particular, simulations of thoracoscopic surgery scenarios within a VR environment have been developed, aiming to create training exercises that serve as an initial step before advancing to laparoscopic simulation boxes and, ultimately, actual surgical procedures on patients. For reproducing the behaviour of the soft lung tissues, we used a recent version of the Extended Position-Based Dynamics (XPBD) model, which solves some of the known problems encountered in the traditional PBD system, thereby enhancing the realism of the simulation.

Keywords: Position-based Dynamics | Training | Virtual Reality

Abstract: Objectives: The research's purpose is to develop a software that automatically integrates and overlay 3D virtual models of kidneys harboring renal masses into the Da Vinci robotic console, assisting surgeon during the intervention. Introduction: Precision medicine, especially in the field of minimally-invasive partial nephrectomy, aims to use 3D virtual models as a guidance for augmented reality robotic procedures. However, the co-registration process of the virtual images over the real operative field is performed manually. Methods: In this prospective study, two strategies for the automatic overlapping of the model over the real kidney were explored: the computer vision technology, leveraging the super-enhancement of the kidney allowed by the intraoperative injection of Indocyanine green for superimposition and the convolutional neural network technology, based on the processing of live images from the endoscope, after a training of the software on frames from prerecorded videos of the same surgery. The work-team, comprising a bioengineer, a software-developer and a surgeon, collaborated to create hyper-accuracy 3D models for automatic 3D-AR-guided RAPN. For each patient, demographic and clinical data were collected. Results: Two groups (group A for the first technology with 12 patients and group B for the second technology with 8 patients) were defined. They showed comparable preoperative and post-operative characteristics. Concerning the first technology the average co-registration time was 7 (3–11) seconds while in the case of the second technology 11 (6–13) seconds. No major intraoperative or postoperative complications were recorded. There were no differences in terms of functional outcomes between the groups at every time-point considered. Conclusion: The first technology allowed a successful anchoring of the 3D model to the kidney, despite minimal manual refinements. The second technology improved kidney automatic detection without relying on indocyanine injection, resulting in better organ boundaries identification during tests. Further studies are needed to confirm this preliminary evidence.

Keywords: artificial intelligence | kidney cancer | nephron-sparing surgery | renal cell carcinoma | robotic surgery | three-dimensional imaging

Abstract: Robotic surgery, also known as robotic-assisted surgery (RAS), has rapidly evolved during the last decade. RAS systems are developed to assist surgeons to perform complex minimally invasive surgeries, and necessitate augmented interfaces for precise execution of these image-guided procedures. Extended Reality (XR) technologies, augmenting the real-world perception via integrating digital contents, show promise in enhancing RAS efficacy in various studies. Despite multiple reviews on technological and medical aspects, the crucial elements of human-robot interaction (HRI) and user experience (UX) remain underexplored. This review fills this gap by elucidating HRI dynamics within XR-aided RAS systems, emphasizing their impact on UX and overall surgical outcomes. By synthesizing existing literature, this systematic review study identifies challenges and opportunities, paving the way for improved XR-enhanced robotic surgery, ultimately enhancing patient care and surgical performance.

Keywords: extended reality | human-computer interaction | robotic-assisted surgery | surgical robots | user experience

Abstract: Background: Addressing intraoperative bleeding remains a significant challenge in the field of robotic surgery. This research endeavors to pioneer a groundbreaking solution utilizing convolutional neural networks (CNNs). The objective is to establish a system capable of forecasting instances of intraoperative bleeding during robot-assisted radical prostatectomy (RARP) and promptly notify the surgeon about bleeding risks. Methods: To achieve this, a multi-task learning (MTL) CNN was introduced, leveraging a modified version of the U-Net architecture. The aim was to categorize video input as either “absence of blood accumulation” (0) or “presence of blood accumulation” (1). To facilitate seamless interaction with the neural networks, the Bleeding Artificial Intelligence-based Detector (BLAIR) software was created using the Python Keras API and built upon the PyQT framework. A subsequent clinical assessment of BLAIR’s efficacy was performed, comparing its bleeding identification performance against that of a urologist. Various perioperative variables were also gathered. For optimal MTL-CNN training parameterization, a multi-task loss function was adopted to enhance the accuracy of event detection by taking advantage of surgical tools’ semantic segmentation. Additionally, the Multiple Correspondence Analysis (MCA) approach was employed to assess software performance. Results: The MTL-CNN demonstrated a remarkable event recognition accuracy of 90.63%. When evaluating BLAIR’s predictive ability and its capacity to pre-warn surgeons of potential bleeding incidents, the density plot highlighted a striking similarity between BLAIR and human assessments. In fact, BLAIR exhibited a faster response. Notably, the MCA analysis revealed no discernible distinction between the software and human performance in accurately identifying instances of bleeding. Conclusion: The BLAIR software proved its competence by achieving over 90% accuracy in predicting bleeding events during RARP. This accomplishment underscores the potential of AI to assist surgeons during interventions. This study exemplifies the positive impact AI applications can have on surgical procedures.

Keywords: artificial intelligence | complications | prostate cancer | robotics

Abstract: More than ever, precision surgery is making its way into modern surgery for functional organ preservation. This is possible mainly due to the increasing number of technologies available, including 3D models, virtual reality, augmented reality, and artificial intelligence. Intraoperative surgical navigation represents an interesting application of these technologies, allowing to understand in detail the surgical anatomy, planning a patient-tailored approach. Automatic superimposition comes into this context to optimally perform surgery as accurately as possible. Through a dedicated software (the first version) called iKidney, it is possible to superimpose the images using 3D models and live endoscopic images during partial nephrectomy, targeting the renal mass only. The patient is 31 years old with a 28 mm totally endophytic right-sided renal mass, with a PADUA score of 9. Thanks to the automatic superimposition and selective clamping, an enucleoresection of the renal mass alone was performed with no major postoperative complication (i.e., Clavien–Dindo < 2). iKidney-guided partial nephrectomy is safe, feasible, and yields excellent results in terms of organ preservation and functional outcomes. Further validation studies are needed to improve the prototype software, particularly to improve the rotational axes and avoid human help. Furthermore, it is important to reduce the costs associated with these technologies to increase its use in smaller hospitals.

Keywords: 3D models | artificial intelligence | augmented reality | kidney cancer | robotics

Abstract: Objective: To evaluate the accuracy of our new three-dimensional (3D) automatic augmented reality (AAR) system guided by artificial intelligence in the identification of tumour's location at the level of the preserved neurovascular bundle (NVB) at the end of the extirpative phase of nerve-sparing robot-assisted radical prostatectomy. Methods: In this prospective study, we enrolled patients with prostate cancer (clinical stages cT1c–3, cN0, and cM0) with a positive index lesion at target biopsy, suspicious for capsular contact or extracapsular extension at preoperative multiparametric magnetic resonance imaging. Patients underwent robot-assisted radical prostatectomy at San Luigi Gonzaga Hospital (Orbassano, Turin, Italy), from December 2020 to December 2021. At the end of extirpative phase, thanks to our new AAR artificial intelligence driven system, the virtual prostate 3D model allowed to identify the tumour's location at the level of the preserved NVB and to perform a selective excisional biopsy, sparing the remaining portion of the bundle. Perioperative and postoperative data were evaluated, especially focusing on the positive surgical margin (PSM) rates, potency, continence recovery, and biochemical recurrence. Results: Thirty-four patients were enrolled. In 15 (44.1%) cases, the target lesion was in contact with the prostatic capsule at multiparametric magnetic resonance imaging (Wheeler grade L2) while in 19 (55.9%) cases extracapsular extension was detected (Wheeler grade L3). 3D AAR guided biopsies were negative in all pathological tumour stage 2 (pT2) patients while they revealed the presence of cancer in 14 cases in the pT3 cohort (14/16; 87.5%). PSM rates were 0% and 7.1% in the pathological stages pT2 and pT3 (<3 mm, Gleason score 3), respectively. Conclusion: With the proposed 3D AAR system, it is possible to correctly identify the lesion's location on the NVB in 87.5% of pT3 patients and perform a 3D-guided tailored nerve-sparing even in locally advanced diseases, without compromising the oncological safety in terms of PSM rates.

Keywords: Artificial intelligence | Augmented reality | Prostate cancer | Radical prostatectomy | Robotics

Abstract: The 6D pose estimation of an object from an image is a central problem in many domains of Computer Vision (CV) and researchers have struggled with this issue for several years. Traditional pose estimation methods (1) leveraged on geometrical approaches, exploiting manually annotated local features, or (2) relied on 2D object representations from different points of view and their comparisons with the original image. The two methods mentioned above are also known as Feature-based and Template-based, respectively. With the diffusion of Deep Learning (DL), new Learning-based strategies have been introduced to achieve the 6D pose estimation, improving traditional methods by involving Convolutional Neural Networks (CNN). This review analyzed techniques belonging to different research fields and classified them into three main categories: Template-based methods, Feature-based methods, and Learning-Based methods. In recent years, the research mainly focused on Learning-based methods, which allow the training of a neural network tailored for a specific task. For this reason, most of the analyzed methods belong to this category, and they have been in turn classified into three sub-categories: Bounding box prediction and Perspective-n-Point (PnP) algorithm-based methods, Classification-based methods, and Regression-based methods. This review aims to provide a general overview of the latest 6D pose recovery methods to underline the pros and cons and highlight the best-performing techniques for each group. The main goal is to supply the readers with helpful guidelines for the implementation of performing applications even under challenging circumstances such as auto-occlusions, symmetries, occlusions between multiple objects, and bad lighting conditions.

Keywords: 6D position estimation | Computer vision | Deep learning | RGB Input

Abstract: The current study aimed to propose a Deep Learning (DL) based framework to retrieve in real-time the position and the rotation of an object in need of maintenance from live video frames only. For testing the positioning performances, we focused on intervention on a generic Fused Deposition Modeling (FDM) 3D printer maintenance. Lastly, to demonstrate a possible Augmented Reality (AR) application that can be built on top of this, we discussed a specific case study using a Prusa i3 MKS FDM printer. This method was developed using a You Only Look Once (YOLOv3) network for object detection to locate the position of the FDM 3D printer and a subsequent Rotation Convolutional Neural Network (RotationCNN), trained on a dataset of artificial images, to predict the rotations’ parameters for attaching the 3D model. To train YOLOv3 we used an augmented dataset of 1653 real images, while to train the RotationCNN we utilized a dataset of 99.220 synthetic images, showing the FDM 3D Printer with different orientations, and fine-tuned it using 235 real images tagged manually. The YOLOv3 network obtained an AP (Average Precision) of 100% with Intersection Over Unit parameter of 0.5, while the RotationCNN showed a mean Geodesic Distance of 0.250 (σ = 0.210) and a mean accuracy to detect the correct rotation r of 0.619 (σ = 0.130), considering as acceptable the range [r − 10, r + 10]. We then evaluate the CAD system performances with 10 non-expert users: the average speed improved from 9.61 (σ = 1.53) to 5.30 (σ = 1.30) and the average number of actions to complete the task from 12.60 (σ = 2.15) to 11.00 (σ = 0.89). This work is a further step through the adoption of DL and AR in the assistance domain. In future works, we will overcome the limitations of this approach and develop a complete mobile CAD system that could be extended to any object that presents a 3D counterpart model.

Keywords: Augmented reality | CAD assistance | Deep learning | Neural network

Abstract: Point cloud 3D models are becoming more and more popular thanks to the spreading of scanning systems employed in many fields, like autonomous vehicles and robotics. When used for rendering purposes, point clouds are usually displayed with their original color acquired at scan time, without considering the lighting condition of the scene where the model is placed. This leads to a lack of realism in many contexts, especially in the case of animated point clouds employed in eXtended Reality applications where it would be desirable to have the model reacting to incoming light and integrating with the surrounding environment. This paper proposes the application of Physically Based Rendering (PBR), a rendering technique widely used in Real-Time Computer Graphics applications, to animated point cloud models for reproducing specular reflections, and achieving a photo-realistic and physically accurate look under any lighting condition. Firstly, we consider the extension of commonly used animated point cloud formats, to include normal vectors, and PBR parameters (such as roughness and Metalness), as well as the encoding of the animated environment maps required by the technique. Then, an animated point cloud model is rendered with a shader implementing the proposed PBR method. Finally, the proposed PBR pipeline is compared to traditional renderings of the same point cloud obtained with commonly used shaders, under different lighting conditions in different environments. It will be shown how, using the proposed PBR method, the point cloud better integrates visually with its surroundings. Moreover, it will be shown that using this rendering technique it is possible to render different materials, by exploiting the features of PBR and the encoding of the surrounding environment.

Keywords: Cloud computing | Lighting | Rendering (computer graphics) | Signal encoding

Abstract: The evaluation of hip implantation success remains one of the most relevant problems in orthopaedics. There are several factors that can cause its failure, e.g.: aseptic loosening and dislocations of the prosthetic joint due to implant impingement. Following a total hip arthroplasty, it is fundamental that the orthopaedist can evaluate which may be the possible risk factors that would lead to dislocation, or in the worst cases, to implant failure. A procedure has been carried out with the aim of evaluating the Range of Movement (ROM) of the implanted prosthesis, to predict whether the inserted implant is correctly positioned or will be prone to dislocation or material wear due to the malposition of its components. Leveraging on a previous patented methodology that consists in the 3D reconstruction and movement simulation of the hip joint, this work aims to provide a more effective visualization of the simulation results through Mixed Reality (MR). The use of MR for the representation of hip kinematics and implant position can provide the orthopaedic surgeon with a deeper understanding of the orientation and position of implanted components, as well as the consequences of such placements while looking directly at the patient. To this end, an anchoring system based on a body-tracking recognition library was developed, so that both completely automatic and human-assisted options are available without additional markers or sensors. An Augmented Reality (AR) prototype has been developed in Unity 3D and used on HoloLens 2, integrating the implemented human-assisted anchoring system option.

Keywords: Computer-aided surgery | HoloLens 2 | Mixed reality | THA assessment | Total hip arthroplasty

Abstract: Purpose: To evaluate the role of 3D models on positive surgical margin rate (PSM) rate in patients who underwent robot-assisted radical prostatectomy (RARP) compared to a no-3D control group. Secondarily, we evaluated the postoperative functional and oncological outcomes. Methods: Prospective study enrolling patients with localized prostate cancer (PCa) undergoing RARP with mp-MRI-based 3D model reconstruction, displayed in a cognitive or augmented-reality fashion, at our Centre from 01/2016 to 01/2020. A control no-3D group was extracted from the last two years of our Institutional RARP database. PSMr between the two groups was evaluated and multivariable linear regression (MLR) models were applied. Finally, Kaplan–Meier estimator was used to calculate biochemical recurrence at 12 months after the intervention. Results: 160 patients were enrolled in the 3D Group, while 640 were selected for the Control Group. A more conservative NS approach was registered in the 3D Group (full NS 20.6% vs 12.7%; intermediate NS 38.1% vs 38.0%; standard NS 41.2% vs 49.2%; p = 0.02). 3D Group patients had lower PSM rates (25 vs. 35.1%, p = 0.01). At MLR models, the availability of 3D technology (p = 0.005) and the absence of extracapsular extension (ECE, p = 0.004) at mp-MRI were independent predictors of lower PSMr. Moreover, 3D model represented a significant protective factor for PSM in patients with ECE or pT3 disease. Conclusion: The availability of 3D models during the intervention allows to modulate the NS approach, limiting the occurrence of PSM, especially in patients with ECE at mp-MRI or pT3 PCa.

Keywords: 3D modeling | Augmented reality | Prostate cancer | Robotic surgery | Surgical margins

Abstract: Following an oncological resection or trauma it may be necessary to reconstruct the normal anatomical and functional mandible structures to ensure the effective and complete social reintegration of patients. In most surgical procedures, reconstruction of the mandibular shape and its occlusal relationship is performed through the free fibula flap using a surgical guide which allows the surgeon to easily identify the location and orientation of the cutting plane. In the present work, we present a Mixed Reality (MR)-based solution to support professionals in surgical guide positioning. The proposed solution, through the use of a Head-Mounted Display (HMD) such as that of the HoloLens 2, visualizes a 3D virtual model of the surgical guide, positioned over the patient’s real fibula in the correct position as identified by the medical team before the procedure. The professional wearing the HMD is then assisted in positioning the real guide over the virtual one by our solution, which is capable of tracking the real guide during the whole process and computing its distance from the final position. The assessment results highlight that Mixed Reality is an eligible technology to support surgeons, combining the usability of the device with an improvement of the accuracy in fibula flap removal surgery.

Keywords: HoloLens 2 | mandibular reconstruction | maxillofacial surgery | mixed reality | surgical guide

Abstract: Introduction: The current study presents a deep learning framework to determine, in real-time, position and rotation of a target organ from an endoscopic video. These inferred data are used to overlay the 3D model of patient's organ over its real counterpart. The resulting augmented video flow is streamed back to the surgeon as a support during laparoscopic robot-assisted procedures. Methods: This framework exploits semantic segmentation and, thereafter, two techniques, based on Convolutional Neural Networks and motion analysis, were used to infer the rotation. Results: The segmentation shows optimal accuracies, with a mean IoU score greater than 80% in all tests. Different performance levels are obtained for rotation, depending on the surgical procedure. Discussion: Even if the presented methodology has various degrees of precision depending on the testing scenario, this work sets the first step for the adoption of deep learning and augmented reality to generalise the automatic registration process.

Keywords: abdominal | Kidney | prostate