Towards Automated Construction: Visual-based Pose Reconstruction for Tower Crane Operations using Differentiable Rendering and Network-based Image Segmentation

This study focuses on visual-based pose reconstruction, aimed at automating construction sites. It addresses the pivotal challenge of camera-based pose reconstruction, crucial for robotic operations, as discussed in this work for tracking objects maneuvered by tower cranes. Central to this research is the formulation of a gradient-based optimization problem, with the objective of enhancing the alignment between synthetic model renderings and actual image captures through the use of differentiable rendering. Additionally, the study presents the design of a neural network tailored for image segmentation, intending to simplify the network architecture to reduce latency times and meet real-time operational demands. Although the entire procedure is motivated and discussed for the application of object tracking of a load by a tower crane, the presented framework is extendable, without loss of generality, to all camera-based object pose reconstructions, where a predefined geometric model of the target object is available. The adaptability and innovative applications of the framework highlight its significant contributions to advancing robotic vision and automation within the construction industry and beyond.