Machine Vision for Construction Equipment by Transfer Learning with Scale Models

Machine vision is required by autonomous heavy construction equipment to navigate and interact with the environment. Wheel loaders need the ability to identify different objects and other equipment to perform the task of automatically loading and dumping material on dump trucks, which can be achieved using deep neural networks. Training such networks from scratch requires the iterative collection of potentially large amounts of video data, which is challenging at construction sites because of the complexity of safely operating heavy equipment in realistic environments. Transfer learning, for which pretrained neural networks can be retrained for use at construction sites, is thus attractive, especially if data can be acquired without full-scale experiments. We investigate the possibility of using scalemodel data for training and validating two different pretrained networks and use real-world test data to examine their generalization capability. A dataset containing 268 images of a 1:16 scale model of a Volvo A60H dump truck is provided, as well as 64 test images of a full-size Volvo A25G dump truck. The code and dataset are publicly available
¹
. The networks, both pretrained on the MS-COCO dataset, were fine-tuned to the created dataset, and the results indicate that both networks can learn the features of the scale-model dump truck (validation mAP of 0.82 for YOLOv3 and 0.95 for RetinaNet). Both networks can transfer these learned features to detect objects on a full-size dump truck with no additional training (test mAP of 0.70 for YOLOv3 and 0.79 for RetinaNet).