ITS Publications

ITS Group

• ITS Research Lab. ITS Magazine (2019). The ACFR Centre: ITS Group. Intelligent Transportation System Magazine, 2019.
  • Paper

• W. Zhou, J.S. Berrio, et al., 2020,  Developing and Testing Robust Autonomy: The University of Sydney Campus Data Set, IEEE Intelligent Transportation Systems Magazine, vol. 12, no. 4, pp. 23-40, winter 2020, doi: 10.1109/MITS.2020.2990183.
  • Paper
  • Abstract
    Vision and lidar are complementary sensors that are incorporated into many applications of intelligent transportation systems. These sensors have been used to great effect in research related to perception, navigation and deep-learning applications. Despite this success, the validation of algorithm robustness has recently been recognised as a major challenge for the massive deployment of these new technologies. To achieve this validation, it is necessary to evaluate the performance of the algorithms against a sufficiently large and diverse dataset to gain the confidence for deployment in a real-world application. Towards this goal, this paper presents a long-term, large-scale dataset collected over the period of 1.5 years on a weekly basis over the University of Sydney (USyd) campus and surrounds. The USyd Campus Dataset has been designed to aid in the development, validation and testing of algorithm robustness under a wide variety of conditions. It includes multiple sensor modalities and covers various environmental conditions as well as diverse changes to illumination, scene structure, and pedestrian/vehicle traffic volumes. The dataset also includes a semantic segmentation dataset with around 300 images covering different illumination conditions and camera perspectives and an automatically generated `Road’ class image dataset. To facilitate the dataset use by researchers and practitioners, we also provide a set of development tools for the analysis and visualisation of the dataset.
  • Video
  • BibTex
    @ARTICLE{zhou_berrio_Dataset2020,
    author={W. {Zhou} and J. S. {Berrio} and C. {De Alvis} and M. {Shan} and S. {Worrall} and J. {Ward} and E. {Nebot}},  journal={IEEE Intelligent Transportation Systems Magazine},   title={Developing and Testing Robust Autonomy: The University of Sydney Campus Data Set},
    year={2020},
    volume={12},  number={4},
    pages={23-40},
    doi={10.1109/MITS.2020.2990183}
    }

Path Planning

• Narula, K., Worrall, S., Nebot, E. (2020) Two-Level Hierarchical Planning in a Known Semi-Structured Environment. IEEE International Conference on Intelligent Transportation Systems (ITSC 2020).
  • Paper
  • Abstract
    The application of motion planning for autonomous vehicles has been primarily focused either in highly structured or unstructured environments. However, many environments in the real-world share the characteristics of both and can be classified as semi-structured. The adaptation of the strategies from other environments to that of semi-structured, although possible, do not produce trajectories with the required characteristics, especially when the environment is dynamic. To that end, this paper introduced a two-level hierarchical planning strategy consisting of a discrete lane-network-based global planner and a Hybrid A *local planner that: (i) generates a smooth, safe and kinematically feasible path in real-time; (ii) considers structural constraints of the environment from an a priori map. Furthermore, a valid lane-network-based sub-goaling strategy is proposed for providing a reference goal during the local planning process. Simulation and live tests have been conducted to evaluate the functionality of the strategy in several case studies. The implementation choices of: (i) using an open-source highly automated driving (HAD) map framework, Lanelet2, (ii) developing as plugins to an open-source navigation stack, Move Base Flex (MBF), allow the proposed strategy to be easily adopted on other autonomous platforms.
  • BibTex
    @INPROCEEDINGS{karan_2020,
    author={K. {Narula} and S. {Worrall} and E. {Nebot}},
    booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    title={Two-Level Hierarchical Planning in a Known Semi-Structured Environment},
    year={2020},
    volume={},
    number={},
    pages={1-6},
    doi={10.1109/ITSC45102.2020.9294373}}

Safety

• Karunakaran, D., Worrall, S., & Nebot, E. (2020). Efficient statistical validation with edge cases to evaluate Highly Automated Vehicles. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 1-8.
  • Paper
  • Abstract
    The widescale deployment of Autonomous Vehicles (AV) seems to be imminent despite many safety challenges that are yet to be resolved. It is well known that there are no universally agreed Verification and Validation (VV) methodologies to guarantee absolute safety, which is crucial for the acceptance of this technology. Existing standards focus on deterministic processes where the validation requires only a set of test cases that cover the requirements. Modern autonomous vehicles will undoubtedly include machine learning and probabilistic techniques that require a much more comprehensive testing regime due to the non-deterministic nature of the operating design domain. A rigorous statistical validation process is an essential component required to address this challenge. Most research in this area focuses on evaluating system performance in large-scale real-world data gathering exercises (number of miles travelled), or randomised test scenarios in simulation. This paper presents a new approach to compute the statistical characteristics of a system’s behaviour by biasing automatically generated test cases towards the worst case scenarios, identifying potential unsafe edge cases. We use reinforcement learning (RL) to learn the behaviours of simulated actors that cause unsafe behaviour measured by the well established RSS safety metric. We demonstrate that by using the method we can more efficiently validate a system using a smaller number of test cases by focusing the simulation towards the worst case scenario, generating edge cases that correspond to unsafe situations.
  • BibTex
    @article{Karunakaran2020EfficientSV,
    title={Efficient statistical validation with edge cases to evaluate Highly Automated Vehicles},
    author={Dhanoop Karunakaran and S. Worrall and E. Nebot},
    journal={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    year={2020},
    pages={1-8}}}

Perception

Calibration

• D. Tsai, S. Worrall, M. Shan, A. Lohr and E. Nebot. (2021). Optimising the selection of samples for robust lidar camera calibration, IEEE International Intelligent Transportation Systems Conference (IEEE ITSC 2021).
  • Paper
  • Abstract
    We propose a robust calibration pipeline that optimises the selection of calibration samples for the estimation of calibration parameters that fit the entire scene. We minimise user error by automating the data selection process according to a metric, called Variability of Quality (VOQ) that gives a score to each calibration set of samples. We show that this VOQ score is correlated with the estimated calibration parameter’s ability to generalise well to the entire scene, thereby overcoming the overfitting problems of existing calibration algorithms. Our approach has the benefits of simplifying the calibration process for practitioners of any calibration expertise level and providing an objective measure of the quality for our calibration pipeline’s input and output data. We additionally use a novel method of assessing the accuracy of the calibration parameters. It involves computing reprojection errors for the entire scene to ensure that the parameters are well fitted to all features in the scene. Our proposed calibration pipeline takes 90s, and obtains an average reprojection error of 1-1.2cm, with standard deviation of 0.4-0.5cm over 46 poses evenly distributed in a scene. This process has been validated by experimentation on a high resolution, software definable lidar, Baraja Spectrum-Scan™; and a low, fixed resolution lidar, Velodyne VLP-16. We have shown that despite the vast differences in lidar technologies, our proposed approach manages to estimate robust calibration parameters for both. Our code and data set used for this paper are made available at https://github.com/acfr/cam_lidar_calibration.
  • Video
  • Code
  • BibTex
    @INPROCEEDINGS{Darren_2021,author={Tsai, Darren and Worrall, Stewart and Shan, Mao and Lohr, Anton and Nebot, Eduardo},
    booktitle={2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
    title={Optimising the selection of samples for robust lidar camera calibration},
    year={2021},
    volume={},
    number={},pages={2631-2638},
    doi={10.1109/ITSC48978.2021.9564700}}
• Verma, S., Berrio Perez, J., Worrall, S.., Nebot, E. (2019). Automatic extrinsic calibration between a camera and a 3D Lidar using 3D point and plane correspondences. IEEE 22nd International Conference on Intelligent Transportation Systems (IEEE ITSC 2019), Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    This paper proposes an automated method to obtain the extrinsic calibration parameters between a camera and a 3D lidar with as low as 16 beams. We use a checkerboard as a reference to obtain features of interest in both sensor frames. The calibration board centre point and normal vector are automatically extracted from the lidar point cloud by exploiting the geometry of the board. The corresponding features in the camera image are obtained from the camera’s extrinsic matrix. We explain the reasons behind selecting these features, and why they are more robust compared to other possibilities. To obtain the optimal extrinsic parameters, we choose a genetic algorithm to address the highly non-linear state space. The process is automated after defining the bounds of the 3D experimental region relative to the lidar, and the true board dimensions. In addition, the camera is assumed to be intrinsically calibrated. Our method requires a minimum of 3 checkerboard poses, and the calibration accuracy is demonstrated by evaluating our algorithm using real world and simulated features.
  • BibTex
    @INPROCEEDINGS{surabhi_calib_2019,
    author={S. {Verma} and J. S. {Berrio} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Transportation Systems Conference (ITSC)},
    title={Automatic extrinsic calibration between a camera and a 3D Lidar using 3D point and plane correspondences},
    year={2019},
    volume={},
    number={},
    pages={3906-3912},
    doi={10.1109/ITSC.2019.8917108}}
  • Video
  • Code
• De Alvis, C., Shan, M., Worrall, S., Nebot, E. (2019) Uncertainty Estimation for Projecting Lidar Points onto Camera Images for Moving Platforms. IEEE International Conference on Robotics and Automation (ICRA 2019).
  • Paper
  • Abstract
    Combining multiple sensors for advanced perception is a crucial requirement for autonomous vehicle navigation. Heterogeneous sensors are used to obtain rich information about the surrounding environment. The combination of the camera and lidar sensors enables precise range information that can be projected onto the visual image data. This gives a high level understanding of the scene which can be used to enable context based algorithms such as collision avoidance and navigation. The main challenge when combining these sensors is aligning the data into a common domain. This can be difficult due to the errors in the intrinsic calibration of the camera, extrinsic calibration between the camera and the lidar and errors resulting from the motion of the platform. In this paper, we examine the algorithms required to provide motion correction for scanning lidar sensors. The error resulting from the projection of the lidar measurements into a consistent odometry frame is not possible to remove entirely, and as such it is essential to incorporate the uncertainty of this projection when combining the two different sensor frames. This work proposes a novel framework for the prediction of the uncertainty of lidar measurements (in 3D) projected in to the image frame (in 2D) for moving platforms. The proposed approach fuses the uncertainty of the motion correction with uncertainty resulting from errors in the extrinsic and intrinsic calibration. By incorporating the main components of the projection error, the uncertainty of the estimation process is better represented. Experimental results for our motion correction algorithm and the proposed extended uncertainty model are demonstrated using real-world data collected on an electric vehicle equipped with wide-angle cameras covering a 180-degree field of view and a 16-beam scanning lidar..
  • BibTex
    @INPROCEEDINGS{charika_2019,
    author={C. {De Alvis} and M. {Shan} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 International Conference on Robotics and Automation (ICRA)},
    title={Uncertainty Estimation for Projecting Lidar Points onto Camera Images for Moving Platforms},
    year={2019},
    volume={},
    number={},
    pages={6637-6643},
    doi={10.1109/ICRA.2019.8794424}}

Sensor Fusion

• J. S. Berrio, M. Shan, S. Worrall and E. Nebot, “Camera-LIDAR Integration: Probabilistic Sensor Fusion for Semantic Mapping,” in IEEE Transactions on Intelligent Transportation Systems.
  • Paper
  • Abstract
    An automated vehicle operating in an urban environment must be able to perceive and recognise objects and obstacles in a three-dimensional world for navigation and path planning. In order to plan and execute accurate and sophisticated driving maneuvers, a high-level contextual understanding of the surroundings is essential. Due to the recent progress in image processing, it is now possible to obtain high definition semantic information in 2D from monocular cameras, though cameras cannot reliably provide the high accuracy 3D information provided by lasers. The fusion of these two sensor modalities can overcome the shortcomings of each individual sensor, though there are a number of important challenges that need to be addressed in a probabilistic manner. In this paper we address the common, yet challenging, LIDAR/camera/semantic fusion problems which are seldom approached in a wholly probabilistic manner. Our approach is capable of using a multi-sensor platform to build a three-dimensional semantic voxelized map that considers the uncertainty of all of the processes involved. We present a probabilistic pipeline that incorporates uncertainty from the sensor readings (cameras, LIDAR, IMU and wheel encoders), compensation for the motion of the vehicle, and heuristic label probabilities for the semantic images depicted in Fig. 1. We also present a novel and efficient viewpoint validation algorithm to check for occlusions within the camera frame. A probabilistic projection is performed from the camera images to the LIDAR point cloud. Each labelled LIDAR scan then feeds into an octree map-building algorithm that updates the class probabilities of the map voxels every time a new observation is available. We validate our approach using a set of qualitative and quantitative experiments using the USyd Campus Dataset.1 These tests demonstrate the usefulness of a probabilistic sensor fusion approach by evaluating the performance of the perception system in a typical autonomous vehicle application.
  • BibTex
    @ARTICLE{Steph_2021, author={Berrio, Julie Stephany and Shan, Mao and Worrall, Stewart and Nebot, Eduardo},
    journal={IEEE Transactions on Intelligent Transportation Systems},
    title={Camera-LIDAR Integration: Probabilistic Sensor Fusion for Semantic Mapping},
    year={2021},
    volume={},
    number={},
    pages={1-16},
    doi={10.1109/TITS.2021.3071647}}
• Berrio Perez, J., Ward, J., Worrall, S., Zhou, W., Nebot, E. (2017). Fusing Lidar and Semantic Image Information in Octree Maps. Australasian Conference on Robotics and Automation (ACRA 2017), Sydney: Australian Robotics and Automation Association (ARAA).
  • Paper
  • Abstract
    Current autonomous driving applications requirenot only the occupancy information of the close environment butalso reactive maps to represent dynamic surroundings. Thereis also benefit from incorporating semantic classification intothe map to assist the path planning in changing scenarios. Thispaper presents an approach to building a multi-label semantic3D octree map based on the Octomap mapping framework.Current state-of-the-art point cloud classification methods suchas conditional random fields (CRF), random forest (RF) andsupport vector machines (SVM) use dense point clouds in orderto train the model that assign a label to each point. This workutilizes images from a convolutional neural network to providethe semantic context of the local environment and projectsthe classification into a 3D lidar point cloud. The resultingpoint cloud feeds into the octree map building algorithmand computes the corresponding probabilities (occupancy andclassification) for every 3D voxel. We also propose a method toincorporate the uncertainty of semantic labels based on the pixeldistance to the label boundaries. The algorithm is validatedusing data collected by our mobile vehicle platform drivingaround the University of Sydney campus.
  • BibTex
    @inproceedings{berrio2017fusing,
    title={Fusing Lidar and Semantic Image Information in Octree Maps},
    author={Berrio, Julie Stephany and Ward, James and Worrall, Stewart and Zhou, Wei and Nebot, Eduardo},
    booktitle={ACRA Australasian Conference on Robotics and Automation 2017},
    year={2017} }
  • Video
• Berrio Perez, J., Zhou, W., Ward, J., Worrall, S., Nebot, E. (2018). Octree map based on sparse point cloud and heuristic probability distribution for labeled images. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018), Piscataway: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    To navigate through urban roads, an automated vehicle must be able to perceive and recognize objects in a three-dimensional environment. A high level contextual understanding of the surroundings is necessary to execute accurate driving maneuvers. This paper presents a novel approach to build three dimensional semantic octree maps from lidar scans and the output of a convolutional neural network (CNN) to obtain the labels of the environment. We present a heuristic method to associate uncertainties to the labels from the images based on a combination of the labels themselves, score maps retrieved by the CNN and the raw images. These uncertainties and the camera-lidar calibration parameters for multiple cameras are considered in the projection of the labels and their uncertainties into the point cloud. Every labeled lidar scan works as an input to an octree map building algorithm that calculates and updates the label probabilities of the voxels in the map. This paper also presents a qualitative and quantitative evaluation of accuracy, analyzing projection in single lidar scans and complete maps built with our probabilistic octree framework.
  • BibTex
    @inproceedings{berrio2018octree,
    title={Octree map based on sparse point cloud and heuristic probability distribution for labeled images},
    author={Berrio, Julie Stephany and Zhou, Wei and Ward, James and Worrall, Stewart and Nebot, Eduardo},
    booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    pages={3174–3181}, year={2018},
    organization={IEEE} }
  • Video
• M. Shan, J. S. Berrio, S. Worrall and E. Nebot, (2020), Probabilistic Egocentric Motion Correction of Lidar Point Cloud and Projection to Camera Images for Moving Platforms, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, pp. 1-8, doi: 10.1109/ITSC45102.2020.9294601.
  • Paper
  • Abstract
    The fusion of sensor data from heterogeneous sensors is crucial for robust perception in various robotics applications that involve moving platforms, for instance, autonomous vehicle navigation. In particular, combining camera and lidar sensors enables the projection of precise range information of the surrounding environment onto visual images. It also makes it possible to label each lidar point with visual segmentation/classification results for 3D mapping, which facilitates a higher level understanding of the scene. The task is however considered non-trivial due to intrinsic and extrinsic sensor calibration, and the distortion of lidar points resulting from the ego-motion of the platform. Despite the existence of many lidar ego-motion correction methods, the errors in the correction process due to uncertainty in egomotion estimation are not possible to remove completely. It is thus essential to consider the problem a probabilistic process where the ego-motion estimation uncertainty is modelled and considered consistently. The paper investigates the probabilistic lidar ego-motion correction and lidar-to-camera projection, where both the uncertainty in the ego-motion estimation and time jitter in sensory measurements are incorporated. The proposed approach is validated both in simulation and using real-world data collected from an electric vehicle retrofitted with wide-angle cameras and a 16-beam scanning lidar.
  • BibTex
    @INPROCEEDINGS{shan_berrio_motion_c_2020,
    author={M. {Shan} and J. S. {Berrio} and S. {Worrall} and E. {Nebot}},
    booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    title={Probabilistic Egocentric Motion Correction of Lidar Point Cloud and Projection to Camera Images for Moving Platforms},
    year={2020},
    volume={},
    number={},
    pages={1-8},
    doi={10.1109/ITSC45102.2020.9294601}}

Semantic Segmentation

• Zhou, W., Arroyo, R., Zyner, A., Ward, J., Worrall, S., Nebot, E., Bergasa, L. (2017). Transferring visual knowledge for a robust road environment perception in intelligent vehicles. IEEE 20th International Conference on Intelligent Transportation Systems (IEEE ITSC 2017), Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    Vision-based urban scene recognition is capable of providing semantic information that will have a significant impact in intelligent transportation systems. Semantic segmentation can provide high-level information from images of urban scenes, but we have discovered that existing models trained on public datasets often do not adapt well to other environments. This work explores the transferability of Convolution Neural Networks (CNNs) features by retraining the network using a minimal dataset incorporating training data specific to the local environment. A new local dataset is manually annotated and used to train a neural network for pixel-level semantic image information. Since data annotation is time-consuming, we evaluate the transferability of CNNs and the performance of different data augmentation methods for dataset expansion. Small datasets are normally considered not sufficient for training a neural network from scratch. This paper presents an incremental fine-tunning algorithm to update the pre-trained network. The experimental results show that it is possible to successfully transfer semantic features to a different environment by incorporating a relatively small number of local images.
  • BibTex
    @inproceedings{zhou2017transferring,
    title={Transferring visual knowledge for a robust road environment perception in intelligent vehicles},
    author={Zhou, Wei and Arroyo, Roberto and Zyner, Alex and Ward, James and Worrall, Stewart and Nebot, Eduardo and Bergasa, Luis M},
    booktitle={2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)},
    pages={1–6},
    year={2017},
    organization={IEEE}
    }
• Zhou, W., Worrall, S., Zyner, A., Nebot, E. (2018). Automated Process for Incorporating Drivable Path into Real-time Semantic Segmentation. IEEE International Conference on Robotics and Automation (ICRA 2018), Brisbane: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    Vision systems are widely used in autonomous vehicle systems due to the rich information that camera sensors provide of the surrounding environment. This paper presents an automatic algorithm to obtain the drivable path of a vehicle operating in urban roads with or without clear lane markings. The developed system projects trajectories obtained during human operation of the vehicle and utilizes these to generate automatic labels for training a semantic based path prediction model. The system segments an urban scenario into 13 categories including vehicles, pedestrian, undrivable road, other categories relevant to urban roads, and a new class for a path proposal. The drivable path information is essential particularly in unstructured scenarios, and is critical for an intelligent vehicle system to make sound driving decisions. The path proposal category is a car-width drivable lane estimated to be safe to drive for the vehicle under consideration. The data collection, model training and inference process requires only images from a monocular camera and odometry from a low-cost IMU combined with a wheel encoder. The algorithm has been successfully demonstrated on the Sydney University campus, which is a challenging environment without clear road markings. The algorithm was demonstrated to run in real-time, proving its applicability for intelligent vehicles.
  • BibTex
    @inproceedings{zhou2018automated,
    title={Automated Process for Incorporating Drivable Path into Real-Time Semantic Segmentation},
    author={Zhou, Wei and Worrall, Stewart and Zyner, Alex and Nebot, Eduardo},
    booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)},
    pages={1–6},
    year={2018},
    organization={IEEE}
    }
  • Video
• Zhou, W., Zyner, A., Worrall, S., Nebot, E. (2019). Adapting Semantic Segmentation Models for Changes in Illumination and Camera Perspective. IEEE Robotics and Automation Letters, 4(2), 461-468.
  • Paper
  • Abstract
    Semantic segmentation using deep neural networks has been widely explored to generate high-level contextual information for autonomous vehicles. To acquire a complete 180° semantic understanding of the forward surroundings, we propose to stitch semantic images from multiple cameras with varying orientations. However, previously trained semantic segmentation models showed unacceptable performance after significant changes to the camera orientations and the lighting conditions. To avoid time-consuming hand labeling, we explore and evaluate the use of data augmentation techniques, specifically skew and gamma correction, from a practical real-world standpoint to extend the existing model and provide more robust performance. The experimental results presented have shown significant improvements with varying illumination and camera perspective changes. A comparison of the results from a high-performance network (PSPNet), and a realtime capable network (ENet) is provided.
  • BibTex
    @article{zhou2019adapting,
    title={Adapting semantic segmentation models for changes in illumination and camera perspective},
    author={Zhou, Wei and Zyner, Alex and Worrall, Stewart and Nebot, Eduardo},
    journal={IEEE Robotics and Automation Letters},
    volume={4},
    number={2},
    pages={461–468},
    year={2019},
    publisher={IEEE}
    }
• Zhou, W., Berrio, J.S., Worrall, S. and Nebot, E., (2019). Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems. 
  • Paper
  • Abstract
    One of the fundamental challenges in the design of perception systems for autonomous vehicles is validating the performance of each algorithm under a comprehensive variety of operating conditions. In the case of vision-based semantic segmentation, there are known issues when encountering new scenarios that are sufficiently different to the training data. In addition, even small variations in environmental conditions, such as illumination and precipitation, can affect the classification performance of the segmentation model. Given the reliance on visual information, these effects often translate into poor semantic pixel classification which can potentially lead to catastrophic consequences when driving autonomously. This paper presents a novel method for analyzing the robustness of semantic segmentation models and provides a number of metrics to evaluate the classification performance over a variety of environmental conditions. The process incorporates an additional sensor (lidar) to automate the process and improve the system integrity, eliminating the need for labor-intensive hand labeling of validation data. The experimental results are presented based on multiple datasets collected at different times of the year with different environmental conditions. We extract the “Road” class using the lidar to demonstrate the concepts, but this could be extended to other classes with different feature detection algorithms. These results show that the semantic segmentation performance varies depending on the weather, camera parameters, and existence of shadows. The results also demonstrate how the metrics can be used to compare and validate the performance after making improvements to a model, and compare the performance of different networks.
  • BibTex
    @article{zhou2019automated,
    title={Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving},
    author={Zhou, Wei and Berrio, Julie Stephany and Worrall, Stewart and Nebot, Eduardo},
    journal={IEEE Transactions on Intelligent Transportation Systems},
    year={2019},
    publisher={IEEE}
    }

Detection

• Gomez-Donoso, E. Cruz, M. Cazorla, S. Worrall and E. Nebot, 2020, Using a 3D CNN for Rejecting False Positives on Pedestrian Detection, 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, pp. 1-6, doi: 10.1109/IJCNN48605.2020.9207432.
  
  • Paper
  • Abstract
    Self-driving cars are becoming slowly but surely the future of transport. Nonetheless, in order to achieve fully automatic operation, several challenges are still needed to be tackled. One of the main goals that is currently being pursued is a very accurate scene understanding and object detection. In this regard, the most accurate object detectors are image-based. However, these methods yield critical flaws that make them prone to error in some specific scenarios. For instance, actual objects would be detected in depictions of such objects. The urban environment is strewn with these cases. Namely, in billboards and advertisements. However, most of the self-driving cars feature a lidar that provides 3D perception. This sensor could help to disambiguate the cases mentioned before.In this paper, we combine the accuracy of 2D deep learning object detectors with a 3D Convolutional Neural Network (3D CNN) for rejecting false positives on pedestrian detection. First, the object detector provides all the detected pedestrians in the scene, and then the 3D CNN is in charge of rejecting or verify the detections. Our proposal is tested on two well-known publicly available datasets and provides up to 84% accuracy
  • BibTex
    @INPROCEEDINGS{gomez-2020,
    author={F. {Gomez-Donoso} and E. {Cruz} and M. {Cazorla} and S. {Worrall} and E. {Nebot}},
    booktitle={2020 International Joint Conference on Neural Networks (IJCNN)},
    title={Using a 3D CNN for Rejecting False Positives on Pedestrian Detection},
    year={2020},
    volume={},
    number={},
    pages={1-6},
    doi={10.1109/IJCNN48605.2020.9207432}

Classification

• H. C. Sánchez, N. H. Parra, I. P. Alonso, E. Nebot and D. Fernández-Llorca, Are We Ready for Accurate and Unbiased Fine-Grained Vehicle Classification in Realistic Environments?, in IEEE Access, vol. 9, pp. 116338-116355, 2021.
  • Paper
  • Abstract
    Predicting the condition of the road is an important task for autonomous vehicles to make driving decisions. Vehicles are expected to slow down or stop for potential road risks such as road cracks, bumps and potholes. Vision systems are widely used to provide such information given the rich colours and textures carried by images. This paper presents a weakly-supervised deep learning method to classify road images into two category sets. The first category identifies the existence of bumps or ramps in the image. The second category determines the road roughness given an input image. These two outputs are combined into a single convolutional neural network (CNN) to classify the camera image simultaneously. As a supervised learning method, deep learning algorithms normally require a large amount of training data with manually annotated labels. The annotation process is, however, very time-consuming and labour-intensive. This paper presents a method to avoid this costly process using a pipeline to automatically generate ground-truth labels by incorporating IMU and wheel encoder data. This automated pipeline does not require human effort to label images and will not be impeded by adverse environmental or illumination conditions. The experimental results presented show that after training the model using the automatically generated labels, the two-output CNN is capable to achieve good accuracy for classifying road conditions.
  • BibTex
    @ARTICLE{Hector_2021,
    author={Sánchez, Héctor Corrales and Parra, Noelia Hernández and Alonso, Ignacio Parra and Nebot, Eduardo and Fernández-Llorca, David},
    journal={IEEE Access},
    title={Are We Ready for Accurate and Unbiased Fine-Grained Vehicle Classification in Realistic Environments?},
    year={2021},
    volume={9}, number={},
    pages={116338-116355},
    doi={10.1109/ACCESS.2021.3104340}}
• Zhou, Wei & Cruz, Edmanuel & Worrall, Stewart & Gomez-Donoso, Francisco & Cazorla, Miguel & Nebot, Eduardo. (2020). Weakly-supervised Road Condition Classification Using Automatically Generated Labels. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) 1-7. 10.1109/ITSC45102.2020.9294207.
  • Paper
  • Abstract
    Predicting the condition of the road is an important task for autonomous vehicles to make driving decisions. Vehicles are expected to slow down or stop for potential road risks such as road cracks, bumps and potholes. Vision systems are widely used to provide such information given the rich colours and textures carried by images. This paper presents a weakly-supervised deep learning method to classify road images into two category sets. The first category identifies the existence of bumps or ramps in the image. The second category determines the road roughness given an input image. These two outputs are combined into a single convolutional neural network (CNN) to classify the camera image simultaneously. As a supervised learning method, deep learning algorithms normally require a large amount of training data with manually annotated labels. The annotation process is, however, very time-consuming and labour-intensive. This paper presents a method to avoid this costly process using a pipeline to automatically generate ground-truth labels by incorporating IMU and wheel encoder data. This automated pipeline does not require human effort to label images and will not be impeded by adverse environmental or illumination conditions. The experimental results presented show that after training the model using the automatically generated labels, the two-output CNN is capable to achieve good accuracy for classifying road conditions.
  • BibTex
    @INPROCEEDINGS{9294207,
    author={W. {Zhou} and E. {Cruz} and S. {Worrall} and F. {Gomez-Donoso} and M. {Cazorla} and E. {Nebot}},
    booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    title={Weakly-supervised Road Condition Classification Using Automatically Generated Labels},
    year={2020},
    volume={},
    number={},
    pages={1-7},
    doi={10.1109/ITSC45102.2020.9294207}}

Collaborative Perception

• Shan, M.; Narula, K.; Wong, Y.F.; Worrall, S.; Khan, M.; Alexander, P.; Nebot, E. (2021) Demonstrations of Cooperative Perception: Safety and Robustness in Connected and Automated Vehicle Operations. Sensors 2021, 21, 200. https://doi.org/10.3390/s21010200
  • Paper
  • Abstract
    Predicting the condition of the road is an important task for autonomous vehicles to make driving decisions. Vehicles are expected to slow down or stop for potential road risks such as road cracks, bumps and potholes. Vision systems are widely used to provide such information given the rich colours and textures carried by images. This paper presents a weakly-supervised deep learning method to classify road images into two category sets. The first category identifies the existence of bumps or ramps in the image. The second category determines the road roughness given an input image. These two outputs are combined into a single convolutional neural network (CNN) to classify the camera image simultaneously. As a supervised learning method, deep learning algorithms normally require a large amount of training data with manually annotated labels. The annotation process is, however, very time-consuming and labour-intensive. This paper presents a method to avoid this costly process using a pipeline to automatically generate ground-truth labels by incorporating IMU and wheel encoder data. This automated pipeline does not require human effort to label images and will not be impeded by adverse environmental or illumination conditions. The experimental results presented show that after training the model using the automatically generated labels, the two-output CNN is capable to achieve good accuracy for classifying road conditions.
  • BibTex
    @Article{shan_coop_2021,
    AUTHOR = {Shan, Mao and Narula, Karan and Wong, Yung Fei and Worrall, Stewart and Khan, Malik and Alexander, Paul and Nebot, Eduardo},
    TITLE = {Demonstrations of Cooperative Perception: Safety and Robustness in Connected and Automated Vehicle Operations},
    JOURNAL = {Sensors},
    VOLUME = {21},
    YEAR = {2021},
    NUMBER = {1},
    ARTICLE-NUMBER = {200},
    URL = {https://www.mdpi.com/1424-8220/21/1/200},
    ISSN = {1424-8220},
    DOI = {10.3390/s21010200}}

Tracking and prediction

• K. Li, M. Shan, S. Eiffert, S. Worrall and E. Nebot, Towards Collision-Free Probabilistic Pedestrian Motion Prediction for Autonomous Vehicles, 2022 IEEE 33rd IEEE Intelligent Vehicles Symposium.
  • Abstract
    Autonomous vehicle navigation in shared pedestrian environments requires the ability to predict future crowd motion as well as understand human behaviour. However, most existing methods predict pedestrian future motion without considering potential collisions within the crowd. Furthermore, most current predictive models are tested on datasets that assume full observability of the crowd by relying on a top-down view, which does not reflect the real-world use case of autonomous vehicles due to the inherent limitations of on-board sensors such as visual occlusion. Inspired by prior works, we propose a pedestrian motion prediction model trained via contrastive learning, improving prediction accuracy as well as forecasting collision-free trajectories. Additionally, we propose a method for implementing a predictor using a multi-pedestrian probabilistic tracker, which fuses multiple on-board sensors to track pedestrians in 3D space. Through comprehensive experiments on both aerial view and driving datasets collected in a real-world urban environment, we show that our proposed method improves on state of art methods with better prediction accuracy and more socially acceptable prediction trajectories.
  • BibTex
    @inproceedings{li2022towards,
    title={Towards Collision-Free Probabilistic Pedestrian Motion Prediction for Autonomous Vehicles},
    author={Li, Kunming and Shan, Mao and Stuart Eiffert and Worrall, Stewart and Nebot, Eduardo},
    booktitle={2022 IEEE 33rd IEEE Intelligent Vehicles Symposium},
    pages={},
    year={2022},
    organization={IEEE} }
  • video
• K. Li, S. Eiffert, M. Shan, F. Gomez-Donoso, S. Worrall and E. Nebot, Attentional-GCNN: Adaptive Pedestrian Trajectory Prediction towards Generic Autonomous Vehicle Use Cases, 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
  • Paper
  • Abstract
    Autonomous vehicle navigation in shared pedestrian environments requires the ability to predict future crowd motion both accurately and with minimal delay. Understanding the uncertainty of the prediction is also crucial. Most existing approaches however can only estimate uncertainty through repeated sampling of generative models. Additionally, most current predictive models are trained on datasets that assume complete observability of the crowd using an aerial view. These are generally not representative of real-world usage from a vehicle perspective, and can lead to the underestimation of uncertainty bounds when the on-board sensors are occluded. Inspired by prior work in motion prediction using spatio-temporal graphs, we propose a novel Graph Convolutional Neural Network (GCNN)-based approach, Attentional-GCNN, which aggregates information of implicit interaction between pedestrians in a crowd by assigning attention weight in edges of the graph. Our model can either output a probabilistic distribution or faster deterministic prediction, demonstrating applicability to autonomous vehicle use cases where either speed or accuracy with uncertainty bounds are required. To further improve the training of predictive models, we propose an automatically labelled pedestrian dataset collected from an intelligent vehicle platform representative of real-world use. Through experiments on a number of datasets, we show our proposed method achieves an improvement over the state of the art by 10% on Average Displacement Error (ADE) and 12% on Final Displacement Error (FDE) with fast inference speeds.
  • BibTex
    @INPROCEEDINGS{9561480,
    author={Li, Kunming and Eiffert, Stuart and Shan, Mao and Gomez-Donoso, Francisco and Worrall, Stewart and Nebot, Eduardo},
    booktitle={2021 IEEE International Conference on Robotics and Automation (ICRA)},
    title={Attentional-GCNN: Adaptive Pedestrian Trajectory Prediction towards Generic Autonomous Vehicle Use Cases},
    year={2021},
    volume={}, number={},
    pages={14241-14247},
    doi={10.1109/ICRA48506.2021.9561480}}
• Konrad, S., Shan, M., Masson, F., Worrall, S., Nebot, E. (2018). Pedestrian Dynamic and Kinematic Information Obtained from Vision Sensors. 29th IEEE Intelligent Vehicle Symposium, Piscataway: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    The estimation and prediction of pedestrian motion is of fundamental importance in ITS applications. Most existing solutions have utilized a particular type of sensor for perception such as cameras (stereo, monocular, infrared) or other modalities such as a laser range finder or radar. The advent of wearable devices with inertial sensors have led to the development of systems capable of the robust inference of pedestrian intention. Unfortunately, these devices do not have communications capabilities to broadcast this information to all vehicles in proximity, and also this strategy requires functioning devices on all pedestrians to work. This paper presents a robust perception method that is able to extract dynamic pedestrian information with accuracy comparable to that of typical gyroscopes and accelerometers installed in wearable devices. Experimental results are presented to demonstrate the potential for obtaining very comprehensive dynamic information from limbs representing the skeleton of a pedestrian. This work also demonstrates the accuracy of vision based systems by comparing these results to the rotation and acceleration measured directly on the pedestrian using a wearable device. The contributions of this paper demonstrate that it is possible to significantly improve both the detection and estimation of pedestrian intention by incorporating dynamic information obtained from vision sensors.
  • BibTex
    @inproceedings{konrad2018pedestrian,
    title={Pedestrian dynamic and kinematic information obtained from vision sensors},
    author={Konrad, Santiago Gerling and Shan, Mao and Masson, Favio R and Worrall, Stewart and Nebot, Eduardo},
    booktitle={2018 IEEE Intelligent Vehicles Symposium (IV)},
    pages={1299–1305},
    year={2018},
    organization={IEEE} }
• Gerling Konrad, S., Masson, F., Nebot, E. (2018). Analysis of accuracy of pedestrian inertial data obtained from cameras images. IEEE Biennial Congress of Argentina (ARGENCON 2018), San Miguel de Tucuman: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    Wearable devices have inertial sensors that provide useful information to estimate and predict pedestrian motion and intention, which is of fundamental importance in ITS applications. These devices are usually placed in the limbs, such as wrist, ankles and feet and they provide rotation rate and acceleration information. This information is essential for the successful development of systems capable of inferencing pedestrian intentions. Unfortunately these devices do not have the capabilities to broadcast information to all vehicles in proximity and require all pedestrian to be retrofitted with such capability. This is the fundamental reason why all existing approaches are based on sensing installed directly in the vehicles. Intelligent vehicles have different types of sensors to perceive the environment in proximity, the most common being cameras. This work demonstrates that vision from cameras is capable of obtaining pedestrian dynamics with similar accuracy of wearables devices. It compares rotation ratios and acceleration obtained with wearables installed in pedestrian wrists with similar information obtained by vision. The vision dynamic information is obtained using robust methods that combine skeleton representation with semantic information. The experimental results presented demonstrate the strong correlation between the wearable measured and vision observed rates and acceleration information. The outcomes of this work will enable the solution of one of the fundamental issues in pedestrian safety that is inference of intent.
  • BibTex
    @inproceedings{konrad2018analysis,
    title={Analysis of accuracy of pedestrian inertial data obtained from camera’s images},
    author={Konrad, S Gerling and Masson, FR and Nebot, E},
    booktitle={2018 IEEE Biennial Congress of Argentina (ARGENCON)},
    pages={1–5},
    year={2018},
    organization={IEEE} }
• Eiffert, S., Li, K., Shan, M., Worrall, S., Sukkarieh, S., Nebot, E. (2020) Probabilistic Crowd GAN: Multimodal Pedestrian Trajectory Prediction Using a Graph Vehicle-Pedestrian Attention Network. IEEE Robotics and Automation Letters.
  • Paper
  • Abstract
    Understanding and predicting the intention of pedestrians is essential to enable autonomous vehicles and mobile robots to navigate crowds. This problem becomes increasingly complex when we consider the uncertainty and multimodality of pedestrian motion, as well as the implicit interactions between members of a crowd, including any response to a vehicle. Our approach, Probabilistic Crowd GAN, extends recent work in trajectory prediction, combining Recurrent Neural Networks (RNNs) with Mixture Density Networks (MDNs) to output probabilistic multimodal predictions, from which likely modal paths are found and used for adversarial training. We also propose the use of Graph Vehicle-Pedestrian Attention Network (GVAT), which models social interactions and allows input of a shared vehicle feature, showing that inclusion of this module leads to improved trajectory prediction both with and without the presence of a vehicle. Through evaluation on various datasets, we demonstrate improvements on the existing state of the art methods for trajectory prediction and illustrate how the true multimodal and uncertain nature of crowd interactions can be directly modelled.
  • BibTex
    @ARTICLE{Kunming_gan_2020,
    author={S. {Eiffert} and K. {Li} and M. {Shan} and S. {Worrall} and S. {Sukkarieh} and E. {Nebot}},
    journal={IEEE Robotics and Automation Letters},
    title={Probabilistic Crowd GAN: Multimodal Pedestrian Trajectory Prediction Using a Graph Vehicle-Pedestrian Attention Network},
    year={2020},
    volume={5},
    number={4},
    pages={5026-5033},
    doi={10.1109/LRA.2020.3004324}}
• Li, K., Shan, M., Narula, K., Worrall, S., Nebot, E. (2020) Socially Aware Crowd Navigation with Multimodal Pedestrian Trajectory Prediction for Autonomous Vehicles. IEEE International Conference on Intelligent Transportation Systems (ITSC 2020).
  • Paper
  • Abstract
    Seamlessly operating an autonomous vehicles in a crowded pedestrian environment is a very challenging task. This is because human movement and interactions are very hard to predict in such environments. Recent work has demonstrated that reinforcement learning-based methods have the ability to learn to drive in crowds. However, these methods can have very poor performance due to inaccurate predictions of the pedestrians’ future state as human motion prediction has a large variance. To overcome this problem, we propose a new method, SARL-SGAN-KCE, that combines a deep socially aware attentive value network with a human multimodal trajectory prediction model to help identify the optimal driving policy. We also introduce a novel technique to extend the discrete action space with minimal additional computational requirements. The kinematic constraints of the vehicle are also considered to ensure smooth and safe trajectories. We evaluate our method against the state of art methods for crowd navigation and provide an ablation study to show that our method is safer and closer to human behaviour.

  • BibTex
    @INPROCEEDINGS{Kunming_nav_2020,
    author={K. {Li} and M. {Shan} and K. {Narula} and S. {Worrall} and E. {Nebot}},
    booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    title={Socially Aware Crowd Navigation with Multimodal Pedestrian Trajectory Prediction for Autonomous Vehicles},
    year={2020},
    volume={},
    number={},
    pages={1-8},
    doi={10.1109/ITSC45102.2020.9294304}}

• Shan, M., De Alvis, C., Worrall, S., Nebot, E. (2019) Extended Vehicle Tracking with Probabilistic Spatial Relation Projection and Consideration of Shape Feature Uncertainties. The 30th IEEE Intelligent Vehicles Symposium (IV 2019).
  • Paper
  • Abstract
    This work focuses on a novel probabilistic approach for extended vehicle tracking, where multiple spatially distributed measurements can originate from the target, and kinematic state and geometry variables are estimated jointly. Prominent shape features extracted from raw measurement points contain spatial uncertainties due to noise in sensor measurements, the feature extraction process, approximation error of shape hypothesis, partial vision occlusion, to name a few. This work proposes a novel tracking paradigm that respects the variant spatial measurement model subject to changes in target pose and sensor viewpoint. This is achieved through probabilistic projection of the spatial measurement points to the predicted measurement sources on the visible side(s) of the target shape. The spatial uncertainties in the shape features are probabilistically modelled and incorporated in the unscented Kalman filter based estimation. The proposed approach is validated with field experiment results using cameras and a laser range scanner.
  • BibTex
    @INPROCEEDINGS{shan_tracking_2019,
    author={M. {Shan} and C. {De Alvis} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Vehicles Symposium (IV)}, title={Extended Vehicle Tracking with Probabilistic Spatial Relation Projection and Consideration of Shape Feature Uncertainties},
    year={2019},
    volume={},
    number={},
    pages={1477-1483},
    doi={10.1109/IVS.2019.8814064}}

Driver Intention

• Zyner, A., Worrall, S., Ward, J., Nebot, E. (2017). Long Short Term Memory for Driver Intent Prediction. 28th IEEE Intelligent Vehicles Symposium (IV 2017), Piscataway: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    Advanced Driver Assistance Systems have been shown to greatly improve road safety. However, existing systems are typically reactive with an inability to understand complex traffic scenarios. We present a method to predict driver intention as the vehicle enters an intersection using a Long Short Term Memory (LSTM) based Recurrent Neural Network (RNN). The model is learnt using the position, heading and velocity fused from GPS, IMU and odometry data collected by the ego-vehicle. In this paper we focus on determining the earliest possible moment in which we can classify the driver’s intention at an intersection. We consider the outcome of this work an essential component for all levels of road vehicle automation.
  • BibTex
    @inproceedings{zyner2017long,
    title={Long short term memory for driver intent prediction},
    author={Zyner, Alex and Worrall, Stewart and Ward, James and Nebot, Eduardo},
    booktitle={2017 IEEE Intelligent Vehicles Symposium (IV)},
    pages={1484–1489},
    year={2017},
    organization={IEEE}
    }
• Zyner, A., Worrall, S., Nebot, E. (2018). A Recurrent Neural Network Solution for Predicting Driver Intention at Unsignalized Intersections. IEEE Robotics and Automation Letters, 3(3), 1759-1764.
  • Paper
  • Abstract
    In this letter, we present a system capable of inferring intent from observed vehicles traversing an unsignalized intersection, a task critical for the safe driving of autonomous vehicles, and beneficial for advanced driver assistance systems. We present a prediction method based on recurrent neural networks that takes data from a Lidar-based tracking system similar to those expected in future smart vehicles. The model is validated on a roundabout, a popular style of unsignalized intersection in urban areas. We also present a very large naturalistic dataset recorded in a typical intersection during two days of operation. This comprehensive dataset is used to demonstrate the performance of the algorithm introduced in this letter. The system produces excellent results, giving a significant 1.3-s prediction window before any potential conflict occurs.
  • BibTex
    @article{zyner2018recurrent,
    title={A recurrent neural network solution for predicting driver intention at unsignalized intersections},
    author={Zyner, Alex and Worrall, Stewart and Nebot, Eduardo},
    journal={IEEE Robotics and Automation Letters},
    volume={3},
    number={3},
    pages={1759–1764},
    year={2018},
    publisher={IEEE}
    }
• Zyner, A., Worrall, S., & Nebot, E. (2019). Naturalistic driver intention and path prediction using recurrent neural networks. IEEE Transactions on Intelligent Transportation Systems.
  • Paper
  • Abstract
    Understanding the intentions of drivers at intersections is a critical component for autonomous vehicles. Urban intersections that do not have traffic signals are a common epicenter of highly variable vehicle movement and interactions. We present a method for predicting driver intent at urban intersections through multi-modal trajectory prediction with uncertainty. Our method is based on recurrent neural networks combined with a mixture density network output layer. To consolidate the multi-modal nature of the output probability distribution, we introduce a clustering algorithm that extracts the set of possible paths that exist in the prediction output and ranks them according to probability. To verify the method’s performance and generalizability, we present a real-world dataset that consists of over 23,000 vehicles traversing five different intersections, collected using a vehicle-mounted lidar-based tracking system. An array of metrics is used to demonstrate the performance of the model against several baselines.
  • BibTex
    @article{zyner2019naturalistic,
    title={Naturalistic driver intention and path prediction using recurrent neural networks},
    author={Zyner, Alex and Worrall, Stewart and Nebot, Eduardo},
    journal={IEEE Transactions on Intelligent Transportation Systems},
    year={2019},
    publisher={IEEE} }
• Zyner, A., Worrall, S., & Nebot, E. (2019). ACFR Five Roundabouts Dataset: Naturalistic Driving at Unsignalised Intersections.
  IEEE Intelligent Transportation Systems Magazine, vol. 11, no. 4, pp. 8-18, winter 2019. doi: 10.1109/MITS.2019.2907676
  
  • Abstract
    Understanding the intentions of drivers at intersections is a critical component for autonomous vehicles. Urban intersections that do not have traffic signals are a common epicenter of highly variable vehicle movement and interactions. We present a method for predicting driver intent at urban intersections through multi-modal trajectory prediction with uncertainty. Our method is based on recurrent neural networks combined with a mixture density network output layer. To consolidate the multi-modal nature of the output probability distribution, we introduce a clustering algorithm that extracts the set of possible paths that exist in the prediction output and ranks them according to probability. To verify the method’s performance and generalizability, we present a real-world dataset that consists of over 23,000 vehicles traversing five different intersections, collected using a vehicle-mounted lidar-based tracking system. An array of metrics is used to demonstrate the performance of the model against several baselines.
  • Early Access Paper
  • Dataset
• Bender A., Agamennoni G., Worrall, S., Ward, J., Nebot, E. (2017). An Unsupervised Approach for Inferring Driver Behavior From Naturalistic Driving Data. IEEE Transactions on Intelligent Transport Systems, 2019.
  • Paper
  • Abstract
    Intelligent transportation systems are able to collect large volumes of high-resolution data. The amount of data collected by these systems can quickly overwhelm the ability of human analysts to draw meaningful conclusions from the data, particularly in large-scale multivehicle field trials. As advanced driver assistance systems develop, they will also be required to form a rich and high-level understanding of the world from the data they receive, including the behavior of the driver. These applications motivate the need for unsupervised tools capable of forming a high-level summary of low-level driving data. This paper presents an unsupervised method for converting naturalistic driving data into high-level behaviors. The proposed method works in two steps. In the first step, inertial data are automatically decomposed into linear segments. In the second step, the segments are assigned to high-level driving behaviors. The proposed method is computationally efficient and completely unsupervised and requires minimal preprocessing. Although the method is unsupervised, the clusters produced exhibit high-level patterns that can easily be associated with driving behaviors such as braking, turning, accelerating, and coasting. The effectiveness of the proposed algorithms is demonstrated in an offline application where the objective is to summarize inertial data into driving behaviors. The method is also demonstrated in an online application where the aim is to infer the current driving behavior using only inertial data. Both experiments were conducted using driving data collected in natural driving conditions.
  • BibTex
    @article{BenderAWWN15,
    title = {An Unsupervised Approach for Inferring Driver Behavior From Naturalistic Driving Data},
    author = {Asher Bender and Gabriel Agamennoni and James R. Ward and Stewart Worrall and Eduardo Mario Nebot}, journal={IEEE Transactions on Intelligent Transportation Systems},
    year={2017},
    pages = {3325–3336},
    publisher={IEEE} }

Localisation and Mapping

Localisation

• Yi, Siqi, Stewart Worrall, and Eduardo Nebot., Integrating Vision, Lidar and GPS Localization in a Behavior Tree Framework for Urban Autonomous Driving, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 2021
  • Paper
  • Abstract
    Robust and persistent global localisation is essential for the safe operation of autonomous vehicles. When operating in large and diverse urban driving environments, localisation systems are frequently exposed to environments that violate sensor and algorithm assumptions. Global localisation algorithms relying on specific sensors have known failure modes, generating position information that can be accurate and robust in suitable areas, while unreliable or unavailable in other areas. To guarantee the continuous global localisation in a large and diverse area, it is required that there are redundant sources of global observations available at any location within the area. We present a behavior tree framework that can intelligently pick the best available global localisation method from amongst visual features, lidar landmarks and GPS. Compared to our previous work, the visual component makes available a source of global updates in challenging environments such as underground tunnels or covered carparks using a prior map and a single front facing camera. We present a method to make a geographically registered visual feature map by incorporating scale and global adjustment, a visual SLAM component that can repeatedly localise within the prebuilt geographically consistent map, and a behavior tree framework that makes intelligent state transition decisions and supervises localisation performance. We demonstrate the localisation performance of the framework using 53 datasets from the University of Sydney Campus dataset acquired over an 18 month period.
  • BibTex
    @INPROCEEDINGS{siqi_itsc2021,
    author={Yi, Siqi and Worrall, Stewart and Nebot, Eduardo},
    booktitle={2021 IEEE International Intelligent Transportation Systems Conference (ITSC)},
    title={Integrating Vision, Lidar and GPS Localization in a Behavior Tree Framework for Urban Autonomous Driving},
    year={2021},
    volume={},
    number={},
    pages={3774-3780},
    doi={10.1109/ITSC48978.2021.9564875}}
• Yi, Siqi, Stewart Worrall, and Eduardo Nebot. Metrics for the Evaluation of localisation Robustness. In 30th IEEE Intelligent Vehicles Symposium (IV 2019), (IEEE).
  • Paper
  • Abstract
    Robustness and safety are crucial properties for the real-world application of autonomous vehicles. One of the most critical components of any autonomous system is localisation. During the last 20 years there has been significant progress in this area with the introduction of very efficient algorithms for mapping, localisation and SLAM. Many of these algorithms present impressive demonstrations for a particular domain, but fail to operate reliably with changes to the operating environment. The aspect of robustness has not received enough attention and localisation systems for self-driving vehicle applications are seldom evaluated for their robustness. In this paper we propose novel metrics to effectively quantify localisation robustness with or without an accurate ground truth. The experimental results present a comprehensive analysis of the application of these metrics against a number of well known localisation strategies.
  • BibTex
    @INPROCEEDINGS{siqi_metrics_2019,
    author={S. {Yi} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Vehicles Symposium (IV)},
    title={Metrics for the Evaluation of localisation Robustness},
    year={2019},
    volume={},
    number={},
    pages={1247-1253},
    doi={10.1109/IVS.2019.8814106}}
• Yi, Siqi, Stewart Worrall, and Eduardo Nebot. Geographical Map Registration and Fusion of Lidar-Aerial Orthoimagery in GIS. 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 2019, pp. 128-134. doi: 10.1109/ITSC.2019.8917305
  • Paper
  • Abstract
    Centimeter level globally accurate and consistent maps for autonomous vehicles navigation has long been achieved by on board real-time kinematic(RTK)-GPS in open areas. However when dealing with urban environments, GPS will experience multipath and blockage in urban canyon, under bridges, inside tunnels and in underground environments. In this paper we present strategies to efficiently register local maps in geographical coordinate systems through the tactical integration of GPS and information extracted from precisely geo-referenced high resolution aerial orthogonal imagery. Dense lidar point clouds obtained from moving vehicle are projected down to horizontal plane, accurately registered and overlaid on aerial orthoimagery. Sparse, robust and long-term pole-like landmarks are used as anchor points to link lidar and aerial image sensing, and constrain the spatial uncertainties of remaining lidar points that cannot be directly measured and identified. We achieved 15-75cm absolute average global accuracy using precisely geo-referenced aerial imagery as ground truth. This is valuable in enabling the fusion of ground vehicle on-board sensor features with features extracted from aerial images such as traffic and lane markings. It is also useful for cooperative sensing to have an unbiased and accurate global reference. Experimental results are presented demonstrating the accuracy and consistency of the maps when operating in large areas.
  • BibTex
    @INPROCEEDINGS{siqi_map_2019,
    author={S. {Yi} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Transportation Systems Conference (ITSC)},
    title={Geographical Map Registration and Fusion of Lidar-Aerial Orthoimagery in GIS},
    year={2019},
    volume={},
    number={},
    pages={128-134},
    doi={10.1109/ITSC.2019.8917305}}

Map Maintenance

• Introduction video
• J. S. Berrio, S. Worrall, M. Shan and E. Nebot, Long-Term Map Maintenance Pipeline for Autonomous Vehicles, in IEEE Transactions on Intelligent Transportation Systems..
  • Paper
  • Abstract
    For autonomous vehicles to operate persistently in a typical urban environment, it is essential to have high accuracy position information. This requires a mapping and localisation system that can adapt to changes over time. A localisation approach based on a single-survey map will not be suitable for long-term operation as it does not incorporate variations in the environment. In this paper, we present new algorithms to maintain a featured-based map. A map maintenance pipeline is proposed that can continuously update a map with the most relevant features taking advantage of the changes in the surroundings. Our pipeline detects and removes transient features based on their geometrical relationships with the vehicle’s pose. Newly identified features became part of a new feature map and are assessed by the pipeline as candidates for the localisation map. By purging out-of-date features and adding newly detected features, we continually update the prior map to more accurately represent the most recent environment. We have validated our approach using the USyd Campus Dataset, which includes more than 18 months of data. The results presented demonstrate that our maintenance pipeline produces a resilient map which can provide sustained localisation performance over time.
  • BibTex
    @ARTICLE{Steph_map_2021,
    author={Berrio, Julie Stephany and Worrall, Stewart and Shan, Mao and Nebot, Eduardo},
    journal={IEEE Transactions on Intelligent Transportation Systems},
    title={Long-Term Map Maintenance Pipeline for Autonomous Vehicles},
    year={2021}, volume={}, number={},
    pages={1-14},
    doi={10.1109/TITS.2021.3094485}} 

• Berrio Perez, J., Ward, J., Worrall, S., Nebot, E. (2019). Identifying robust landmarks in feature-based maps. 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 2019, pp. 1166-1172.
  doi: 10.1109/IVS.2019.8814289
  • Paper
  • Abstract
    To operate in an urban environment, an automated vehicle must be capable of accurately estimating its position within a global map reference frame. This is necessary for optimal path planning and safe navigation. To accomplish this over an extended period of time, the global map requires long-term maintenance. This includes the addition of newly observable features and the removal of transient features belonging to dynamic objects. The latter is especially important for the long-term use of the map as matching against a map with features that no longer exist can result in incorrect data associations, and consequently erroneous localisation. This paper addresses the problem of removing features from the map that correspond to objects that are no longer observable/present in the environment. This is achieved by assigning a single score which depends on the geometric distribution and characteristics when the features are re-detected (or not) on different occasions. Our approach not only eliminates ephemeral features, but also can be used as a reduction algorithm for highly dense maps. We tested our approach using half a year of weekly drives over the same 500-metre section of road in an urban environment. The results presented demonstrate the validity of the long-term approach to map maintenance.
  • BibTex
    @INPROCEEDINGS{berrio_landmarks_2019,
    author={J. S. {Berrio} and J. {Ward} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Vehicles Symposium (IV)},
    title={Identifying robust landmarks in feature-based maps},
    year={2019},
    volume={},
    number={},
    pages={1166-1172},
    doi={10.1109/IVS.2019.8814289}}
• Berrio Perez, J., Ward, J., Worrall, S., Nebot, E. (2019). Updating the visibility of a feature-based map for long-term maintenance. 30th IEEE Intelligent Vehicles Symposium (IV 2019), Piscataway: Institute of Electrical and Electronics Engineers (IEEE).
  • Paper
  • Abstract
    Mobile vehicles operating in urban navigation applications can achieve high integrity localisation with high accuracy by using maps of the surroundings. To accomplish this, the map should always have an accurate representation of the environment. Thus, it is necessary to detect and remove the map components that no longer exist in the current environment. This maintains the map compactness and dependability while simplifying the data association problem. This paper addresses the problem of deletion of transient map components by taking advantage of the geometric connection between the map and agent poses in order to establish and update the visibility of each feature. Once the map is created an initial visibility vector is associated with every map element and updated over time. The visibility of a map element which no longer exists is reduced and ultimately removed from the map. We demonstrate our approach in a 2D feature-based map composed of poles and corners extracted from information provided by a Iidar sensor. The experimental results show the map update using a seven-month data set collected in the University of Sydney campus.
  • BibTex
    @INPROCEEDINGS{berrio_visility_2019,
    author={J. S. {Berrio} and J. {Ward} and S. {Worrall} and E. {Nebot}},
    booktitle={2019 IEEE Intelligent Vehicles Symposium (IV)}, title={Updating the visibility of a feature-based map for long-term maintenance},
    year={2019},
    volume={},
    number={},
    pages={1173-1179},
    doi={10.1109/IVS.2019.8814189}}
  • Video

Human-machine Interfaces

• Hoggenmüller, Marius & Tomitsch, Martin & Hespanhol, Luke & Tran, Tram & Worrall, Stewart & Nebot, Eduardo. (2021). Context-Based Interface Prototyping: Understanding the Effect of Prototype Representation on User Feedback. 10.1145/3411764.3445159. .
  • Paper
  • Abstract
    The rise of autonomous systems in cities, such as automated vehicles (AVs), requires new approaches for prototyping and evaluating how people interact with those systems through context-based user interfaces, such as external human-machine interfaces (eHMIs). In this paper, we present a comparative study of three prototype rep- resentations (real-world VR, computer-generated VR, real-world video) of an eHMI in a mixed-methods study with 42 participants. Quantitative results show that while the real-world VR representa- tion results in higher sense of presence, no significant differences in user experience and trust towards the AV itself were found. How- ever, interview data shows that participants focused on different experiential and perceptual aspects in each of the prototype repre- sentations. These differences are linked to spatial awareness and perceived realism of the AV behaviour and its context, affecting in turn how participants assess trust and the eHMI. The paper offers guidelines for prototyping and evaluating context-based interfaces through simulations.
  • BibTex
    @inproceedings{inproceedings,
    author = {Hoggenmüller, Marius and Tomitsch, Martin and Hespanhol, Luke and Tran, Tram and Worrall, Stewart and Nebot, Eduardo},
    year = {2021},
    month = {05},
    pages = {},
    title = {Context-Based Interface Prototyping: Understanding the Effect of Prototype Representation on User Feedback},
    doi = {10.1145/3411764.3445159}}

• Hoggenmüller, Marius & Tomitsch, Martin & Parker, Callum & Nguyen, Trung & Zhou, Dawei & Worrall, Stewart & Nebot, Eduardo. (2020). A Tangible Multi-Display Toolkit to Support the Collaborative Design Exploration of AV Pedestrian Interfaces. 10.1145/3441000.3441031.
  • Paper
  • Abstract
    The advent of cyber-physical systems, such as robots and autonomous vehicles (AVs), brings new opportunities and challenges for the domain of interaction design. Though there is consensus about the value of human-centred development, there is a lack of documented tailored methods and tools for involving multiple stakeholders in design exploration processes. In this paper we present a novel approach using a tangible multi-display toolkit. Orchestrating computer-generated imagery across multiple displays, the toolkit enables multiple viewing angles and perspectives to be captured simultaneously (e.g. top-view, first-person pedestrian view). Participants are able to directly interact with the simulated environment through tangible objects. At the same time, the objects physically simulate the interface’s behaviour (e.g. through an integrated LED display). We evaluated the toolkit in design sessions with experts to collect feedback and input on the design of an AV-pedestrian interface. The paper reports on how the combination of tangible objects and multiple displays supports collaborative design explorations.
  • BibTex
    @inproceedings{10.1145/3441000.3441031,
    author = {Hoggenm\”{u}ller, Marius and Tomitsch, Martin and Parker, Callum and Nguyen, Trung Thanh and Zhou, Dawei and Worrall, Stewart and Nebot, Eduardo},
    title = {A Tangible Multi-Display Toolkit to Support the Collaborative Design Exploration of AV-Pedestrian Interfaces},
    year = {2020},
    isbn = {9781450389754},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3441000.3441031},
    doi = {10.1145/3441000.3441031},
    booktitle = {32nd Australian Conference on Human-Computer Interaction},
    pages = {25–35},
    numpages = {11},
    keywords = {design tools, cyber-physical systems, interdisciplinary collaboration, automated vehicles, human-machine interfaces, HCI toolkit},
    location = {Sydney, NSW, Australia},
    series = {OzCHI ’20} }