TRI Authors: Jonathan DeCastro, Soonho Kong

All Authors: Daniel Jackson, Jonathan DeCastro, Soonho Kong, Dimitrios Koutentakis, Angela Leong Feng Ping, Armando Solar-Lezama, Mike Wang, Xin Zhang

Certified control is a new architectural pattern for achieving high assurance of safety in autonomous cars. As with a traditional safety controller or interlock, a separate component oversees safety and intervenes to prevent safety violations. Tis component (along with sensors and actuators) comprises a trusted base that can ensure safety even if the main controller fails. But in certified control, the interlock does not use the sensors directly to determine when to intervene. Instead, the main controller is given the responsibility of presenting the interlock with a certificate that provides evidence that the proposed next action is safe. Te interlock checks this certificate, and intervenes only if the check fails. Because generating such a certificate is usually much harder than checking one, the interlock can be smaller and simpler than the main controller, and thus assuring its correctness is more feasible. Read More

Citation: Jackson, Daniel, Jonathan DeCastro, Soonho Kong, Dimitrios Koutentakis, Angela Leong Feng Ping, Armando Solar-Lezama, Mike Wang, and Xin Zhang. "Certified Control for Self-Driving Cars," in DARS 2019 Workshop at CAV 2019

TRI Authors: Wadim Kehl, Adrien Gaidon

All Authors: Fabian Manhardt, Wadim Kehl, Adrien Gaidon

We present a deep learning method for end-to-end monocular 3D object detection and metric shape retrieval. We propose a novel loss formulation by lifting 2D detection, orientation, and scale estimation into 3D space. Instead of optimizing these quantities separately, the 3D instantiation allows to properly measure the metric misalignment of boxes. We experimentally show that our 10D lifting of sparse 2D Regions of Interests (RoIs) achieves great results both for 6D pose and recovery of the textured metric geometry of instances. This further enables 3D synthetic data augmentation via inpainting recovered meshes directly onto the 2D scenes. We evaluate on KITTI3D against other strong monocular methods and demonstrate that our approach doubles the AP on the 3D pose metrics on the official test set, defining the new state of the art. Read More

Citation: Manhardt, Fabian, Wadim Kehl, and Adrien Gaidon. "Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2069-2078. 2019.

TRI Author: Nikos Arechiga

All Authors: Nikos Arechiga

We develop a set of contracts for autonomous control software that ensures that if all traffic participants follow the contracts, the overall traffic system will be collision-free. We express our contracts in Signal Temporal Logic (STL), a lightweight specification language that enables V &V methodologies. We demonstrate how the specification can be used for evaluation of the performance of autonomy software, and We provide preliminary evidence that our contracts are not excessively conservative, i.e., they are not more restrictive than existing guidelines for safe driving by humans. Read more

TRI Author: Nikos Arechiga

All Authors: Karen Leung, Nikos Arechiga, Marco Pavone

This paper proposes a method to evaluate Signal Temporal Logic (STL) robustness formulas using computation graphs. This method results in efficient computations and enables the use of backpropagation for optimizing over STL parameters. Inferring STL formulas from behavior traces can provide powerful insights into complex systems, such as longterm behaviors in time-series data. It can also be used to augment existing prediction and planning architectures by ensuring specifications are met. However, learning STL formulas from data is challenging from a theoretical and numerical standpoint. By evaluating and learning STL formulas using computation graphs, we can leverage the computational efficiency and utility of modern machine learning libraries. The proposed approach is particularly effective for solving parameteric STL (pSTL) problems, the problem of parameter fitting for a given signal. We provide a relaxation technique that makes this method tractable when solving general pSTL formulas. Through a traffic-weaving case-study, we show how the proposed approach is effective in learning pSTL parameters, and how it can be applied for scenario-based testing for autonomous driving and other complex robotic systems. Read More

Citation: Leung, Karen, Nikos Aréchiga, and Marco Pavone. "Backpropagation for Parametric STL." In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 185-192. IEEE, 2019.

TRI Authors: Kuan-Hui Lee, Takaaki Tagawa, Jia-En M. Pan, Adrien Gaidon, Bertrand Douillard

All Authors: Kuan-Hui Lee, Takaaki Tagawa, Jia-En M. Pan, Adrien Gaidon, Bertrand Douillard

Vehicle taillight recognition is an important application for automated driving, especially for intent prediction of ado vehicles and trajectory planning of the ego vehicle. In this work, we propose an end-to-end deep learning framework to recognize taillights, i.e. rear turn and brake signals, from a sequence of images. The proposed method starts with a Convolutional Neural Network (CNN) to extract spatial features, and then applies a Long Short-Term Memory network (LSTM) to learn temporal dependencies. Furthermore, we integrate attention models in both spatial and temporal domains, where the attention models learn to selectively focus on both spatial and temporal features. Our method is able to outperform the state of the art in terms of accuracy on the UC Merced Vehicle Rear Signal Dataset, demonstrating the effectiveness of attention models for vehicle taillight recognition.  Read More

Citation: Lee, Kuan-Hui, Takaaki Tagawa, Jia-En M. Pan, Adrien Gaidon, and Bertrand Douillard. "An Attention-based Recurrent Convolutional Network for Vehicle Taillight Recognition." In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 2365-2370. IEEE, 2019.

TRI Authors: Guy Rosman

All Authors: Alexander Amini, Guy Rosman, Sertac Karaman and Daniela Rus

Deep learning has revolutionized the ability to learn "end-to-end" autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-to-point navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior. Read More

Citation: Amini, Alexander, Guy Rosman, Sertac Karaman, and Daniela Rus. "Variational end-to-end navigation and localization." In 2019 International Conference on Robotics and Automation (ICRA), pp. 8958-8964. IEEE, 2019.

TRI Authors: Stephen McGill, Luke Fletcher, Guy Rosman

All Authors: Xin Huang, Stephen McGill, Brian C Williams, Luke Fletcher, Guy Rosman

Deep learning has revolutionized the ability to learn "end-to-end" autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-to-point navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior. Read more 

Citation: Huang, Xin, Stephen G. McGill, Brian C. Williams, Luke Fletcher, and Guy Rosman. "Uncertainty-aware driver trajectory prediction at urban intersections." In 2019 International Conference on Robotics and Automation (ICRA), pp. 9718-9724. IEEE, 2019.

TRI Author: Vitor Guizilini

All Authors: Vitor Guizilini, Ransalu Senanayake, Fabio Ramos

This paper addresses the problem of learning instantaneous occupancy levels of dynamic environments and predicting future occupancy levels. Due to the complexity of most real environments, such as urban streets or crowded areas, the efficient and robust incorporation of temporal dependencies into otherwise static occupancy models remains a challenge. We propose a method to capture the uncertainty of moving objects and incorporate this uncertainty information into a continuous occupancy map represented in a rich high-dimensional feature space. This data-efficient model not only allows us to learn the occupancy states incrementally, but also makes predictions about what the future occupancy states will be. Experiments performed using 2D and 3D laser data collected from crowded unstructured outdoor environments show that the proposed methodology can accurately predict occupancy states for areas of around 1000 m 2 at 10 Hz, making the proposed framework ideal for online applications under real-time constraints. Read More

Citation: Guizilini, Vitor, Ransalu Senanayake, and Fabio Ramos. "Dynamic Hilbert Maps: Real-Time Occupancy Predictions in Changing Environments." In 2019 International Conference on Robotics and Automation (ICRA), pp. 4091-4097. IEEE, 2019.

TRI Authors: Hongkai Dai, Russ Tedrake

All Authors: Hongkai Dai, Gregory Izatt, Russ Tedrake

In this paper, we present a novel formulation of the inverse kinematics (IK) problem with generic constraints as a mixed-integer convex optimization program. The proposed approach can solve the IK problem globally with generic task space constraints: a major improvement over existing approaches, which either solve the problem in only a local neighborhood of the user initial guess through nonlinear non-convex optimization, or address only a limited set of kinematics constraints. Specifically, we propose a mixed-integer convex relaxation of non-convex SO(3) rotation constraints, and apply this relaxation on the IK problem. Our formulation can detect if an instance of the IK problem is globally infeasible, or produce an approximate solution when it is feasible. We show results on a seven-joint arm grasping objects in a cluttered environment, an 18-degree-of-freedom quadruped standing on stepping stones, and a parallel Stewart platform. Moreover, we show that our approach can find a collision free path for a gripper in a cluttered environment, or certify such a path does not exist. We also compare our approach against the analytical approach for a six-joint manipulator. The open-source code is available at http://drake.mit.edu.  Read More

Citation: Dai, Hongkai, Gregory Izatt, and Russ Tedrake. "Global inverse kinematics via mixed-integer convex optimization." The International Journal of Robotics Research 38, no. 12-13 (2019): 1420-1441.

TRI Author: Simon Stent

All Authors: Robert Semmens, Nikolas Martelaro, Pushyami Kaveti, Simon Stent, Wendy Ju

Advances in automotive sensing systems and speech interfaces provide new opportunities for smarter driving assistants or infotainment systems. For both safety and consumer satisfaction reasons, any new system which interacts with drivers must do so at appropriate times. We asked 63 drivers, ''Is now a good time?'' to receive non-driving information during a 50-minute drive. We analyzed 2,734 responses and synchronized automotive and video data, and show that while the chances of choosing a good time can be determined with better success using easily accessible automotive data, certain nuances in the problem require a richer understanding of the driver and environment states in order to achieve higher performance. We illustrate several of these nuances with quantitative and qualitative analyses to contribute to the understanding of how to design a system that might simultaneously minimize the risk of interacting at a bad time while maximizing the window of allowable interruption. Read More

Citation: Semmens, Rob, Nikolas Martelaro, Pushyami Kaveti, Simon Stent, and Wendy Ju. "Is Now A Good Time? An Empirical Study of Vehicle-Driver Communication Timing." In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12. 2019.