Context: “the circumstances that form the setting for an event, statement, or idea, an in terms of which it can be fully understood”
In software engineering, the context is the setting in which statements are interpreted. As developers, we usually prefer a closed context – clearly defining the surroundings of our product, the interaction with all external interfaces. If the closed context does not exist we tend to limit the context by making explicit or implicit assumptions. The evidence that those assumptions are correct for the use case of the product is what we generally call validation – do we develop the right product? Or are there any explicit assumptions leading to a wrong product. An implicit assumption could also mean to leave out certain aspects in our software that could result in the same wrong product – not fitting to the intended use during its lifetime.
Driver assistance and autonomous driving systems operate within an open context.
Both systems comprise the following parts: sensing, the processing including perception, functional behavior, planning and finally HMI and actuators. Perception as an example does an interpretation of the environment identifying relevant objects.
Due to the open context, we have to accept the presence of implicit performance limitations in our products. Considering driver assistance systems, the driver is still in control of the vehicle and just supported by the systems – for an example, an autonomous emergency braking system for crash avoidance. The performance of those systems is described by the Receiver-Operator-Curve (ROC). Due to the open context, we won’t be able to achieve a 100 percent true positive rate without accepting the same for the false positive rate. So the development task for perception is to find the function specific optimum between true and false positives.
For autonomous driving, the challenge is to push the curve into the upper left corner, e.g. by sensor fusion. Currently one can find various publications and demonstrations showing the performance of perception algorithms for dedicated problems, partially using artificial intelligence.
Coming from driver assistance systems validation evidence was widely provided by enhanced black box testing. While driving worldwide thousands of hours with full data recording the question came up: is black box testing possible for AD products? Due to several reasons, it is at least challenging and in terms of product release and the required number of hours for a statistical evidence is tremendous. So industry moves forward and changes the validation process from black-box testing to making a statement on different levels of the AD system. Coming back to the perception: the performance of the perception is rated.
We can do many showcases, bring prototypes to the streets, but we need to put much more effort into understanding how the system should behave. During development, we need to start thinking about validation. And we need to make sure that we define our systems as good as possible for all upcoming situations.
Validation is far more important than any fancy algorithm! 99,9% fit rate is far from being enough.
Authors: Alexander Ulrich, Fabian Timm & Mirko Franke, Robert Bosch GmbH, Division of Chassis Systems Control
To gain more insights join Bosch at the WeAreDevelopers World Congress 2018!