notes/docs/lectures/ethics/05_dependable_computing.md

Dependable Computing

What is a dependable System

Another way of putting it is that computing systems, especially systems built into societal infrastructure, and which are otherwise safety-critical as the London ambulance system was, are dependable.

Dependability is defined by Brian Randell as the trustworthiness of a computer system such that reliance can justifiably be placed on the service it delivers. Dependability thus includes such properties as:

• Reliability
• Integrity
• Privacy
• Safety
• Security
• Maintainability

And provides a convenient means of subsuming these various concerns within a single conceptual framework.

Reliability means that a system provides continuity of correct service during its useful lifetime, from commissioning, through operation, to decommissioning.

Safety means that a system is engineered to avoid catastrophic consequences for users and the environment and that the life-critical system behaves as needed, even if components fail.

Integrity means that a system’s source code or state cannot be altered improperly, i.e., it is secure, or its data cannot be corrupted.

Maintainability means that a system is engineered to permit adaptive maintenance, ease of modification and repair of defects.

Dependability

Uber’s self-driving car accident

• Backup driver charged with negligent homicide
• However, the National Transport Safety Board finds Uber’s system to be at fault
• While Uber’s radar and Lidar detected Elaine 6 seconds before the impact, their system did not have the capacity to classify the object as a pedestrian unless they were near a crosswalk
  • It classified Elaine as a vehicle, bicycle and an unknown object
  • It assumed Elaine would be travelling in the same direction as the car and therefore did not slow down
• Furthermore, the car had its own in-built automatic braking system which was capable of detecting and stopping for Elaine, but it was disabled by Uber engineers as they thought it would interfere with Uber’s self-driving sensors
• When the car was just a second away from Elaine, Uber’s system finally recognised that the object could not be avoided
• Now at this point, Uber’s system could have slammed on the brakes to mitigate the impact; instead, an action suppression component kicked in.
  • This was implemented to avoid extreme manoeuvres in response to false alarms.
• Uber couldn’t supply documents showing checks performed on the backup driver

Computing failures are not restricted to 1 car and 2 plane crashes

The FDA reports that medical device recalls are at an all-time high and that defective software is a major cause. One in every three medical devices that use software for operations has been recalled because of failures in their software.

As the Uber and Boeing cases clearly demonstrate, dependability is still a critical issue in computing today.

• Apart from the direct human cost, the failure of computing systems costs a great deal of money.
• The 5th edition of the Software Fail Watch identified 606 recorded software failures, impacting half of the world’s population (3.7 billion people) and 314 companies to the cost of 1.7 trillion dollars, and noted that “this is just scratching the surface – there are far more software defects in the world than we will likely ever know about.”

We have an ethical duty to the public to minimise these harms. I purposefully say minimise and not eradicate, as it is inevitable that things will go wrong sometimes due to unforeseen circumstances, but if we exercise due diligence in our work then we should be able to significantly reduce the harms caused through what are euphemistically called “software bugs”.

Software Bugs

A software bug is defined as an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.

Debugging and Testing

Waterfall Model

The waterfall model places testing after requirements, analysis and specification, software design and implementation.

• Placing testing here is problematic as it means testing only takes place during the later stages of development
• The waterfall model is inflexible and has been widely blamed for a great many large-scale projects running over budget, over time and failing to deliver on requirements

V Model

The V Model adapts the waterfall by placing an emphasis on early testing

• V model is often criticised for squeezing testing into tight windows at the end of development phases when earlier stages have overrun but implementation dates remain fixed.

Spiral Model

Spiral model provides a major alternative and places testing in iterative requirements, design, implement and test sequences that spiral out from one another and are marked by the development of increasingly high-fidelity prototypes

Testing Methodologies

Static Testing

• Static testing takes place early in a software system’s development and examines source code and accompanying documentation but doesn’t execute the program.
• It may be done manually, though increasingly relies on automated analysis tools.

Dynamic Testing

• Dynamic testing checks the behaviour of software code when it is executed.

• Testers compare outputs with expected behaviour to determine whether or not the software works as intended.

White Box Testing

White box testing digs into the inner workings of the software.

• It tests each statement, object, and function on an individual basis
• Identifies broken or poorly structured paths in coding processes
• Internal security holes
• It also verifies the flow of specific inputs through the code and expected outputs.

Black Box Testing

Black box testing on the other hand examines the outer workings of the software and that the software does what it’s supposed to do.

• Knowledge of coding isn’t necessary, and testers work at the user-interface level checking inputs and outputs.

GUI Testing

Graphical user interface or GUI testing

• Checks user interface works as per the GUI specification.
• It tests the software control dialogues, including:
  • screen layouts
  • menus
  • buttons
  • icons, pop-up windows, text boxes, text formatting, colours, fonts, font sizes, etc.

Testing Levels

Unit Testing

Component or Module Testing

Integration Testing

System Testing

Alpha, Beta and acceptance Testing

Testing and Dependability

As Linda Rosenberg and her colleagues told us in their award-winning 1998 IEEE paper on software reliability,

“Metrics to measure software reliability exist and can be used starting in the requirements phase. At each phase of the development life cycle, metrics can identify potential areas of problems that may lead to problems or errors. Finding these areas in the phase they are developed decreases the cost and prevents potential ripple effects from the changes, later in the development life cycle by at least a factor of 14.”

Limits of Testing

Brian Randell tells us that

“a system failure occurs when the delivered service no longer complies with the specification, the latter being an agreed description of the system's expected function and/or service.”

Daniel Jackson and colleagues elaborate the point, saying that,

“Software, according to a popular view, fails because of bugs: errors in the code that cause the software to fail to meet its specification. In fact, only a tiny proportion of failures due to the mistakes of software developers can be attributed to bugs – 3% in one study that focused on fatal accidents. As is well known to software engineers (but not to the general public), by far the largest class of problems arises from errors made in the eliciting, recording, and analysis of requirements.

Boeing 737 MAX 8

The bug at work here was a faulty angle of attack or AOA sensor, which indicated the angle at which the aircraft was positioned in flight.

The Ethiopian accident investigation report says that Boeing’s engineers determined that no piloted simulation was required for take-off or low-speed flight. This meant that specific failures that could lead to MCAS activation, such as false AOA input, were not simulated as part of the aircraft’s functional hazard assessment and validation tests.

Boeing assumed that the worst that could happen would be single fault-driven MCAS activation that flight crew would correct as per “trained memory procedures” acquired during flight training for previous 737 models. As the graph showing the plane going up and down in the Vox video makes painfully visible, the MAX 8 crashes involved multiple MCAS activations, caused by the faulty AOA sensor.

Poor specification requirements: Input was only required from one AOA sensor to activate MCAS, despite two sensors being fitted.

• This means the faulty sensor constantly triggered MCAS
• No information about MCAS was given in the flight crew manuals and MCAS was not included in flight crew training.
• Boeing assumed that pilots certified to fly on earlier versions of the 737 didn’t need any extra training.
  • The lack of documentation and training meant that flight crews were unaware of MCAS and its effects
  • The lack of information about MCAS in the flight crew manual meant that there were no procedures for mitigating erroneous input from the AOA sensors
• An AOA disagree warning light would flash if the two sensors were at odds with each other
  • These indicators were sold as optional extras
  • These extras were not found on either aircraft
  • The Indonesian crash report finds that the flight crew were not aware that the AOA DISAGREE warning would not appear if AOA DISAGREE conditions were met, and that in failing to install the warning lights Boeing denied the flight crew valid information about the abnormal conditions they faced

It becomes apparent then that the AOA sensor bug wasn’t really the problem. It could well have been handled

• Had MCAS not been designed to activate off input from a single sensor
• If flight crew had been informed about MCAS
• Its effects built into difference training and the flight crew manual,
• Had the planes been fitted (like their predecessors) with the AOA warning lights.

The crashes are as much, if not more, a failure of poor requirements, including both poor technical and usability specifications, and inadequate, indeed non-existent, documentation and training, which are also key parts of the user interface to and usability of a system.

There are limits to software testing.

Dependability relies as much on sound requirements specifications as it does good code and rigorous testing.