Tidy up
This commit is contained in:
103 files changed
+3663
-3779
No files matched your search
@@ -1,64 +1,69 @@
|
||||
# Why do we need Professional Ethics
|
||||
|
||||
Computers enable social harm:
|
||||
|
||||
|
||||
## Illegal content and activity
|
||||
- Terrorism
|
||||
- Crypto-currencies can finance this
|
||||
- Organised crime
|
||||
- Phishing and fraud
|
||||
- Information stealing malware
|
||||
- Ransomware and DDoS extortion
|
||||
- Domestic Abuse
|
||||
- Abusers can look at devices connected to the internet to control their partner even after they have left the house to establish control
|
||||
- Cyber-Bullying
|
||||
- Sending harmful messages/photos to people
|
||||
- Promoting hate
|
||||
- Impersonating another person
|
||||
- *No legal definition of cyber-bullying but still prosecutable*
|
||||
- Child sexual exploitation and Abuse
|
||||
- Solicitation, Grooming, Distribution of images & videos
|
||||
- Trafficking
|
||||
|
||||
## Impact on health and well being
|
||||
- Terrorism
|
||||
- Crypto-currencies can finance this
|
||||
- Organised crime
|
||||
- Phishing and fraud
|
||||
- Information stealing malware
|
||||
- Ransomware and DDoS extortion
|
||||
- Domestic Abuse
|
||||
- Abusers can look at devices connected to the internet to control their partner even after they have left the house to establish control
|
||||
- Cyber-Bullying
|
||||
- Sending harmful messages/photos to people
|
||||
- Promoting hate
|
||||
- Impersonating another person
|
||||
- *No legal definition of cyber-bullying but still prosecutable*
|
||||
- Child sexual exploitation and Abuse
|
||||
- Solicitation, Grooming, Distribution of images & videos
|
||||
- Trafficking
|
||||
|
||||
## Impact on health and well-being
|
||||
|
||||
- Computers can affect physical, social and mental health
|
||||
- Lower physical activity
|
||||
- Increases loneliness
|
||||
- Lower physical activity
|
||||
- Increases loneliness
|
||||
- Designed for addiction
|
||||
- Click bait
|
||||
- Infinite scroll
|
||||
- Short term dopamine-driven feedback loops - Chamath Palihapitya (ex Facebook VP)
|
||||
- Clickbait
|
||||
- Infinite scroll
|
||||
- Short-term dopamine-driven feedback loops - Chamath Palihapitya (ex Facebook VP)
|
||||
- Self-harm
|
||||
- Enables people to research self harm methods
|
||||
- Validates negative feelings
|
||||
- Legitimise suicide as an acceptable course of action
|
||||
- Enables people to research self-harm methods
|
||||
- Validates negative feelings
|
||||
- Legitimises suicide as an acceptable course of action
|
||||
|
||||
## Threats to our way of life
|
||||
|
||||
- Manipulating public opinion
|
||||
- Can be state sanctioned
|
||||
- Distribution of inaccurate information, disinformation and fake news
|
||||
- Can be state-sanctioned
|
||||
- Distribution of inaccurate information, disinformation and fake news
|
||||
- The Oxford internet institute found 26 countries including China, Turkey and Russia were using computational propaganda to suppress human rights and discredit political opposition
|
||||
|
||||
### Risk to critical national infrastructure
|
||||
|
||||
- Cyber attacks on nuclear power stations, electricity grids, banking communications
|
||||
- WannaCry targeting the NHS
|
||||
- WannaCry targeting the NHS
|
||||
|
||||
## Environmental Impact
|
||||
|
||||
- Data centres consume huge amounts of energy
|
||||
- Consumed 416.2 TWH of electricity - more than the total UK’s power consumption
|
||||
- 3% of global electricity supply
|
||||
- 2% of greenhouse gas emissions
|
||||
- Consumed 416.2 TWH of electricity - more than the UK’s total power consumption
|
||||
- 3% of global electricity supply
|
||||
- 2% of greenhouse gas emissions
|
||||
|
||||
## GDPR
|
||||
|
||||
- Data protection
|
||||
- Data is the oil of the digital economy
|
||||
- **GDPR** applies to the processing of personal data by automated means, regardless of whether the processing takes place in the EU or not relating to:
|
||||
- The offering of goods or services to EU citizens
|
||||
- The monitoring of their behaviour
|
||||
- There are stiff fines for those who break GDPR
|
||||
- £20,000,000 or 4% of total annual turnover - whichever is greater.
|
||||
- Data is the oil of the digital economy
|
||||
- **GDPR** applies to the processing of personal data by automated means, regardless of whether the processing takes place in the EU or not relating to:
|
||||
- The offering of goods or services to EU citizens
|
||||
- The monitoring of their behaviour
|
||||
- There are stiff fines for those who break GDPR
|
||||
- £20,000,000 or 4% of total annual turnover - whichever is greater.
|
||||
|
||||
# A world under attack
|
||||
|
||||
It’s not computer scientists who do harm, but the way the technology is designed, who designed it and the outcomes it is trying to achieve influence how it impacts its users and wider society.
|
||||
@@ -5,23 +5,27 @@
|
||||
## Morally permissible
|
||||
|
||||
- Morality is ubiquitous, as moral standards apply to everyone
|
||||
- Professional ethics only apply to the members of particular groups (such as lawyers, doctors etc)
|
||||
- Professional ethics only apply to the members of particular groups (such as lawyers, doctors, etc.)
|
||||
|
||||
**Ethical does not equal moral**
|
||||
|
||||
>For example it is against ethical standards in the USA for doctors to advertise prices for their services, but there is nothing inherently immoral about advertising prices for services.
|
||||
|
||||
- An action may be morally permissible but unethical
|
||||
- It is also possible to behave ethically but apparently immorally
|
||||
- Professional ethics requires that one behaves consistently with the standards of the group.
|
||||
|
||||
**Professional ethics is a subset of moral concerns**
|
||||
|
||||
- Morality encompasses societal reasoning and norms of conduct as to what constitutes right and wrong
|
||||
- Professional ethics govern professional practice with respect to particular moral issues or challenges like *algorithmic decisions*
|
||||
- As the broader social-moral order evolves so do professional ethics, like ACM Code of Ethics
|
||||
- As the broader social-moral order evolves, so do professional ethics, like the ACM Code of Ethics
|
||||
|
||||
## Standards
|
||||
|
||||
Govern professional practice
|
||||
Standards consist of:
|
||||
|
||||
- Principles
|
||||
- Rules of Conduct
|
||||
- Embedded in code of conduct or code of ethics
|
||||
@@ -29,11 +33,12 @@ Standards consist of:
|
||||
>A professional puts profession first. When a conflict arises between the professional's code and the policy of an employer or the law, the professional's code must take precedence - Brinkman & Sanders, *Ethics in Computing Culture.* Boston: Cengage Learning, 2013.
|
||||
|
||||
### Shared by a Group
|
||||
|
||||
Standards are shared by a cohort of people engaged in professional activity
|
||||
|
||||
**What constitutes professional activity?**
|
||||
|
||||
- Provides an important service to soceity
|
||||
- Provides an important service to society
|
||||
- Requires extensive training
|
||||
- Involves significant intellectual effort
|
||||
- Organisation of members
|
||||
@@ -41,15 +46,19 @@ Standards are shared by a cohort of people engaged in professional activity
|
||||
- Certification or Licensing
|
||||
|
||||
#### Is computing a profession?
|
||||
|
||||
The problematic static of computing
|
||||
|
||||
- Lack of accreditation, certification or licensing
|
||||
+ No single organisation of members for the computing profession
|
||||
Question is immaterial:
|
||||
The harms enabled by computing mean that computing professionals still have important ethical obligations
|
||||
- No single organisation of members for the computing profession
|
||||
|
||||
Question is immaterial:
|
||||
The harms enabled by computing mean that computing professionals still have important ethical obligations
|
||||
|
||||
>Programmers need ethics when designing the technologies that influence people's lives - President of the ACM
|
||||
|
||||
We still need professional ethics in computing even if computings professional status is dubitable.
|
||||
We still need professional ethics in computing even if computing’s professional status is dubitable.
|
||||
|
||||
- We need ethics if we are to be considered professionals
|
||||
|
||||
> It is impossible to satisfy the definition of profession without a code of ethics, impossible to teach 'professionalism' without teaching the code, and indeed impossible to understand professions without understanding them as bound by such a code. Without a code of ethics, there are only honest occupations, trade associations, and the like - Micheal Davis
|
||||
@@ -72,18 +81,18 @@ These standards require:
|
||||
- Only undertake to do work or provide a service that is within your professional competence
|
||||
- Do not claim a level of competence that you do not possess
|
||||
- Continue to develop professional knowledge relevant to your field
|
||||
- Ensure that you have the knowledge and understanding of relevent legislation
|
||||
- Ensure that you have the knowledge and understanding of relevant legislation
|
||||
- Respect and value alternate viewpoints
|
||||
- Avoid injuring others
|
||||
- Reject and will not make any offer of bribery or unethical inducement
|
||||
- Reject and do not make any offer of bribery or unethical inducement
|
||||
|
||||
##### Duty to relevant authority
|
||||
|
||||
- Carry out your professional responsiblities with due care and diligence
|
||||
- Carry out your professional responsibilities with due care and diligence
|
||||
- Avoid situations that conflict with the interests of relevant authorities
|
||||
- Accept professioal responsibilities for your work
|
||||
- Accept professional responsibilities for your work
|
||||
- Do not disclose confidential information
|
||||
- Do not misrepresent or withhold information on the performance of products, system or services
|
||||
- Do not misrepresent or withhold information on the performance of products, systems or services
|
||||
|
||||
##### Duty to Profession
|
||||
|
||||
@@ -105,7 +114,7 @@ Covers about half of what the BCS covers, little attention to duty to relevant a
|
||||
25 principles governing professional conduct
|
||||
|
||||
- 7 general ethical principles
|
||||
- 9 principles governing professional responsiblities
|
||||
- 9 principles governing professional responsibilities
|
||||
- 7 principles of professional leadership
|
||||
- 2 principles of compliance
|
||||
|
||||
@@ -117,25 +126,25 @@ Covers about half of what the BCS covers, little attention to duty to relevant a
|
||||
- Be fair and take action not to discriminate
|
||||
- Respect the work of others
|
||||
- Respect privacy
|
||||
- Honor confidentiality
|
||||
- Unless in cases in which it is evidence of the violation of law or the code itself
|
||||
- Honour confidentiality
|
||||
- Unless in cases in which it is evidence of the violation of law or the code itself
|
||||
|
||||
This links to the BCS public interest requirement
|
||||
|
||||
##### Professional responsibilities
|
||||
|
||||
- Strive to achieve high quality work
|
||||
- Maintain high standards to professional competence
|
||||
- Strive to achieve high-quality work
|
||||
- Maintain high standards of professional competence
|
||||
- Know and respect rules pertaining to professional work
|
||||
- Accept and provide appropriate professional review
|
||||
- Evaluate computer systems and possible risks
|
||||
- Providing objective evaluations for employers or clients
|
||||
- Providing objective evaluations for employers or clients
|
||||
- Perform work only in areas of competence
|
||||
- Foster public awareness and understanding of computing
|
||||
- Access computing only when authorised or for public good
|
||||
- Basically **do not hack**, unless it is to disrupt or inhibit malicious systems
|
||||
- Basically **do not hack**, unless it is to disrupt or inhibit malicious systems
|
||||
- Design and implement robust and secure systems
|
||||
- Does not link to BCS code however important
|
||||
- Does not link to BCS code however important
|
||||
|
||||
##### Professional leadership Principles
|
||||
|
||||
@@ -143,8 +152,8 @@ This links to the BCS public interest requirement
|
||||
- Promote social responsibility
|
||||
- Enhance quality of working life
|
||||
- Support the principles of the code
|
||||
- Create oppotunities for professional development
|
||||
- User care when modifying or retiring systems
|
||||
- Create opportunities for professional development
|
||||
- Use care when modifying or retiring systems
|
||||
- Take special care of systems integrated in societal infrastructure
|
||||
|
||||
##### Compliance with the Code
|
||||
@@ -158,8 +167,3 @@ This links to the BCS public interest requirement
|
||||
|
||||
- More to the ACM code
|
||||
- But a strong relationship between the two exists, although it is not always direct
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The coursework issue is about a class action lawsuit against Ring.
|
||||
|
||||
file: <studentID>_Surname
|
||||
file: `<studentID>_Surname`
|
||||
|
||||
## Example of applying Codes
|
||||
|
||||
@@ -10,7 +10,7 @@ The example is taken from the ACM code of ethics - case study 5
|
||||
|
||||
> ###### Malicious Input to Content Filters
|
||||
>
|
||||
> **The US. Children’s Internet Protection Act (CIPA) mandates that public schools and libraries employ mechanisms to block inappropriate material that is deemed harmful to minors.**
|
||||
> **The US. Children’s Internet Protection Act (CIPA) mandates that public schools and libraries employ mechanisms to block inappropriate material that is deemed harmful to minors.**
|
||||
>
|
||||
> Blocker Plus is an automated Internet content filter designed to help these institutions comply with CIPA’s requirements. To accomplish this task, Blocker Plus has a centrally controlled blacklist maintained by the software maker. In addition, Blocker Plus provides a user-friendly interface that makes it a popular product for home use by parents.
|
||||
>
|
||||
@@ -18,30 +18,30 @@ The example is taken from the ACM code of ethics - case study 5
|
||||
>
|
||||
> During a recent review session, the development team reviewed several recent complaints about content being blocked inappropriately. An increasing amount of content regarding gay and lesbian marriage, vaccination, climate change, and other topics not covered by CIPA, had been added to the blacklist. Initial investigations into these incidents suggested that some activist groups had exploited Blocker Plus’s feedback mechanism to provide input that corrupted the classification model.
|
||||
>
|
||||
> **ANALYSIS SUMMARY:**
|
||||
> **ANALYSIS SUMMARY:**
|
||||
>
|
||||
> Blocker Plus is a system designed to block content legally designated as harmful to children. While this filtering constitutes a form of censorship, children are considered a protected vulnerable class. To reduce the impact on adults, CIPA also mandates that these filters must be disabled on request. Given that Blocker Plus is complying with US. federal regulations to facilitate socially responsible uses of computers, the system is consistent with Principles 1.1 and 2.3. Given the complexity and risk involved in Blocker Plus’s use of machine learning techniques, Principle 2.5 calls for extraordinary care. Principle 2.9 suggests that Blocker Plus should have included better protections against the intentional misuse by the activist groups. Blocker Plus’s deployment of machine learning causes harm by suppressing information of legitimate public interest and safety, as well as by discriminating based on sexual orientation, raising concerns for both Principles 1.2 and 1.4. In addition, Blocker Plus provides an example of a system becoming integrated into the educational infrastructure of society. Principle 3.7 emphasises that the developers of such systems have an added responsibility to provide good stewardship and Blocker Plus must correct these issues.
|
||||
|
||||
#### Which principles apply to Blocker Plus?
|
||||
|
||||
- `1.1` Contribute to society and human well-being
|
||||
- Socially responsible uses of computing
|
||||
- Socially responsible uses of computing
|
||||
- `2.3` Know and respect rules pertaining to professional work
|
||||
- This is broken as a federal law is being broken
|
||||
- This is broken as a federal law is being broken
|
||||
- `2.5` Evaluate computer systems and their impacts, including risks
|
||||
- Extraordinary care be taken to identify and mitigate potential risks. Blocker Plus violates this principle by allowing its feedback algorithm to be manipulated by activists to corrupt the classification model.
|
||||
- Extraordinary care should be taken to identify and mitigate potential risks. Blocker Plus violates this principle by allowing its feedback algorithm to be manipulated by activists to corrupt the classification model.
|
||||
- `2.9` Design and implement robustly and usably secure systems
|
||||
- 2.9 requires that computing professionals should perform due diligence to ensure systems function as intended, and take appropriate action to secure resources against accidental and intentional misuse, modification or denial of service. That the activists were able to intentionally misuse Blocker Plus means that the system violates this principle
|
||||
- 2.9 requires that computing professionals should perform due diligence to ensure systems function as intended, and take appropriate action to secure resources against accidental and intentional misuse, modification or denial of service. That the activists were able to intentionally misuse Blocker Plus means that the system violates this principle
|
||||
- `1.2` Avoid harm
|
||||
- Avoid harm applies as the corruption of the machine learning model means that information of legitimate public interest (gay & lesbian marriage) and safety (vaccinations and climate change) is suppressed by the activists' intentional misuse of the system
|
||||
- Avoid harm applies as the corruption of the machine learning model means that information of legitimate public interest (gay & lesbian marriage) and safety (vaccinations and climate change) is suppressed by the activists' intentional misuse of the system
|
||||
- `1.4` Be fair and do not discriminate
|
||||
- This applies in the respect of suppression of information of legitimate public interest enables discrimination of the basis of sex and sexual orientation
|
||||
- This applies in the respect that suppression of information of legitimate public interest enables discrimination on the basis of sex and sexual orientation
|
||||
- `3.7` Take special care of systems integrated into societal infrastructure
|
||||
- Applies as Blocker Plus is designed for educational purposes. In failing to prevent intentional misuse of the system, the leadership of Blocker Plus have failed in their responsibility to be good stewards of the system and enabling fair access.
|
||||
- Applies as Blocker Plus is designed for educational purposes. In failing to prevent intentional misuse of the system, the leadership of Blocker Plus have failed in their responsibility to be good stewards of the system and enable fair access.
|
||||
|
||||
Codes for the coursework only apply in negative reasons, e.g. 1.1 may apply as amazon wished to contribute to society and human well being. However this will not be marked.
|
||||
Codes for the coursework only apply for negative reasons, e.g. 1.1 may apply as Amazon wished to contribute to society and human well-being. However, this will not be marked.
|
||||
|
||||
There is one code in the amazon ring that there is no evidence of, however it is inferred by a *lack* of action.
|
||||
There is one code in the Amazon Ring case that there is no evidence of; however, it is inferred by a *lack* of action.
|
||||
|
||||
## The ACM CARE Framework
|
||||
|
||||
@@ -55,19 +55,18 @@ What were the observable effects of Amazon's actions or decisions for Ring users
|
||||
|
||||
##### Analyse
|
||||
|
||||
What stakeholder rights (legal, natural or social) were impacted and to what extent, and ask what principles of the code are relevent here.
|
||||
What stakeholder rights (legal, natural or social) were impacted and to what extent, and ask what principles of the code are relevant here.
|
||||
|
||||
> What stakeholder rights (legal, natural, or social) were impacted and to what extent? What technical facts are most relevant to the actors’ decision? What principles of the Code were most relevant? What personal, institutional, or legal values should be considered?
|
||||
|
||||
##### Review
|
||||
|
||||
What potential actions could changed the outcomes
|
||||
What potential actions could have changed the outcomes
|
||||
|
||||
> What responsibilities, authority, practices, or policies shaped the actors’ choices? What potential actions could have changed the outcomes?
|
||||
|
||||
##### Evaluate
|
||||
|
||||
What actions (or lack of actions) supported or violated the Code. Are the actions taken in this case justified, particularly when considering the rights of and impact on all stakeholders.
|
||||
What actions (or lack of actions) supported or violated the Code? Are the actions taken in this case justified, particularly when considering the rights of and impact on all stakeholders?
|
||||
|
||||
> How might the decision in this case be used as a foundation for similar future cases? What actions (or lack of action) supported or violated the Code? Are the actions taken in this case justified, particularly when considering the rights of and impact on all stakeholders?
|
||||
|
||||
@@ -20,7 +20,7 @@ This means:
|
||||
- Appropriate steps are taken to avoid harm
|
||||
- Systems are robust, secure and respect privacy
|
||||
- Rules are followed
|
||||
- Special care is taken when modifying or retiring systems or systems are integrated in societal infrastructure
|
||||
- Special care is taken when modifying or retiring systems or when systems are integrated into societal infrastructure
|
||||
|
||||
### Public Good
|
||||
|
||||
@@ -32,17 +32,17 @@ This means:
|
||||
- Entirely natural
|
||||
- Can be mitigated
|
||||
- Draws our attention to micro-issues
|
||||
- for example discriminate against people of tattoos, or people with piercings
|
||||
- Can have an squally detrimental effect as the big issues
|
||||
- Design to minimise unconscious bias
|
||||
- For example, discriminating against people with tattoos or people with piercings
|
||||
- Can have an equally detrimental effect as the big issues
|
||||
- Design to minimise unconscious bias
|
||||
|
||||
### Respect the Work of Others
|
||||
|
||||
- Do no harm
|
||||
- Do not hack
|
||||
- Unless public good requires it or you are authorised to do so
|
||||
- Unless public good requires it or you are authorised to do so
|
||||
- Respect intellectual property rights (IPR)
|
||||
- Relevant types of IPR: trade marks, industrial designs, patents, trade secrets, databases & domain names
|
||||
- Relevant types of IPR: trade marks, industrial designs, patents, trade secrets, databases & domain names
|
||||
|
||||
#### IPR
|
||||
|
||||
@@ -61,16 +61,16 @@ Distinctive elements of a product
|
||||
Used where products have a short design life e.g. fashion
|
||||
|
||||
- Two types of protection
|
||||
- Registered Community designs (RCD)
|
||||
- Protection lasts **5** years, renewed up to **25** years
|
||||
- Unregistered Community designs (UCD)
|
||||
- Protection lasts for **3** years
|
||||
- Registered Community designs (RCD)
|
||||
- Protection lasts **5** years, renewed up to **25** years
|
||||
- Unregistered Community designs (UCD)
|
||||
- Protection lasts for **3** years
|
||||
|
||||
###### Patent
|
||||
|
||||
- An exclusive right granted to protect an invention
|
||||
- Prevents others from making, using, offering for sale, selling or importing invention without owner's permission
|
||||
- Lasts for **20** years from date of filed
|
||||
- Prevents others from making, using, offering for sale, selling or importing invention without owner's permission
|
||||
- Lasts for **20** years from the filing date
|
||||
- Costs between $3,000 and \$6,000
|
||||
- Can't patent a computer program only a "computer-implemented invention"
|
||||
|
||||
@@ -85,31 +85,31 @@ Used where products have a short design life e.g. fashion
|
||||
|
||||
- Confidential business information that provides a competitive advantage
|
||||
- Must put reasonable measures in place to keep it a secret
|
||||
- Store safely, implement NDAs
|
||||
- Store safely, implement NDAs
|
||||
- Do not confer proprietary rights
|
||||
- Protected by law for an unlimited time period
|
||||
|
||||
###### Copyright
|
||||
|
||||
- Author's or creator's right to protection over uses of their work
|
||||
- Ideas cannot be copyrighted, only the concrete implementation of the idea
|
||||
- Ideas cannot be copyrighted, only the concrete implementation of the idea
|
||||
- Obtained automatically
|
||||
- Includes economic rights (renumeration for use by others)]
|
||||
- Includes economic rights (remuneration for use by others)
|
||||
- Fair use allowed
|
||||
- Covers life-time of owners plus **50-70** years
|
||||
- Covers lifetime of owners plus **50-70** years
|
||||
|
||||
###### Databases
|
||||
|
||||
- A systematic arrangement of data, works or materials
|
||||
- Two forms:
|
||||
- Original
|
||||
- Protection lasts lifetime + 50-70 years
|
||||
- Non-original (like a phone directory)
|
||||
- Protected by *sui generis* database right which lasts for **15** years
|
||||
- Original
|
||||
- Protection lasts lifetime + 50-70 years
|
||||
- Non-original (like a phone directory)
|
||||
- Protected by *sui generis* database right which lasts for **15** years
|
||||
|
||||
###### Domain Names
|
||||
|
||||
- Registered by ICANN registars
|
||||
- Registered by ICANN registrars
|
||||
- Not protected by copyright
|
||||
- May be protected by a registered trade mark
|
||||
- Last up to **10** years, renewed indefinitely
|
||||
@@ -118,11 +118,11 @@ Used where products have a short design life e.g. fashion
|
||||
|
||||
Don't go too far in protecting your own works
|
||||
|
||||
###### Sony Rookit
|
||||
###### Sony Rootkit
|
||||
|
||||
They produced CDs that when entered into a computer downloaded a rootkit which gained administrator control on the victims computer.
|
||||
They produced CDs that, when inserted into a computer, downloaded a rootkit which gained administrator control on the victim’s computer.
|
||||
|
||||
Rookit modified the victims OS, limiting the users ability to use the CD.
|
||||
The rootkit modified the victim’s OS, limiting the user’s ability to use the CD.
|
||||
|
||||
**Profoundly unethical and illegal**
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
### What is a dependable System
|
||||
|
||||
Another way of putting it is that computing systems, especially systems built into societal infrastructure, and which are otherwise safety-critical as London ambulance system was, are **dependable**.
|
||||
Another way of putting it is that computing systems, especially systems built into societal infrastructure, and which are otherwise safety-critical as the London ambulance system was, are **dependable**.
|
||||
|
||||
**Dependability** is defined by Brian Randell as the **trustworthiness** of a computer system such that reliance can justifiably be placed on the service it delivers. Dependability thus includes such properties as:
|
||||
|
||||
@@ -13,13 +13,13 @@ Another way of putting it is that computing systems, especially systems built in
|
||||
- Security
|
||||
- Maintainability
|
||||
|
||||
And provides a convenient means of subsuming these various concerns within a single conceptual framework.
|
||||
And provides a convenient means of subsuming these various concerns within a single conceptual framework.
|
||||
|
||||
**Reliability** means that a system provides continuity of correct service during its useful lifetime, from commisioning, through operation, to decomissioning.
|
||||
**Reliability** means that a system provides continuity of correct service during its useful lifetime, from commissioning, through operation, to decommissioning.
|
||||
|
||||
**Safety** means that a system is engineered to avoid catastrophic consequences for user and the environment and that the life-critical system behaves as needed, even if components fail.
|
||||
**Safety** means that a system is engineered to avoid catastrophic consequences for users and the environment and that the life-critical system behaves as needed, even if components fail.
|
||||
|
||||
**Integrity** means that a system’s source code or state cannot be altered improperly, i.e., it is secure, or its data be corrupted.
|
||||
**Integrity** means that a system’s source code or state cannot be altered improperly, i.e., it is secure, or its data cannot be corrupted.
|
||||
|
||||
**Maintainability** means that a system is engineered to permit adaptive maintenance, ease of modification and repair of defects.
|
||||
|
||||
@@ -27,27 +27,27 @@ And provides a convenient means of subsuming these various concerns within a sin
|
||||
|
||||
##### Uber’s self-driving car accident
|
||||
|
||||
- Back up drivber charged with negligent homicide
|
||||
- However the National Transport Safety Board finds ubers system to be at fault
|
||||
- Backup driver charged with negligent homicide
|
||||
- However, the National Transport Safety Board finds Uber’s system to be at fault
|
||||
- While Uber’s radar and Lidar detected Elaine 6 seconds before the impact, their system did not have the capacity to **classify** the object as a pedestrian unless they were near a crosswalk
|
||||
- It classified Elaine as a vehicle, bicycle and an unknown object
|
||||
- It assumed Elaine would be travelling in the same direction as the car and therefore did not slow down
|
||||
- Furthermore, the car had its own in-built automatic braking system which was capable of detecting and stopping for Elaine, but it was disabled by Uber engineers as they thought it would interfere with Uber’s self driving sensors
|
||||
- It classified Elaine as a vehicle, bicycle and an unknown object
|
||||
- It assumed Elaine would be travelling in the same direction as the car and therefore did not slow down
|
||||
- Furthermore, the car had its own in-built automatic braking system which was capable of detecting and stopping for Elaine, but it was disabled by Uber engineers as they thought it would interfere with Uber’s self-driving sensors
|
||||
- When the car was just a second away from Elaine, Uber’s system finally recognised that the object could not be avoided
|
||||
- Now at this point, Uber’s system could have slammed on the brakes to migate the imapact, instead an *action supression* component kicked in.
|
||||
- This was implemented to avoid extreme manoeuvers in response to false alarms.
|
||||
- Now at this point, Uber’s system could have slammed on the brakes to mitigate the impact; instead, an *action suppression* component kicked in.
|
||||
- This was implemented to avoid extreme manoeuvres in response to false alarms.
|
||||
- Uber couldn’t supply documents showing checks performed on the backup driver
|
||||
|
||||
Computing failures are not restricted to 1 car and 2 plane crashes
|
||||
|
||||
The FDA reports, that medical device recalls are at an all time high and that defective software is a major cause. One in every three medical devices that use software for operations have been **recalled** because of **failures in their software**.
|
||||
The FDA reports that medical device recalls are at an all-time high and that defective software is a major cause. One in every three medical devices that use software for operations has been **recalled** because of **failures in their software**.
|
||||
|
||||
As the Uber and Boeing cases clearly demonstrate, dependability is still a critical issue in computing today.
|
||||
|
||||
- Apart from the direct human cost, the failure of computing systems costs a great deal of money.
|
||||
- Apart from the direct human cost, the failure of computing systems costs a great deal of money.
|
||||
- The 5th edition of the Software Fail Watch identified 606 recorded software failures, impacting half of the world’s population (3.7 billion people) and 314 companies to the cost of 1.7 trillion dollars, and noted that “this is just scratching the surface – there are far more software defects in the world than we will likely ever know about.”
|
||||
|
||||
We have an ethical duty to the public to minimise these harms. I purposefully say minimise and not eradicate, as it is inevitable that things will go wrong some-times due to unforeseen circumstances, but if we exercise due diligence in our work then we should be able to significantly reduce the harms caused through what are euphemistically called “software bugs”.
|
||||
We have an ethical duty to the public to minimise these harms. I purposefully say minimise and not eradicate, as it is inevitable that things will go wrong sometimes due to unforeseen circumstances, but if we exercise due diligence in our work then we should be able to significantly reduce the harms caused through what are euphemistically called “software bugs”.
|
||||
|
||||
#### Software Bugs
|
||||
|
||||
@@ -70,14 +70,14 @@ The V Model adapts the waterfall by placing an emphasis on early testing
|
||||
|
||||
###### Spiral Model
|
||||
|
||||
Spiral model provides a major alternative and places testing, in iterative requirements, design, implement and test sequences that spiral out from one another and are marked by the development of increasingly high fidelity prototypes
|
||||
Spiral model provides a major alternative and places testing in iterative requirements, design, implement and test sequences that spiral out from one another and are marked by the development of increasingly high-fidelity prototypes
|
||||
|
||||
##### Testing Methodologies
|
||||
|
||||
###### Static Testing
|
||||
|
||||
- Static testing takes place early in a software system’s development and examines source code and accompanying documentation but doesn’t execute the program.
|
||||
- It may be done manually, though increasingly relies on automated analysis tools.
|
||||
- It may be done manually, though increasingly relies on automated analysis tools.
|
||||
|
||||
###### Dynamic Testing
|
||||
|
||||
@@ -87,7 +87,7 @@ Spiral model provides a major alternative and places testing, in iterative requi
|
||||
|
||||
###### White Box Testing
|
||||
|
||||
White box testing digs into the inner workings of the software.
|
||||
White box testing digs into the inner workings of the software.
|
||||
|
||||
- It tests each statement, object, and function on an individual basis
|
||||
- Identifies broken or poorly structured paths in coding processes
|
||||
@@ -96,20 +96,20 @@ White box testing digs into the inner workings of the software.
|
||||
|
||||
###### Black Box Testing
|
||||
|
||||
Black box testing on the other hand examines the outer workings of the software and that the software does what it’s supposed to do.
|
||||
Black box testing on the other hand examines the outer workings of the software and that the software does what it’s supposed to do.
|
||||
|
||||
- Knowledge of coding isn’t necessary, and testers work at the user-interface level checking inputs and outputs.
|
||||
|
||||
###### GUI Testing
|
||||
|
||||
Graphical user interface or GUI testing
|
||||
Graphical user interface or GUI testing
|
||||
|
||||
- Checks user interface works as per the GUI specification.
|
||||
- Checks user interface works as per the GUI specification.
|
||||
- It tests the software control dialogues, including:
|
||||
- screen layouts
|
||||
- menus
|
||||
- buttons
|
||||
- icons, pop-up windows, text boxes, text formatting, colours, fonts, font sizes, etc.
|
||||
- screen layouts
|
||||
- menus
|
||||
- buttons
|
||||
- icons, pop-up windows, text boxes, text formatting, colours, fonts, font sizes, etc.
|
||||
|
||||
##### Testing Levels
|
||||
|
||||
@@ -125,13 +125,13 @@ Graphical user interface or GUI testing
|
||||
|
||||
### Testing and Dependability
|
||||
|
||||
As Linda Rosenberg and her colleagues told us in their award-winning 1998 IEEE paper on software reliability,
|
||||
As Linda Rosenberg and her colleagues told us in their award-winning 1998 IEEE paper on software reliability,
|
||||
|
||||
> “Metrics to measure software reliability exist and can be used starting in the requirements phase. At each phase of the development life cycle, metrics can identify potential areas of problems that may lead to problems or errors. Finding these areas in the phase they are developed decreases the cost and prevents potential ripple effects from the changes, later in the development life cycle by at least a factor of 14.”
|
||||
|
||||
#### Limits of Testing
|
||||
|
||||
Brian Randell tells us that
|
||||
Brian Randell tells us that
|
||||
|
||||
> “a system **failure** occurs when the delivered service no longer complies with the **specification**, the latter being an agreed description of the system's expected function and/or service.”
|
||||
|
||||
@@ -143,31 +143,31 @@ Daniel Jackson and colleagues elaborate the point, saying that,
|
||||
|
||||
The bug at work here was a **faulty** angle of attack or AOA **sensor**, which indicated the angle at which the aircraft was positioned in flight.
|
||||
|
||||
The Ethiopian accident investigation report says that Boeing’s engineers determined that no piloted simulation, was required for take-off or low speed flight. This meant that specific failures that could lead to MCAS activation, such as false AOA input, were not simulated as part of the aircraft’s functional hazard assessment and validation tests.
|
||||
The Ethiopian accident investigation report says that Boeing’s engineers determined that no piloted simulation was required for take-off or low-speed flight. This meant that specific failures that could lead to MCAS activation, such as false AOA input, were not simulated as part of the aircraft’s functional hazard assessment and validation tests.
|
||||
|
||||
Boeing assumed that the worse that could happen would be single fault-driven MCAS activation that flight crew would correct as per “trained memory procedures” acquired during flight training for previous 737 models. As the graph showing the plane going up and down in the Vox video makes painfully visible, the MAX 8 crashes involved multiple MCAS activations, caused by the faulty AOA sensor.
|
||||
Boeing assumed that the worst that could happen would be single fault-driven MCAS activation that flight crew would correct as per “trained memory procedures” acquired during flight training for previous 737 models. As the graph showing the plane going up and down in the Vox video makes painfully visible, the MAX 8 crashes involved multiple MCAS activations, caused by the faulty AOA sensor.
|
||||
|
||||
Poor specification requirements: Input was only required from one AOA sensor to activate MCAS, depsite two sensors being fitted.
|
||||
Poor specification requirements: Input was only required from one AOA sensor to activate MCAS, despite two sensors being fitted.
|
||||
|
||||
- This means the faulty sensor constantly triggered MCAS
|
||||
- No information about MCAS was given in the flight crew manuals and MCAS was not included in flight crew training.
|
||||
- Boeing assumed that pilots certified to fly on earlier versions of the 737 didn’t need any extra training.
|
||||
- The lack of documentation and training meant that flight crews were unaware of MCAS and its effects
|
||||
- The lack of information about MCAS in the flight crew manual meant that there were no procedures for mitigating erroneous input from the AOA sensors
|
||||
- The lack of documentation and training meant that flight crews were unaware of MCAS and its effects
|
||||
- The lack of information about MCAS in the flight crew manual meant that there were no procedures for mitigating erroneous input from the AOA sensors
|
||||
- An AOA disagree warning light would flash if the two sensors were at odds with each other
|
||||
- These indicators were sold as optional extras
|
||||
- These extras were not found on either aircraft
|
||||
- The Indonesian crash report finds that the flight crew were **not aware** that the AOA DISAGREE warning would not appear if AOA DISAGREE conditions were met, and that in failing to install the warning lights **Boeing denied the flight crew valid information** about the abnormal conditions they faced
|
||||
- These indicators were sold as optional extras
|
||||
- These extras were not found on either aircraft
|
||||
- The Indonesian crash report finds that the flight crew were **not aware** that the AOA DISAGREE warning would not appear if AOA DISAGREE conditions were met, and that in failing to install the warning lights **Boeing denied the flight crew valid information** about the abnormal conditions they faced
|
||||
|
||||
It becomes apparent then that the **AOA sensor bug wasn’t really the problem**. It could well have been handled
|
||||
It becomes apparent then that the **AOA sensor bug wasn’t really the problem**. It could well have been handled
|
||||
|
||||
- Had MCAS not been designed to activate off input from a single sensor
|
||||
- If flight crew had been informed about MCAS
|
||||
- Its effects built into difference training and the flight crew manual,
|
||||
- Its effects built into difference training and the flight crew manual,
|
||||
- Had the planes been fitted (like their predecessors) with the AOA warning lights.
|
||||
|
||||
The crashes are as much, if not more, a failure of poor requirements, including both poor technical and usability specifications, and inadequate, indeed non-existent, documentation and training, which are also key parts of the user interface to and usability of a system.
|
||||
|
||||
There are limits to software testing.
|
||||
There are limits to software testing.
|
||||
|
||||
Dependability relies as much on **sound requirements specifications** as it does **good code** and **rigorous testing**.
|
||||
Dependability relies as much on **sound requirements specifications** as it does **good code** and **rigorous testing**.
|
||||
@@ -10,81 +10,81 @@ Security is legally required for systems that process personal data.
|
||||
|
||||
#### Why is Security so Important
|
||||
|
||||
In the UK 46% of businesses and 26% of charities have delt with cyber attacks
|
||||
In the UK 46% of businesses and 26% of charities have dealt with cyber attacks
|
||||
|
||||
Ransomware is the fastest growing type of cybercrime and costs are predicted to reach 20 billion dollars by 2021, which is 57 times greater than it was in 2015.
|
||||
Ransomware is the fastest-growing type of cybercrime and costs are predicted to reach 20 billion dollars by 2021, which is 57 times greater than it was in 2015.
|
||||
|
||||
Cyber security breaches have increased globally by 67% since 2014. They essentially operate in 2 ways:
|
||||
|
||||
1. Through bad actors, particularly people who try to phish for and otherwise elicit usernames and passwords to access systems
|
||||
2. Through bad computing, particularly the use of viruses, malware and denial of service attacks that compromise systems.
|
||||
|
||||
It is broadly acknowledged that IoT devices, which typically exploit low cost sensors, suffer from extremely poor and indeed non-existent security.
|
||||
It is broadly acknowledged that IoT devices, which typically exploit low-cost sensors, suffer from extremely poor and indeed non-existent security.
|
||||
|
||||
#### Causes of poor Security
|
||||
|
||||
In addition to internal reasons to do with poor coding and testing, and poor specification of technical and usability requirements, poor security has also been attributed to the law and limits of liability.
|
||||
In addition to internal reasons to do with poor coding and testing, and poor specification of technical and usability requirements, poor security has also been attributed to the law and limits of liability.
|
||||
|
||||
In the US, for example, the courts have consistently interpreted software licenses in a way that allows vendors to disclaim almost all liability for software defects.
|
||||
In the US, for example, the courts have consistently interpreted software licences in a way that allows vendors to disclaim almost all liability for software defects.
|
||||
|
||||
**The economic loss**: rule states that if a product causes no personal injury or property damage, other than to the product itself, then such damages are determined by contract law and limited to a breach of contract claim.
|
||||
|
||||
- This prevents customers from suing as most often claims consist of
|
||||
- Loss of sensitive & personal data
|
||||
- Loss of sensitive & personal data
|
||||
|
||||
Then there is the fact that any data entered into a computer system by the user is **not considered part of the software**, and hence **not part of the product**. The data and the software are separate. The data can be read and manipulated by the software, but it is created by the user or a third party, not the software vendor. Therefore, destruction of data due to insecure software is not deemed damage to or destruction of the software itself.
|
||||
Then there is the fact that any data entered into a computer system by the user is **not considered part of the software**, and hence **not part of the product**. The data and the software are separate. The data can be read and manipulated by the software, but it is created by the user or a third party, not the software vendor. Therefore, destruction of data due to insecure software is not deemed damage to or destruction of the software itself.
|
||||
|
||||
Now GDPR, the EU’s updated data protection regulation, goes some way towards incentivising secure treatment of personal data with its 20 million euro fines for anyone who **fails to put adequate technical and organisational safeguards in place**, but that of course only **applies to the parties who process such data**, and **not to those who build**, **distribute**, **sell**, or **maintain** the software they use.
|
||||
|
||||
#### National Cyber Security Strategy
|
||||
|
||||
UK Govement invested £1.9 bn in its National Cyber Security strategy in 2016.
|
||||
UK Government invested £1.9 bn in its National Cyber Security strategy in 2016.
|
||||
|
||||
The UK’s National Cyber Security Strategy stands on 3 pillars:
|
||||
|
||||
1. **DEFEND**: the country against evolving cyber threats, which involves responding effectively to incidents, ensuring UK networks, systems and data are protected and resilient, and providing UK citizens and businesses with the knowledge needed to defend themselves.
|
||||
2. **DETER**, which involves detecting, investigating and disrupting hostile action, and pursuing and prosecuting offenders.
|
||||
3. **DEVELOP** a self-sustaining pipeline of talent providing the skills to meet national needs across the public and private sectors.
|
||||
1. **DEFEND**: the country against evolving cyber threats, which involves responding effectively to incidents, ensuring UK networks, systems and data are protected and resilient, and providing UK citizens and businesses with the knowledge needed to defend themselves.
|
||||
2. **DETER**, which involves detecting, investigating and disrupting hostile action, and pursuing and prosecuting offenders.
|
||||
3. **DEVELOP** a self-sustaining pipeline of talent providing the skills to meet national needs across the public and private sectors.
|
||||
|
||||
#### Secure By Design
|
||||
|
||||
Cyber-physical systems include software systems that not only compute but also act in the world, e.g., IoT devices such as smart thermostats or smart door locks or autonomous systems such as self-driving cars.
|
||||
Cyber-physical systems include software systems that not only compute but also act in the world, e.g., IoT devices such as smart thermostats or smart door locks or autonomous systems such as self-driving cars.
|
||||
|
||||
**Secure by design:** software has been designed from its foundations up to be secure.
|
||||
**Secure by design:** software has been designed from its foundations up to be secure.
|
||||
|
||||
NCSC articulates **5 core secure by design principles**. These include:
|
||||
|
||||
1. Establishing the context before designing a system
|
||||
- Risk analysis is **critical**
|
||||
- Component-driven analysis and system-driven analysis (see below)
|
||||
- Risk analysis is **critical**
|
||||
- Component-driven analysis and system-driven analysis (see below)
|
||||
2. Making compromise difficult
|
||||
- External data inputs cannot be trusted
|
||||
- Data inputs must be sanitised, validated
|
||||
- Attack surfaces should be minimised, exposing as few components as possible
|
||||
- Read-only views should be enforced where ever possible
|
||||
- All privileged actions should be accessed through control functions and must be attributed to individuals
|
||||
- External data inputs cannot be trusted
|
||||
- Data inputs must be sanitised, validated
|
||||
- Attack surfaces should be minimised, exposing as few components as possible
|
||||
- Read-only views should be enforced wherever possible
|
||||
- All privileged actions should be accessed through control functions and must be attributed to individuals
|
||||
3. Making disruption difficult
|
||||
- Identify system bottlenecks
|
||||
- Test systems with unreasonably high loads and Ddos attacks
|
||||
- Understanding how the system responds to failure
|
||||
- Monkey testing
|
||||
- Identify system bottlenecks
|
||||
- Test systems with unreasonably high loads and DDoS attacks
|
||||
- Understanding how the system responds to failure
|
||||
- Monkey testing
|
||||
4. Making compromise detection easier
|
||||
- Monitoring system behaviour
|
||||
- Logging security events
|
||||
- Like a log of all logins and logouts
|
||||
- Ensuring the monitoring is independent of the software itself
|
||||
- Monitoring system behaviour
|
||||
- Logging security events
|
||||
- Like a log of all logins and logouts
|
||||
- Ensuring the monitoring is independent of the software itself
|
||||
5. Reducing the impact of compromise.
|
||||
- Removing unnecessary functionality such as debug or test functionality
|
||||
- Segmenting assets on networks to contain breaches to particular segments
|
||||
- Designing systems so that they can be quickly rebuilt to a known clean state
|
||||
- Removing unnecessary functionality such as debug or test functionality
|
||||
- Segmenting assets on networks to contain breaches to particular segments
|
||||
- Designing systems so that they can be quickly rebuilt to a known clean state
|
||||
|
||||
###### Component-driven Analysis
|
||||
|
||||
Focuses on the technical components a system is composed of, the threats and vulnerabilities that may effect those components, and the impact caused if any of the components was compromised.
|
||||
Focuses on the technical components a system is composed of, the threats and vulnerabilities that may affect those components, and the impact caused if any of the components was compromised.
|
||||
|
||||
This type of analysis allows the specific risks faced by specific components within a system to be identified and prioritised
|
||||
|
||||
1. According to the **ease** with which a vulnerablity could be exploited and a component comprimised.
|
||||
1. According to the **ease** with which a vulnerability could be exploited and a component compromised.
|
||||
2. According to the **severity** of impact.
|
||||
|
||||
The purpose of prioritising risks in this way is to mitigate the worst risks first.
|
||||
@@ -97,52 +97,52 @@ NCSC suggests we rarely consider what a system should not do at the beginning of
|
||||
|
||||
### Securing the IoT
|
||||
|
||||
There are more the 10 billion IoT devices as of 2021. This inevitably creates an exponential increase in the attack surface and opens up society to cyber attack on an unprecedented scale, especially as IoT devices are broadly recognised to have very poor cyber security.
|
||||
There are more than 10 billion IoT devices as of 2021. This inevitably creates an exponential increase in the attack surface and opens up society to cyber attack on an unprecedented scale, especially as IoT devices are broadly recognised to have very poor cyber security.
|
||||
|
||||
#### Guidelines
|
||||
|
||||
1. **No longer set default passwords**
|
||||
|
||||
- Many IoT devices are compromised by the Mirai botnet, which exploits default passwords set by manufacturers.
|
||||
- Many IoT devices are compromised by the Mirai botnet, which exploits default passwords set by manufacturers.
|
||||
|
||||
- All IoT device passwords should be unique and should not reset to a universal factory default.
|
||||
- All IoT device passwords should be unique and should not reset to a universal factory default.
|
||||
|
||||
2. **Vulnerability disclosure policy**
|
||||
|
||||
- Provide a public point of contact to enable security researchers and users to report issues.
|
||||
- This enables the continual monitoring, identification and rectification of security vulnerabilities as part of a device’s security lifecycle.
|
||||
- Provide a public point of contact to enable security researchers and users to report issues.
|
||||
- This enables the continual monitoring, identification and rectification of security vulnerabilities as part of a device’s security lifecycle.
|
||||
|
||||
3. **Keep their software updated**
|
||||
|
||||
- Security patches should be delivered over a secure channel and their provenance be assured.
|
||||
- Security patches should be delivered over a secure channel and their provenance be assured.
|
||||
|
||||
4. **Secure data storage**
|
||||
|
||||
- Sensitive data, including cryptographic keys, device identifiers and initialisation vectors, should be **stored securely** using mechanisms provided by a Trusted Execution Environment.
|
||||
- Sensitive data, including cryptographic keys, device identifiers and initialisation vectors, should be **stored securely** using mechanisms provided by a Trusted Execution Environment.
|
||||
|
||||
5. **Secure Communications**
|
||||
|
||||
- All data should be encrypted in transit to ensure **secure communications**.
|
||||
- All data should be encrypted in transit to ensure **secure communications**.
|
||||
|
||||
6. **Minimise the attack surface of devices**
|
||||
|
||||
- Device manufacturers and service providers should ensure hardware does not unnecessarily expose access points
|
||||
- Unused ports should be closed, services should not be available if they are not used, and code should be minimised to the functionality necessary for the service to operate.
|
||||
- All devices should operate on the principle of least **privilege**
|
||||
- Giving users or processes only those privileges essential to the performance of their intended function.
|
||||
- Device manufacturers and service providers should ensure hardware does not unnecessarily expose access points
|
||||
- Unused ports should be closed, services should not be available if they are not used, and code should be minimised to the functionality necessary for the service to operate.
|
||||
- All devices should operate on the principle of least **privilege**
|
||||
- Giving users or processes only those privileges essential to the performance of their intended function.
|
||||
|
||||
7. **Ensure software integrity**
|
||||
|
||||
- Using secure boot mechanisms to verify software.
|
||||
- If an unauthorised change is detected, the device should alert the consumer and not connect to wider networks, other than those necessary to perform the alerting function.
|
||||
- Using secure boot mechanisms to verify software.
|
||||
- If an unauthorised change is detected, the device should alert the consumer and not connect to wider networks, other than those necessary to perform the alerting function.
|
||||
|
||||
8. **Resilient to outages**
|
||||
|
||||
- Whenever possible, IoT systems should remain operating and be **locally functional** in the case of a loss of network connectivity and should recover cleanly in the case of restoration of a loss of power.
|
||||
- Whenever possible, IoT systems should remain operating and be **locally functional** in the case of a loss of network connectivity and should recover cleanly in the case of restoration of a loss of power.
|
||||
|
||||
9. **Easy to install and maintain**
|
||||
|
||||
- User interfaces should be easy to use and clear guidance should be provided to users to set up devices securely and reduce their exposure to threats.
|
||||
- User interfaces should be easy to use and clear guidance should be provided to users to set up devices securely and reduce their exposure to threats.
|
||||
|
||||
10. **Monitor telemetry data**
|
||||
|
||||
@@ -161,6 +161,3 @@ There are more the 10 billion IoT devices as of 2021. This inevitably creates an
|
||||
13. **Delete personal data**
|
||||
|
||||
- Users should be able to **delete personal data** easily if they wish to, when there is a transfer of ownership, or when they dispose of a device.
|
||||
|
||||
|
||||
|
||||
@@ -13,20 +13,20 @@
|
||||
|
||||
And so on
|
||||
|
||||
> Privacy allows us to negotiate who we are and how we want to interact with the world around us, and is essential to who we are as human beings. It gives us a space to be ourselves without judgement, allows us to think freely without discrimination, and is essential to individual autonomy and the protection of human dignity.
|
||||
> Privacy allows us to negotiate who we are and how we want to interact with the world around us, and is essential to who we are as human beings. It gives us a space to be ourselves without judgement, allows us to think freely without discrimination, and is essential to individual autonomy and the protection of human dignity.
|
||||
|
||||
https://privacyinternational.org/explainer/56/what-privacy
|
||||
https://privacyinternational.org/explainer/56/what-privacy
|
||||
|
||||
> “No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.” **Article 12 of the UN declaration**
|
||||
> “No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.” **Article 12 of the UN declaration**
|
||||
|
||||
> **Article 8.1 of the EU convention – the right to respect for private and family life**
|
||||
> **Article 8.1 of the EU convention – the right to respect for private and family life**
|
||||
>
|
||||
> 1. Everyone has the right to respect for his private and family life, his home and his correspondence;
|
||||
> 2. There shall be no interference by a public authority with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others.
|
||||
|
||||
Privacy is a fundamental human right and underpins many other human rights including freedom of association and free speech.
|
||||
|
||||
It’s politically contentious status makes it an ethical imperative in professional computing and key to ensuring public confidence and trust.
|
||||
Its politically contentious status makes it an ethical imperative in professional computing and key to ensuring public confidence and trust.
|
||||
|
||||
> That’s why the BCS and ACM include “respect for privacy” as a requirement in their ethics codes, and the IEEE has a separate Data Access and Use policy to align it with industry best practice and ensure compliance with international regulations including the European Union’s General Data Protection Regulation or GDPR
|
||||
|
||||
@@ -40,7 +40,7 @@ Warren and Brandeis argued that technology enabled harms to privacy including in
|
||||
- unwanted publicity
|
||||
- misuse of a person’s name or likeness for financial advantage.
|
||||
|
||||
Informational privacy is thus a concern with the protection of personal or private information from unauthorised disclosure and misuse. https://plato.stanford.edu/entries/privacy
|
||||
Informational privacy is thus a concern with the protection of personal or private information from unauthorised disclosure and misuse. https://plato.stanford.edu/entries/privacy
|
||||
|
||||
### Relevant Authority
|
||||
|
||||
@@ -55,7 +55,7 @@ The **data subject** is a natural person, an individual who can be identified, d
|
||||
**Personal data** is **any** information relating to an identified **or** identifiable person (i.e., the ‘data subject’), **either directly or indirectly**. Personal data includes a bunch of technical information including such things as account handles, IP or MAC addresses, cookies, RFID frequencies, device fingerprints, etc.
|
||||
|
||||
- The key point here is that personal data may not directly link to a *data subject* as say a passport might
|
||||
- But may relate indirectly to a person once the data has been procesed
|
||||
- But may relate indirectly to a person once the data has been processed
|
||||
|
||||
**Processing** means any operation or set of operations which is performed on personal data or on sets of personal data, whether or not by automated means.
|
||||
|
||||
@@ -77,21 +77,21 @@ Similarly, **processor** does not refer to a CPU on a computer, but to the perso
|
||||
|
||||
**Controller** means the person, legal entity, public authority, agency or other body which, alone or jointly with others, determines the purposes for which personal data will be processed and the means of processing them.
|
||||
|
||||
**Data protection** officer or **DPO**, who may be an employee of the controller or processor or an independent contractor who has expert knowledge of data protection law and must be consulted by the controller or processor in a timely manner in all issues which relate to the protection of personal data. A DPO must be appointed if a controller or processor’s core activities involve the processing of personal data on a large scale or involve large scale, regular and systematic monitoring of individuals.
|
||||
**Data protection** officer or **DPO**, who may be an employee of the controller or processor or an independent contractor who has expert knowledge of data protection law and must be consulted by the controller or processor in a timely manner in all issues which relate to the protection of personal data. A DPO must be appointed if a controller or processor’s core activities involve the processing of personal data on a large scale or involve large-scale, regular and systematic monitoring of individuals.
|
||||
|
||||
#### GDPR
|
||||
|
||||
GDPR places specific legal requirements on controllers, which directly impact processors.
|
||||
|
||||
> **Article 23 of GDPR** says that, “1. Taking into account the state of the art, the cost of implementation and the nature, scope, context and purposes of processing as well as the risks … posed by the processing, the controller shall, both at the time of the determination of the means for processing and at the time of the processing itself, implement appropriate technical and organisational measures … in an effective manner and … integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects. 2. The controller shall **implement** appropriate technical and organisational measures … **by default** …”
|
||||
> **Article 23 of GDPR** says that, “1. Taking into account the state of the art, the cost of implementation and the nature, scope, context and purposes of processing as well as the risks … posed by the processing, the controller shall, both at the time of the determination of the means for processing and at the time of the processing itself, implement appropriate technical and organisational measures … in an effective manner and … integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects. 2. The controller shall **implement** appropriate technical and organisational measures … **by default** …”
|
||||
|
||||
> The European Data Protection Board or EDPD, which furnishes guidance on GDPR tells us that, “a ‘default’, as commonly defined in computer science, refers to the pre-existing or preselected value of a configurable setting that is assigned to a software application, computer program or device. Such settings are also called ‘presets’ or ‘factory presets’.” EDPB Guidelines
|
||||
|
||||
So the term **implement by default** in GDPR refers to the design of preset technical and organisational measures to ensure that data processing operations meet the requirements of GDPR and thus protects the legal rights of data subjects. We’ll take a look at what those presets are about shortly.
|
||||
So the term **implement by default** in GDPR refers to the design of preset technical and organisational measures to ensure that data processing operations meet the requirements of GDPR and thus protect the legal rights of data subjects. We’ll take a look at what those presets are about shortly.
|
||||
|
||||
The controller is legally **accountable** for the choice of presets and implementing data protection by design and default. (Article 5 GDPR)
|
||||
|
||||
This means that the controller must be able to **demonstrate** to themselves, to data subjects and to supervisory authorities alike that the technical and organisational measures they have put in place are a) appropriate and b) effective in ensuring data protection by design and default.
|
||||
This means that the controller must be able to **demonstrate** to themselves, to data subjects and to supervisory authorities alike that the technical and organisational measures they have put in place are a) appropriate and b) effective in ensuring data protection by design and default.
|
||||
|
||||
### Data Protection by Design and default
|
||||
|
||||
@@ -100,16 +100,16 @@ By default controllers must be **transparent** about how they collect, use and s
|
||||
These include:
|
||||
|
||||
- the right to access any personal data held by the controller that relates to the data subject (Article 15)
|
||||
- to object to the processing of personal data (Article 21)
|
||||
- to object to the processing of personal data (Article 21)
|
||||
- obtain human intervention when querying automated decisions (Article 22)
|
||||
- to restrict processing (Article 18)
|
||||
- to rectify inaccuracies (Article 16)
|
||||
- to export data in a commonly used and machine-readable format (Article 20)
|
||||
- to have data erased and be forgotten (Article 17).
|
||||
|
||||
> **Recital 63** which says, “Where possible, the controller should be able to provide remote access to a secure system which would provide the data subject with direct access to his or her personal data.”
|
||||
> **Recital 63** which says, “Where possible, the controller should be able to provide remote access to a secure system which would provide the data subject with direct access to his or her personal data.”
|
||||
|
||||
So transparency is something that needs to built into systems in the long term and not simply be seen as a matter of appending documentation to their use.
|
||||
So transparency is something that needs to be built into systems in the long term and not simply be seen as a matter of appending documentation to their use.
|
||||
|
||||
The controller must also by default identify and declare a **valid legal basis** for the processing. Six legal grounds exist including:
|
||||
|
||||
@@ -126,13 +126,13 @@ Fairness is especially important with respect to data processing operations that
|
||||
|
||||
The controller must also ensure that data is only collected for **specific, explicitly stated purposes** and that data is not further processed in a manner that is incompatible with the purposes for which they were initially collected.
|
||||
|
||||
This is called **purpose limitation**. It means a controller cannot simply collect as much data as they like and do with it what they want. Data collection must be limited by default to specific purposes which are transparent to the data subject.
|
||||
This is called **purpose limitation**. It means a controller cannot simply collect as much data as they like and do with it what they want. Data collection must be limited by default to specific purposes which are transparent to the data subject.
|
||||
|
||||
**Data minimisation**: the controller must ensure that data collection is limited to what is necessary to meet the purposes for which they are being processed.
|
||||
|
||||
Data minimisation requires that the controller verify whether the purposes can be achieved by processing less personal data, or having less detailed or aggregated personal data or without having to process personal data at all. Such verification should take place before any processing takes place, and be carried out at any during the processing lifecycle.
|
||||
Data minimisation requires that the controller verify whether the purposes can be achieved by processing less personal data, or having less detailed or aggregated personal data or without having to process personal data at all. Such verification should take place before any processing takes place, and be carried out during the processing lifecycle.
|
||||
|
||||
Data minimisation also refers to the degree of identification. If the purpose does not require the final set of data to refer to an individual (such as statistics) - then the controller should delete or anonymise personal data as soon as possible. If continued identification is needed for other processing activities, personal data should be pseudonymized to mitigate risks for the data subjects’ rights.
|
||||
Data minimisation also refers to the degree of identification. If the purpose does not require the final set of data to refer to an individual (such as statistics) - then the controller should delete or anonymise personal data as soon as possible. If continued identification is needed for other processing activities, personal data should be pseudonymised to mitigate risks for the data subjects’ rights.
|
||||
|
||||
By default, the controller must **limit** the period for which personal data kept in a form which permits identification of data subjects are **stored** and retain data in such a form for no longer than is necessary to meet the purposes for which it has been collected.
|
||||
|
||||
@@ -152,44 +152,44 @@ DPIA - **D**ata **P**rotection **I**mpact **A**ssessments
|
||||
|
||||
A DPIA is also required by law where large amounts of special category data are processed.
|
||||
|
||||
Special category data is data that reveal racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, bio-metric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person's sex life or sexual orientation.
|
||||
Special category data is data that reveal racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person's sex life or sexual orientation.
|
||||
|
||||
DPIAs are legally required for these areas of personal data processing, but they are generally recommended as “good practice” for any processing of personal data. https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/accountability-and-governance/data-protection-impact-assessments/
|
||||
DPIAs are legally required for these areas of personal data processing, but they are generally recommended as “good practice” for any processing of personal data. https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/accountability-and-governance/data-protection-impact-assessments/
|
||||
|
||||
### How to know when processing is high risk
|
||||
|
||||
There are 4 critieria specified in GDPR article 35
|
||||
There are 4 criteria specified in GDPR article 35
|
||||
|
||||
1. The use of new technologies to process personal data
|
||||
2. Automated-decision making with legal or significant effect
|
||||
2. Automated decision-making with legal or significant effect
|
||||
3. Processing of special category data
|
||||
4. Systematic monitoring of public spaces
|
||||
|
||||
There are additional criteria
|
||||
|
||||
5. **Evaluation or scoring, including profiling and predicting**
|
||||
- especially of data concerning the data subject's performance at work, economic situation, health, personal preferences or interests, reliability or behavior, location or movements.
|
||||
- Examples of this are financial institutions that screen customers against a credit reference database
|
||||
5. **Evaluation or scoring, including profiling and predicting**
|
||||
- especially of data concerning the data subject's performance at work, economic situation, health, personal preferences or interests, reliability or behaviour, location or movements.
|
||||
- Examples of this are financial institutions that screen customers against a credit reference database
|
||||
6. **The processing of sensitive data or data of a highly personal nature**
|
||||
- Not only special categories of personal data, but also any data considered as sensitive as the term is commonly understood
|
||||
- e.g., data linked to household and private activities (such as electronic communications), or data that impact the exercise of a fundamental right (such as location data whose collection may impact freedom of movement), financial data, personal documents, personal information contained in life-logging applications, etc.
|
||||
- Not only special categories of personal data, but also any data considered as sensitive as the term is commonly understood
|
||||
- e.g., data linked to household and private activities (such as electronic communications), or data that impact the exercise of a fundamental right (such as location data whose collection may impact freedom of movement), financial data, personal documents, personal information contained in life-logging applications, etc.
|
||||
7. **The processing of personal data on a large scale**
|
||||
- which is determined by the number of data subjects concerned
|
||||
- the volume of data and/or the range of different data items being processed
|
||||
- the duration or permanence of the data processing activity
|
||||
- the geographical extent of the processing activity
|
||||
- which is determined by the number of data subjects concerned
|
||||
- the volume of data and/or the range of different data items being processed
|
||||
- the duration or permanence of the data processing activity
|
||||
- the geographical extent of the processing activity
|
||||
8. **Matching or combining datasets**
|
||||
- data originating from two or more data processing operations performed for different purposes and/or by different data controllers in a way that would exceed the reasonable expectations of the data subject.
|
||||
- data originating from two or more data processing operations performed for different purposes and/or by different data controllers in a way that would exceed the reasonable expectations of the data subject.
|
||||
9. **Data is processed that relates to vulnerable data subjects**
|
||||
- For example, children, employees, and vulnerable persons requiring special protection such as mentally ill persons, asylum seekers, the elderly, patients, etc.
|
||||
- Indeed any personal data where an imbalance in the relationship between the data subject and the controller can be identified and processing increases the power imbalance between them.
|
||||
- For example, children, employees, and vulnerable persons requiring special protection such as mentally ill persons, asylum seekers, the elderly, patients, etc.
|
||||
- Indeed any personal data where an imbalance in the relationship between the data subject and the controller can be identified and processing increases the power imbalance between them.
|
||||
10. **Data processing that prevents data subjects from exercising a right, using a service or entering into a contract**
|
||||
- This includes processing operations that permit, modify or refuse data subjects’ access to a service or entry into a contract.
|
||||
- An example of this is where a bank screens its customers against a credit reference database in order to decide whether to offer them a loan.
|
||||
|
||||
**If a processing operation meets 2 of these criteria, then a DPIA is required by law.**
|
||||
**If a processing operation meets 2 of these criteria, then a DPIA is required by law.**
|
||||
|
||||
#### Whats involved in carrying out a DPIA?
|
||||
#### What’s involved in carrying out a DPIA?
|
||||
|
||||
###### Step 1
|
||||
|
||||
@@ -205,14 +205,14 @@ Specify the nature of the processing including the source of the data
|
||||
- how it will be collected, used, stored and deleted
|
||||
- the amount of data to be collected
|
||||
- the frequency and duration of collection and storage, and the geographical area covered
|
||||
- the flow of data and if it will be shared, how and with who
|
||||
- any types of processing that are identified as high risk.
|
||||
- the flow of data and if it will be shared, how and with whom
|
||||
- any types of processing that are identified as high risk.
|
||||
|
||||
Also involves specifying the purpose or purposes of the processing and what the controller wants to achieve by processing the data, including the intended effect on data subjects (if any), the benefits of the processing to the controller and more broadly.
|
||||
|
||||
###### Step 3
|
||||
|
||||
Is consider the need for consultation
|
||||
Consider the need for consultation
|
||||
|
||||
1. when and how the views of data subjects will be sought
|
||||
2. justifying why it is not appropriate to do so
|
||||
@@ -221,11 +221,11 @@ Third & external parties need to be consulted to ensure data protection by desig
|
||||
|
||||
###### Step 4
|
||||
|
||||
Accessing necessity and proportionality, which involves specifying how the processing will actually achieve the purpose and that there is no other way to achieve the same outcome.
|
||||
Assessing necessity and proportionality, which involves specifying how the processing will actually achieve the purpose and that there is no other way to achieve the same outcome.
|
||||
|
||||
- the lawful basis for processing
|
||||
- the lawful basis for processing
|
||||
- how data minimisation and data quality will be ensured
|
||||
- how function creep will be prevented; what information will be given to data subjects and their rights will be supported
|
||||
- how function creep will be prevented; what information will be given to data subjects and how their rights will be supported
|
||||
- measures that will be taken to ensure processors are in compliance with DPbDD
|
||||
- how any international data transfers will be safeguarded.
|
||||
|
||||
@@ -234,29 +234,29 @@ Accessing necessity and proportionality, which involves specifying how the proce
|
||||
Identify and assess risks and involves identifying sources of risk and specifying
|
||||
|
||||
1. risks to data subjects
|
||||
2. corporate risks
|
||||
3. compliance risks
|
||||
2. corporate risks
|
||||
3. compliance risks
|
||||
|
||||
and the potential impact of each.
|
||||
|
||||
###### Step 6
|
||||
|
||||
Identify and specify measures to mitigate the risks, including the options available to
|
||||
Identify and specify measures to mitigate the risks, including the options available to
|
||||
|
||||
1. reduce risk
|
||||
2. eliminate risk
|
||||
|
||||
###### Step 7
|
||||
|
||||
Have the DPAI signed off and outcomes recorded. If the DPO’s advice is overruled, justification must be provided, as must the reasons for not abiding by consultation outcomes. A **review date must also be specified** for the DPIA and done so over the lifetime of a processing operation.
|
||||
Have the DPIA signed off and outcomes recorded. If the DPO’s advice is overruled, justification must be provided, as must the reasons for not abiding by consultation outcomes. A **review date must also be specified** for the DPIA and done so over the lifetime of a processing operation.
|
||||
|
||||
You cannot do a DPIA on your own. IBM’s Dave Whitelegg says you must have the following invovled
|
||||
You cannot do a DPIA on your own. IBM’s Dave Whitelegg says you must have the following involved
|
||||
|
||||
> - The developer lead or project manager, who is responsible for managing the DPIA process.
|
||||
> - A data protection officer who must be consulted about and sign off on the DPIA process*.*
|
||||
> - A data protection officer who must be consulted about and sign off on the DPIA process*.*
|
||||
> - A security specialist who must verify that best practises are adopted throughout development.
|
||||
> - A risk manager to advise on privacy risk management.
|
||||
> - Project sponsors and business directors, who are accountable for privacy risks.
|
||||
> - A risk manager to advise on privacy risk management.
|
||||
> - Project sponsors and business directors, who are accountable for privacy risks.
|
||||
> - And where processing operations are developed for external organisations, who must be able to verify that the processing is compliant with GDPR.
|
||||
|
||||
### Relevance of DPbDD and DPIA to computing
|
||||
@@ -273,13 +273,13 @@ However, we should not forget that documentation is a key part of the software e
|
||||
|
||||
###### OWASP’s Security Principles
|
||||
|
||||
1. Data anonymisation methods include: nulling, deletion and redaction, which involves removing all direct and indirect identifier fields in a dataset,
|
||||
- removing names or postcodes.
|
||||
1. Data anonymisation methods include: nulling, deletion and redaction, which involves removing all direct and indirect identifier fields in a dataset,
|
||||
- removing names or postcodes.
|
||||
2. Substitution, which involves overwriting personal data identifier fields with fake personal data.
|
||||
3. Data masking, which involves substituting identifier field characters with a ‘mask’ character,
|
||||
- e.g., inserting X’s instead numbers on a credit card field.
|
||||
3. Data masking, which involves substituting identifier field characters with a ‘mask’ character,
|
||||
- e.g., inserting X’s instead of numbers on a credit card field.
|
||||
4. Scrambling / shuffling, which involves moving the contents of identifier fields around
|
||||
- e.g. moving surnames up or down.
|
||||
- e.g. moving surnames up or down.
|
||||
5. Aggregation / generalisation, which involves rendering data in statistical form.
|
||||
6. Hashing provides a method of pseudonymisation and involves using an algorithm to transform personal data fields into alphanumeric strings.
|
||||
7. Penetration testing is recommended to verify whether these methods enable reidentification in any actual case.
|
||||
@@ -287,4 +287,3 @@ However, we should not forget that documentation is a key part of the software e
|
||||
https://owasp.org/www-project-top-ten/
|
||||
|
||||
Privacy engineering may help you implement the presets and meet the requirements, but it is your **ethical responsibility** to know and respect the rules that pertain to professional work. You now know what rules you need to follow to respect people’s privacy and protect their data.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Automonous Systems
|
||||
# Autonomous Systems
|
||||
|
||||
Autonomous systems include robots and cyber physical systems that actuate or perform actions in the world, and algorithmic systems particularly machine learning systems or AI.
|
||||
Autonomous systems include robots and cyber-physical systems that actuate or perform actions in the world, and algorithmic systems particularly machine learning systems or AI.
|
||||
|
||||
The UK robotics and autonomous systems or RAS network identifies 7 key ethical challenges that confront autonomous systems. These include
|
||||
|
||||
@@ -14,17 +14,17 @@ The UK robotics and autonomous systems or RAS network identifies 7 key ethical c
|
||||
|
||||
> “The race between job creation through new products and job destruction from new technologies has in the past been won by the job-creating effects of innovation. There is no guarantee for a happy end this time; however, an important lesson from the past is that we tend to under-estimate the job-creating potential of fundamental technological transformations, because we lack sufficient knowledge and imagination about the types of jobs that will be created under the new technological paradigm.
|
||||
>
|
||||
> https://www.europarl.europa.eu/RegData/etudes/STUD/2018/614539/EPRS_STU(2018)614539_EN.pdf
|
||||
> https://www.europarl.europa.eu/RegData/etudes/STUD/2018/614539/EPRS_STU(2018)614539_EN.pdf
|
||||
|
||||
Alan Winfield and Marina Jirotka in their Royal Society paper on building societal trust in autonomous systems: http://dx.doi.org/10.1098/rsta.2018.0085. They thus propose 5 pillars of good governance, which include establishing a machine intelligence commission to address public fears, including the impact of autonomous systems on jobs. The UK Government established an AI Council in 2019. Regulation is seen as the second pillar of good governance, as are standards, such as those established by professional bodies including the BCS, ACM and IEEE.
|
||||
|
||||
###### The Third Pillar
|
||||
|
||||
Recommends we take particular care about the use of AI in safety critical systems. Of particular concern, as we will take a closer look at later in this lecture, are artificial neural networks, whose decision-making cannot easily be verified. Neural networks learn for themselves and how they arrive at particular decisions is extremely difficult if not impossible to determine.
|
||||
Recommends we take particular care about the use of AI in safety-critical systems. Of particular concern, as we will take a closer look at later in this lecture, are artificial neural networks, whose decision-making cannot easily be verified. Neural networks learn for themselves and how they arrive at particular decisions is extremely difficult if not impossible to determine.
|
||||
|
||||
###### Fourth Pillar
|
||||
|
||||
Good governance, transparency not only of product, i.e., how an autonomous system arrived at a decision, but also of process and how such machines are developed. The concern with process involves
|
||||
Good governance, transparency not only of product, i.e., how an autonomous system arrived at a decision, but also of process and how such machines are developed. The concern with process involves
|
||||
|
||||
- developing ethical codes
|
||||
- ensuring ethical training for everyone involved in development
|
||||
@@ -35,7 +35,7 @@ Good governance, transparency not only of product, i.e., how an autonomous syste
|
||||
|
||||
Build ethical governors into autonomous systems which would enable a robot or AI system to evaluate the consequences of its actions and modify its actions according to a set of ethical rules.
|
||||
|
||||
This is a longstanding ideal in AI, which must address the fundamental problem of encoding and implementing ethics, all of which begs the question of who’s ethics get encoded and implemented? Pillar five is then the most idealistic, problematic and challenging of Winfield and Jirotka’s proposals.
|
||||
This is a longstanding ideal in AI, which must address the fundamental problem of encoding and implementing ethics, all of which begs the question of whose ethics get encoded and implemented? Pillar five is then the most idealistic, problematic and challenging of Winfield and Jirotka’s proposals.
|
||||
|
||||
### Deception
|
||||
|
||||
@@ -46,17 +46,17 @@ For example, Babyclon’s animatronic babies and the strong emotions they evoke
|
||||
The issue of deception is part of a broader set of ethical principles governing the development of robots advocated by the UK’s Engineering and Physical Sciences Research Council or EPSRC
|
||||
|
||||
- **Principle 1** states that robots should not be designed solely or primarily to kill or harm humans, except in the interests of national security.
|
||||
- **Principe 2** states that humans, not robots, are responsible agents and that robots should therefore be designed and operated in compliance with existing laws and respect the fundamental rights and freedoms of human beings, including privacy.
|
||||
- **Principle 3** states that robots should be designed to be safe and secure.
|
||||
- **Principle 2** states that humans, not robots, are responsible agents and that robots should therefore be designed and operated in compliance with existing laws and respect the fundamental rights and freedoms of human beings, including privacy.
|
||||
- **Principle 3** states that robots should be designed to be safe and secure.
|
||||
- **Principle 4** states that robots are manufactured artefacts and their machine nature should therefore be transparent so as to avoid deception.
|
||||
- **Principe 5** states that the party with legal responsibility for a robot should always be attributed, which is to say that it should always be possible to find out who is responsible for any robot.
|
||||
- This of course is not a straightforward matter as the disruption of flights at airports by drones demonstrates.
|
||||
- **Principle 5** states that the party with legal responsibility for a robot should always be attributed, which is to say that it should always be possible to find out who is responsible for any robot.
|
||||
- This of course is not a straightforward matter as the disruption of flights at airports by drones demonstrates.
|
||||
|
||||
### Algorithmic Bias
|
||||
|
||||
Bias is a concern with the validity of outputs or decisions made by autonomous systems, particularly with whether or not those outputs or decisions **discriminate** against individuals and/or social groups and thus treat them unfairly.
|
||||
|
||||
Discrimination is rife in computing today:
|
||||
Discrimination is rife in computing today:
|
||||
|
||||
- webcams that fail to track black people’s faces
|
||||
- auto-tagging of black people and women as animals or gorillas
|
||||
@@ -64,39 +64,39 @@ Discrimination is rife in computing today:
|
||||
- systematic discrimination against female job candidates and black patients in need of healthcare
|
||||
- the A-Level debacle in the UK
|
||||
|
||||
Discrimination, is a specific form of harm based on a personal characteristics including gender identity, marital status, sexual orientation, colour, race, ethnic origin, nationality, religion, age, union membership, political affiliation, military status, and disability.
|
||||
Discrimination is a specific form of harm based on personal characteristics including gender identity, marital status, sexual orientation, colour, race, ethnic origin, nationality, religion, age, union membership, political affiliation, military status, and disability.
|
||||
|
||||
These characteristics are otherwise called **“special categories of personal data”** or **“protected characteristics”** and are regulated by GDPR and equality legislation, which would appear to provide a relatively straightforward way of tackling algorithmic bias.
|
||||
|
||||
#### Sources of Algorithmic Bias
|
||||
|
||||
Selena Silva and Martin Kenney identify 9 sources of algorithmic bias within the ML life cycle. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3246252
|
||||
Selena Silva and Martin Kenney identify 9 sources of algorithmic bias within the ML life cycle. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3246252
|
||||
|
||||
1. **Training bias**
|
||||
- The data used to train the algorithm may be unrepresentive or prejudiced
|
||||
- A facial recognition algorithm is trained on data which primarily consists of white faces, it will be worse at recognising black faces and may even categorise them wrongly.
|
||||
- The data used to train the algorithm may be unrepresentative or prejudiced
|
||||
- If a facial recognition algorithm is trained on data which primarily consists of white faces, it will be worse at recognising black faces and may even categorise them wrongly.
|
||||
2. **Algorithmic focus bias**
|
||||
- The attributes it takes into account and either includes or excludes
|
||||
- The exclusion of gender or race in a health diagnostic algorithm can lead to inaccurate and harmful outcomes.
|
||||
- Whereas the inclusion of gender or race in a sentencing algorithm can lead to discrimination against protected groups.
|
||||
- The attributes it takes into account and either includes or excludes
|
||||
- The exclusion of gender or race in a health diagnostic algorithm can lead to inaccurate and harmful outcomes.
|
||||
- Whereas the inclusion of gender or race in a sentencing algorithm can lead to discrimination against protected groups.
|
||||
3. **Algorithmic processing bias**
|
||||
- Thomas Guskey and Lee Ann Jung found, for example, that when an ML algorithm processed student grades across a learning module, it scored students based on the average marks for their assignments, but when teachers were given the same data, they adjusted the students’ score according to their progress and understanding of the material and provided a fairer assessment of students learning. https://core.ac.uk/download/pdf/232576892.pdf
|
||||
- Thomas Guskey and Lee Ann Jung found, for example, that when an ML algorithm processed student grades across a learning module, it scored students based on the average marks for their assignments, but when teachers were given the same data, they adjusted the students’ scores according to their progress and understanding of the material and provided a fairer assessment of students’ learning. https://core.ac.uk/download/pdf/232576892.pdf
|
||||
4. **Non-transparency bias**
|
||||
- The lack of transparency about algorithmic decision-making.
|
||||
- This is not only to do with how decisions were arrived, but also concerns IPR and trade secrets and what developers are willing and expected to divulge about their ML systems and AI
|
||||
- The lack of transparency about algorithmic decision-making.
|
||||
- This is not only to do with how decisions were arrived at, but also concerns IPR and trade secrets and what developers are willing and expected to divulge about their ML systems and AI
|
||||
5. **Transfer context bias**
|
||||
- The use of ML systems in inappropriate or unintended contexts is also a source of bias. The use of credit scores as a variable in employment provides a ready example of what is called “**transfer context bias**”
|
||||
- Employer’s request credit checks on job candidates, which effectively means that bad credit is being equated with bad job performance.
|
||||
- The use of ML systems in inappropriate or unintended contexts is also a source of bias. The use of credit scores as a variable in employment provides a ready example of what is called “**transfer context bias**”
|
||||
- Employers request credit checks on job candidates, which effectively means that bad credit is being equated with bad job performance.
|
||||
6. **Automation bias**
|
||||
- A human bias which involves the users of algorithmic systems treating outputs as objectively true, rather than as statistical probabilities.
|
||||
- Such as the COMPAS system used by judges in sentencing criminals in the US, provides a good example, where a judge might take the output at face value and apply it uncritically, without reference to other information
|
||||
- Automation bias is very much a case of “computer says so …”
|
||||
- A human bias which involves the users of algorithmic systems treating outputs as objectively true, rather than as statistical probabilities.
|
||||
- The COMPAS system used by judges in sentencing criminals in the US provides a good example, where a judge might take the output at face value and apply it uncritically, without reference to other information
|
||||
- Automation bias is very much a case of “computer says so …”
|
||||
7. **Consumer bias**
|
||||
- Is bias expressed by the users of digital platforms
|
||||
- Great care needs to be taken with ML systems trained on such data, as they will reflect consumer bias and be inherently prejudiced in one way or another.
|
||||
- Is bias expressed by the users of digital platforms
|
||||
- Great care needs to be taken with ML systems trained on such data, as they will reflect consumer bias and be inherently prejudiced in one way or another.
|
||||
8. **Feedback loop bias**
|
||||
- Where ML systems learn from user behaviour, including discriminatory behaviour.
|
||||
- So even though an ML system may have been developed without bias in its training, focus and initial processing of data, over time bias may be introduced through use.
|
||||
- Twitter taught Microsoft’s AI chatbot Tay to be a racist in less than a day.
|
||||
- Where ML systems learn from user behaviour, including discriminatory behaviour.
|
||||
- So even though an ML system may have been developed without bias in its training, focus and initial processing of data, over time bias may be introduced through use.
|
||||
- Twitter taught Microsoft’s AI chatbot Tay to be a racist in less than a day.
|
||||
9. **Interpretation bias**
|
||||
- Occurs when users interpret outputs according to their own prejudices. For example, it is ultimately up to a judge to interpret the score provided by a recidivism prediction system such as COMPAS, and to decide what action to take. However, a judge may interpret a risk score of 6 as high in a particular case, while they may treat it as indicator of medium or even low risk in another.
|
||||
- Occurs when users interpret outputs according to their own prejudices. For example, it is ultimately up to a judge to interpret the score provided by a recidivism prediction system such as COMPAS, and to decide what action to take. However, a judge may interpret a risk score of 6 as high in a particular case, while they may treat it as an indicator of medium or even low risk in another.
|
||||
@@ -9,12 +9,12 @@ An 80 billion euro programme to tackle:
|
||||
- Health, demographic change and the well-being of citizens
|
||||
- Food security and sustainable agriculture
|
||||
- Sustainable and efficient energy
|
||||
- Smart, green transport
|
||||
- Smart, green transport
|
||||
- Climate action, resource efficiency and raw materials
|
||||
- Inclusive and innovative society
|
||||
- Secure society protecting the rights and freedoms of citizens
|
||||
|
||||
What RRI seeks to achieve with respect to these grand challenges is **situate** science and technology development in its **social context**. Fundamentally, RRI aims to drive high quality innovations in science and technology that are in the public interest and create a society in which research and innovation practices work towards **ethically acceptable, socially desirable and sustainable outcomes**.
|
||||
What RRI seeks to achieve with respect to these grand challenges is to **situate** science and technology development in its **social context**. Fundamentally, RRI aims to drive high-quality innovations in science and technology that are in the public interest and create a society in which research and innovation practices work towards **ethically acceptable, socially desirable and sustainable outcomes**.
|
||||
|
||||
#### Responsible Innovation
|
||||
|
||||
@@ -38,13 +38,13 @@ Reflexivity is particularly important at an institutional or organisational leve
|
||||
|
||||
This is called “second-order reflexivity” and contrasts with “first-order reflexivity”, where individuals reflect on and scrutinise themselves privately. Second-order reflexivity seeks to make reflexivity a public matter and leads to kinds of consideration of ethical governance proposed by Alan Winfield and Marina Jirotka we discussed in lecture 7.
|
||||
|
||||
Reflexivity is key to the development of ethically acceptable and socially desirable innovations. It requires researchers and innovators see beyond organisational boundaries and responsibilities and consider their wider, moral responsibilities.
|
||||
Reflexivity is key to the development of ethically acceptable and socially desirable innovations. It requires researchers and innovators to see beyond organisational boundaries and responsibilities and consider their wider, moral responsibilities.
|
||||
|
||||
**Inclusion**
|
||||
|
||||
**Inclusion** recognises the need to open up anticipatory visions of future social worlds to public dialogue in ways that critically interrogate the social, political and ethical viewpoints implicated in technology development.
|
||||
|
||||
Inclusion requires that we are sensitive to
|
||||
Inclusion requires that we are sensitive to
|
||||
|
||||
- a) the ‘intensity’ of public engagement – i.e., how early members of the public and other stakeholders are consulted in the innovation process
|
||||
- b) ‘openness’ – i.e., how diverse the sample is and who is represented
|
||||
@@ -60,58 +60,58 @@ Stilgoe et al. also place emphasis on the role of governance approaches in R&I,
|
||||
|
||||
https://www.epsrc.ac.uk/research/framework
|
||||
|
||||
The framework is called **AREA** and reflects the 4 dimensions of Stilgoe et als responsible innovation framework, reframed as Anticipate, Engage, Reflect and Act.
|
||||
The framework is called **AREA** and reflects the 4 dimensions of Stilgoe et al.’s responsible innovation framework, reframed as Anticipate, Engage, Reflect and Act.
|
||||
|
||||
**Anticipate** asks researchers to describe and analyse any economic, social and / or environmental impacts, intended or otherwise, that might arise from the proposed research. The aim is not to predict the actual impact of the proposed research, but to explore potential impacts and implications of the research that may otherwise remain ignored during the research the process.
|
||||
**Anticipate** asks researchers to describe and analyse any economic, social and/or environmental impacts, intended or otherwise, that might arise from the proposed research. The aim is not to predict the actual impact of the proposed research, but to explore potential impacts and implications of the research that may otherwise remain ignored during the research process.
|
||||
|
||||
**Reflect** asks researchers to reflect on the purposes, motivations, and potential implications of their research, and the associated uncertainties, areas of ignorance, assumptions, framings, questions, dilemmas and social transformations these may occasion.
|
||||
|
||||
**Engage** asks researchers to open up their research visions and their potential impacts to broader deliberation, dialogue, engagement and debate with stakeholders and the public in an inclusive way.
|
||||
|
||||
**Act** asks researchers to using the processes of Anticipation, Reflection and Engagement to influence the direction and trajectory of the research and innovation process itself.
|
||||
**Act** asks researchers to use the processes of Anticipation, Reflection and Engagement to influence the direction and trajectory of the research and innovation process itself.
|
||||
|
||||
So RRI is an important part of the EU and UK research and innovation pipeline and will become much more so now that the UK research councils have been brought together under the umbrella of UK Research and Innovation or UKRI.
|
||||
|
||||
## How does RRI work?
|
||||
|
||||
The focus of RRI is not only on achieving ethically acceptable, socially desirable and sustainable outcomes. It also and fundamentally concerned with *how* research and innovation is conducted and the parties involved in the process. RRI can thus be broken down into four key elements: **policy**, **stakeholders**, **outcomes**, **process**.
|
||||
The focus of RRI is not only on achieving ethically acceptable, socially desirable and sustainable outcomes. It is also and fundamentally concerned with *how* research and innovation is conducted and the parties involved in the process. RRI can thus be broken down into four key elements: **policy**, **stakeholders**, **outcomes**, **process**.
|
||||
|
||||
###### Policy
|
||||
|
||||
The EU sets out six key policies to shape responsible research and innovation processes, which are target at governments, funding agencies and R&I organisations.
|
||||
The EU sets out six key policies to shape responsible research and innovation processes, which are targeted at governments, funding agencies and R&I organisations.
|
||||
|
||||
1. Robust goverence
|
||||
- RRI principles should, as a matter of policy, be **embedded in robust** **governance** frameworks. These frameworks should be flexible and adapt to change so as to be capable of responding to the unpredictable nature of research and innovation.
|
||||
1. Robust governance
|
||||
- RRI principles should, as a matter of policy, be **embedded in robust** **governance** frameworks. These frameworks should be flexible and adapt to change so as to be capable of responding to the unpredictable nature of research and innovation.
|
||||
2. Gender equality
|
||||
- It is also a matter of policy that research and innovation take the perspectives of both men and women into account to ensure outcomes are relevant to the whole population.
|
||||
- Decision-making bodies and R&I organisations should have balanced gender representation and strive to ensure **gender equality** in research and innovation.
|
||||
- It is also a matter of policy that research and innovation take the perspectives of both men and women into account to ensure outcomes are relevant to the whole population.
|
||||
- Decision-making bodies and R&I organisations should have balanced gender representation and strive to ensure **gender equality** in research and innovation.
|
||||
3. Integrity
|
||||
- Honesty, accountability, fairness and good stewardship should be core principles of research and innovation and are key to ensuring the **integrity** of R&I.
|
||||
4. Public and stakeholder engagment
|
||||
- The **public and other stakeholders** should, as a matter of policy, be **engaged in research** and innovation processes as early as possible to avoid tokenism, ensure outcomes align with the values, needs and expectations of society and to avert societal backlash
|
||||
- as, for example, happened with the attempted introduction of GM crops into the UK
|
||||
- Honesty, accountability, fairness and good stewardship should be core principles of research and innovation and are key to ensuring the **integrity** of R&I.
|
||||
4. Public and stakeholder engagement
|
||||
- The **public and other stakeholders** should, as a matter of policy, be **engaged in research** and innovation processes as early as possible to avoid tokenism, ensure outcomes align with the values, needs and expectations of society and to avert societal backlash
|
||||
- as, for example, happened with the attempted introduction of GM crops into the UK
|
||||
5. Open Access (FAIR)
|
||||
- publicly funded research should be **open access** in order to catalyse broader innovation, encourage collaboration and improve the quality of research
|
||||
- Scientific results and data should follow the FAIR principle
|
||||
- results and data should be **F**indable, **A**ccessible, **I**nteroperable, and **R**eusable
|
||||
- publicly funded research should be **open access** in order to catalyse broader innovation, encourage collaboration and improve the quality of research
|
||||
- Scientific results and data should follow the FAIR principle
|
||||
- results and data should be **F**indable, **A**ccessible, **I**nteroperable, and **R**eusable
|
||||
6. Science and technology education
|
||||
- The demand for highly qualified people continues to rise globally and there is also need as a matter of policy for improved **science and technology education** to build the necessary capacity to enable R&I at scale and to provide citizens with the knowledge they need to engage with research and innovation.
|
||||
- The demand for highly qualified people continues to rise globally and there is also need as a matter of policy for improved **science and technology education** to build the necessary capacity to enable R&I at scale and to provide citizens with the knowledge they need to engage with research and innovation.
|
||||
|
||||
###### Stakeholders
|
||||
|
||||
RRI involves a range of stakeholders, who should in one way or another be involved in permanent and ongoing dialogue with one another. These stakeholders include:
|
||||
|
||||
**Policymakers**, who have the ability to bring stakeholders to the table and foster debate. This not only includes government but funding agencies, the directors R&I organisations and anyone else involved in making decisions that shape research and innovation locally, nationally and internationally.
|
||||
**Policymakers**, who have the ability to bring stakeholders to the table and foster debate. This not only includes government but funding agencies, the directors of R&I organisations and anyone else involved in making decisions that shape research and innovation locally, nationally and internationally.
|
||||
|
||||
The **research community** is obviously a key stakeholder in research and innovation and includes everyone in the research and innovation pipeline from science advocates and communicators, to research managers, researchers, technicians and support staff.
|
||||
|
||||
**Business and industry**, from start ups to SMEs to large corporates and transnational companies, are all key to research and bringing innovations to bear on social life.
|
||||
**Business and industry**, from start-ups to SMEs to large corporates and transnational companies, are all key to research and bringing innovations to bear on social life.
|
||||
|
||||
**The education community**, from primary school to university, science centres and museums, and including teachers, students and their families, play a key role in building capacity and promoting public understanding of science and technology.
|
||||
|
||||
**Civil society organisations**, such as trade unions, NGOs and the media, also play important roles in shaping research and innovation.
|
||||
|
||||
RRI seeks to involve these stakeholders in shaping ethically acceptable, socially desirable and sustainable outcomes. Indeed, in recognising that research and innovation reaches beyond the lab, RRI seeks to foster **shared** **responsibility** for research and innovation and ensure that it that serves the public good.
|
||||
RRI seeks to involve these stakeholders in shaping ethically acceptable, socially desirable and sustainable outcomes. Indeed, in recognising that research and innovation reaches beyond the lab, RRI seeks to foster **shared** **responsibility** for research and innovation and ensure that it serves the public good.
|
||||
|
||||
###### Process
|
||||
|
||||
@@ -141,15 +141,15 @@ Abma Tineke and Jacqueline Broerse’s ‘dialogue model’ of participatory res
|
||||
|
||||
**Exploration** is the first phase of the dialogue model and aims to identify and make contact with the different stakeholder organisations, groups, and individuals that should be involved in the research.
|
||||
|
||||
**Consultation** does at it suggests and engages stakeholders separately in a dialogue about the research to ensure their voices are heard. Tineke and Broerse emphasize the importance of paying attention to diversity (age, gender, ethnicity, etc.) and being sensitive to asymmetries in power in doing this.
|
||||
**Consultation** does as it suggests and engages stakeholders separately in a dialogue about the research to ensure their voices are heard. Tineke and Broerse emphasise the importance of paying attention to diversity (age, gender, ethnicity, etc.) and being sensitive to asymmetries in power in doing this.
|
||||
|
||||
- They underscore the need to empower stakeholders who are not used to actively participating in research to enable “more equal interaction with professionals” and that researchers should pay particular attention to the issues that matter to specific stakeholders.
|
||||
- Consultation also involves determining appropriate methods of conducting research dialogues with stakeholders, e.g., interviews, focus groups, questionnaires, observations, etc.
|
||||
|
||||
**Prioritisation** as the name suggests is about identifying which research themes that emerge from the consultation process should be take priority.
|
||||
**Prioritisation** as the name suggests is about identifying which research themes that emerge from the consultation process should take priority.
|
||||
|
||||
- This often an iterative process involving further consultation with stakeholders to ensure the right themes are being prioritised appropriately.
|
||||
- Importantly it involves consideration of what can reasonably be expected to be achieved within the lifetime of project, which means that while a theme may have high priority for stakeholders, it may not be technically achievable in the available timeframes, which may lead to it being de-prioritised.
|
||||
- This is often an iterative process involving further consultation with stakeholders to ensure the right themes are being prioritised appropriately.
|
||||
- Importantly it involves consideration of what can reasonably be expected to be achieved within the lifetime of the project, which means that while a theme may have high priority for stakeholders, it may not be technically achievable in the available timeframes, which may lead to it being de-prioritised.
|
||||
- Prioritisation is a matter of compromise between what stakeholders want and what can be technically delivered.
|
||||
|
||||
**Integration** seeks to combine the prioritised research themes into a coherent research agenda.
|
||||
@@ -159,7 +159,7 @@ Abma Tineke and Jacqueline Broerse’s ‘dialogue model’ of participatory res
|
||||
|
||||
The **programming** phase involves specifying a research plan to enable the research agenda to be implemented.
|
||||
|
||||
- It involves setting a programming committee involving stakeholder representatives to ensure the research addresses the concerns of all stakeholders as it proceeds into implementation.
|
||||
- It involves setting up a programming committee involving stakeholder representatives to ensure the research addresses the concerns of all stakeholders as it proceeds into implementation.
|
||||
|
||||
And **implementation** obviously involves putting the plan into practice.
|
||||
|
||||
@@ -169,19 +169,19 @@ And **implementation** obviously involves putting the plan into practice.
|
||||
|
||||
The collective resources approach led to action-based and experience-based design methods that leveraged prototypes as vehicles for participatory research.
|
||||
|
||||
Prototyping was established as an alternative approach to requirements specification in the 1970s, replacing a written document subject to the vagaries of interpretation with a functioning version of a computing system.
|
||||
Prototyping was established as an alternative approach to requirements specification in the 1970s, replacing a written document subject to the vagaries of interpretation with a functioning version of a computing system.
|
||||
|
||||
The **problem** with prototyping is that it is by its very nature a technical exercise, all too often preoccupied with demonstrating technical features to stakeholders and having them sign-off on them.
|
||||
The **problem** with prototyping is that it is by its very nature a technical exercise, all too often preoccupied with demonstrating technical features to stakeholders and having them sign off on them.
|
||||
|
||||
The challenge that Cooperative Design set out tackle was how to *involve* ordinary people – users and other non-technical stakeholders – in the actual development of prototypes.
|
||||
The challenge that Cooperative Design set out to tackle was how to *involve* ordinary people – users and other non-technical stakeholders – in the actual development of prototypes.
|
||||
|
||||
Prototyping is a common feature of many design models today, from the spiral model to agile. The contribution of Cooperative Design is to use it as a vehicle for put stakeholder viewpoints and experience at the centre of the design process, not technical specifications and feature demonstrations, and it provides us with a tried and tested way of doing participatory research in computing.
|
||||
Prototyping is a common feature of many design models today, from the spiral model to agile. The contribution of Cooperative Design is to use it as a vehicle for putting stakeholder viewpoints and experience at the centre of the design process, not technical specifications and feature demonstrations, and it provides us with a tried and tested way of doing participatory research in computing.
|
||||
|
||||
### RRI self-reflection tool
|
||||
|
||||
Perhaps the most useful tool in the RRI Toolkit is the self-reflection tool: https://rri-tools.eu/self-reflection-tool
|
||||
|
||||
- It helps you determine whether or not your research is responsible.
|
||||
- It helps you determine whether or not your research is responsible.
|
||||
|
||||
The public engagement section asks you 10 questions about stakeholder involvement.
|
||||
|
||||
|
||||
@@ -12,11 +12,11 @@ According to code 1.4 from the ACM code of ethics, computing professionals shoul
|
||||
|
||||
#### b)
|
||||
|
||||
Algorithmic bias is a series of systematic and repeatable errors, that over the course of the systems runtime, produces output that dis-proportionally discriminates against individuals and/or social groups. Selena Silva and Martin Kenny found 9 sources of algorithmic bias in their research paper, all of which capable of discriminating and producing bias
|
||||
Algorithmic bias is a series of systematic and repeatable errors that, over the course of the system’s runtime, produces output that disproportionately discriminates against individuals and/or social groups. Selena Silva and Martin Kenny found 9 sources of algorithmic bias in their research paper, all of which are capable of discriminating and producing bias
|
||||
|
||||
Bias can be introduced in the development of a machine learning system. Training bias is where data used to train the algorithm may be unrepresentive or prejudiced, this can cause the system to unfairly associate one trait to another even though they have no effect on one another. This can be through the developers own bias by only including data sets representative to their own socitak group or through systemic bias where minority groups are under represented in national and global data sets. Developers can also introduce bias by including or excluding certain attributes. This is called algorithmic focus bias and developers must take variables supplied to the algorithm into careful consideration, evaluating why each variable needs to be included in the system. Similarly bias can arise from the way data is processed, for example this can be from weighting quantitative attributes higher than qualitative ones simply as quantitative data is easier to manipulate, this is called algorithmic processing bias. Non-transparency bias is where companies do not divulge or explain how they came to certain decisions, what their rationale was for different design choices. In the best case this can introduce bias in an unforeseen way as all the developers may come from similar social groups and in the worse case scenario developers can obstruct reviews of the algorithm, allowing discrimination to take place.
|
||||
Bias can be introduced in the development of a machine learning system. Training bias is where data used to train the algorithm may be unrepresentative or prejudiced; this can cause the system to unfairly associate one trait with another even though they have no effect on one another. This can be through the developers’ own bias by only including data sets representative of their own social group or through systemic bias where minority groups are under-represented in national and global data sets. Developers can also introduce bias by including or excluding certain attributes. This is called algorithmic focus bias and developers must take variables supplied to the algorithm into careful consideration, evaluating why each variable needs to be included in the system. Similarly, bias can arise from the way data is processed, for example from weighting quantitative attributes higher than qualitative ones simply because quantitative data is easier to manipulate; this is called algorithmic processing bias. Non-transparency bias is where companies do not divulge or explain how they came to certain decisions or what their rationale was for different design choices. In the best case this can introduce bias in an unforeseen way as all the developers may come from similar social groups and in the worst-case scenario developers can obstruct reviews of the algorithm, allowing discrimination to take place.
|
||||
|
||||
Bias can also arise in the use of computing systems. Transfer context bias is where machine learning systems are used inappropriately. This can happen in job applications where credit checks are required or in justice systems where race needs to be explicitly stated. The assumption job performance correlates to wealth or criminal charges correlates to race is unfair and biased. Therefore the use of computer systems particularly in subjective use cases should be scrutinised to ensure the potential benefits outweigh the increased chance of discriminating or additional steps are taken after the system outputs to mitigate any potential harms. Similarly automation bias is where humans hold the output of a system in high regard and don’t question or apply additional thought. Computer systems used in subjective context such as justice systems should be treated as a second opinion or a statistical model and disregarded readily when an unsuitable result is returned. Consumer bias is where bias is introduced to the system via the training data. Humans are inherently flawed and biased and therefore extra care and additional review steps should be added to check the neutrality of the training data. Likewise feedback loop bias affects systems that learn from user behaviour, which again is prone to being discriminatory. This requires special attention has even when a system has been developed without bias, bias is introduced through the systems use lifetime. Lastly interpretation bias is where humans introduce bias from interpreting results from the algorithm. For example if the algorithm agrees with someones own bias, they might be more likely to give a more extreme verdict however if it opposes their own opinion, the result may be completely disregarded.
|
||||
Bias can also arise in the use of computing systems. Transfer context bias is where machine learning systems are used inappropriately. This can happen in job applications where credit checks are required or in justice systems where race needs to be explicitly stated. The assumption that job performance correlates with wealth or criminal charges correlate with race is unfair and biased. Therefore, the use of computer systems particularly in subjective use cases should be scrutinised to ensure the potential benefits outweigh the increased chance of discriminating or additional steps are taken after the system outputs to mitigate any potential harms. Similarly, automation bias is where humans hold the output of a system in high regard and don’t question it or apply additional thought. Computer systems used in subjective contexts such as justice systems should be treated as a second opinion or a statistical model and disregarded readily when an unsuitable result is returned. Consumer bias is where bias is introduced to the system via the training data. Humans are inherently flawed and biased and therefore extra care and additional review steps should be added to check the neutrality of the training data. Likewise, feedback loop bias affects systems that learn from user behaviour, which again is prone to being discriminatory. This requires special attention as even when a system has been developed without bias, bias is introduced through the system’s useful lifetime. Lastly, interpretation bias is where humans introduce bias from interpreting results from the algorithm. For example, if the algorithm agrees with someone’s own bias, they might be more likely to give a more extreme verdict; however, if it opposes their own opinion, the result may be completely disregarded.
|
||||
|
||||
## Question 3
|
||||
|
||||
@@ -28,17 +28,16 @@ The application scope is worldwide, the regulation states “This Regulation app
|
||||
|
||||
#### b)
|
||||
|
||||
To enable proper data protection by design and default, a number of presets must be implemented.
|
||||
To enable proper data protection by design and default, a number of presets must be implemented.
|
||||
|
||||
Firstly controllers must be transparent about how and why they are collecting and using data, how they use and share personal data and how data subjects can exercise their legal rights over data processing. This includes the right to: access, object, intervene, restrict, rectify, export and erase. This allows for data subjects to have full knowledge and control over their data and on top of this, recital 63 of GDPR states “where possible controller[s] should … provide remote access … with direct access to his or her personal data”. Controllers must also by default declare a valid legal basis for the processing. This ensures transparency as there is full disclosure of how data subjects legal rights are being maintained.
|
||||
Firstly, controllers must be transparent about how and why they are collecting and using data, how they use and share personal data and how data subjects can exercise their legal rights over data processing. This includes the right to: access, object, intervene, restrict, rectify, export and erase. This allows for data subjects to have full knowledge and control over their data and on top of this, recital 63 of GDPR states “where possible controller[s] should … provide remote access … with direct access to his or her personal data”. Controllers must also by default declare a valid legal basis for the processing. This ensures transparency as there is full disclosure of how data subjects’ legal rights are being maintained.
|
||||
|
||||
Controllers must ensure their data processing operations are fair. This principle requires personal data should not be processed in ways that are unjustifiably detrimental, unexpected or misleading to the data subject. Fairness is especially prevalent in dealing with AI systems since these do not operate on predefined instructions written by humans, therefore controllers should be able to demonstrate fairness through the inputs and outputs of the system.
|
||||
|
||||
Controllers must explicitly state what the data collected on data subjects will be used for. These must be specific tasks and cannot be processed in way that doesn’t align with the initial reason given. This is called purpose limitation and prevents controllers from collecting as much data as possible for monetary gain or nefarious purposes. This allows data subjects to only give their data to controllers who’s vision aligns with their own.
|
||||
Controllers must explicitly state what the data collected on data subjects will be used for. These must be specific tasks and the data cannot be processed in a way that doesn’t align with the initial reason given. This is called purpose limitation and prevents controllers from collecting as much data as possible for monetary gain or nefarious purposes. This allows data subjects to only give their data to controllers whose vision aligns with their own.
|
||||
|
||||
Controllers must practise data minimisation, this is a practice where the controller must review the data being asked and verifying all pieces of data are needed to meet the purposes for which they are being processed. This can also include the degree of identification, if the purpose is statistical this likely does not require any immediate identifying attributes. If continued identification is needed, data should be pseudonoymised to migrate damages caused from a data breach. Similarly data must be deleted once it has fulfilled it’s purpose. GDPR places no time limit on data storage of anonymised data however this can be reversed engineered and this data should be treated analogous to raw personal data.
|
||||
Controllers must practise data minimisation; this is a practice where the controller must review the data being requested and verify that all pieces of data are needed to meet the purposes for which they are being processed. This can also include the degree of identification; if the purpose is statistical this likely does not require any immediate identifying attributes. If continued identification is needed, data should be pseudonymised to mitigate damage caused by a data breach. Similarly, data must be deleted once it has fulfilled its purpose. GDPR places no time limit on data storage of anonymised data; however, this can be reverse-engineered and this data should be treated analogously to raw personal data.
|
||||
|
||||
Controllers must also ensure data is accurate, and if not it is the controllers duty to rectify or erase mistakes immediately. This is important as data subjects could be relying on this data for employment, housing or other civic needs and not being able to obtain this could cause harm to the data subject and family.
|
||||
Controllers must also ensure data is accurate, and if not it is the controller’s duty to rectify or erase mistakes immediately. This is important as data subjects could be relying on this data for employment, housing or other civic needs and not being able to obtain this could cause harm to the data subject and family.
|
||||
|
||||
Lastly controllers must put substantial measures in place to prevent unauthorised access, accidental loss and destruction or damage. Regular reviews should be conducted, testing security and inviting professional hackers to further test how the system stands up to new hacking methods.
|
||||
|
||||
Reference in new issue
Block a user