Add acn

Reviewed-on: #1
Co-authored-by: Jay <jayo60013@gmail.com>
Co-committed-by: Jay <jayo60013@gmail.com>

.gitea/workflows/deploy.yaml

@@ -1,44 +0,0 @@
-name: Build and Deploy MkDocs
-
-on:
-  push:
-    branches:
-      - main
-
-jobs:
-  deploy:
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 0 # Important for switching branches
-
-      - name: Build and Extract Site
-        run: |
-          docker build -t mkdocs-temp -f ci/mkdocs/Dockerfile .
-          docker create --name temp-container mkdocs-temp
-          # Copying content to a folder named 'output_content' to avoid naming collisions
-          docker cp temp-container:/build/site ./output_content
-          docker rm temp-container
-
-      - name: Deploy to docs-static Branch
-        run: |
-          git config user.name "gitea-actions[bot]"
-          git config user.email "actions@noreply.gitea.io"
-          
-          # Increase buffer to handle larger media files
-          git config http.postBuffer 524288000
-
-          mv output_content /tmp/site_final
-          git checkout --orphan docs-static
-          git rm -rf .
-          cp -r /tmp/site_final/. .
-
-          # Optional: Remove source maps to save space
-          find . -name "*.map" -type f -delete
-
-          git add .
-          git commit -m "Automated MkDocs build"
-          git push origin docs-static --force

ci/mkdocs/Dockerfile

@@ -1,9 +0,0 @@
-FROM ghcr.io/squidfunk/mkdocs-material:latest
-
-WORKDIR /build
-
-COPY . .
-
-RUN pip install --no-cache-dir mkdocs-minify-plugin
-
-RUN mkdocs build

ci/mkdocs/requirements.txt

@@ -1,3 +0,0 @@
-mkdocs==1.6.1
-mkdocs-material==9.7.3
-pymdown-extensions==10.21

docs/lectures/acn/01_intro.md

@@ -0,0 +1,79 @@
+### Module contents
+
+##### Part 1
+
+* Mobile Ad Hoc Networks (MANETs)
+* Delay/Disconnection Tolerant Networks (DTNs)
+* Vehicular Ad Hoc Networks (VANETs)
+
+##### Part 2
+
+* Network experimentation, criteria and evaluations. This part is to help with coursework
+
+##### Part 3
+
+* Peer to Peer (P2P)
+* Content Centric Networks (CCNs)
+* Information Centric Networks (ICNs)
+
+##### Part 4
+
+* Software Defined Networks (SDNs) and Applications
+
+# Mobile Social Networks
+
+They have two parts: physical part and a social part
+
+Social structures are vital for these networks - think covid tracking networks
+
+![p2p](/lectures/acn/img/a.png)
+
+Clouds have multiple layers
+
+* Network interfaces
+  * Request & accept sensor data
+  * resource management
+  * communicate with other clouds 
+* Processing layer
+  * Store raw data
+  * filter noise
+* Analysing layer
+  * produce trend chart
+  * learn & predict user behaviour
+* Services
+  * Interactive dashboard
+  * notification service
+  * sharing access
+
+
+
+## Vehicle Ad Hoc Networks
+
+Have social characteristics as they are driven by humans
+
+VANETs may refer to robots or drones.
+
+This can be used to exchange warning and beacon messages via V2V (vehicle to vehicle) as well as V2I (vehicle to infrastructure) channels.
+
+### Fully autonomous Vehicles
+
+![img](/lectures/acn/img/b.png)
+
+Vehicles can connect to the cloud and share & request information to help other vehicles.
+
+![img](/lectures/acn/img/c.png)
+
+An example of transient clouds - in this case vehicular clouds.
+
+This can be useful for informing cars behind about congestion. This is real time communication (order of ms which is needed for when cars are moving at 70 mph), cloud communication is not fast enough, due to the data needing to be processed before shared.
+
+## Challenges
+
+* Optimal forwarding/routing
+* Congestion avoidance and control
+* Security and privacy aware communications
+  * black & grey hole attacks
+* Energy efficient communications
+  * Important for mobile devices & electric cars
+* Service provision
+  * Location based services

docs/lectures/acn/02_MANET_and_DTN.md

@@ -0,0 +1,70 @@
+# Mobile Ad Hoc Networks (MANETs)
+
+* An infrastructure-less network formed by mobile wireless nodes
+* Nodes in MANET can communicate via single or multi-hop approach (due to absence of centralised network infrastructure)
+* Nodes operate as clients, routers and servers at the same time to forward packets
+* The mobility of nodes results in frequent and unpredictable changes in network topology
+
+One of the core features of a MANET node is the ability to autonomously connect to other nodes and configure itself for data transmission over the network.
+
+
+#### MANET Routing
+
+* Mobile wireless nodes create a temporary connection between them to forward data
+* Because some nodes may not be cooperative or faulty, they may drop/compromise packets
+* Typically routing is split into **route discovery** and **actual data transmission**.
+  * Nodes have to self organise in order to route.
+
+![img](/lectures/acn/img/d.png)
+
+(green boxes is route chosen)
+
+The source has a limited range of nodes it can detect, it cannot send it direct to the destination as it doesn't know where the destination is. Hops are decided by communication protocols.
+
+#### Proactive MANETs
+
+* Also known as table driven routing protocol
+* Nodes in the network maintain a comprehensive routing information of the network
+  * This is done by spreading network status information to nodes and tracking changes in network topology - think the network is constantly pinged
+  * These status updates can slow the network with the traffic
+* Useful if the network is not that large
+
+#### Reactive MANETs
+
+* Also known as on-demand routing
+* Network nodes only store information of paths to destination nodes
+* Nodes delay the search for routes to new destinations in order to reduce communication overheads
+  * i.e. if a route is found between A and B, this route will be stored and not recalculated
+* May be slower, as a shorter path may not be used
+
+#### Hybrid MANETs
+
+* Hybrid protocols combine the advantages of proactive and reactive protocols to reduce traffic overheads and route discovery delays
+
+Table showing all different protocols of MANETs
+
+![img](/lectures/acn/img/e.png)
+
+### Delay/Disconnection Tolerance
+
+Traditional MANET routing protocols like DSR and AODV (both reactive) cannot work in intermittent infrastructure-less environments because they require a complete path from source to destination for communication.
+
+* Messages get dropped at intermediate nodes when the link to the next hop is none existent in MANETs
+* DTNs expand MANETs to allow more intermittent and sparse connections of nodes caused by node mobility or low transmission  range.
+
+#### Store-carry-forward Paradigm
+
+* DTN routing protocols allow forwarding of messages by using a 'store-carry-forward' approach.
+  * messages are stored by nodes and moved in hops throughout the network until messages reach their destination
+  * This approach is used by DTN routing protocols to increase the probability of message delivery.
+
+#### DTN Protocol Classifications
+
+##### Flooding based
+
+* Flooding based routing protocols spread a message and have multiple copies of the message in the network. 
+* This is done to increase the probability of messages reaching their destination and also decrease the time of delivery
+
+##### Forwarding based
+
+* Forwarding based routing protocols gather information about the nodes in a network to select the best path to forward messages with the aim of enhancing message delivery networks with limited resources.

docs/lectures/acn/03_VANET_and_DTN.md

@@ -0,0 +1,102 @@
+# Vehicular Ad Hoc Networks
+
+* VANETs are a special type of Mobile Ad Hoc network which is used to
+  * provide communication between vehicles that are nearby (V2V)
+  * between vehicles on the road and fixed infrastructures on the roadside (V2I)
+* VANETs provide complementary approach for intelligent transport system (ITS) and are characterised by **high node mobility** and the limited degree of freedom in the mobility patterns.
+
+##### Categories of information
+
+1. Safety application information
+   * e.g. information regarding an accident that has just occurred
+   * the current conditions of the road
+2. Convenience application
+   * traffic information
+   * parking availability
+3. Commercial application for pleasure
+   * games
+   * real-time video relay
+
+### Why do VANETs need different protocols to MANETs
+
+###### Large scale
+
+> All vehicles on the road are potential nodes in the VANET.
+
+###### Predictive Mobility
+
+> The nodes in a VANET cannot follow arbitrary direction, they have to stay on the road and cannot suddenly change their direction.
+
+###### High Mobility
+
+> The network mobility in a VANET changes rapidly due to vehicular speeds.
+
+###### Partitioned Network
+
+> The ranges of wireless communication used in V2V networks is near 1 km but vehicles can get disconnected. Can be thought of many disconnected networks.
+
+The nodes in the VANET can move at **high speeds** which **reduces transmission capacity**, this causes the following issues:
+
+* **Rapid changes in the network topology** because the state of connectivity between nodes is dynamically changing.
+
+* **Occasional disconnections due to low traffic density**. This keeps the nodes distant from each other and results to **link failure** that could last for awhile.
+
+* **Node congestion**, a high traffic situation which affects protocol performance.
+
+### WAVE IEEE 802.11p
+
+WAVE - Wireless Access for Vehicular Environment
+
+* In WAVE vehicles communicate in a **hop by hop** manner with each other
+* The area of coverage for the WAVE node is limited to 300m-800m
+  * Beyond this range cars cannot communicate
+
+If there is dense traffic in the coverage region, **nodes become easily congested** because all nodes will be transmitting the same message to every other node.
+
+> To overcome the limitation of restricted coverage region, the use of DTNs was implemented which uses a **store-carry-forward paradigm**.
+>
+> With the store-carry-forward approach, a vehicle stores a message in a buffer and carries the message with it. When it comes into contact with another node, it forwards the message.
+>
+> * This introduced the idea of the **Vehicular Delay Tolerant Network (VDTN)** concept
+
+#### Vehicular Delay Tolerant Network (VDTN)
+
+VDTNs enable communication in the face of connectivity issues such as
+
+* long and variable delay
+* sparse and intermittent connectivity
+* high error rates
+* high latency
+* high asymmetric data rate
+
+Communication is made possible in the network when intermediate nodes become **custodians** of the message being transmitted and then forward the message only when a opportunity arises.
+
+###### Fixed DTN nodes
+
+* The stationary or relay nodes have store and forward capabilities and are located at **road-side intersections** (road side units)
+* They allow mobile nodes that pass by to collect and leave data on them.
+* They contribute to increasing the frequency of node contacts and improve **delivery ratio** and **delivery delay**.
+
+![img](/lectures/acn/img/f.png)
+
+## Categories of VANETs
+
+##### Pure cellular/WLAN
+
+> Pure cellular VANETs may use **fixed cellular gateways and WiMAX access points at road** intersections to gather information
+>
+> * note these road side gateways may not be feasible due to cost of infrastructure
+>
+> The information collected from sensors of a vehicle in the VANET can become valuable in notifying other nodes about the situation of the traffic in the network.
+
+##### Pure Ad-Hoc
+
+> Pure Ad-Hoc architecture is **not reliant** on infrastructure nodes
+>
+> In this architecture, nodes perform vehicle to vehicle (V2V) communication with each other.
+
+##### Hybrid
+
+> The hybrid category is a combination of the first two. It provides a richer content and offers great **flexibility in the sharing of data**
+>
+> * Some vehicles with WLAN and cellular capabilities may be used as **gateways** and **mobile routers** so that vehicles with only WLAN capabilities can interact and communicate effectively with them via multi-hop links.

docs/lectures/acn/04_DTN_Protocols.md

@@ -0,0 +1,65 @@
+# DTN Protocols
+
+### Forwarding Based
+
+Where each message may only be under the custody of a single node.
+
+* Upon forwarding the message, the receiving node also takes on the responsibility of custody. 
+* This means there will exist only one copy of the message within the network at any period of time.
+
+#### Direct Transmission
+
+* Direct transmission is the simplest single-copy forwarding protocol possible.
+* Once the source has generated a message, it will retain custody and carry it until it encounters the destination.
+* Once a connection with the destination is established, the message is forwarded directly
+  * This uses minimal resources
+  * Has unbounded amounts of latency
+  * Probability of a message being delivered is only as likely as the probability of the node encountering the destination node
+
+#### First Contact
+
+* First contact is a single-copy based forwarding protocol - it randomly chooses a node out of all possible nodes and forwards as many messages as possible to that node.
+* If no connections are available, the first encountered node will be used.
+* Once the message(s) are sent, the messages on the original node are deleted, relinquishing custody to the new node.
+* This protocol routes messages throughout the network via a random walk pattern.
+  * This can lead to packets being routed to dead ends.
+  * Packets can make negative progress or getting stuck in a loop.
+
+### Replication Based
+
+Replication-based protocols disseminate messages throughout the network via replication of the messages.
+
+* When one node encounters another, it will forward the message while retaining the local copy it has.
+  * The existence of multiple copies increases the probability of message delivery and reduces latency.
+  * The more nodes carrying the message, the more chance one node encounters the destination.
+* However this also means there are many redundant messages on the network - therefore more resources are needed.
+
+#### Epidemic
+
+* Utilising  the flooding concept, Epidemic aims to achieve message delivery by flooding the network with message copies.
+* When any two nodes meet, they compare messages.
+  * They then exchange messages they do not have in common
+  * This is repeated allowing the messages to spread similar to an epidemic.
+* This method achieves minimal latency & high delivery probabilities however suffers from limited resources.
+
+#### MaxProp
+
+* Like epidemic, maxprop floods the network, however each message has a priority.
+  * Messages stored in a **ordered-queue** in the **message buffer**.
+  * Messages with a higher probability of being delivered have a higher priory of being forwarded first.
+* To determine the probability, it looks at **history of encounters**, maintaining a vector with **tracks the likelihood of the node encountering any other node in the network**.
+  * When two nodes meet, they exchange messages and vectors, updating their own local copy.
+  * These vectors are then used to compute the shortest path for each message, messages are then ordered within the buffer by destination cost.
+* MaxProp uses overhead messages to acknowledge when a message has reached it destination
+  * Once this ACK signal is received, all local copies of redundant messages are dropped.
+
+#### PROPHET
+
+Probabilistic Routing Protocol using History of Encounters and Transitivity (PRoPHET) 
+
+* PROPHET maintains a vector that keeps track of a history of the encountered nodes.
+* It uses this vector to calculate the probability of a message copy reaching its destination by being forwarded to a particular node.
+  * When a source node forwards a message copy, it selects a subset of nodes that it can possibly send to.
+  * The algorithm then **ranks these nodes** based on the calculated probabilities, with the copy being forwarded to the highest ranked nodes first.
+* This is effective however the routing tables **rapidly grow** as a result of the amount of information on the nodes required to calculate the probability predictions.
+

docs/lectures/acn/05_DTN_advanced_protocols.md

@@ -0,0 +1,44 @@
+# DTN Protocols - Advanced
+
+**Rate control** - addresses easing network congestion by controlling the rate of traffic on the network.
+
+**Adaptive forwarding** - to direct traffic away from congestion hot spots.
+
+#### Spray and Focus
+
+* Spray and focus replicates an allowable number of messages from source in the spray phase.
+* **The focus phase allows** each node to forward a copy of its messages to other potential nodes until the messages gets to its destination.
+* The protocol uses a single-copy utility based routing scheme to forward a copy of the message further.
+* Forwarding decisions are made based on **timers** which record the times nodes come in communication range of each other.
+* Node $$A$$ forwards message with destination $$D$$ to node $$B$$ , **if and only if** $$B$$ has a higher potential of delivering the message to $$D$$.
+
+#### SimBet
+
+* A source node with no prior knowledge of the destination node will forward a message to a more central node that has the potential of finding a suitable relay node.
+  * A central node has the ease of connecting other nodes in a network.
+  * This is known as **centrality** a measure of the **structural importance** of a node in a network.
+* A central node uses **similarity and betweenness centrality** to avoid unnecessary information exchange in the entire network.
+* SimBet maintains a single copy of each message in the network to reduce resource overheads.
+
+### Replication Management
+
+Replication Management refers to easing network congestion by managing the amount and frequency that messages are replicated.
+
+* This is particularly notable concern as it is often the replication of messages that leads to congestion in DTNs, with surplus and redundant messages causing wastage within node message buffers.
+
+#### Café
+
+* Congestion Aware Forwarding Algorithm (Café)
+  * Single-copy
+  * Adaptive forwarding techniques - to reduce network congestion by directing traffic away from nodes experiencing congestion to less congested areas of the network.
+  * Uses **Contact Manager** and **Congestion Manager**
+
+**Contact Manager** - deals with nodes forwarding heuristics, updating statistics for each contact such as frequency and duration's.
+
+**Congestion Manager** - focuses on calculating the availability of nodes, keeping and updating a record of information such as the amount of available buffer and delays expected from each contacted node.
+
+#### CafREP
+
+* Congestion Aware Forwarding and Replication (CafREP)
+  * replication-based
+  * builds on Cafe protocol by coalescing the proposed **adaptive forwarding algorithm with an adaptive replication management technique**

docs/lectures/acn/06_DTN_congestion_control.md

@@ -0,0 +1,96 @@
+# Framework for Congestion Control in Delay Tolerant Opportunistic Networks
+
+DTNs mainly focus on increasing the probability to deliver to the destination and on minimising delays
+
+* Using complex graph theory techniques
+* Where load is unfairly distributed towards the better connected nodes
+* May lead to network congestion
+
+## CAFREP
+
+CAFREP or Congestion Aware Forwarding and Replication
+
+* Detects the congested nodes and parts of the network
+* Moves the traffic away from hot-spots and spreads it around while preserving the directionality of the traffic and not overwhelming non-interested nodes with unwanted content
+* Adaptively change message replication rates
+
+When deciding on the best carrier and the optimal number of messages, CAFREP dynamically combines three heuristics
+
+1. **Contact** analytics
+2. Predictive **node congestion** (node storage and in-network delays)
+3. Predictive **ego network congestion** 
+
+![img](/lectures/acn/img/g.png)
+
+Each layer you go up, the more information is exchanged between the nodes.
+
+### Metrics
+
+###### Node Retentiveness
+
+- Aims to avoid or replicate proportionally less at the **nodes** that have lower buffer availability.
+
+$$
+Ret(X) = B_c(X) - \sum^N_{i=1} \space M^i_{size}(X)
+$$
+
+For a node $$X$$, it has buffer of size $$B_c(X)$$. When a message of size $$M^i_{size}$$ is sent to node $$X$$, it's buffer size is the total buffer minus the memory taken by the sum of all messages in the buffer.
+
+###### Node Receptiveness
+
+- Aims to avoid or decrease sending rates to the **nodes** that have higher in network delays
+
+$$
+Rec(X) = \sum^N_{i=1}(T_{now} - M^i_{received}(X))
+$$
+
+How long a node keeps a message before forwarding it on. If a high level of receptiveness is found on a node, it means the node isn't useful as messages aren't forwarded. Could mean the node has limited connections.
+
+###### Node Congestion Rate
+
+- Aims to avoid or decrease sending rates to **nodes** that congest at the higher rate
+
+$$
+CR(X) = \frac{100\cdot T_{FullBuffer}(X)/T_{TotalTime}(X)}{\frac{1}{N}\cdot \sum^N_{i=1}(T_iend(X) - T_istart(X))}
+$$
+
+Estimates the time between a node being full and full again. Measures the time the node is unusable.
+
+#### Ego Network Congestion Metrics
+
+###### Ego Network Retentiveness
+
+* Aims to replicate less at the **parts of the network** with lower buffer availability.
+
+$$
+EN_{Ret}(X) = \frac{1}{N}\sum^N_{i=1}Ret(C_i(X))
+$$
+
+Gets the average of the retentiveness of node $$X$$ and it's neighbours $$c_i(X)$$
+
+###### Ego Network Receptiveness
+
+* Aims to replicate less at **parts of the network** with higher delays.
+
+$$
+EN_{Rec}(X) = \frac{1}{N}\sum^N_{i=1}Rec(c_i(X))
+$$
+
+###### Ego Network Congestion Rate
+
+- Aims to send less to the **parts of the network** that have higher congestion rates.
+- This is useful as if a node isn't congested, but all connected nodes are. It stops it from being used.
+
+$$
+EN_{CR}(X) = \frac{1}{N}\sum^N_{i=1}CR_i(X)
+$$
+
+#### Contents of CAFREP Node
+
+![img](img/h.png)
+$$
+Replication\space rate = M \times \frac{TotalUtil(Y)}{TotalUtil(X) + TotalUtil(Y)}
+$$
+Total utility, changes constantly. The replication limit grows to take advantage of all available resources, and backs off when congestion increases.
+
+Social utility prevents replication at a high rate on free nodes that are not on the path to the destination.

docs/lectures/acn/07_information_centric_networks.md

@@ -0,0 +1,102 @@
+# Information Centric Networks
+
+#### Problems with today's Networks
+
+* URLs and IP addresses are overloaded with locator and identifier functionality.
+* No consistent way to keep track of *identical copies*.
+* Information dissemination is inefficient.
+  * Cannot benefit from existing copies
+  * Can lead to problems like Flash-Crowd effect and Denial of service
+* Can't trust a copy received from an un-trusted node
+  * Security is host-Centric
+  * Based on *securing channels* (encryption) and trusting servers (authentication)
+* Application and content providers are independent of each other
+  * CDNs focus on web content distributions for major players
+
+![img](/lectures/acn/img/i.png)
+
+**Important requirements for ICNs** (Information Centric Networks)
+
+1. Accessing named resources - not hosts
+2. Scalable distribution through replication and caching
+3. Good control of resolution / routing and access
+
+## Content-based Routing for ICNs
+
+Apart from routing protocols that use direct identifiers of nodes, networking can take place based directly on content.
+
+* Content can be **collected** from the network, **processed** in the network and **stored** in the network.
+* The goal is to provide a network infrastructure capable of providing services better suited to today's application requirements
+  * Content distribution and mobility
+  * More resilience to disruption and failures
+
+#### Network Evolution
+
+**Traditional networking**
+
+- Host-Centric communications, addressing and end-points
+
+**ICNs**
+
+- Data-Centric communications addressing information
+- Decoupling in space - neither sender nor receiver need to know their partner.
+- Decoupling in time - *answer* not necessarily directly triggered by a *question*. **asynchronous communication**.
+
+#### Approach
+
+* Named Data Objects (NDOs)
+* In-network caching/storage
+* Multi-party communication through replication
+* Senders decoupled from receivers
+
+### Dissemination Networking
+
+* Data is requested by name, using any and all means available (IP, VPN tunnels, multi-cast, proxies etc)
+* Anything that hears the request and has a valid copy of the data can respond.
+* The returned data is signed, and optionally secured, so its integrity & association with name can be validated (data-Centric security)
+
+![ICN Stack](img/j.png)
+
+* Change of network abstraction from **named host** to **named content** (content chunks).
+* Security is built in - **secures content** and **not the hosts**.
+* **Mobility** is present by design.
+* Can handle **static** and **dynamic** content.
+
+#### Naming Data
+
+###### Solution 1 - Name the data
+
+- **Flat** - non human readable identifiers
+  - `1HJKRH535KJH252JLH3424JLBNL`
+- **Hierarchical** - meaningful structured names
+  - `/nytimes/sport/baseball/mets/game0224143`
+
+###### Solution 2 - Describe the data
+
+- With a set of tags
+  - `baseball, new york, mets`
+- With schema that defines attributes, values and relations among attributes
+
+##### Using Names in CCNs (Content Centric Networks)
+
+- The hierarchical structure is used to do *longest match look-ups* which guarantees $$log(n)$$ state scaling for globally accessible data.
+- Although CCN names are longer than IP identifiers, their **explicit structure** allows look-ups as efficient as IP's.
+
+### ICN Forwarding
+
+* Consumer *broadcasts* and *interest* over all available communication media
+* Interest identifies a *collection of data* whose name has the interest as a prefex.
+* Anything that hears the interest and has an element of the collection can respond with that data.
+
+### ICN Transport
+
+* Data that matches an interest, *consumes* it.
+* Interest must be re-expressed to get new data.
+  * Controlling re-expressions allows for traffic management and congestion control.
+* Multiple (distinct) interests in the same collection may be expressed 
+
+### ICN Caching
+
+* Storage and caching are integral part of the ICN service
+* All nodes potentially have caches. Requests for data can be satisfied by any node holding a copy in it's cache.
+* ICN combines caching at the network edge with in-network caching.

docs/lectures/acn/08_content_centric_networks.md

@@ -0,0 +1,96 @@
+# Content Centric Networks
+
+A Brief History of Networking
+
+- Gen 1. The **phone system** (focus on the **wires**)
+  - The utility of the system depends on running wires to every home & office.
+  - Wires are the dominant cost.
+  - A *call* is not the conversation, its the **PATH** between two end-office line cards.
+  - A *phone number* is not the name/address of the caller, its a **program** for the end-office switch fabric to build a path to the destination line card.
+  - <img src="/lectures/acn/img/k.png" alt="switch board" style="zoom:50%;" />
+  - Path building is **non-local** and **encourages centralisation** and **monopoly**.
+  - Calls fail is any element in the path fails so reliability goes down exponentially as the system scales up.
+  - Data cannot flow until the path is set up so efficiency decreases with setup time.
+
+- Gen 2. The **Internet** (focus on the **endpoints**)
+  - Data sent in independent chunks and each chunk contains the name of the final destination.
+  - Nodes forward packets onward using routing tables.
+  - **ARPAnet** was built on top of the existing phone system.
+
+- Gen 3. **dissemination** (focus on the **data**)
+
+#### TCP/IP
+
+###### Pros
+
+- Adaptive routing lets system **repair failures**
+- **Reliability increases exponentially** with **system size**.
+- **No call setup** means **high efficiency** at any bandwidth and scale.
+- Distributed routing supports any topology and tends to spread load and avoid a hierarchy's hot spots.
+
+###### Cons
+
+- *Connected* is a binary attribute. 
+- Becoming part of the internet requires a globally unique, globally know IP address that's topologically stable on routing time scales.
+  - Connecting is a heavy weight operation
+  - The net struggles with moving nodes
+
+#### Conversation and Dissemination
+
+Acquiring chunks of data (web pages, emails, videos etc) is not a conversation, it's *dissemination*.
+
+In a dissemination **the data matters**, not the supplier.
+
+- Data is request by name.
+- Anything that hears the request, and has a valid copy can respond.
+- The return data is signed, so integrity and association can be validated.
+
+CCN can run over and be run over anything e.g. IP.
+
+#### CCN Packets
+
+![img](/lectures/acn/img/l.png)
+
+**Interest** - similar to HTTP `GET`
+
+**Data** - similar to HTTP response
+
+#### Content Based Security
+
+Data packets are authenticated with digital signatures.
+
+![img](/lectures/acn/img/m.png)
+
+#### CCN Forwarding
+
+Consumer *broadcasts* and *interest* over all available communication media
+
+- e.g. `get '/parc.com/van/presentation.pdf'`
+- response: `heres '/parc.com/van/presentation.pdf/p1' <data>`
+
+##### Names and Meaning
+
+* Like IP, CCN nodes imposes no semantics on names
+* Meaning comes from **application**, **institution** and **global conventions** reflected in prefix forwarding rules.
+* Globally meaningful name leveraging the DNS global naming structure
+  * `/parc.com/van/presentation.pdf`
+* Local and context sensitive, it refers to different objects depending on the room you're in.
+  * `/thisRoom/projector`
+
+#### Strategy Layer
+
+* When you do not care who you are talking to, you don't care if they change
+* When you are not having a conversation, there's no need to migrate conversation state.
+* Multi-point gives you multi-interface for free.
+* When all communication is locally flow balanced, your stack knows exactly whats working and how well.
+
+In the current Internet, Quality of Service (QoS) Problems are highly localised
+
+* Roughly half the problems are from serial dependencies created by queues
+* The other half are caused from a lack of receiver based control over bottle-necked links.
+
+Unlike IP, CCN is **local**, don't have queues and receivers have complete control
+
+![img](/lectures/acn/img/n.png)
+
+Tree serves as transport state

docs/lectures/acn/09_DTN_security.md

@@ -0,0 +1,138 @@
+# Delay Tolerant Networks Security
+
+#### Applications of DTNs
+
+##### Interplanetary communication
+
+<img src="/lectures/acn/img/o.png" alt="DTN in space" style="zoom:50%;" />
+
+> **Characteristics**
+>
+> * High intermittent connectivity
+> * Extremely long message travel time
+>   * Delay: finite speed of light
+> * Low Transmission reliability
+>   * Inaccurate position
+>   * Limited visibility
+> * Low asymmetric Data Rate
+>
+> **Security**
+>
+> - CCSDS protocol
+>   - space End to End security
+>   - space end to end reliability
+
+##### Military
+
+> No consistent network infrastructure and frequent disruptions
+>
+> **Characteristics**
+>
+> * High intermittent connectivity
+> * Mobility, destruction, noise & attacks, interference
+> * Low transmission reliability
+>   * positioning inaccuracy
+>   * limited visibility
+> * Low data rate
+>
+> **Security**
+>
+> - Mainly MANET security
+> - Distribution of CAs (Certificate Authorities) in mobile ad hoc networks cannot provide military level security
+> - Combining a self-organised approach with an off-line trusted third-party
+
+##### Rural Areas
+
+>Providing internet connectivity to rural/developing areas
+>
+>**Characteristics**
+>
+>- Intermittent connectivity
+>- Mobility - sparse development
+>- High propagation delay
+>- Asymmetric data rate
+>
+>![img](/lectures/acn/img/p.png)
+>
+>**Security**
+>
+>- Standard cryptographic techniques such as PKI and transparent encrypted file systems
+
+- Disaster struck areas
+- Disconnected kiosks in rural areas
+- Remote sensing applications
+
+But also
+
+- Bulk data distribution in urban areas
+- Sharing of individual contents in urban areas
+- Mobile location-aware sensing application
+- Social mobile applications
+
+#### DTN Security Goals
+
+Due to the resource-causticity that DTNs have, the focus is on protecting the DTN infrastructure from unauthorised access and use.
+
+* Prevent **access** by unauthorised applications.
+* Prevent unauthorised applications from asserting control over DTN infrastructure.
+* Prevent authorised applications from sending bundles at a rate or class of service for which they **don't have permissions for**.
+* Detect and discard bundles that were sent from unauthorised applications/users.
+* Detect and discard bundles who's headers have been modified.
+* Detect and discard compromised entities.
+
+Secondary emphasis is on providing optional end-to-end security services to bundle applications.
+
+#### DTN Security Challenges
+
+* High round-trip times and disconnections
+  * Do not allow frequent distribution of a large number of certificates and encryption keys end-to-end.
+  * More scalable to use user's keys and credentials at neighbouring or nearby nodes.
+* Delays or loss of connectivity to a key or certificate server
+  * Multiple certificate authorities desirable but not sufficient and certificate revocation not appropriate
+* Long delays
+  * Messages may be valid for days/weeks, so message expiration may not be able to be depended on to rid the network of unwanted messages as efficiently as in other types of networks.
+* Constrained Bandwidth
+  * Need to minimise the cost of security in terms of network overhead (header bits).
+
+###### Traditional PKI not applicable
+
+* Traditional symmetric cryptography approaches are not suitable for DTNs for two major reasons
+  * In PKI a user authenticates another users public key using a certificate
+    * This is not possible without online access to the receivers public key or certificates
+  * PKIs implement key revocation based on frequently updated online certificate revocation lists
+    * In the absence of instant online access to CAs servers, a receiver cannot authenticate the sender's certificate.
+
+###### Identity Based Cryptography not applicable
+
+Identity Based Cryptography (IBC) schemes where the public key of each entity is replaced by its identity and associated public formatting policies are not suitable for the security in DTNs
+
+- IBC does not solve the key management problem in DTNs
+- It is not scalable because it assumes that a user must know the public parameters for all the trusted parties.
+
+###### Mobile ad hoc Key Management Proposals not applicable
+
+- Virtual Certificate Authority 
+  - Not applicable due to no  trusted third parties
+- Certificate chaining based on pretty good privacy (PGP)
+  - Not applicable due to insufficient density of certificate graphs
+- Peer-to-peer key management based on mobilty
+  - Not applicable due to certificate revocation mechanism
+
+#### Existing Mandatory DTN Security
+
+Based on the *bundle* protocol
+
+* Hop-by-hop bundle integrity
+* Hop-by-hop bundle sender authentication
+* Access Control (only legit users with right permissions)
+* Limited protection from DoS attacks
+
+![img](/lectures/acn/img/q.png)
+
+- Payload Security Header is computed once at the source bundle agent, carried unchanged, and checked at the destination bundle agent (and possibly also security boundary bundle agents)
+
+- Bundle Authentication Header is computed at every sending bundle agent and checked at every receiving hop along the way from the source to the destination.
+
+Current DTN security initiative is based on pre-shared secrets and involves no trust dynamics mechanisms
+
+- Works well against external threats but not applicable to internal threats

docs/lectures/acn/10_drones.md

@@ -0,0 +1,40 @@
+# Enabling Real-Time communications and Services in Heterogeneous Networks of Drones and Vehicles
+
+### Real World Experiments
+
+#### Agricultural Monitoring Application
+
+- Agricultural context in UK
+- The production of potatoes or livestock has always been a major part of farming
+- There has always been a need for farmers to be able to observe their field crops or animals as often as possible so that they are informed quickly about potential deep rooted problems in the fields.
+
+Enable mobile reliable multi-hop DTN communications (in field near Nottingham)
+
+- 2 Flying drones
+- 1 Vehicle
+- 2 Static ground sensing nodes
+  - Raspberry Pis
+  - These capture and send data to the drones, which forward it to a node with higher computational output
+
+The two static sensing nodes are deployed on two different sides of the field out of reach of each other while the drone acts as a intermediaries.
+
+All sensing nodes could measure
+
+- Air temp
+- wind speed
+- soil temp
+
+We measure average edge to edge (E2E) delays of content query and dissemination in the network
+
+- Two drones and one vehicle (3 intermediaries) result in lower delays
+
+#### Smart City Applications
+
+- More focused on single hop
+  - 1 Hovering drone (publisher)
+  - Moving vehicle (subscriber)
+  - The drone continuously sends information such as sensors readings, street images, traffic videos to the vehicle which monitors road conditions
+- Single hop communications is significantly affected by physical obstructions
+  - Therefore the latency went down in suburbs compared to city centres
+  - Height of the drone is important as well
+

docs/lectures/acn/11_connecting.md

@@ -0,0 +1,157 @@
+# Connecting
+
+#### Elasticity: Supply and Demand
+
+This is about **resource management**
+
+- **Supply** - Available link capacity on path
+- **Demand** - Host transmitting and receiving traffic
+- **Elastic** - capacity reduces -> demand is scaled back
+  - Hosts stop sending / send less
+- **Inelastic** - applications can’t handle this
+
+TCP manages resource usage based on observed loss and latency
+
+#### Quality of Service
+
+If capacity > demand, there is no need for quality of service
+
+If capacity < demand, we need to keep queuing minimal
+
+- As queuing directly impacts latency, jitter and loss
+  - In stable networks
+  - **Jitter**: The difference in delays, a measure of stability
+
+#### IP Type of Service
+
+- Single IP header byte
+
+```
+Bits 0-2:  Precedence.
+Bit    3:  0 = Normal Delay,       1 = Low Delay.
+Bits   4:  0 = Normal Throughput,  1 = High Throughput.
+Bits   5:  0 = Normal Reliability, 1 = High Reliability.
+Bit  6-7:  Reserved for Future Use.
+```
+
+- Precedence for *special* traffic
+
+```
+0     1     2     3     4     5     6     7
++-----+-----+-----+-----+-----+-----+-----+-----+
+|                 |     |     |     |     |     |
+|   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
+|                 |     |     |     |     |     |
++-----+-----+-----+-----+-----+-----+-----+-----+
+
+Precedence
+
+111 - Network Control
+110 - Internetwork Control
+101 - CRITIC/ECP
+100 - Flash Override
+011 – Flash
+010 – Immediate
+001 – Priority
+000 - Routine
+```
+
+### Differentiated Services (DiffServ)
+
+- Operates on *traffic aggregates*
+  - Label packets with desired class via ToS
+  - Routers apply different queuing as operator sees fit
+- Four service classes, or *per-hop behaviour*
+  - **Default**: best effort
+    - No QoL applied
+  - **Expedited Forwarding**: low delay, loss & jitter
+  - **Assured Forwarding**: low loss if within rate
+  - **Class Selector**: use ToS precedence bits
+
+##### Problems
+
+- End to end semantics
+- Mapping to service level agreement
+  - If an internet company sells a network with a certain speed, this might have legal repercussions if QoS are enacted
+- Mapping to application demands
+
+### Integrated Services (IntServ)
+
+- Operates on explicitly signalled *flows*
+  - Think phone switchboards
+  - The network signals exactly what it can and can’t do to the destination nodes
+- Flow setup specifies some quality of service
+- Routers perform **C**onnection **A**dmission **C**ontrol
+  - CDA can accept and reject traffic based on whether or not the route/path is available
+
+##### Problems
+
+- Complexity
+  - Hard to scale
+- Mapping requirements to parameters
+  - This was easier when ATM did it as they owned all the infrastructure
+  - Whereas now it is difficult to map across all different companies
+- Per-flow state
+  - Extremely difficult
+
+## NAT
+
+### Address Shortages
+
+**IPv4** supports 32 bit addresses
+
+- 95% allocated already (440,000 netblocks)
+
+**IPv6** supports 128 bit address
+
+- Loads of addresses :white_check_mark:
+- Routing protocols need to ported :negative_squared_cross_mark:
+- Associated services needing to move :negative_squared_cross_mark:
+
+### Network Address Translation
+
+Because IPv6 did not magically solve address shortage problem and not all routers are ipv6 aware, we had to rely on NAT.
+
+- Private Addressing, `RFC1918`
+  - `172.16/12`, `192.168/16`, `10/8`
+  - Devices with these local IPs should never be externally routed
+  - Not for security reasons - just for getting more addresses
+- Traditional NAT, `RFC3022` is the standard
+  - Use private addresses internally (within the local network)
+  - Map into a (small) set of routable addresses
+  - Use source ports to distinguish connections
+  - For large scale **carrier grade NAT** [`RFC6598`] on `100.64/10`
+
+#### Implementation
+
+- Requires IP, TCP/UDP header rewriting
+  - Addresses, ports and checksums all need to be recalculated
+- Behaviours
+  - Network Address Translation
+  - Network Address and Port Translation
+
+###### Full Cone
+
+![image-20220124180451508](/home/jay/.config/Typora/typora-user-images/image-20220124180451508.png)
+```
+ea:ep - NAT address : NAT port
+```
+
+When client receives packet from server 1 `da:dp`, the NAT translates the NAT address `ea:ep` to the clients internet address and port `ia:ip`.
+
+###### Address Restricted Cone NAT
+
+![image-20220124181009792](/home/jay/.config/Typora/typora-user-images/image-20220124181009792.png)
+In this case server 2 is not trusted and therefore any request will be dropped.
+
+###### Port Restricted Cone NAT
+
+![image-20220124181127801](/home/jay/.config/Typora/typora-user-images/image-20220124181127801.png)
+
+If the router receives a packet from a bad IP or bad port, it will be dropped.
+
+###### Symmetric NAT
+
+![image-20220124181325935](/home/jay/.config/Typora/typora-user-images/image-20220124181325935.png)
+
+Here the internal address is obfuscated from the external servers, same client can use different ports for different communications.

docs/lectures/acn/12_naming.md

@@ -0,0 +1,139 @@
+# Naming
+
+IPs are not human readable.
+
+Not always the appropriate granularity
+
+- The address names an interface
+- This however does not give information about the kind of service / hardware
+
+A file maps names to addresses
+
+- Unix & Linux
+  - `/etc/hosts`
+- Windows
+  - `C:\Windows\System32\drivers\etc\hosts`
+
+These are simple but neither automatic or scalable which led to **DNS**.
+
+- Was initially `RFC882`
+- Now is `RFC1035, 1987`
+
+DNS is a consistent namespace
+
+- No reference to addresses, routes etc
+- Is hierarchical, distributed & cache
+  - All of which to help with scalability
+- **Federated** - sources control trade-off
+  - This just means DNS are worldwide
+- **Flexible** - many record
+- Simple client-server name resolution protocol
+
+#### Components
+
+- *Domain name space* and *resource records*
+  - Tree structured name space
+  - Data associated with names
+- *Name server*
+  - Contains records for a sub tree
+  - May cache information about any part of the tree
+- Resolver
+  - Extract information from tree upon client requests
+  - `gethostbyname()`
+
+![img](/lectures/acn/img/aa.png)
+
+###### Root
+
+- Ultimate authority with the US Dept. of commerce (NITA)
+- Managed by IANA, operated by ICANN, maintained by Verisign
+- Started with only thirteen root server clusters
+  - Now much more
+- Top level Domains, TLDs
+  - Operated by registrars, delegated by ICANN
+- Delegate zones to other registrars
+  - and so on down the hierarchy
+- Eventually customer rents a name - their **zone**
+  - Registrar installs appropriate *resource records*
+  - Associated with names within the zone
+
+#### Query
+
+- Query generated by resolver
+  - e.g. call to `gethostbyname()`, `gethostbyaddr()`
+- Carried in single UDP/53 packet
+  - Or more rarely TCP/53 in case of truncation
+  - UDP is not smart and therefore does not follow traffic routing (it is selfish)
+    - It is beneficial for the internet as a whole to use UDP sometimes
+- Header followed by question
+  - ID, Q/R, opcode, AA/TC/RD/RA, response code, counts
+  - Query type, query class, query name
+
+Response consists of three RRsets following the header and question
+
+- **Answers**: RRs that the server had for the QNAME
+- **Authoritatives**: RRs pointing to an authority for the name
+- **Additionals**: RRs related to the question but don’t answer it
+
+###### Common Resource Records
+
+- `A` / `CNAME` / `PTR`
+
+```
+www.cs.nott.ac.uk.			61272	IN	CNAME	pat.cs.nott.ac.uk.
+
+pat.cs.nott.ac.uk.			68622	IN	A		128.243.20.9
+
+pat.cs.nott.ac.uk.			68622	IN	A		128.243.21.19
+
+9.20.243.128.in-addr.arpa.	39617	IN	PTR		pat.cs.nott.ac.uk.
+```
+
+`cname` refers to the mapping of the domain name to its IP (or another domain) & ports
+
+Can have 2 authoritative records
+
+- `NS`
+
+```
+cs.nott.ac.uk.	10585	IN	NS	ns1.nottingham.ac.uk.
+cs.nott.ac.uk. 	10585	IN	NS	ns2.nottingham.ac.uk.
+cs.nott.ac.uk. 	10585	IN	NS	marian.cs.nott.ac.uk.
+cs.nott.ac.uk.	10585	IN	NS	extdns1.warwick.ac.uk.
+cs.nott.ac.uk.	10585	IN	NS	extdns2.warwick.ac.uk.
+```
+
+It is good practice to have an external DNS, UoN uses Warwick as an external DNS.
+
+- `MX`
+
+```
+nott.ac.uk.	3600	IN	MX	1 mx191.emailfiltering.com.
+nott.ac.uk.	3600	IN	MX	2 mx192.emailfiltering.com.
+nott.ac.uk	3600	IN	MX	3 mx193.emailfiltering.com.
+```
+
+What happens when the resolver queries a server that doesn't know the answer? two solutions:
+
+1. **Iterative** (required)
+   - Server responds indicating who to ask next
+   - This method is slower and more difficult to retrieve an answer
+1. **Recursive** (optional)
+   - Server generates a new query to the next server
+
+![img](/lectures/acn/img/ab.png)
+
+#### Load Balancing
+
+DNS may have multiple servers, when a query comes various algorithms can be used to choose the best one, this can be geographical location.
+
+#### Operational & Security Issues
+
+- Usually need primary and secondary servers
+  - Separate IP netblocks, physical networks - more robust
+  - DNS is a *very* common single point of failure
+- Cache poisoning
+  - Caching and soft-state means bad data propagates and can persist for some time
+  - Even if through simple mistakes (or of course malicious attacks)
+- Man-in-the-middle attacks
+  - Can happen with both iterative & recursive queries

docs/lectures/acn/13_reliability.md

@@ -0,0 +1,235 @@
+# Reliability
+
+Achieving reliability:
+
+- Re-transmitting lost data
+  - This is done by detecting lost via explicit acknowledgment
+  - These can be positive or negative
+
+### Stop ‘n’ Wait
+
+Simplest possible paradigm
+
+- Transmit `seq(x)`
+- Wait for `ack(x)`
+- Transmit `seq(x+1)`
+
+![img](/lectures/acn/img/a.jpeg)
+
+This has really poor performance in high latency and uses high bandwidth (half the bandwidth is overhead (acknowledgements))
+
+**Rate control**: Never sending too fast for the network
+
+**Sliding window**: allow unacknowledged data in flight (data to be sent)
+
+**Retransmission TimeOut**: how long to wait to decide a segment is lost
+
+- This requires estimates of dynamic quantities
+
+- Permit N segments in flight
+- Timeout implies loss
+- Retransmit from lost packet onward
+  - This is bad as imagine if only packet 3 is lost out of 5, this means client will resend 3-5.
+
+##### Congestion Collapse
+
+When network load is too high, it causes *congestion collapse*
+
+Why?
+
+- The routers buffers fill up, traffic is discarded, hosts retransmit
+- Retransmit rates increase since more data was lost
+- This was solved in “Congestion Avoidance and Control”
+
+#### Stability of the Internet
+
+Flows and protocols **include some sort of congestion control** and adaptation so that they moderate their bandwidth use, limit packet loss as well as get approximately fair share of available network bandwidth
+
+1. **Responsiveness** defined as a number of round-trip times of sustained congestion required to reduce the rate by half
+2. **Stability and smoothness** defined as the largest reduction of the sending rate in one round trip time in a steady state scenario
+3. **Fairness** towards other flows when competing for bandwidth
+
+Mimicking TCP behaviour for multimedia congestion control results in fairness towards TCP but also in significant oscillations in bandwidth
+
+- Multimedia streaming applications need to **have much lower variation at throughput** over time compared to TCP to result in relatively smooth sending rates that are of importance to the end-user perceived quality.
+- The penalty for having smoother throughput than TCP while competing for bandwidth is that multimedia congestion control responds slower than TCP to changes in available bandwidth.
+- Thus, if multimedia traffic wants smooth throughput, it needs to avoid TCP’s halving of the sending rate in response to a single packet drop.
+
+###### Packet Loss
+
+- When choosing the method for packet loss detection, it is important to choose a method that **detects packet losses as early and accurately as possible**
+  - Incorrect detection & late packet delivery can lead to incorrect packet loss estimation
+  - This causes unresponsive & unfair behaviour
+- Calculating packet loss rates can be done over various lengths of time intervals.
+  - Shorter intervals result in more responsive behaviour but are more susceptible to noise
+  - Longer intervals = smoother but less responsive
+  - It is important to find a balance
+- In order to guarantee sufficient responsiveness to congestion and preserver smoothness, methods for detecting & calculating packet loss must be chosen carefully.
+  1. What mechanism can be used for packet loss detection?
+  2. What algorithm can be used for packet loss rate calculation?
+  3. Where can packet loss detection and calculation happen?
+
+###### Approach
+
+- All sent packets are marked with consecutive sequence of numbers
+- When a packet is sent a timeout value for this packet is computed and an entry containing the sequence number and the timeout value is inserted into a list and kept there until packet delivery is acknowledged or considered to be lost
+  - If the timeout expires before the packet is acknowledged, the corresponding packet is considered to be lost
+- In order to adapt to varying and unpredictable network conditions, the timeout is not fixed, but computed based on one of the algorithms for TCP timeout computation
+
+##### Timeout Based Approaches
+
+This is mostly used for multimedia situations
+
+RTT - round trip times
+
+- Before the first packet is ACK and RTT measurement is made, the sender sets the TIMEOUT to a certain initial value
+  - This value is usually **2.5-3 seconds for TCP**
+  - For real time interactive multimedia traffic, the timeout value should be set to **0.5 seconds** as this is the time where audio delay affects media
+- When the first `RTT` measurement is taken the sender sets the smoothed `RTT` (`SRTT`), `RTT` variance (`RTTVAR`) and `TIMEOUT` in the following way
+  - `SRTT = RTT`
+  - `RTTVAR = RTT/2`
+  - `TIMEOUT = `$\Mu\cdot$`SRTT + 4*RTTVAR`
+  - Where $\Mu$ is a constant, which in this implementation is 1.08 (obtained experimentally)
+- When subsequent `RTT` measurements are made the sender sets the `RTTVAR`, `SRTT`, TIMEOUT 
+  - `RTTVAR`$= (1 - \frac{1}{4}) \times$`RTTVAR`$+ \frac14 \times |$`SRTT`$-$`RTT`$|$
+  - `SRTT`$= (-\frac18)\times$`SRTT`$+\frac18\times$`RTT`
+  - `TIMEOUT`$= \Mu\times$`SRTT`$+ 4\times$`RTTVAR`
+
+###### Packet loss rate calculation
+
+- Real time interactive multimedia approaches typically use the **weighted Loss Interval Average (WLIA)**
+- It relies on **using loss events** and **loss intervals** for correct computation of packet loss rate and is in accordance with how TCP performs packet loss calculation
+- A **loss event** is defined as a number of packets lost within a single RTT
+
+This can be done either on the sending or receiving side
+
+**Sender-side**: if the packet loss detection is done in the sender, the sender can use timeout mechanism for each packet or gap in sequence numbers of the acknowledged
+
+- The receiver has to acknowledge either every packet or every packet not received
+- Acknowledging every packet can introduce high levels of traffic between between sender and receiver
+- This is solved by having receivers send report summaries of losses every nth packet or nth RTT
+
+**Receiver-side**: Packet loss is detected in the receiver and explicitly reported back to the sender
+
+- Noticing the gap in the sequence number - a loss event can be assumed
+- A loss event is directly forwarded to the sender
+
+##### Sender vs Receiver Detection
+
+Receiver driven packet loss discovery is preferred.
+
+- This is because loss events are sent early as possible
+- This means high responsiveness
+
+In the case of very high congestion - where there is no feedback from the receiver
+
+- The pure receiver based loss detection is useless because the sender has no way of calculating packet loss
+- In these cases sender enters **self-limitation** - where packet loss is assumed and sending rate is decreased or even stopped
+
+#### Adaption
+
+Once the parameters of a given link are measured (packet loss and round trip times), there is a range of approaches that could be followed when choosing rate adaptation scheme(s).
+
+**Equation-based control** uses a control equation that explicitly gives the maximum acceptable sending rate as a function of the recent loss event rate (loss rates).
+
+**Additive Increase Multiplicative Decrease (AIMD) control** of in response to a single congestion indication.
+
+###### Decision Function
+
+Options for Decision function:
+
+- **On congestion** (overload/packet loss/packet loss increase), **decrease the rate immediately, or periodically****
+- On absence of congestion** (underload/no packet loss, packet loss decrease), **increase the rate immediately**
+
+###### Increase/decrease function
+
+Options for **increase phase**: (during underload)
+
+- constant additive increase rate,
+- straight jump to the expected value or value calculated by the formula
+- multiplicative increase rate
+
+The default for the Internet is **constant linear increase**.
+
+One could argue that a loss estimate of zero indicates that there is no congestion and thus the sending rate should be increased with the maximum possible increase factor until a loss event occurs. 
+
+- However, this approach **causes instabilities** in the sending rate and is very susceptible to a noisy packet drop rates. 
+
+Options for **decrease phase**
+
+- constant multiplicative decrease factor, TCP-like or TCP-similar like.
+- linear decrease
+- straight jump to the expected value (calculated by the formula)
+
+The default for the internet is multiplicative decrease (halving)
+
+- Because congestion recovery should be exponential and not linear
+
+Options for **decision frequency**
+
+Decision frequency specifies **how often to change the rate.** **Based on system control theory, optimal adjustment frequency depends on the feedback delay.**
+
+- The feedback delay **is the time between changing the rate and detecting the network’s reaction to that change.**
+  - It is suggested that equation-based schemes adjust their rates **not more than once per RTT**.
+  - Changing the rate too often results in oscillation
+  - Infrequent change of the rate leads to an unresponsive behaviour.
+
+###### Self Clocking
+
+Aim is that transmission spacing matches bottleneck rate
+
+- Avoids consistent queuing at bottleneck
+- Queue to smooth out short-term variation
+
+##### Congestion Control
+
+Aim to obey **conversation of packets**
+
+- In equilibrium flow is conservative
+- New packet doesn't enter until one leaves
+
+This fails in three ways:
+
+1. Connection doesn't reach equilibrium
+2. Sender transmits too soon
+3. Resource limits prevent equilibrium being reached
+
+Solutions:
+
+**Slow-start**
+
+- Each ACK opens congestion window by 1 packet
+  - Every ACK, `cwnd += 1`
+  - Every RTT `cwnd *= 2`
+- If a stop occurs, stop or `cwnd == ssthresh`
+- Else multiplicative increase
+
+![img](img/ac.png)
+
+**Congestion Avoidance**
+
+1. Network signals congestion occurring
+   - Detect loss
+2. Host responds by reducing sending rate
+   - `ssthresh := cwnd/2` multiplicative decrease
+   - `cwnd    := 1` initialises slow start
+
+Avoid congestion by slow increase
+
+`cwnd += 1/cwnd` window increases 1 per window
+
+TCP is not always useful
+
+- Reliability can cause untimely delivery
+
+Audio/Video codecs usually produce frames (not continuous bytestream)
+
+- Losing a frame is better than delaying all subsequent data
+
+UDP encapsulates media using **R**eal **T**ime **P**rotocol
+
+- Sequencing, time stamping, delivery monitoring, no quality of service
+- Adds a control channel
+  - Back channel to report statistics & participants
+- Transport only
+  - Leaves encodings & floor control to application

docs/lectures/acn/14_bgp.md

@@ -0,0 +1,163 @@
+# Border Gateway Protocol
+
+#### Routing Protocols
+
+The main job is to distribute the data to build forwarding tables
+
+- These are **intra-domain routing protocols**
+  - Or **Interior gateway protocols**
+  - When the source and destination are **inside** the **same network**
+
+It is important to distinguish between local and global protocols
+
+- Interior vs Exterior Gateway Protocol (IGP vs EGP)
+
+##### BGPv4
+
+The internet inter-domain routing protocol
+
+- Derives from GGP & EGP
+- Deals in IP prefixes and **autonomous systems**
+  - autonomous systems are purely administrative
+  - Purpose is to enable *policy* to be applied
+  - Only prefixes matter in the data-plane
+- Internet policy domains
+  - Logical construct only
+  - No meaning outside BGP
+  - Do not map simply onto ISPs or networks
+- Currently ~493,000 prefixes & ~46,000 ASs
+  - Because we have less ASs, the routing is easily -> less complex
+  - Reduces complexity
+  - Speeds up performance
+
+BGP uses TCP as transport
+
+- `OPEN`, `UPDATE`, `KEEPALIVE`, `NOTIFICATION`
+
+Sessions between peers have:
+
+- Simple capability negotiation
+- Manage simultaneous `OPEN` connections
+- Lose everything on session failure
+
+#### Sessions and Routing Information Base (RIBs)
+
+A BGP peer typically has many sessions
+
+- Logically, for each peer, it receives the information about routing from peers, this is sorted into `Adj-RIB-in` table.
+- After processing, it produces a `Adj-RIB-out` table which it sends to other peers
+  - Advertisements received and to be sent
+- Generates a local RIB table from `Adj-RIB-in`
+  - Routes to use and potentially distribute
+  - Resolved into per-port forwarding tables
+- Generate `Adj-RIB-out` from `Loc-RIB` and policy
+
+#### Update messages
+
+- Incremental - indicate *changes* to state
+  - These updates could be:
+  - Withdrawn routes
+  - Path attributes, common to all advertised routes
+  - Advertised routes, known as NLRI
+- There are ~27 path attributes
+  - Only ~12 are in common use
+  - Communicate information about prefixes
+  - Used to apply policy in BGP *decision process*
+
+##### Path Attributes
+
+Mandatory - every AS has to inform the other ASs about these 3 attributes:
+
+- Next hop
+- AS Path
+- Origin
+
+Discretionary
+
+- Local preferences
+  - Allows prioritisation of ASs 
+
+Optional & transitive
+
+- Aggregator
+- Community
+- Extended Communities
+
+Optional & non-transitive
+
+- Multi-exit discriminator
+- Originator ID
+
+### Path Vectors
+
+**Distance Vector** - prefer lowest cost path (not always)
+
+**Path Vector**
+
+- How do we know if an AS has seen this advert before
+- Store the list of ASs in the packet
+  - This is called the `AS_PATH`
+- This way loops can be broken
+  - If our ASN appears in a received `AS_PATH`, drop the advert
+
+##### Decision Process
+
+Drop prefix if:
+
+- `NEXT_HOP` is unreachable via local routing table
+- Local AS appears in `AS_PATH` (packet in a loop)
+
+Then (commonly) apply following preference:
+
+1. Higher `weight` (local to this router)
+2. Highest `LOCAL_PREF`
+3. Shortest `AS_PATH`
+4. Lowest `ORIGIN`
+5. Lowest `MED`
+6. `EGP` to `IGP` (hot potato)
+7. Shortest internal path
+8. Prefer oldest route
+   - Oldest routes are often most stable
+9. Lowest interface IP address
+
+### Consistency
+
+Learn external routes on `EBGP` sessions
+
+- `EBGP` defined as peers having different ASNs
+- Must ensure every router knows all external routes
+- Redistribute external routes inside the network
+  - Via `IGP` - only in small networks
+  - via `IBGP` - gives full control over route distribution
+
+###### Scaling
+
+Can distribute `IBGP` routes on `IBGP` sessions
+
+- Have to maintain $N\cdot \frac{(N-1)}{2}$ `IBGP` sessions
+  - Each carrying up to 490k routes x2 tables
+- Two standard solutions
+  1. **Route Reflectors**
+     - Super nodes re-advertising `IBGP` routes
+     - Allows for hierarchy
+  2. **AS Confederations**
+     - split AS up into mini-ASs
+
+###### Failures
+
+- Handling link failures
+  - Bind to loopback
+    - If it cant talk to other nodes, will only support communication internally
+  - Flap damping
+    - A warning message saying don't send traffic to me
+    - This can make things worse if this message is delayed
+- Process failures
+  - Out of memory error due to too many routes
+
+##### Network Inter-connection
+
+- Networks interconnect via `EBGP` sessions
+  - POPs - points of presence or IX internet exchanges
+- Multi-homing
+  - This is all logical
+  - http://0x0.st/ooCs.png

docs/lectures/dms/01_java_collections.md

@@ -0,0 +1,135 @@
+# Java Collections
+
+Week 3 (Oct 5th)
+
+**Part 1**
+
+In java a *collection* is an object that represents a group of objects.
+The collections API is a unified framework for representing and manipulating collections independently of their implementation.
+
+An *API* (application programming interface) is an interface protocol between a client and a server, intended to simplify the client side software.
+
+A *library* contains re-usable chunks of code.
+
+**Java Collections framework**
+- We have container objects that contain objects
+- All containers are either "collections" or "maps"
+- All containers provide a common set of method signatures, in addition of their unique set of method signatures
+
+*Collection* - Something that holds a dynamic collection of objects
+*Map* - Defines mapping between keys and objects (two collections)
+*Iterable* - Collections are able to return an iterator objects that can scan over the contents of a collection one object at a time
+
+NOTE: Vector is a legacy structure in Java replaced with *ArrayList*
+      Stack is now *ArrayDeque*
+
+`LinkedList(Collection<? extends E> c)` - means some type that either is E or a subtype of E. The `?` is a wildcard.
+
+```java
+public static void main(String[] args) {
+     LinkedList list = new LinkedList();
+     list.add("string");
+     String s = (String)list.getFirst();
+     System.out.println(s);
+}
+```
+
+This is bad coding practice, the collection constructor are not able to specify the type of objects the collection is intended to contain. A `ClassCastException` will be thrown if we attempt to cast the wrong type.
+
+```java
+public static void main(String[] args) {
+     LinkedList<String> list = new LinkedList<>();
+     list.add("string");
+     String s = list.getFirst();
+     System.out.println(s);
+}
+```
+
+This is a type safe collection using generics.
+- Classes support generics by allowing a type variable to be included in their declaration.
+- The `<>` show the same type as stated (in this case string)
+- You cannot type a collection with a primitive data type eg int
+
+**HashMap Class**
+- A HashMap is a hash table based implementation of the map interface. This implementation provides all if the optional map operations, and permits null values and the null key.
+
+```java
+public static void main(String[] args) {
+     HashMap<String, Integer> userData = new HashMap<>();
+	 userData.put("Emma", 30);
+	 userData.put("John", null);
+	 userData.put("Millie", 17);
+
+	 Set<String> keys = userData.keySet();
+
+	 for (String key:keys){
+	 	System.out.println(key + "=" userData.get(key));
+	 }
+}
+```
+
+```
+Emma = 30
+John = null
+Millie = 17
+```
+
+**Part 2**
+
+__Relationships between objects__
+
+*Aggregation* - The object exists outside the other. It is created outside so it is passed as an argument.
+An animal object *is part of* a compound object (semantically) but the animal object can be shared and if the  compound object is deleted, the animal object isn't deleted.
+
+```java
+public class Compound {
+	private Animal dog;
+
+	public void setAnimal(Animal dog){
+		this.dog = dog;
+	}
+}
+```
+
+![Image](/lectures/dms/assets/1.png)
+
+*Composition* - The object only exists if the parent object exists, if the parent object is deleted then so is the child object.
+The zoo object owns the compound object. If the zoo object is deleted then the compound object is also deleted.
+
+```java
+public class Zoo {
+	private Compound dogArea = new Compound();
+}
+```
+
+![Image](/lectures/dms/assets/2.png)
+
+**Inheritance**
+A way of forming new classes based on existing classes. Has a "is-a" relationship.
+
+*Polymorphism* - A concept in object oriented programming. Method overloading and method overriding are two types of polymorphism.
+- *Method Overloading* - Methods with the same name co-exist in the same class but they must have different method signatures. Resolved during compile time (static binding).
+- *Method Overriding* - Methods with the same name is declared in parent and child class. Resolved during runtime (dynamic binding).
+
+```java
+public class Child extends Parent {
+	public Child(String name){
+		super(name);
+	}
+	
+	@Override
+	Public void eat() {
+		chew();
+	}
+}
+```
+
+The super keyword called the parent class' constructor.
+
+![Image](/lectures/dms/assets/3.png)
+
+**What is the difference between an abstract class and an interface**
+- Java abstract class can have instance methods that implement a default behaviour. May contain non-final variables.
+- Java interfaces have methods that are implicitly abstract and cannot have implementations. Variables are declared final by default.
+
+Interfaces are less restrictive when it comes to inheritance, interfaces can have many levels of inheritance where as a class can only have one level.

docs/lectures/dms/02_uml.md

@@ -0,0 +1,92 @@
+# Unified Modelling Language
+
+12/10/20
+
+**UML**: A specification defining a graphical language for visualising, specifying, constructing and documenting the artefacts of distributed object systems.
+
+Latest version: **2.6**
+
+**Benefits of UML**
+
+- Enhances communication and ensures the right communication 
+- Captures the logical software architecture independent of the implementation language.
+- Helps to manage the complexity
+- Enables reuse of design
+
+<img src="/lectures/dms/assets/4.png" alt="img" style="zoom:80%;" />
+
+
+## Object Orientated Analysis
+
+**Use case diagrams**
+
+- Describe a set of actions that some system should or can perform in collaboration with one or more external users of the system.
+- No attempt to represent an order or a number of executions.
+
+**Use case diagram components**
+
+`Actors` - Entities that interface with the system. Can be people or other systems.
+
+`Use case` - Based on user stories and represent what the actor wants your system to do for them. In the use case diagram only the use case name is represented.
+
+`Subject` - Classifier representing a business, software system, physical system or device under analysis design, or consideration.
+
+`Relationships` 
+
+> Relationships between use case and actor
+>
+> 1. Association indicates which actor indicates which use case
+>
+> Relationship between two use cases
+>
+> 1. Specifying common functionality and simplifying use case flows
+> 2. Using <<include>>  or <<extend>>
+
+**`<<include>>`**- multiple use cases share a piece of same functionality which is placed in a separate use case.
+
+**`<<extend>>`** - Used when activities might be performed as part of another activity but are not mandatory for a use case to run successfully.
+
+
+**Use case diagram of a fleet logistics management company**
+
+![image](/lectures/dms/assets/5.png)
+
+
+**Base Path** - The optimistic path (best case scenario)
+
+**Alternative Path** - Every other possible way the system can be used/abused. Includes perfectly normal alternate use, but also errors and failures.
+
+
+Use Case: `Borrow copy of book`
+
+> **Purpose**: The book borrower (BB) borrows a book from the library using the Library Booking System (LBS)
+>
+> **Pre-conditions**: 
+>
+> 1. The book must exist
+> 2. The book must be available
+>
+> **Base Path**:
+>
+> 1. LBS requests membership card
+> 2. BB provides membership card
+> 3. BB is logged in by LBS 
+> 4. LBS checks permissions / debts 
+> 5. LBS asks for presenting a book 
+> 6. BB presents a book 
+> 7. LBS scans RFID tag inside book 
+> 8. LBS updates records accordingly 
+> 9. LBS disables anti-theft device 
+> 10. BB is logged out by LBS 
+> 11. LBS confirms that process has been completed successfully
+>
+> **Alternative Path**
+>
+> 1. BB's card has expired: Step 3a: LBS must provide a message that card has expired; LBS must exit the use case
+>
+> **Post conditions for base path**
+>
+> **Base path** - BB has successfully borrowed the book & system is up to date.
+>
+> **Alternate Path 1** - BB was NOT able to borrow the book & system is up to date.
+

docs/lectures/osc/01_scheduling_algorithms.md

@@ -0,0 +1,124 @@
+05/10/20
+---
+The OS is responsible for *managing* and *scheduling processes*
+>Decide when to admit processes to the system (new -> ready)
+>
+>Decide which process to run next (ready -> run)
+>
+>Decide when and which processes to interrupt (running -> ready)
+
+It relies on the *scheduler* (dispatcher) to decide which process to run next, which uses a scheduling algorithm to do so.
+
+The type of algorithm used by the scheduler is influenced by the type of operating system e.g. real time vs batch.
+
+**Long Term**
+- Applies to new processes and controls the degree of multi-programming by deciding which processes to admit to the system when:
+    - A good mix of CPU and I/O bound processes is favourable to keep all resources as bust as possible
+    - Usually absent in popular modern OS
+
+**Medium Term**
+
+>Controls swapping and the degree of multi-programming
+
+**Short Term**
+- Decide which process to run next
+	- Manages the *ready queue*
+	- Invoked very frequency, hence must be fast
+	- Usually called in response to *clock interrupts*, *I/O interrupts*, or *blocking system calls*
+
+![Image](/lectures/osc/assets/1.png)
+
+**Non-preemptive** processes are only interrupted voluntarily (e.g. I/O operation or "nice" system call `yield()`)
+>Windows 3.1 and DOS were non-preemptive
+>
+>The issue with this is if the process in control goes wrong or gets stuck in a infinite loop then the CPU will never regain control.
+
+**Preemptive** processes can be interrupted forcefully or voluntarily
+
+>This required context switches which generate *overhead*, too many of them show me avoided.
+>
+>Prevents processes from monopolising the CPU
+>
+>Most popular modern OS use this kind.
+
+Overhead - wasted CPU cycles
+How can we objectively critic the OS?
+
+**User Oriented criteria**
+*Response time* minimise the time between creating the job and its first execution (time between clicking the button and it starting)
+*Turnaround time* minimise the time between creating the job and finishing it
+*Predictability* minimise the variance in processing times
+
+**System oriented criteria**
+*Throughput*: maximise the number of jobs processed per hour
+*Fairness*:
+
+    > Are processing power/waiting time equally distributed?
+    > Are some processes kept waiting excessively long - **starvation**
+
+### Different types of Scheduling Algorithms
+[NOTE: FCFS = FIFO]
+
+**First come first serve**
+Concept: a non-preemptive algorithm that operates as a strict queuing mechanism and schedules the processes in the same order that they were added to the queue.
+
+| Pros | Cons |
+| ----------- | ----------- |
+| Positional fairness | Favours long processes over short ones (think supermarket checkout) ||  |
+| Easy to implement | Could compromise resource utilisation |
+
+![Image](/lectures/osc/assets/2.png)
+
+**Shortest job first**
+A non-preemptive algorithm that starts processes in order of ascending processing time using a provided estimate of the processing
+
+| Pros | Cons |
+| ----------- | ----------- |
+| Always results an optimal turnaround time | Starvation might occur |
+| - | Fairness and predictability are compromised |
+| - | Processing times need to be known in advanced |
+
+![Image](/lectures/osc/assets/3.png)
+
+**Round Robin**
+A preemptive version of FCFS that focuses context switches at periodic intervals or time slices
+
+>Processes run in order that they were added to the queue.
+>Processes are forcefully interrupted by the timer. 
+
+| Pros | Cons |
+| ----------- | ----------- |
+| Improved response time | Increased context switching and overhead |
+| Effective for general purpose interactive/time sharing systems | Favours CPU processes over I/O |
+| - | Can reduce to FCFS |
+
+Exam 2013: Round Robin is said to favour CPU bound processes over I/O bound processes. Explain why this may be the case.
+>I/O processes will spend a lot of their allocated time waiting for data to come back from memory, therefore less processing can occur before the time slice runs out.
+
+If the time slice is only used partially the next process starts immediately
+The length of the time slice must be carefully considered.
+>A small time slice (~ 1ms) gives a good response time.
+>A large time slice (~ 1000ms) gives a high throughput.
+
+![Image](/lectures/osc/assets/4.png)
+
+**Priority Queue**
+A preemptive algorithm that schedules processes by priority 
+
+>A round robin is used for processes with the same priority level
+>The process priority is saved in the process control block
+
+| Pros | Cons |
+| ----------- | ----------- |
+|Can prioritise I/O bound jobs | Low priority processes might suffer from starvation |
+
+Low priority starvation only happens when a static priority level is used.
+You could give higher priority processes a larger time slice to improve efficiency
+
+![Image](/lectures/osc/assets/5.png)
+
+Exam Q 2013: Which algorithms above lead to starvation? 
+>Shortest job first and highest priority first. 
+
+![Image](/lectures/osc/assets/6.png)
+![Image](/lectures/osc/assets/7.png)

docs/lectures/osc/02_threads.md

@@ -0,0 +1,128 @@
+08/10/20
+---
+
+A process consists of two **fundamental** units
+
+1. Resources
+	- A logical address space containing the process image (program, data, heap, stack)
+	- Files, I/O devices, I/O channels
+2. Execution trace e.g. an entity that gets executed
+
+A process can share its resources between multiple execution traces, e.g multiple threads running in the same resource environment.
+
+![Image](/lectures/osc/assets/9.png)
+
+Every thread has its own *execution context* (e.g. program counter, stack, registers).
+All threads have **access** to the process' **shared resources**
+
+>e.g. Files; if one thread opens a file then all threads have access to it
+>
+>Same with global variables, memory etc
+
+Similar to processes, threads have:
+**States**, **transitions** and a **thread control block**
+
+![Image](/lectures/osc/assets/a.png)
+
+The *registers*, *stack* and *state* are all specific to the registers. When a context switch occurs they must be stored in the **thread control block**.
+
+Threads incur less overhead to create/terminate/switch processes. This is because the address space remains the same for threads of the same process.
+>When switching from thread A to thread B, the computer doesn't need to worry about updating the memory management unit as they're using the same memory layout.
+>
+>This makes switching threads very quick
+
+Some CPU's have direct **hardware support** for **multi-threading**.
+>With hyper threading and multi-threading, the thread's execution context isn't saved to the thread control block. Instead the CPU stops using one thread and starts using another.
+>
+>This decreases overhead as the execution context doesn't need to be saved and reloaded.
+
+1. **Inter-thread communication** is easier and faster that **inter-process** communication (threads share memory by default)
+2. **No protection boundaries** are required in the address space (threads are cooperating, they belong to the same user and have the same goal)
+3. Synchronisation has to be considered carefully.
+
+If you opened word and excel, you wouldn't want them running on threads as you don't want word to have access to the memory excel is accessing. However if you just had word open the spell check and graphics libraries would all run on threads as they work towards a common goal.
+
+### Why use threads
+
+1. Multiple **related activities** apply to the **same resources**, these resources should be accessible.
+2. Processes will often contain multiple **blocking tasks**
+	1. I/O operations (thread blocks, interrupt marks completion)
+	2. Memory access: pages faults are result in blocking
+
+Such activities should be carried out in parallel on threads. e.g. web-servers, word processors, processing large data volumes etc
+
+**User** threads - happen inside the user space, the OS doesn't need to do anything.
+
+>**Thread management** (creating, destroying, scheduling, thread control block manipulation) is carried out in user space with the help of a user library.
+>
+>The process maintains a thread table managed by the run-time system without the kernel's knowledge (similar to a process table and used for thread switching)
+
+**Kernel** threads - ask the OS to create a tread for the user and give it to the user.
+**Hybrid** implementations - is what is used in windows 10
+
+![Image](/lectures/osc/assets/d.png)
+
+**Pros and cons of user threads**
+
+| Pros | Cons |
+| ----------- | ----------- |
+| Threads in user space don't require mode switches | Blocking system calls suspend all running threads   |
+| Full control over the thread scheduler | No true parallelism (the processes still scheduled on a single CPU) |
+| OS independent | Clock interrupts (user threads are non-preemptive) |
+|  -  | Page faults result in blocking the process|
+
+The user threads don't share the memory management unit therefore if a thread tries to access memory that isn't loaded in the MMU then a page fault will occur, these occur often.
+
+**Kernel Threads**
+The kernel manages the threads, user application accesses threading facilities through **API** and **system calls**
+>The **thread table** is in the kernel, containing the thread control blocks.
+>
+>If a thread blocks, the kernel chooses a thread from the same or different process.
+
+Advantages:
+>**True parallelism** can be achieved
+>No run time system needed
+
+However frequent **mode switches** take place, resulting in a lower performance.
+
+![Image](/lectures/osc/assets/e.png)
+
+Kernel threads are slower to create and sync that user level however user level cannot exploit parallelism.
+
+**Hybrid Implementation**
+>User threads are **multiplexed** onto kernel threads
+>
+>Kernel sees and schedules the kernel threads
+>
+>User application sees user threads and creates/schedules these (an unrestricted number)
+
+![Image](/lectures/osc/assets/f.png)
+
+Thread libraries provide an API for managing threads
+Thread libraries can be implemented
+>Entirely in user space (user threads)
+>
+>Based off system calls (rely on the kernel)
+
+Examples of thread APIs include **POSIX PThreads**, windows threads and Java threads
+
+`pthread_create`  - Create new thread
+`pthread_exit`   - Exit existing thread
+`pthread_join`   - Wait for thread with ID
+`pthread_yield`   - Release CPU
+`pthread_attr_init`   - Thread Attributes (e.g. priority)
+`pthread_attr_destroy`   - Release Attributes
+
+$ ~ man `pthread_create` returns the help page
+
+![img](/lectures/osc/assets/g.png)
+
+```
+$ ~ HELLO from thread 10
+$ ~ HELLO from thread 10
+$ ~ HELLO from thread 10
+etc
+```
+
+This is because by the time the thread is created `i` has already iterated to 10.
+You cannot guarantee the first thread you create will be the first to run.

docs/lectures/osc/03_processes4.md

@@ -0,0 +1,141 @@
+09/10/20
+
+
+**Multi-level scheduling algorithms**
+>Nothing is stopping us from using different scheduling algorithms for individual queues for each different priority level.
+>
+> - **Feedback queues** allow priorities to change dynamically i.e. jobs can move between queues
+	1. Move to **lower priority queue** if too much CPU time is used
+	2. Move to **higher priority queue** to prevent starvation and avoid inversion of control.
+
+Exam 2013: Explain how you would prevent starvation in a priority queue algorithm?
+
+![alt text](/lectures/osc/assets/h.png)
+
+The solution to this is to momentarily boost thread A's priority level, this will let A do what it what's to do and release resource X so that B and C can run. 
+
+Priority boosting helps avoid control inversion.
+
+**Defining characteristics of feedback queues**
+
+1. The **number of queues**
+2. The scheduling algorithms used for individual queues
+3. **Migration policy** between queues
+4. Initial **access** to the queues
+
+Feedback queues are highly configurable and offer significant flexibility.
+
+<ins>**Windows 7**</ins>
+
+> An interactive system using a pre-emptive scheduler with dynamic priority levels.
+>
+> Two priority classes with 16 different priority levels exist.
+>
+>   1. **Real time** processes/threads have a fixed priority level. (These are the most important)
+>   2. **Variable** processes/threads can have their priorities **boosted temporarily**.
+>
+> A **round robin** is used within the queues.
+
+
+
+![alt text](/lectures/osc/assets/i.png)
+
+![alt text](/lectures/osc/assets/j.png)
+
+If you give a couple of the threads the highest priority level, you can freeze your computer. (causes starvation for low priority threads)
+
+
+
+<ins>**Scheduling in Linux**</ins>
+
+> Process scheduling has evolved over different versions of Linux to account for multiple processors/cores, processor affinity, and **load balancing** between cores.
+>
+> Linux distinguishes between two types of tasks for scheduling:
+>
+> 1. **Real time tasks** (to be POSIX compliant)
+>    1. Real time FIFO tasks
+>    2. Real time Round Robin tasks
+> 2. **Time sharing tasks** using a pre-emptive approach (similar to variable in Windows)
+>
+> The most recent scheduling algorithm in Linux for time sharing tasks is the **completely fair scheduler**
+
+**Real time FIFO** have the highest priority and are scheduled with a **FCFS approach** using a pre-emption if a higher priority job shows up.
+
+**Real time round robin tasks** are preemptable by clock interrupts and have a time slice associated with them.
+
+Both ways *cannot* guarantee hard deadlines.
+
+**Time sharing tasks**
+
+> The CFS (completely fair scheduler) **divides the CPU time** between all processes and threads.
+>
+> <ins>If all N processes/threads have the same priority. </ins>
+>
+> ​	They will be allocated a time slice equal to 1/N times the available CPU time.
+>
+> The length of the **time slice** and the available CPU time are based on the **targeted latency** (every process/thread should run at least once in this time)
+>
+> If N is very large, the **context switch time will be dominant**, hence a lower bound on the time slice is imposed by the minimum granularity.
+>
+> ​	A process/thread's time slice can be no less than the **minimum granularity.**
+
+A **weighting scheme** is used to take difference priorities into account.
+
+<img src="/lectures/osc/assets/k.png" alt="alt text" style="zoom:60%;" />
+
+The tasks with the **lowest proportional amount** of "used CPU time" are selected first. (Shorter tasks picked first if Wi is the same).
+
+
+
+**Shared Queues**
+
+A single of multi-level queue **shared** between all CPUs
+
+| Pros                         | Cons                                                     |
+| ---------------------------- | -------------------------------------------------------- |
+| Automatic **load balancing** | Contention for the queues (locking is needed)            |
+|                              | **Cache** becomes invalid when moving to a different CPU |
+
+Windows will allocate the **highest priority threads** to the individual CPUs/cores.
+
+
+
+**Private Queues**
+
+> Each CPU has a private (set) of queues
+
+| Pros                                         | Cons                                                         |
+| -------------------------------------------- | ------------------------------------------------------------ |
+| CPU affinity is automatically satisfied      | **Less load balancing** (one queue could have 1000 tasks while the other has 4) |
+| **Contention** for shared queue is minimised |                                                              |
+
+**Related vs. Unrelated threads**
+
+> **Related**: multiple threads that communicated with one another and **ideally run** together
+>
+> **Unrelated** processes threads that are **independent**, possibly started by **different users** running different programs.
+
+![alt text](/lectures/osc/assets/l.png)
+
+Threads belong to the same process are are cooperating e.g. they **exchange messages** or **share information**
+
+The aim is to get threads running as much as possible, at the **same time across multiple CPU**s.
+
+**Space Sharing**
+
+> N threads are allocated to N **dedicated CPUs**
+>
+> N threads are kept waiting until N CPUs are available
+>
+> **Non pre-emptive** i.e. blocking calls result in idle CPUs 
+>
+> N can be dynamically adjusted to match processor capacity.
+
+**Gang Scheduling**
+
+> Time slices are synchronised and the scheduler groups threads together to run simultaneously
+>
+> A pre-emptive algorithm
+>
+> **Blocking threads** result in idle CPU (If a thread blocks, the rest of the time slice will be unused due the time slice synchronisation across all CPUs)
+

docs/lectures/osc/04_concurrency1.md

@@ -0,0 +1,172 @@
+12/10/20
+
+1. Threads and processes execute concurrently or in parallel and can **share resources** (like devices, memory, variables, data structures etc)
+2. A process/thread can be interrupted at any point in time. The process "state" (including registers) is saved in the **process control block**
+
+The outcome of programs may be unpredictable:
+
+> Sharing data can lead to **inconsistencies**.
+>
+> The **outcome of execution** may **depend on the order** in which instructions are carried out.
+
+```c
+#include <stdio.h>
+#include <pthread.h>
+int counter = 0;
+void * calc(void * number_of_increments) {
+	int i;
+	for(i = 0; i < *((int*) number_of_increments);i++)
+		counter++;
+	}
+int main() {
+	int iterations = 50000000;
+	pthread_t tid1,tid2;
+	pthread_create(&tid1, NULL, calc, (void *) &iterations);
+	pthread_create(&tid2, NULL, calc, (void *) &iterations);
+	pthread_join(tid1,NULL);
+	pthread_join(tid2,NULL);
+	printf("The value of counter is: %d\n", counter);
+}
+```
+
+This piece of code creates two threads, and points them towards the `calc` function. The `pthread_join(tid1,NULL);`  line is waiting until thread 1 is finished until the code moves on.
+
+
+
+Counter++ consists of three separate actions.
+
+1. *read* the value of counter from memory and **store it in a register**
+2. *add* one to the value in the register
+3. *store* the value of the register **in counter** in memory
+
+The above actions are **not** "atomic". This means they can be interrupted by the timer.
+
+![image](/lectures/osc/assets/p.png)
+
+TCB - *Thread Control Block*
+
+This is what could happen if the threads are not interrupted.
+
+However the thread control block could be out of date by the time the thread starts running again. For example *counter* could be 2 but the thread control block still has the old value of *counter*.
+
+
+
+The problem is that simple instructions in c are actually multiple instructions in assembly code. Another example is `print()`
+
+```c	
+void print() {
+	chin = getchar();
+	chout = chin;
+	putchar(chout);
+}
+```
+
+If the two threads are **interleaved** one after the other, there is no issue.
+
+![img](/lectures/osc/assets/q.png)
+
+However if **interleaved** like this they do interact. The global variable used to store the character in thread 1, is overwritten when thread 2 runs. This means 1+1+1 = 2 
+
+## Bounded Buffer
+
+> Consider a **bounded buffer** in which N items can be stored
+>
+> A **counter** is maintained to count the number of items currently in the buffer. **Increment** when something is added and **decremented** when an item is removed.
+>
+> Similar **concurrency problems** as with the calculation of sums happen in the bounded buffer which is a consumer problem.
+
+```c	
+// producer
+while (true) {
+	//while buffer is full
+	while (counter == BUFFER SIZE); /* do nothing */
+	// Produce item
+	buffer[in] = new_item;
+	in = (in + 1) % BUFFER_SIZE;
+	counter++;
+}
+// consumer
+while (true) {
+	// wait until items in buffer
+	while (counter == 0); /* do nothing */
+	// Consume item
+	consumed = buffer[out];
+	out = (out + 1) % BUFFER_SIZE;
+	counter--;
+}
+```
+
+This is a circular queue, there's a start and end pointer (*in* and *out*). The shared counter is being manipulated from 2 different places which can go wrong.
+
+## Race Conditions
+
+A **race conditions occurs** when multiple threads/processes **access shared data** and the result is dependent on **the order in which the instructions are interleaved**.
+
+### Concurrency within the OS
+
+> **Kernels are pre-emptive** 
+> **Multiple processes/threads are running** in the kernel.
+> Kernel processes can be **interrupted** at any point.
+>
+> The kernel maintains **data structures**
+>
+> 1. These data structures are accessed **concurrently.**
+> 2. These can be subject to **concurrency issues.**
+>
+> The OS must make sure that interactions within the OS do not result in race conditions.
+>
+> Processes **share resources** including memory, files, processor time, printers etc.
+>
+> The OS must:
+>
+> 1. provide **locking mechanisms** to implement **mutual exclusion** and **prevent starvation and deadlocks.**
+> 2. Allocate and deallocate these resources safely.
+
+A **critical section** is a set of instructions in which **shared resources** between processes/threads **are changed**.
+
+**Mutual exclusion** must be enforced for **critical sections**.
+
+> Only **one process at a time** should be in the critical section (mutual exclusion)
+>
+> Processes have to **get "permission"** before entering their critical section 
+>
+> 1. Request a lock
+> 2. Hold the lock
+> 3. Release the lock
+
+Any solution to the **critical section problem** must satisfy the following requirements:
+
+1. **Mutual exclusion** - only one process can be in its critical section at any one point in time.
+2. **Progress** - any process must be able to enter its critical section at some point in time. (a process/thread has a right to enter its critical section at a point in time). If there is no thread/process in the critical section there is no reason for the currently thread not to be allowed in the **critical section**.
+3. **Fairness/bounded waiting** - fairly distributed waiting times/processes cannot be made to wait indefinitely.
+
+These requirements have to be satisfied, independent of the order in which sequences are executed.
+
+### Enforcing Mutual Exclusion
+
+**Approaches** for mutual exclusion can be:
+
+1. Software based - Peterson's solution
+2. Hardware based - `test_and_set()` `swap_and_comapare()`
+
+Deadlocks have to be prevented as well.
+
+#### Deadlock Example 
+
+A set of processes/threads is *deadlocked* if each process/thread in the set is waiting for an event that only the other process/thread in the set can cause.
+
+Each **deadlocked process/thread** is waiting for a resource held by another deadlocked process/thread (which cannot run and hence release the resource).
+
+* Assume that X and Y are **mutually exclusive resources**.
+* Thread A and B need to **acquire both resources** and request them in oppose orders.
+
+![img](assets/r.png)
+
+**Four conditions** must hold for a deadlock to occur
+
+1. **Mutual exclusion** - a resource can be assigned to at most one process at a time.
+2. **Hold and wait condition** - a resource can be held while requesting new resources.
+3. **No pre-emption** - resources cannot be forcefully taken away from a process
+4. **Circular wait** - there is a circular chain of two or more processes,, waiting for a resource held by the other processes.
+
+**No deadlocks** can occur if one of the conditions isn't met.

docs/lectures/osc/05_concurrency2.md

@@ -0,0 +1,161 @@
+15/10/20
+
+## Peterson's Solution
+
+**Peterson's solution** is a **software based** solution which worked well on **older machines** 
+
+Two **shared variables** are used
+
+1. *turn* - indicates which process is next to enter its critical section.
+2. *Boolean flag [2]*  - indicates that a process is ready to enter its critical section
+
+* Peterson's solution can be used over multiple processes or threads
+* Peterson's solution for two processes satisfies all **critical section requirements** (mutual exclusion, progress, fairness)
+
+`````c
+do {
+	flag[i] = true; // i wants to enter critical section
+	turn = j; // allow j to access first
+	while (flag[j] && turn == j);
+	// whilst j wants to access critical section
+	// and its j’s turn, apply busy waiting
+	// CRITICAL SECTION
+	    counter++
+	flag[i] = false;
+	// remainder section
+} while (...);
+`````
+
+**Figure**: *Peterson's solution for process i*
+
+```c	
+do {
+	flag[j] = true; // j wants to enter critical section
+	turn = i; // allow i to access first
+	while (flag[i] && turn == i);
+    // whilst i wants to access critical section
+    // and its i’s turn, apply busy waiting
+    // CRITICAL SECTION
+    	counter++
+    flag[j] = false;
+    // remainder section
+} while (...);
+```
+
+**Figure**: *Peterson's solution for process j*
+
+Even when these two processes are interleaved, its unbreakable as there is always a check to see if the other process is in the critical section. 
+
+### Mutual exclusion requirement:
+
+The variable turn can have at most one value at a time.
+
+* Both `flag[i]` and `flag[j]` are *true* when they want to enter their critical section
+* Turn is a **singular variable** that can store only one value
+* Hence `while (flag[i] && turn == i);` or `while (flag[j] && turn == j);` is true and at most one process can enter its critical section (mutual exclusion)
+
+**Progress**: any process must be able to enter its critical section at some point in time
+
+> Processes/threads in the **remaining code** do not influence access to critical sections
+>
+> If a process *j* does not want to enter its critical section
+>
+> * `flag[j] == false`
+> * `white (flag[j] && turn == j)` will terminate for process *i*
+> * *i* enters critical section
+
+### Fairness/bounded waiting
+
+Fairly distributed waiting times/process cannot be made to wait indefinitely.
+
+> If P<sub>i</sub> and P<sub>j</sub> both want to enter their critical section
+>
+> * `flag[i] == flag[j] == true`
+> * `turn` is either *i* or *j* assuming that `turn == i` *i* enters it's critical section
+> * *i* finishes critical section `flag[i] = false` and then *j* enters its critical section.
+
+Peterson's solution works when there is two or more processes. Questions on Peterson's solution with more than two solutions is not in the spec.
+
+
+**Disable interrupts** whilst **executing a critical section** and prevent interruptions from I/O devices etc.
+
+For example we see `counter ++` as one instruction however it is three instructions in assembly code. If there is an interrupt somewhere in the middle of these three instructions bad things happen.
+
+```c
+register = counter;
+register = register + 1;
+counter = register;
+```
+
+Disabling interrupts may be appropriate on a **single CPU machine**, not on a multi-core processor though. This means multiple cores can take a value from memory, manipulate that value (in this example iterating it) whilst not knowing the value has already changed on a different core and write back the wrong value to memory. This can lead to `1+1+1=2`.
+
+### Atomic Instructions
+
+> Implement `test_and_set()` and `swap_and_compare()` instructions as a **set of atomic (uninterruptible) instructions**
+>
+> * Reading and setting the variables is done as **one complete set of instructions**
+> * If `test_and_set()` / `sawp_and_compare()` are called **simultaneously** they will be executed sequentially.
+>
+> They are used in combination with **global lock variables**, assumed to be `true (1) ` is the lock is in use.
+
+#### Test_and_set()
+
+```c
+// Test and set method
+boolean test_and_set(boolean * bIsLocked) {
+	boolean rv = *bIsLocked;
+	*bIsLocked = true;
+	return rv;
+}
+
+// Example of using test and set method
+do {
+	// WHILE the lock is in use, apply busy waiting
+	while (test_and_set(&bIsLocked));
+	// Lock was false, now true
+	// CRITICAL SECTION
+	...
+	bIsLocked = false;
+	...
+	// remainder section
+} while (...)
+```
+
+* `test_and_set()` must be **atomic**.
+* If two processes are using `test_and_set()` and are interleaved, it can lead to two processes going into the critical section.
+
+```c
+// Compare and swap method
+int compare_and_swap(int * iIsLocked, int iExpected, int iNewValue) {
+	int iTemp = *iIsLocked;
+	if(*iIsLocked == iExpected)
+		*iIsLocked = iNewValue;
+	return iTemp;
+}
+
+// Example using compare and swap method
+do {
+	// While the lock is in use (i.e. == 1), apply busy waiting
+	while (compare_and_swap(&iIsLocked, 0, 1));
+	// Lock was false, now true
+	// CRITICAL SECTION
+    ...
+    iIsLocked = 0;
+    ...
+    // remainder section
+} while (...);
+```
+
+
+
+`test_and_set()` and `swap_and_compare()` are **hardware instructions** and **not directly accessible** to the user.
+
+**Disadvantages**:
+
+* **Busy waiting** is used. When the process is doing **nothing** just sitting in a loop, the process is still eating up processor time. If I know the process won't be **waiting for long busy waiting is beneficial** however if it is a long time a blocking signal will be sent to the process.
+* **Deadlock** is possible e.g when two locks are requested in opposite orders in different threads.
+
+The OS uses the hardware instructions to implement higher level mechanisms/instructions for mutual exclusion i.e. **mutexes** and **semaphores**.
+
+
+

docs/lectures/osc/06_concurrency3.md

@@ -0,0 +1,234 @@
+16/10/20 
+
+## Mutexes
+
+**Mutexes** are an approach for mutual exclusion **provided by the operating system** containing a Boolean lock variable to indicate availability.
+
+> The lock variable is set to **true** if the lock is available
+>
+> Two atomic functions are used to **manipulate the mutex**
+>
+> 1. `acquire()` - called **before**  entering a critical section, Boolean set to **false**
+> 2. `release()` - called **after** exiting the critical section, Boolean set to **true** again.
+
+```c
+acquire() {
+	while(!available); // busy wait
+	available = false;
+}
+```
+
+```c
+release() { 
+    available = true; 
+}
+```
+
+`acquire()` and `release()` must be **atomic instructions** .
+
+* No **interrupts** should occur between reading and setting the lock.
+* If interrupts can occur, the follow sequence could occur.
+
+```c
+T_i => lock available
+... 								T_j => lock available
+... 								T_j sets lock
+T_i sets lock 						...
+```
+
+The process/thread that acquires the lock must **release the lock** - in contrast to semaphores.
+
+| Pros                                                         | Cons                                                         |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| Context switches can be **avoided**.                         | Calls to `acquire()` result in **busy waiting**. Shocking performance on single CPU systems. |
+| Efficient on multi-core systems when locks are **held for a short time**. | A thread can waste it's entire time slice busy waiting.      |
+
+![img](/lectures/osc/assets/S.png)
+
+
+
+## Semaphores
+
+> **Semaphores** are an approach for **mutual exclusion** and **process synchronisation** provided by the operating system.
+>
+> * They contain an **integer variable**
+> * We distinguish between **binary** (0-1) and **counting semaphores** (0-N)
+>
+> Two **atomic functions** are used to manipulate semaphores** 
+>
+> 1. `wait()` - called when a resource is **acquired** the counter is decremented.
+> 2. `signal()` / `post()` is called when the resource is **released**.
+>
+> **Strictly positive values** indicate that the semaphore is available, negative values indicate the number of processes/threads waiting.
+
+```c
+//Definition of a Semaphore
+typedef struct {
+	int value;
+	struct process * list;
+} semaphore;
+```
+
+```c
+wait(semaphore * S) {
+	S->value--;
+	if(S->value < 0) {
+		add process to S->list
+		block(); // system call
+    }
+}
+```
+
+```c
+post(semaphore * S) {
+	S->value++;
+	if (S->value <= 0) {
+        remove a process P from S->list;
+        wakeup(P); // system call
+	}
+}
+```
+
+```
+Thread N
+...
+...
+wait(&s)
+...
+(wakeup)
+...
+post(&s)
+```
+
+Calling `wait()` will **block** the process when the internal **counter is negative** (no busy waiting)
+
+1. The process **joins the blocked queue**
+2. The **process/thread state** is changed from running to blocked
+3. Control is transferred to the process scheduler.
+
+Calling `post()` **removes a process/thread** from the blocked queue if the counter is less than or equal to 0.
+
+1. The process/thread state is changed from ***blocked** to **ready**
+2. Different queuing strategies can be employed to **remove** process/threads e.g. FIFO etc
+
+The negative value of the semaphore is the **number of processes waiting** for the resource.
+
+`block()` and `wait()` are system called provided by the OS.
+
+`post()` and `wait()` **must** be **atomic**
+
+> Can be achieved through the use of mutexes (or disabling interrupts in a single CPU system)
+>
+> Busy waiting is moved from the **critical section** to `wait()` and `post()` (which are short anyway - the original critical sections themselves are usually much longer)
+
+We must lock the variable `value`.
+
+```c
+post(semaphore * S) {
+    // lock the mutex
+	S->value++;
+	if (S->value <= 0) {
+        remove a process P from S->list;
+        wakeup(P); // system call
+	}
+    //unlock mutex
+}
+```
+
+Semaphores put your code to sleep. Mutexes apply busy waiting to user code.
+
+Semaphores within the **same process** can be declared as **global variables** of the type `sem_t`
+
+> * `sem_init()` - initialises the value of the semaphore.
+> * `sem_wait()` - decrements the value of the semaphore.
+> * `sem_post()` - increments the values of the semaphore.
+
+![img](/lectures/osc/assets/t.png)
+
+Synchronising code does result in a **performance penalty**
+
+> * Synchronise only **when necessary**
+>
+> * Synchronise as **few instructions** as possible (synchronising unnecessary instructions will delay others from entering their critical section)
+
+```c
+void * calc(void * increments) {
+    int i, temp = 0;
+    for(i = 0; i < *((int*) increments);i++) {
+    	temp++;
+    }
+    sem_wait(&s);
+    sum+=temp;
+    sem_post(&s);
+}
+```
+
+**Figure**: optimised `calc` function
+
+#### Starvation
+
+> Poorly designed **queueing approaches** (e.g. LIFO) may results in fairness violations
+
+#### Deadlock
+
+> Two or more processes are **waiting indefinitely** for an event that can be caused only by one of the waiting processes or thread.
+
+#### Priority Inversion
+
+> Priority inversion happens when a high priority process (`H`) has to wait for a **resource** currently held by a low priority process (`L`)
+>
+> Priority inversion can happen in chains e.g. `H` waits for `L` to release a resource and L is interrupted by a medium priority process `M`. 
+>
+> This can be avoided by implementing priority inheritance to boost `L` to the `H`'s priority.
+
+
+
+## The Producer and Consumer Problem
+
+> * Producer(s) and consumer(s) share N **buffers** (an array) that are capable of holding **one item each** like a printer queue.
+>   * The buffer can be of bounded (size N) or **unbounded size**.
+>   * There can be one or multiple consumers and or producers.
+> * The **producer(s)** add items and **goes to sleep** if the buffer is **full** (only for a bounded buffer)
+> * The **consumer(s)** remove items and **goes to sleep** if the buffer is **empty**
+
+The simplest version of this problem has **one producer**, **one consumer** and a buffer of **unbounded size**.
+
+* A counter (index) variable keeps track of the **number of items in the buffer**.
+* It uses **two binary semaphores:**
+  * `sync` **synchronises** access to the **buffer** (counter) which is initialised to 1.
+  * `delay_consumer` ensures that the **consumer** goes to **sleep** when there are no items available, initialised to 0.
+
+![img](/lectures/osc/assets/U.png)
+
+It is obvious that any manipulations of count will have to be **synchronised**.
+
+**Race conditions** will still exist:
+
+> When the consumer has **exhausted the buffer** (when `items == 0`), it should go to sleep but the producer increments `items` before the consumer checks it.
+>
+> * Consumer has removed the **last element**
+> * The producer adds a **new element**
+> * The consumer should have gone to sleep but no longer will
+> * The consumer consumes **non-existing elements**
+>
+> **Solutions**:
+>
+> * Move the consumers' if statement inside the critical section
+
+
+
+### Producers and Consumers Problem
+
+The previous code (one consumer one producer) is made to work by **storing the value of** `items` 
+
+A different variant of the problem has `n` consumers and `m` producers and a fixed buffer size `N`. The solution is based on **3 semaphores**
+
+1. `sync` - used to **enforce mutual exclusion** for the buffer
+
+2. `empty` - keeps track of the number of empty buffers, initialised to `N`
+
+3. `full` - keeps track of the number of **full buffers** initialised to 0.
+
+The `empty` and `full` are **counting semaphores** and represent **resources**.
+
+![img](/lectures/osc/assets/V.png)

docs/lectures/osc/07_concurrency4.md

@@ -0,0 +1,114 @@
+19/10/20 
+
+## The Dining Philosophers Problem
+
+<img src="/lectures/osc/assets/w.png" alt="img" style="zoom:67%;" />
+
+The problem is defined as:
+
+* **Five philosophers** are sitting on a round table
+* Each one has a plate of spaghetti
+* The spaghetti is too slippery, and each philosopher **needs 2 forks** to be able to eat
+* When hungry, the philosopher tries to acquire the forks on his left and right.
+
+Note that this reflects the general problem of **sharing a limited set** of resources (forks) between a **number of processes** (philosophers).
+
+### Solution 1
+
+**Forks** are represented by **semaphores** (initialised to 1)
+
+* 1 if the fork is available: the philosopher can continue.
+* 0 if the fork is unavailable: the philosopher goes to **sleep** if trying to acquire it.
+
+Solution: Every philosopher picks up one fork and waits for the second fork to become available (without putting the first one down).
+
+This solution will **deadlock** every time.
+
+> * The deadlock can be avoided by exponential decay. This is where a philosopher puts down their fork and waits for a random amount of time. (this is how Ethernet systems avoid data collisions)
+> * Just **add another fork**
+
+### Solution 2
+
+**One global mutex** set by a philosopher when they want to eat (only **one can eat at a time**)
+
+*Question*: Can I initialise the value of the `eating` semaphore to 2 to create more parallelism?
+
+Setting the semaphore to 2 allows the possibility of 2 philosophers to eat at one time. If these two philosophers are sitting next to each other then they will try to grab the same fork. The code will not deadlock, however only one (sometimes two) philosopher(s) is able to eat.
+
+
+
+### Solution 3
+
+A more sophisticated solution is necessary to allow **maximum parallelism**
+
+The solution uses:
+
+> * `state[N]` : one **state variable** for every philosopher (`THINKING` `HUNGRY` and `EATING`)
+> * `phil[N] ` : one **semaphore per philosopher** (i.e. **not forks** initialised to 0)
+>   * The philosopher goes to sleep if one of their neighbours are eating
+>   * The neighbours wake up the philosopher if they have finished eating
+> * `sync` : one **semaphore/mutex** to enforce **mutual exclusion** of the critical section (while updating the **states** of `hungry` `thinking` and `eating`)
+> * A philosopher can only **start eating** if their neighbours are **not eating**.
+
+![img](/lectures/osc/assets/x.png)
+
+#### Code for Solution 3
+
+```c
+#define N 5
+#define THINKING 1
+#define HUNGRY 2
+#define EATING 3
+
+int state[N] = {THINKING, THINKING, THINKING, THINKING, THINKING};
+sem_t phil[N]; // sends philosopher to sleep
+sem_t sync;
+
+void * philosopher(void * id) {
+    int i = *((int *) id);
+    
+    while(1) {
+        printf("%d is thinking\n", i);
+        take_forks(i);
+        printf("%d is eating\n", i);
+        put_forks(i);
+    }
+}
+
+void take_forks(int i) {
+    sem_wait(&sync);
+    state[i] = HUNGRY;
+    test(i); //checks surrounding philosophers to see if its ok to eat
+    sem_post(&sync);
+    sem_wait(&phil[i]); //1 -> 0
+}
+
+void test(int i) {
+    int left = (i + N - 1) % N;
+    int right = (i + 1) % N;
+    if(state[i] == HUNGRY && state[left] != EATING && state[right] != EATING) {
+        state[i] = EATING;
+        sem_post(&phil[i]); //0 -> 1
+    }
+}
+
+void put_forks(int i) {
+    int left = (i + N - 1) % N;
+    int right = (i + 1) % N;
+    sem_wait(&sync);
+    state[i] = THINKING;
+    test(left);
+    test(right);
+    sem_post(&sync);
+}
+
+void test(int i) {
+    int left = (i + N - 1) % N;
+    int right = (i + 1) % N;
+    if(state[i] == HUNGRY && state[left] != EATING && state[right] != EATING) {
+        state[i] = EATING;
+        sem_post(&phil[i]);
+    }
+}
+```
+

docs/lectures/osc/08_concurrency6.md

@@ -0,0 +1,120 @@
+23/10/20 
+
+## The readers-writers Problem
+
+* Reading a record (or a variable) can happen in parallel without problems, **writing needs synchronisation** (or exclusive access).
+
+* Different solutions exist:
+
+  > * Solution 1: naive implementation with limited parallelism
+  > * Solution 2: **readers** receive **priority**. No reader is kept waiting unless a writer already has access (writers may starve).
+  > * Solution 3: **writing** is performed as soon as possible (readers may starve).
+
+### Solution 1: No parallelism
+
+```c
+void * reader(void * arg)
+{
+	while(1)
+    {
+        pthread_mutex_lock(&sync);
+        printf("reading record\n");
+        pthread_mutex_unlock(&sync);
+    }
+}
+void * writer(void * writer)
+{
+    while(1)
+    {
+        pthread_mutex_lock(&sync);
+        printf("writing\n");
+        pthread_mutex_unlock(&sync);
+    }
+}
+```
+
+This prevents **parallel reading**.
+
+### Solution 2: Allows parallel reading
+
+A correct implementation requires:
+
+> `iReadCount`: an integer tracking the number of readers
+>
+> * if `iReadCount` > 0: writers are blocked `sem_wait(rwSync)`
+> * if `iReadCount` == 0: writers are released `sem_post(rwSync)`
+> * if already writing, readers must wait
+>
+> `sync`: a mutex for mutual exclusion of `iReadCount`.
+>
+> `rwSync` : a semaphore that synchronises the readers and writers, set by the first/last reader.
+
+![img](/lectures/osc/assets/y.png)
+
+`sync` is used to `mutex_lock` and `mutex_unlock` when the `iReadCount` is being modified.
+
+When `iReadCount == 1`, the `sem_wait(&rwSync)` is used to block the writer from writing. Further down in the code when `iReadCount == 0`, the `sem_post(&rwSync)` is called to 'wake up' the writer, so that it can write.
+
+When we say 'send process to sleep' or 'wake up a process' we actually mean: move that process from the blocked queue to the ready queue (or visa versa).
+
+If the `iReadCount == 1` is run when the writer is writing. The `sem_wait(&rwSync)` will go from 0 -> -1, forcing the reader to go to sleep. As soon as the writer is done, the `sem_post(&rwSync)` is run meaning it goes from -1 -> 0, which wakes the reader up.
+
+Unless `iReadCount` reaches 0, writing will not happen. **This means writers can easily starve if there are multiple readers**.
+
+### Solution 3: Gives priority to the writer
+
+**Solution 3 uses:**
+
+> * `iReadCount` and `iWriteCount`: to keep track of the number of readers and writers.
+> * `sRead`/`sWrite`: to synchronise the **reader/writer's critical section**.
+> * `sReadTry`: to **stop readers** when there is a **writer waiting**.
+> * `sResource`: to **synchronise** the resource for **reading/writing**.
+
+![img](/lectures/osc/assets/z.png)
+
+[explanation time stamp 43:35]
+
+`sRead` and `sWrite` are used whenever `iReadCount` and `iWriteCount` are used respectively. Unlike the mutex in the last example it is important that the same semaphore variable isn't used for both `iReadCount` and `iWriteCount`.
+
+There is no reason the read and write count cannot be changed at the same time. If you were to use the same semaphore then you would be limiting the parallelism of your code (slowing run time).
+
+In the case `iWriteCount == 1` the `sReadTry` is set from 1 -> 0, meaning that no new readers can attempt to read. For the writer to begin writing, it must wait for the readers to finish reading (due to the `sResource` semaphore.
+
+So when `iReadCount --`, the reader checks if it is the last reader by `iReadCount == 0`, and if it is it unlocks `sResource` (-1->0) so that the writers can write. If more readers show up, they cannot enter as `sReadTry == -1`.
+
+The last writer does the same thing, but instead of unlocking the resource it unlocks the `sReadTry` semaphore.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

docs/lectures/osc/09_mem_management1.md

@@ -0,0 +1,137 @@
+29/10/20
+
+## Memory Overview
+
+Computers typically have memory hierarchies:
+
+> * Registers
+> * L1/L2/L3 cache
+> * Main memory (RAM)
+> * Disks
+
+**Higher Memory** is faster, more expensive and volatile. **Lower Memory** is slower, cheaper and non-volatile.
+
+The operating system provides **memory abstraction** for the user. Otherwise memory can be seen as one **linear array** of bytes/words.
+
+### OS Responsibilities
+
+* Allocate/de-allocate memory when requested by processes, keep track of all used/unused memory.
+* Distribute memory between processes and simulate an **indefinitely large** memory space. The OS must create the illusion of having infinite main memory, processes assume they have access to all main memory.
+* **Control access** when multi programming is applied.
+* **Transparently** move data from **memory** to **disk** and vice versa.
+
+#### Partitioning
+
+![img](/lectures/osc/assets/A.png)
+
+##### Contiguous memory management
+
+Allocates memory in **one single block** without any holes or gaps.
+
+##### Non-contiguous memory management models
+
+Where memory is allocated in multiple blocks, or segments, which may not be placed next to each other in physical memory.
+
+**Mono-programming:** one single partition for user processes.
+
+**Multi-programming** with **fixed partitions**
+
+* Fixed  **equal** sized partitions
+* Fixed non-equal sized partitions
+
+**Multi-programming** with **dynamic partitions**
+
+#### Mono-programming
+
+> * Only one single user process is in memory/executed at any point in time.
+> * A fixed region of memory is allocated to the OS & kernal, the remaining memory is reserved for a single process
+> * This process has direct access to physical memory (no address translation takes place)
+> * Every process is allocated **contiguous block memory** (no holes or gaps)
+> * One process is allocated the **entire memory space** and the process is always located in the same address space.
+> * **No protection** between different user processes required. Also no protection between the running process and the OS, so sometimes that process can access pieces of the OS its not meant to.
+>
+> * Overlays enable the **programmer** to use **more memory than available**.
+
+![img](/lectures/osc/assets/B.png)
+
+##### Short comings of mono-programming
+
+> * Since a process has direct access to the physical memory, it may have access to the OS memory.
+> * The OS can be seen as a process - so we have **two processes anyway**.
+> * **Low utilisation** of hardware resources (CPU, I/O devices etc)
+> * Mono-programming is unacceptable as **multi-programming is excepted** on modern machines
+
+**Direct memory access** and **mono-programming** are common in basic embedded systems and modern consumer electronics eg washing machines, microwaves, cars etc.
+
+##### Simulating Multi-Programming
+
+We can simulate multi-programming through **swapping**
+
+* **Swap process** out to the disk and load a new one (context switches would become **time consuming**)
+
+Why Multi-Programming is better theoretically
+
+> * There are *n* **processes in memory**
+> * A process spends *p* percent of its time **waiting for I/O**
+> * **CPU Utilisation** is calculated as 1 minus the time that all processes are waiting for I/O
+> * The probability that **all** *n* **processes are waitying for I/O is *p*^n^ 
+> * Therefore CPU utilisation is given by $1 - p^{n}$ 
+
+![cpu_util_form](/lectures/osc/assets/C.png)
+
+With an **I/O wait time of 20%** almost **100% CPU utilisation** can be achieved with four processes             ($1-0.2^{4}$)
+
+With an **I/O wait time of 90%**, 10 processes can achieve about **65% CPU utilisation**. ($1-0.9^{10}$)
+
+CPU utilisation **goes up** with the **number of processes** and **down** for **increasing levels of I/O**.
+
+![cpu_util_graph](/lectures/osc/assets/D.png)
+
+**Assume that**:
+
+> * A computer has one megabyte of memory
+> * The OS takes up 200k, leaving room for four 200k processes
+
+**Then:**
+
+> * If we have an I/O wait time of 80%, then we will achieve just under 60% CPU utilisation (1-0.8^4^)
+> * If we add another megabyte of memory, it would allow us to run another five processes. We can now achieve about **87%** CPU utilisation (1-0.8^9^)
+> * If we add another megabyte of memory (14 processes) we find that CPU utilisation will increase to around **96%**
+
+##### Caveats
+
+* This model assumes that all processes are independent, this is not true.
+* More complex models could be built using **queuing theory** but we still use this simplistic model to make **approximate predictions**
+
+#### Fixed Size Partitions
+
+* Divide memory into **static**, **contiguous** and **equal sized** partitions that have a fixed **size and location**.
+  * Any process can take **any** partition. (as long as its large enough)
+  * Allocation of **fixed equal sized partitions to processes is trivial**
+  * Very **little overhead** and **simple implementation**
+  * The OS keeps a track of which partitions are being **used** and which are **free**.
+
+##### Disadvantages
+
+* Partition may be necessarily large
+  * Low memory utilisation
+  * Internal fragmentation
+* **Overlays** must be used if a program does not fit into a partition (burden on the programmer)
+
+#### Fixed Partitions of non-equal size
+
+* Divide memory into **static** and **non-equal sized partitions** that have **fixed size and location**
+  * Reduces **internal fragmentation**
+  * The **allocation** of processes to partitions must be **carefully considered**.
+
+![process_alloc](/lectures/osc/assets/E.png)
+
+**One private queue per partition**:
+
+* Assigns each process to the smallest partition that it would fit in.
+* Reduces **internal fragmentation**.
+* Can reduce memory utilisation (e.g. lots of small jobs result in unused large partitions)
+
+**A single shared queue:**
+
+* Increased internal fragmentation as small processes are allocated into big partitions.

docs/lectures/osc/10_mem_management2.md

@@ -0,0 +1,152 @@
+30/10/20
+
+## Relocation and Protection
+
+```c
+#include <stdio.h>
+
+int iVar = 0;
+
+int main() {
+
+    int i = 0;
+
+    while(i < 10) {
+        iVar++;
+        sleep(2);
+        printf("Address:%x; Value:%d\n",&iVar, iVar);
+        i++;
+    }
+    return 0;
+}
+//If running the code twice (simultaneously):
+//Will the same or different addresses be displayed for iVar?
+//Will the value for iVar in the first run influence the value for iVar in the second run?
+```
+
+The addresses will be the same, as memory management within a process is the same. The process doesn't know where it is in memory, however the process is allocated the same amount of memory and the address is relative to the process.
+
+If the process is run twice, they are allocated two different memory spaces, so the addresses will be the same.
+
+[explanation 8:05]
+
+## Relocation
+
+When a program is run, it does not know in advance which partition it will occupy.
+
+* The program **cannot** simply **generate static addresses** (like jump instructions) that are absolute
+* **Addresses should be relative to where the program has been loaded**.
+* Relocation must be **solved in an operating system** that allows **processes to run at changing memory locations**. 
+
+**Protection**: Once you can have two programs in memory at the same time, protection must be enforced.
+
+![relative_mmu](/lectures/osc/assets/F.png)
+
+**Logical Address**: is a memory address seen by the process
+
+* It is independent of the current physical memory assignment
+* It is relative to the start of the program
+
+**Physical address**: refers to an actual location in main memory
+
+The **logical address space** must be **mapped** onto the **machines physical address space**.
+
+### Static Relocation
+
+This happens at compile time, a process has to be located at the same location every single time (impractical)
+
+### Dynamic Relocation
+
+This happens at load time
+
+* An **offset is added to every logical address** to account for its physical location in memory.
+* **Slows down the loading** of a process, does not account for **swapping**
+
+### Dynamic Relocation at run-time
+
+Two special purpose registers are maintained in the CPU (the **MMU**) containing a **base address** and **limit**
+
+> * The **base register** stores the **start address** of the partition.
+> * The **limit register** holds the **size** of the partition.
+>
+> At **run-time**
+>
+> * The base register is added to the **logical (relative) address** to generate the physical address.
+> * The resulting address is **compared** against the **limit register**. This allows us to see the bounds of where the process exists.
+>
+> NOTE: This requires **hardware support** (which didn't exist in the early days).
+
+<img src="/lectures/osc/assets/G.png" alt="registers" style="zoom:80%;" />
+
+#### Dynamic Partitioning
+
+**Fixed partitioning** results in **internal fragmentation**:
+
+> An exact match between the requirements of the process and the available partitions **may not exist**. 
+>
+> * This means the partition may **not be used in its entirety**.
+
+**Dynamic Partitioning**
+
+> * A **variable number of partitions** of which the **size** and **starting address** can **change over time**.
+> * A process is allocated the **exact amount** of **contiguous memory it requires**, thereby preventing internal fragmentation.
+
+![dynamic_partitioning](/lectures/osc/assets/H.png)
+
+**Swapping** holds some of the **processes** on the drive and **shuttles processes** between the drive and main memory as necessary
+
+Reasons for **swapping**:
+
+> * Some processes only **run occasionally**.
+> * We have more **processes** than **partitions**.
+> * A process's **memory requirements** may have **changed**.
+> * The **total amount of memory that is required** for the process **exceeds the available memory**.
+
+For any given process, we might not know the exact **memory requirements**. This is because processes may involve dynamic parts.
+
+> Solution (??):
+>
+> Allocate the current requirements **AND** a 'bit extra'
+>
+> The 'bit extra' will try and account for the dynamic nature of the process.
+>
+> If the process out grows it's partition, then it is shuttled out onto the main disk and allocated a new partition.
+
+![External Fragmentation](assets/I.png)
+
+##### External Fragmentation
+
+> * Swapping a process out of memory will **create 'a hole'**
+> * A new process may not **use the entire 'hole'**, leaving a small **unused block**
+> * A new process may be **too large for a given 'hole'**
+>
+> The **overhead** of memory **compaction** to **recover holes** can be **prohibitive** and requires **dynamic relocation**.
+
+##### Allocation Structures
+
+**Bitmaps**:
+
+> * The simplest data structure that can be used is a **bitmap**.
+> * **Memory is split into blocks** of 4 Kb size.
+>   * A bitmap is set up so that each **bit is 0** if the memory block is free, and 1 if the **block is being used**
+>     * 32 Mb memory / 4 Kb blocks = 8192 bitmap entries
+>     * 8192 bits occupy 1 Kb of storage (8192 / 8)
+>   * The size of this bitmap will depend on the **size of the memory** and the **size of the allocation unit**.
+> * To find a hole of say 128 K, then a group of **32 adjacent bits set to 0** must be found
+>   * Typically a long operation, the longer it takes, the lower the CPU utilisation is.
+> * A **trade-off exists** between the **size of the bitmap** and the **size of the blocks**
+>   * The size of the bitmaps can become prohibitive for small blocks and may make searching the bitmap slower
+>   * Larger blocks may increase internal fragmentation. 
+> * **Bitmaps are rarely used** because of this trade off
+
+**Linked List**:
+
+A more **sophisticated data structure** is required to deal with a **variable number** of **free and used partitions**.
+
+> * A linked list consists of a **number of entries** (links)
+> * Each link **contains data items** e.g. **start of memory block**, **size** and a flag for free and allocated
+> * It also contains a pointer to the next link.
+
+![Memory Management with linked lists](/lectures/osc/assets/J.png)
+
+![Linked lists for finding free memory](/lectures/osc/assets/K.png)

docs/lectures/osc/11_mem_management3.md

@@ -0,0 +1,110 @@
+05/11/20
+
+## Dynamic Partitioning Management
+
+### Allocating Available Memory
+
+#### First Fit
+
+> * First fit starts scanning **from the start** of the linked list until a link is found, which can fit the process
+>   * If the requested space is **the exact same size** as the 'hole', all the space is allocated
+>   * Otherwise the free link is split into two:
+>     * The first entry is set to the **size requested** and marked **used**
+>     * The second entry is set to **remaining size** and **free**.
+
+#### Next Fit
+
+> * The next fit algorithm maintains a record of where it got to last time and restarts it's search from there
+>   * This gives an even chance to all memory to get allocated (first fit concentrates on the start of the list)
+> * However simulations have been run which show that this is worse than first fit.
+>   * This is because a side effect of **first fit** is that it leaves larger partitions towards the end of memory, which is useful for larger processes.
+
+#### Best Fit
+
+> * The best fit algorithm always **searches the entire linked list** to find the smallest hole that's big enough to fit the memory requirements of the process.
+>   * It is **slower** than first fit
+>   * It also results in more wasted memory. As a exact sized hole is unlikely to be found, this leaves tiny (and useless) holes.
+>
+> Complexity: $O(n)$
+
+#### Worst Fit
+
+> Tiny holes are created when best fit split an empty partition.
+>
+> * The **worst fit algorithm** finds the **largest available empty partition** and splits it.
+>   * The **left over partition** is hopefully **still useful**
+>   * However simulations show that this method **isn't very good**.
+>
+> Complexity: $O(n)$ 
+
+#### Quick Fit
+
+> * Quick fit maintains a **list of commonly used sizes**
+>   * For example a separate list for each of 4 Kb, 8 Kb, 12 Kb etc holes
+>   * Odd sized holes can either go into the nearest size or into a special separate list.
+> * This is much f**aster than the other solutions**, however similar to **best fit** it creates **many tiny holes**.
+> * Finding neighbours for **coalescing** (combining empty partitions) becomes more difficult & time consuming.
+
+### Coalescing
+
+Coalescing (join together) takes place when **two adjacent entries** in the linked list become free.
+
+* Both neighbours are examined when a **block is freed**
+  * If either (or both) are also **free** then the two (or three) **entries are combined** into one larger block by adding up the sizes
+    * The earlier block in the linked list gives the **start point**
+    * The **separate links are deleted** and a **single link inserted**.
+
+### Compacting
+
+Even with coalescing happening automatically, **free blocks** may still be **distributed across memory**
+
+> * Compacting can be used to join free and used memory
+> * However compacting is more **difficult and time consuming** to implement then coalescing.
+>   * Each **process is swapped** out & **free space coalesced**.
+>   * Processes are swapped back in at lowest available location.
+
+## Paging
+
+Paging uses the principles of **fixed partitioning** and **code re-location** to devise a new **non-contiguous management scheme**
+
+> * Memory is split into much **smaller blocks** and **one or multiple blocks** are allocated to a process (e.g. a 11 Kb process would take 3 blocks of 4 Kb)
+> * These blocks **do not have to be contiguous in main memory**, but **the process still perceives them to be contiguous**
+> * Benefits:
+>   * **Internal fragmentation** is reduced to the **last block only** (e.g. previous example the third block, only 3 Kb will be used)
+>   * There is **no external fragmentation**, since physical blocks are **stacked directly onto each other** in main memory.
+
+![Paging in main memory](/lectures/osc/assets/L.png)
+
+![Paging processor's view](/lectures/osc/assets/M.png)
+
+![Paging](/lectures/osc/assets/P.png)
+
+A **page** is a **small block** of **contiguous memory** in the **logical address space** (as seen by the process)
+
+* A **frame** is a **small contiguous block** in **physical memory**.
+* Pages and frames (usually) have the **same size**:
+  * The size is usually a power of 2.
+  * Size range between 512 bytes and 1 Gb. (most common 4 Kb pages & frames)
+
+**Logical address** (page number, offset within page) needs to be **translated** into a **physical address** (frame number, offset within frame)
+
+* Multiple **base registers** will be required
+  * Each logical page needs a **separate base register** that specifies the start of the associated frame
+  * i.e a **set of base registers** has to be maintained for each process
+* The base registers are stored in the **page table**
+
+![Mapping page tables](/lectures/osc/assets/Q.png)
+
+The page table can be seen as a **function**, that **maps the page number** of the logical address **onto the frame number** of the physical address
+$$
+frameNumber = f(pageNumber)
+$$
+
+* The **page number** is used as an **index to the page table** that lists the **location of the associated frame**.
+* It is the OS' duty to maintain a list of **free frames**.
+
+![address translation](/lectures/osc/assets/R.png)
+
+We can see that the **only difference** between the logical address and physical address is the **4 left most bits** (the **page number and frame number**). As **pages and frames are the same size**, then the **offset value will be the same for both**. 
+
+This allows for **more optimisation** which is important as this translation will need to be **run for every memory read/write** call.

docs/lectures/osc/12_mem_management4.md

@@ -0,0 +1,136 @@
+06/11/20
+
+## Paging implementation
+
+Benefits of paging
+
+* **Reduced internal fragmentation**
+* No **external fragmentation**
+* Code execution and data manipulation are usually **restricted to a small subset** (i.e limited number of pages) at any point in time.
+* **Not all pages** have to be **loaded in memory** at the **same time** => **virtual memory** 
+  * Loading an entire set of pages for an entire program/data set into memory is **wasteful**
+  * Desired blocks could be **loaded on demand**.
+  * This is called the **principle of locality**.
+
+#### Memory as a linear array
+
+> * Memory can be seen as one **linear array** of **bytes** (words)
+> * Address ranges from $0 - (N-1)$
+> * N address lines can be used to specify $2^N$ distinct addresses.
+
+### Address Translation
+
+* A **logical address** is relative to the start of the **program (memory)** and consists of two parts:
+  * The **right most** $m$ **bits** that represent the **offset within the page** (and frame) .
+    * $m$ often is 12 bits
+  * The **left most** $n$ **bits** that represent the **page number** (and frame number they're the same thing)
+    * $n$ is often 4 bits
+
+![address composition](/lectures/osc/assets/S.png)
+
+#### Steps in Address Translation
+
+> 1. **Extract the page number** from logical address
+> 2. Use page number as an **index** to **retrieve the frame number** in the **page table**
+> 3. **Add the logical offset within the page** to the start of the physical frame
+>
+> **Hardware Implementation**
+>
+> 1. The CPU's **memory management uni** (MMU) intercepts logical addresses
+> 2. MMU uses a page table as above
+> 3. The resulting **physical address** is put on the **memory bus**.
+>
+> Without this specialised hardware, paging wouldn't be quick enough to be viable.
+
+### Principle of Locality
+
+![virtual memory](/lectures/osc/assets/T.png)
+
+We have more pages here, than we can physically store as frames.
+
+**Resident set**: The set of pages that are loaded in main memory. (In the above image, the resident set consists of the pages not marked with an 'X')
+
+#### Page Faults
+
+> A **page fault** is generated if the processor accesses a page that is **not in memory**
+>
+> * A page fault results in an interrupt (process enters **blocked state**)
+> * An **I/O operation** is started to bring the missing page into main memory
+> * A **context switch** (may) take place.
+> * An **interrupt signal** shows that the I/O operation is complete and the process **enters the ready state**.
+
+```
+1. Trap operating system
+	- Save registers / process state
+	- Analyse interrupt (i.e. identify the interrupt is a page fault)
+	- Validate page reference, determine page location
+	- Issue disk I/O: queueing, seek, latency, transfer
+2. Context switch (optional)
+3. Interrupt for I/O completion
+	- Store process state / registers
+	- Analyse interrupt from disk
+	- Update page table (page in memory) **
+	- Wait for original process to be sceduled
+4. Context switch to original process
+```
+
+### Virtual Memory
+
+#### Benefits
+
+> * Being able to maintain **more processes** in main memory through the use of virtual memory **improves CPU utilisation**
+>   * Individual processes take up less memory since they are only partially loaded
+> * Virtual memory allows the **logical address space** (processes) to be larger than **physical address space** (main memory)
+>   * 64 bit machine => 2^64^ logical addresses (theoretically)
+
+#### Contents of a page entry
+
+> * A **present/absent bit** that is set if the frame is in main memory or not.
+> * A **modified bit** that is set if the page/frame has been modified (only modified pages have to be written back to the disk when evicted. This makes sure the pages and frames are kept in sync).
+> * A **referenced bit** that is set if the page is in use (If you needed to free up space in main memory, move a page, however it is important that a page not in use is moved).
+> * **Protection and sharing bits**: read, write, execute or various different combos of those.
+
+![page entry meta data](assets/U.png)
+
+##### Page Table Size
+
+> * On a **16 bit machine**, the total address space is 2^16^
+>   * Assuming that 10 bits are used for the offset (2^10^)
+>   * 6 bits can be used to number the pages
+>   * This means 2^6^ or 64 pages can be maintained
+> * On a **32 bit machine**, 2^20^ or ~10^6^ pages can be maintained
+> * On a **64 bit machine**, this number increases a lot. This means the page table becomes stupidly large.
+
+Where do we **store page tables with increasing size**?
+
+* Perfect world would be registers - however this isn't possible due to size
+* They will have to be stored in (virtual) **main memory**
+  * **Multi-level** page tables
+  * **Inverted page tables** (for large virtual address spaces)
+
+However if the page table is to be stored in main memory, we must maintain acceptable speeds. The solution is to page the page table.
+
+### Multi-level Page Tables
+
+We use a tree-like structure to hold the page tables
+
+* Divide the page number into
+  * An index to a page table of second level
+  * A page within a second level page table
+
+This means there's no need to keep all the page tables in memory all the time!
+
+![multi level page tables](assets/V.png)
+
+The above image has 2 levels of page tables.
+
+> * The **root page table** is always maintained in memory.
+> * Page tables themselves are **maintained in virtual memory** due to their size.
+>
+> Assume that a **fetch** from main memory takes *T* nano-seconds
+>
+> * With a **single page table level**, access is $2  \cdot T$
+> * With **two page table levels**, access is $3  \cdot T$
+> * and so on...
+>
+> We can have many levels as the address space in 64 bit computers is so massive.

docs/lectures/osc/13_mem_management5.md

@@ -0,0 +1,168 @@
+12/11/20
+
+## Page Tables Optimisations
+
+##### Memory Organisation
+
+* The **root page table** is always maintained in memory.
+* Page tables themselves are maintained in **virtual memory** due to their size.
+* Assume a **fetch** from main memory takes $T$ time - single page table access is now $2\cdot T$  and **two** page table levels access is $3 \cdot T$.
+  * Some optimisation needs to be done, otherwise memory access will create a bottleneck to the speed of the computer.
+
+### Translation Look Aside Buffers
+
+* Translation look aside buffers or TLBs are (usually) located inside the memory management unit
+  * They **cache** the most frequently used page table entries.
+    * As they're stored in cache its super quick.
+  * They can be searched in **parallel**.
+* The principle behind TLBs is similar to other types of **caching in operating systems**. They normally store anywhere from 16 to 512 pages.
+* Remember: **locality** states that processes make a large number of references to a small number of pages.
+
+![TLB diagram](/lectures/osc/assets/W.png)
+
+The split arrows going into the TLB represent searching in parallel. 
+
+* If the TLB gets a hit, it just returns the frame number
+* However if the TLB misses:
+  * We have to account for the time it took to search the TLB
+  * We then have to look in the page table to find the frame number
+* Worst case scenario is a page fault (takes the longest). This is where we have to retrieve a page table from secondary memory, so that it can then be searched.
+
+> Quick maths:
+>
+> * Assume a single-level page table
+>
+> * Assume 20ns associative **TLB lookup time**
+>
+> * Assume a 100ns **memory access time**
+>
+>   * **TLB hit** => 20 + 100 = 120ns
+>   * **TLB miss** => 20 + 100 + 100 = 220ns
+>
+> * Performance evaluation of TLBs
+>
+>   * For an 80% hit rate, the estimated access time is: 
+>     $$
+>     120\cdot 0.8 + 220\cdot (1-0.8)=140ns
+>     $$
+>     (**40% slowdown** relative to absolute addressing)
+>     
+>   * For a 98% hit rate, the estimated access time is: 
+>     $$
+>     120\cdot 0.98 + 220\cdot (1-0.98)=122ns
+>     $$
+>    (**22% slowdown**) 
+>
+> NOTE: **page tables** can be **held in virtual memory** => **further slow down** due to **page faults**.
+
+### Inverted Page Tables
+
+A **normal page table size** is proportional to the number of pages in the virtual address space => this can be prohibitive for modern machines
+
+>An **inverted page table's size** is **proportional** to the size of **main memory**
+>
+>* The inverted table contains one **entry for every frame** (not for every page) and it **indexes entries by frame number** not by page number.
+>* When a process references a page, the OS must search the entire inverted page table for the corresponding entry (which could be too slow)
+>  * It does save memory as there are fewer frames than pages.
+>* To find if your pages is in main memory, you need to iterate through the entire list.
+>* *Solution*: Use a **hash function** that transforms page numbers (*n* bits) into frame numbers (*m* bits) - Remember *n* > *m*
+>  * The has functions turns a page number into a potential frame number.
+
+![inverted page table](/lectures/osc/assets/x.webp)
+
+So when looking for the page's frame location. We have to sequentially search through the table until we hit a match, we then get the frame number from the index - in this case 4.
+
+#### Inverted Page Table Entry
+
+> * The **frame number** will be the index of the inverted page table.
+> * Process Identifier (**PID**) - The process that owns this page.
+> * Virtual Page Number (**VPN**)
+> * **Protection** bits (Read/Write/Execute)
+> * **Chaining Pointer** - This field points towards the next frame that has exactly the same VPN. We need this to solve collisions
+
+![Inverted Page table entry](/lectures/osc/assets/Y.png)
+
+![inverted page table address translation](/lectures/osc/assets/Z.png)
+
+Due to the hash function, we now only have to look through all entries with **VPN**: 1 instead of all the entries.
+
+#### Advantages
+
+* The OS maintains a **single inverted page table** for all processes
+* It **saves lots of space** (especially when the virtual address space is much larger than the physical memory)
+
+#### Disadvantages
+
+* Virtual to physical **translation becomes much slower**
+* Hash tables eliminates the need of searching the whole inverted table, but we have to handle collisions (which also **slows down translation**)
+* TLBs are necessary to improve their performance.
+
+### Page Loading
+
+* Two key decisions have to be made when using virtual memory
+  * What pages are **loaded** and when
+    * Predictions can be made for optimisation to reduce page faults
+  * What pages are **removed** from memory and when
+    * **page replacement algorithms**
+
+#### Demand Paging
+
+> Demand paging starts the process with **no pages in memory**
+>
+> * The first instruction will immediately cause a **page fault**.
+> * **More page faults** will follow but they will **stabilise over time** until moving to the next **locality**
+> * The set of pages that is currently being used is called it's **working set** (same as the resident set)
+> * Pages are only **loaded when needed** (i.e after **page faults**)
+
+#### Pre-Paging
+
+> When the process is started, all pages expected to be used (the working set) are **brought into memory at once**
+>
+> * This **reduces the page fault rate**
+> * Retrieving multiple (**contiguously stored**) pages **reduces transfer times** (seek time, rotational latency, etc)
+>
+> **Pre-paging** loads as many pages as possible **before page faults are generated** (a similar method is used when processes are **swapped in and out**) 
+
+*ma*: memory access time				*p*: page fault rate				*pft*: page fault time
+
+**Effective access time** is given by: $T_{a} = (1-p)\cdot ma+pft\cdot p$
+
+NOTE: This doesn't take into account TLBs.
+
+The expected access time is **proportional to page fault rate** when keeping page faults into account.
+$$
+T_{a} \space\space\alpha \space\space p
+$$
+
+* Ideally, all pages would have to be loaded without demanding paging.
+
+### Page Replacement
+
+> * The OS must choose a **page to remove** when a new one is loaded
+> * This choice is made by **page replacement algorithms** and **takes into account**:
+>   * When the page was **last used** or **expected to be used again**
+>   * Whether the page has been **modified** (this would cause a write).
+> * Replacement choices have to be made **intelligently** to **save time**.
+
+#### Optimal Page Replacement
+
+> * In an **ideal** world
+>   * Each page is labelled with the **number of instructions** that will be executed/length of time before it is used again.
+>   * The page which is **going to be not referenced** for the **longest time** is the optimal one to remove.
+> * The **optimal approach** is **not possible to implement** 
+>   * It can be used for post execution analysis
+>   * It provides a **lower bound** on the number of page faults (used for comparison with other algorithms) 
+
+#### FIFO
+
+> * FIFO maintains a **linked list** of new pages, and **new pages** are added at the end of the list
+> * The **oldest page at the head** of the list is **evicted when a page fault occurs**
+>
+> This is a pretty bad algorithm <s>unsurprisingly</s>
+
+![FIFO page replacement](/lectures/osc/assets/a1.png)
+
+Explanation at 53:40
+
+Shaded squares on the top row are page faults. Shaded squares in the grid are when a new page is brought into memory.
+

docs/lectures/osc/14_mem_management6.md

@@ -0,0 +1,176 @@
+13/11/20
+
+## Virtual Memory & Potential Problems
+
+#### Page Replacement
+
+##### Second chance
+
+> * If a page at the front of the list has **not been referenced** it is **evicted**
+> * If the reference bit is set, the page is **placed at the end** of the list and it's reference bit is unset.
+> * This works better than FIFO and is relatively simple
+> * **Costly to implement** as the list is constantly changing.
+> * Can degrade to FIFO if all pages were initially referenced.
+
+##### Clock Replacement Algorithm
+
+> The second chance implementation can be improved by **maintaining the page list as a circle**
+>
+> * A **pointer** points to the last visited page.
+> * In this form the algorithm is called the one handed clock 
+> * It is faster, but can still be **slow if the list is long**.
+> * The **time spent** on **maintaining** the list is **reduced**.
+
+![clock replacement](/lectures/osc/assets/a2.png)
+
+##### Not Recently Used (NRU)
+
+> For NRU, **referenced** and **modified** bits are kept in the page table
+>
+> * Referenced bits are set to 0 at the start, and **reset periodically** 
+>
+> There are four different **page types** in NRU:
+>
+> 1. Not referenced recently, not modified
+> 2. Not referenced recently, modified
+> 3. Referenced recently, not modified
+> 4. Referenced recently, modified
+>
+> **Page table entries** are inspected upon every **page fault**. This could be implemented in the following way
+>
+> 1. Find a page from **class 0** to be removed.
+> 2. If step 1 fails, scan again looking for **class 1**. During this scan we set the reference bit to 0 on each page that is bypassed
+> 3. If step 2 fails, start again from step 1 (Now we should find pages from class 2&3 have been moved to class 0 or 1)
+>
+> The NRU algorithm provides a **reasonable performance** and is easy to understand and implement.
+
+##### Least Used Recently
+
+> Least recently used **evicts the page** that has **not be used for the longest**
+>
+> * The OS must keep track of when a page was last used.
+> * Every page table entry contains a field for the counter
+> * This is **not cheap to implement** as we need to maintain a **list of pages** which are **sorted** in the order in which they have been used.
+>
+> This algorithm can be **implemented in hardware** using a **counter** that is incremented after each instruction ...
+
+![least used recently visualisation](/lectures/osc/assets/a3.png)
+
+This will look familiar to the FIFO algorithm however, when a page is used, that is like its just come in.
+
+
+### Resident Set
+
+How many pages should be allocated to individual processes:
+
+* **Small resident sets** enable to store **more processes in memory** => improved CPU utilisation.
+* **Small resident sets** may result in **more page faults**
+* **Large resident sets** may  **no longer reduce** the **page fault rate** (**diminishing returns**) 
+
+A trade-off exists between the **sizes of the resident sets** and **system utilisation**.
+
+Resident set sizes may be **fixed** or **variable** (adjusted at run-time)
+
+* For **variable sized** resident sets, **replacement policies** can be:
+  * **Local**: a page of the same process is replaced
+  * **Global**: a page can be taken away from a **different process**
+* Variable sized sets require **careful evaluation of their size** when a **local scope** is used (often based on the **working set** or the **page fault rate**)
+
+### Working Set
+
+The **resident set** comprises the set of pages of the process that are in memory (they have a corresponding frame)
+
+The **working set** is a subset of the resident set that is actually needed for execution.
+
+* The **working set** $W(t, k)$ comprises the set of referenced pages in the last $k$ (working set window) **virtual time units for the process**.
+* $k$ can be defined as **memory references** or as **actual process time**
+  * The set of most recent used pages
+  * The set of pages used within a pre-specified time interval
+* The **working set size** can be used as a guide for the number of frames that should be allocated to a process.
+
+![working set](/lectures/osc/assets/a4.png)
+
+The working set is a **function of time** $t$:
+
+* Processes **move between localities**, hence, the pages that are included in the working set **change over time**
+* **Stable** intervals alternate with intervals of **rapid change**
+
+$|W(t,k)|$ is then a variable in time. Specifically:
+$$
+1\le |W(t,k)| \le min(k, N)
+$$
+where $N$ is the total number of pages of the process. All the maths is saying is that the size of the working set can be as small as **one** or as large as **all the pages in the process**.
+
+Choosing the right value for $k$ is important:
+
+* Too **small**: inaccurate, pages are missing
+* Too **large**: too many unused pages present
+* **Infinity**: all pages of the process are in the working set
+
+Working sets can be used to guide the **size of the resident sets**
+
+* Monitor the working set
+* Remove pages from the resident set that are not in the working set
+
+The working set is costly to maintain => **page fault frequency (PFF)** can be used as an approximation: $PFF\space\alpha\space k$ 
+
+* If the PFF is increased -> we need to increase $k$
+* If PFF is very low -> we could decrease $k$ to allow more processes to have more pages.
+
+#### Global Replacement
+
+> Global replacement policies can select frames from the entire set (they can be taken from other processes)
+>
+> * Frames are **allocated dynamically** to processes
+> * Processes cannot control their own page fault frequency. The PFF of one process is **influenced by other processes**.
+
+#### Local Replacement
+
+> Local replacement policies can only select frames that are allocated to the current process
+>
+> * Every process has a **fixed fraction of memory**
+> * The **locally oldest page** is not necessarily the **globally oldest page**
+
+Windows uses a variable approach with local replacement. Page replacement algorithms can use both policies.
+
+
+### Paging Daemon
+
+It is more efficient to **proactively** keep a number of **free pages** for **future page faults**
+
+* If not, we may have to **find a page** to evict and we **write it to the drive** (if its been modified) first when a page fault occurs.
+
+Many systems have a background process called a **paging daemon**.
+
+* This process **runs at periodic intervals**
+* It inspects the state of the frames and if too few frames are free, it **selects pages to evict** (using page replacement algorithms) 
+
+Paging daemons can be combined with **buffering** (free and modified lists) => write the modified pages **but keep them in main memory** when possible.
+
+**Buffering**: a process that preemptively writes modified pages to the disk. That way when there's a page fault we don't lose the time taken to write to disk
+
+
+### Thrashing
+
+Assume **all available pages are in active use** and a new page needs to be loaded:
+
+* The page that will be evicted will have to be **reloaded soon afterwards**
+
+**Thrashing** occurs when pages are **swapped out** and then **loaded back in immediately**
+
+#### Causes of thrashing include:
+
+* The degree of multi-programming is too high i.e the total **demand** (the sum of all working sets sizes) **exceeds supply** (the available frames)
+* An individual process is allocated **too few pages**
+
+This can be prevented by **using good page replacement algorithms**, reducing the **degree of multi-programming** or adding more memory.
+
+The **page fault frequency** can be used to detect that a system is thrashing.
+
+> * CPU utilisation is too low => scheduler **increases degree of multi-programming**
+>   * Frames are allocated to new processes and taken away from existing processes
+>     * I/O requests are queued up as a consequence of page faults 
+>
+> This is a positive reinforcement cycle.
+
+And when all this comes together, its how memory management working in modern computers.

docs/lectures/osc/15_file_systems1.md

@@ -0,0 +1,198 @@
+19/11/20
+
+## Disk Scheduling
+
+### Hard Drives
+
+#### Construction of Hard Drives
+
+> Disks are constructed as multiple aluminium/glass platters covered with **magnetisable material**
+>
+> * Read/Write heads fly just above the surface and are connected to a single disk arm controlled by a single actuator
+> * **Data** is stored on **both sides**
+> * Hard disks **rotate** at a **constant speed**
+>
+> A hard disk controller sits between the CPU and the drive
+>
+> Hard disks are currently about 4 orders of magnitude slower than main memory.
+
+![hard drive diagram](/lectures/osc/assets/a5.png)
+
+#### Low Level Format
+
+> Disks are organised in:
+>
+> * **Cylinders**: a collection of tracks in the same relative position to the spindle
+> * **Tracks**: a concentric circle on a single platter side
+> * **Sectors**: segments of a track - usually have an **equal number of bytes** in them, consisting of a **preamble, data** and an **error correcting code** (ECC).
+>
+> The number of sectors on each track increases from the inner most track to the outer tracks.
+
+##### Organisation of hard drives
+
+Disks usually have a **cylinder skew** i.e an **offset** is added to sector 0 in adjacent tracks to account for the seek time.
+
+In the past, consecutive **disk sectors were interleaved** to account for transfer time (of the read/write head)
+
+NOTE: disk capacity is reduced due to preamble & ECC
+
+#### Access times
+
+**Access time** = seek time + rotational delay + transfer time
+
+* **Seek time**: time needed to move the arm to the cylinder
+
+* **Rotational latency**: time before the sector appears underneath the read/write head (on average its half a rotation)
+
+* **Transfer time**: time to transfer the data
+
+![hard drive access times](/lectures/osc/assets/a6.png)
+
+Multiple requests may be happening at the same time (concurrently), so access time may be increased by **queuing time**
+
+In this scenario, dominance of seek time leaves room for **optimisation** by carefully considering the order of read operations.
+
+![hard disk delay](/lectures/osc/assets/a7.png)
+
+The **estimated seek time** (i.e to move the arm from one track to another) is approximated by:
+$$
+T_{s} = n \times m + s
+$$
+In which $T_{s}$ denotes the estimated seek time, $n$ the **number of tracks** to be crossed, $m$ the **crossing time per track** and $s$ any **additional startup delay**.
+
+> Let us assume a disk that rotates at 3600 rpm
+>
+> * One rotation = 16.7 ms
+> * The average **rotational latency** $T_{r}$ is then 8.3 ms
+>
+> Let **b** denote the **number of bytes transferred**, **N** the **number of bytes per track**, and **rpm** the **rotation speed in rotations per minute**, the per track,  the transfer time, $T_{t}$, is then given by:
+> $$
+> T_{t} = \frac b N \times \frac {ms\space per\space minute}{rpm}
+> $$
+> $N$ bytes take 1 revolution => $\frac{60000}{3600}$ ms = $\frac {ms\space per\space minute}{rpm}$
+>
+> $b$ contiguous bytes takes $\frac{b}{N}$ revolutions.
+
+> Read a file of **size 256 sectors** with;
+>
+> * $T_{s}$ = 20 ms (average seek time)
+> * 32 sectors per track
+>
+> Suppose the file is stored as compact as possible (its stored contiguously)
+>
+> * The first track takes: seek time + rotational delay + transfer time
+>   $20 + 8.3 + 16.7 = 45ms$
+> * Assuming no cylinder skew and neglecting small seeks between tracks we only need to account for rotational delay + transfer time
+>   $8.3+16.7=25ms$
+>
+> The total time is $45+7\times 25 = 220ms = 0.22s$  
+
+> In case the access is not sequential but at **random for the sectors** we get:
+>
+> * Time per sector = $T_{s}+T_{r}+T_{t} = 20+8.3+0.5=28.8ms$
+>   $T_{t} = 16.7\times \frac {1}{32} = 0.5$
+>
+> It is important to **position the sectors carefully** and **avoid disk fragmentation**
+
+### Disk Scheduling
+
+The OS must use the hardware efficiently:
+
+* The file system can **position/organise files strategically**
+* Having **multiple disk requests** in a queue allows us to **minimise** the **arm movement**
+
+Note that every I/O operation goes through a system call, allowing the **OS to intercept the request and re sequence it**.
+
+If the drive **is free**, the request can be serviced immediately, if not the request is queued.
+
+In a dynamic situation, several I/O requests will be **made over time** that are kept in a **table of requested sectors per cylinder.**
+
+> Disk scheduling algorithms determine the order in which disk events are processed 
+
+#### First-Come First-Served
+
+> Process the requests in the order that they arrive
+>
+> Consider the following sequence of disk requests
+>
+> `11 1 36 16 34 9 12`
+>
+> The total length is: `|11-1|+|1-36|+|36-16|+|16-34|+|34-9|+|9-12|=111`
+>
+> ![FCFS](/lectures/osc/assets/a8.png)
+
+
+
+#### Shortest Seek Time First
+
+> Selects the request that is closest to the current head position to reduce head movement
+>
+> * This allows us to gain **~50%** over FCFS
+>
+> Total length is: `|11-12|+|12-9|+|9-16|+|16-1|+|1-34|+|34-36|=61`
+>
+> ![SSTF](/lectures/osc/assets/a9.png)
+>
+> Disadvantages:
+>
+> * Could result in starvation:
+>   * The **arm stays in the middle of the disk** in case of heavy load, edge cylinders are poorly served - the strategy is biased
+>   * Continuously arriving requests for the same location could **starve other regions**
+
+#### SCAN
+
+> **Keep moving in the same direction** until end is reached
+>
+> * It continues in the current direction, **servicing all pending requests** as it passes over them
+> * When it gets to the **last cylinder**, it **reverses direction** and **services pending requests**
+>
+> Total length: `|11-12|+|12-16|+|16-34|+|34-36|+|36-9|+|9-1|=60`
+>
+> ![scan algorithm](/lectures/osc/assets/b1.png)
+>
+> **Disadvantages**:
+>
+> * The **upper limit** on the waiting time is $2\space\times$ number of cylinders (no starvation)
+> * The **middle cylinders are favoured** if the disk is heavily used.
+
+##### C-SCAN
+
+> Once the outer/inner side of the disk has been reached, the **requests at the other end of the disk** have been **waiting the longest**
+>
+> * SCAN can be improved by using a circular => C-SCAN
+>   * When the disk arm gets to the last cylinder of the disk, it **reverses direction** but **does not service requests** on the return.
+>   * It is **fairer** and equalises **response times on the disk**
+>
+> Total length: `|11-12|+|12-16|+|16-34|+|34-36|+|36-1|+|1-9|=68`
+
+##### LOOK-SCAN
+
+> Look-SCAN moves to the last cylinder containing **the first or last request** (as opposed to the first/last cylinder on the disk like SCAN)
+>
+> * However, seeks are **cylinder by cylinder** and one cylinder contains multiple tracks
+> * It may happen that the arm "sticks" to a cylinder
+
+##### N-Step SCAN
+
+> Only services $N$ requests every sweep.
+
+```bash
+[jay@diablo lecture_notes]$ cat /sys/block/sda/queue/scheduler 
+[mq-deadline] kyber bfq none
+
+# noop: FCFS
+# deadline: N-step-SCAN
+# cfq: Complete Fairness Queueing (from linux)
+```
+
+### Driver caching
+
+For current drives, the time **required to seek a new cylinder** is more than the **rotational time**.
+
+* It makes sense to **read more sectors than actually required**
+  * **Read** sectors during rotational delay (the sectors that just so happen to pass under the control arm)
+  * **Modern controllers read multiple sectors** when asked for the data from one sector **track-at-a-time caching**.
+
+### Scheduling on SSDs
+
+SSDs don't have $T_{seek}$ or rotational delay, we can use FCFS (SSTF, SCAN etc may reduce performace due to no head to move).

docs/lectures/osc/16_file_systems2.md

@@ -0,0 +1,173 @@
+20/11/20
+
+## File System Views
+
+### User View
+
+A **user view** that defines a file system in terms of the **abstractions** that the operating system provides
+
+An **implementation view** that defined the file system in terms of its **low level implementation**
+
+**Important aspects of the user view**
+
+> * The **file abstraction** which **hides** implementation details from the user
+> * File **naming policies**, user file **attributes** (size, protection, owner etc)
+>   * There are also **system attributes** for files (e.g. non-human readable, archive flag, temp flag)
+> * **Directory structures** and organisation
+> * **System calls** to interact with the file system
+>
+> The user view defines how the file system looks to regular users and relates to **abstractions**.
+
+#### File Types
+
+Many OS's support several types of file. Both windows and Unix have regular files and directories:
+
+* **Regular files** contain user data in **ASCII** or **binary** format
+* **Directories** group files together (but are files on an implementation level) 
+
+Unix also has character and block special files:
+
+* **Character special files** are used to model **serial I/O devices** (keyboards, printers etc)
+* **Block special files** are used to model drives
+
+### System Calls
+
+File Control Blocks (FCBs) are kernel data structures (they are protected and only accessible in kernel mode)
+
+* Allowing user applications to access them directly could compromise their integrity
+* System calls enable a **user application** to **ask the OS** to carry out an action on it's behalf (in kernel mode)
+* There are **two different categories** of **system calls**
+  * **File manipulation**: `open()`, `close()`, `read()`, `write()` ...
+  * **Directory manipulation**: `create()`, `delete()`, `rename()`, `link()` ...
+
+### File Structures
+
+**Single level**: all files are in the same directory (good enough for basic consumer electronics)
+
+**Two or multiple level directories**: tree structures
+
+* **Absolute path name**: from the root of the file system
+* **Relative path name**: the current working directory is used as the starting point
+
+**Directed acyclic graph (DAG)**: allows files to be shared (links files or sub-directories) but **cycles are forbidden**
+
+**Generic graph structure**: Links and cycles can exist.
+
+The use of **DAG** and **generic graph structures** results in **significant complications** in the implementation 
+
+* Trees are a DAG with the restriction that a child can only have one parent and don't contain cycles. 
+
+When searching the file system:
+
+* Cycles can result in **infinite loops**
+* Sub-trees can be **traversed multiple times**
+* Files have **multiple absolute file names**
+* Deleting files becomes a lot more complicated
+  * Links may no longer point to a file
+  * Inaccessible cycles may exist
+* A garbage collection scheme may be required to remove files that are no longer accessible from the file system tree.
+
+#### Directory Implementations
+
+Directories contain a list of **human readable file names** that are mapped onto **unique identifiers** and **disk locations**
+
+* They provide a mapping of the logical file onto the physical location
+
+Retrieving a file comes down to **searching the directory file** as fast as possible:
+
+* A **simple random order of directory** entries might be insufficient (search time is linear as a function of the number of entries)
+* Indexes or **hash tables** can be used.
+* They can store all **file related attributes** (file name, disk address - Windows) or they can **contain a pointer** to the data structure that contains the details of the file (Unix)
+
+![directory files](/lectures/osc/assets/b2.png)
+
+##### System Calls
+
+Similar to files, **directories** are manipulated using **system calls**
+
+* `create/delete`: new directory is created/deleted.
+* `opendir, closeddir`: add/free directory to/from internal tables
+* `readdir`: return the next entry in the directory file
+
+**Directories** are **special files** that **group files** together and of which the **structure is defined** by the **file system**
+
+* A bit is set to indicate that they are directories
+* In Linux when you create a directory, two files are in that directory that the user has no control over. These files are represented as `.` and `..`
+  * `.` - a file dealing with file permissions
+  * `..` - represents the parent directory (`cd ..`)
+
+##### Implementation
+
+> Regardless of the type of file system, a number of **additional considerations** need to be made
+>
+> * **Disk Partitions**, **partition tables**, **boot sectors** etc
+> * Free **space management**
+> * System wide and per process **file tables**
+>
+> **Low level formatting** writes sectors to the disk
+>
+> **High level formatting** imposes a file system on top of this (using **blocks** that can cover multiple **sectors**)
+
+### Partitions
+
+Disks are usually divided into **multiple partitions**
+
+* An independent file system may exist on each partiton
+
+**Master Boot Record** 
+
+* Located as the start of the entire drive
+* Used to boot the computer (BIOS reads and executes MBR)
+* Contains **partition table** at its end with **active partition**.
+* One partition is listed as **active** containing a boot block to load the operating system.
+
+![master boot record](assets/b3.png)
+
+#### Unix Partition
+
+> The partition contains
+>
+> * The partition **boot block**:
+>   * Contains code to boot the OS
+>   * Every partition has a boot block - even if it does not contain an OS
+> * **Super block** contains the partitions details e.g. partition size, number of blocks, I-node table etc
+> * **Free space management** contains a bitmap or linked list that indicates the free blocks.
+>   * A linked list of disk blocks (also known as grouping)
+>   * We use free blocks to hold the **number of the free blocks**. Since the free list shrinks when the disk becomes full, this is not wasted space
+>   * **Blocks are linked together**. The size of the list **grows with the size of the disk** and **shrinks with the size of the blocks**
+>   * Linked lists can be modified by **keeping track of the number of consecutive free blocks** for each entry (known as counting)
+> * **I-Nodes**: An array of data structures, one per file, telling all about the files
+> * **Root directory**: the top of the file-system tree
+> * **Data**: files and directories
+
+![unix partition composition](/lectures/osc/assets/b4.png)
+
+![Free block management](/lectures/osc/assets/b5.png)
+
+Free space management with linked list (on the left) and bitmaps (on the right)
+
+**Bitmaps**
+
+* Require extra space
+* Keeping it in main memory is possible but only for small disk
+
+**Linked lists**
+
+* No wasted disk space
+* We only need to keep in memory one block of pointers (load a new block when needed)
+
+Apart from the free space memory tables, there is a number of key data structures stored in memory:
+
+* An in-memory mount table (table with different partitions that have been mounted)
+* An in-memory directory cache of recently accessed directory information
+* A **system-wide open file table**, containing a copy of the FCB for every currently open file in the system, including location on disk, file size and **open count** (number of processes that use the file)
+* A **per-process open file table**, containing a pointer to the system open file table.
+
+![file tables](/lectures/osc/assets/b6.png)
+
+![opening & reading a file](/lectures/osc/assets/b7.png)
+
+
+
+
+

docs/lectures/osc/17_file_systems3.md

@@ -0,0 +1,91 @@
+26/11/20
+
+## File System Implementation
+
+1. Contiguous
+2. Linked Lists
+3. File Allocation Table (FAT)
+4. I-nodes (lookups)
+
+### File access
+
+Files will be composed of a number of blocks. Files are **sequential** or **random access**. Random access is essential for example in database systems.
+
+#### Contiguous Allocation
+
+**Contiguous file systems** are similar to **dynamic partitioning** in memory allocation.
+
+> Each file is stored in a single group of **adjacent blocks** on the hard disk
+>
+> Allocation of free space can be done using **first fit, best fit, next fit**.
+>
+> * However when files are removed, this can lead to external fragmentation.
+>
+> **Advantages**
+>
+> * **Simple** to implement - only location of the first block and the length of the file must be stored
+> * **Optimal read/write performance** - blocks are clustered in nearby sectors, hence the seek time (of the hard drive) is minimised
+>
+> **Disadvantages**
+>
+> * The **exact size** is not known before hand (what if the file size exceeds the initially allocated disk space)
+> * **Allocation algorithms** needed to decide which free blocks to allocate to a given file
+> * Deleting a file results in **external fragmentation**
+>
+> Contiguous allocation is still in use in **CD-ROMS & DVDs**
+>
+> * External fragmentation isn't an issue here as files are written once.
+
+#### Linked List Allocation
+
+To avoid external fragmentation, files are stored in **separate blocks** that are **linked**.
+
+> Only the address of the first block has to be stored to locate a file
+>
+> * Each block contains a **data pointer** to the next block
+>
+> **Advantages**
+>
+> * Easy to maintain (only the first block needs to be maintained in directory entry)
+> * File sizes can **grow and shrink dynamically**
+> * There is **no external fragmentation** - every possible block/sector is used (can be used)
+> * Sequential access is straight forward - although **more seek operations** required
+>
+> **Disadvantages**
+>
+> * **Random access is very slow**, to retrieve a block in the middle, one has to walk through the list from the start
+> * There is some **internal fragmentation** - on average the last half of the block is left unused
+>   * Internal fragmentation will reduce for **smaller block sizes**
+>   * However **larger blocks** will be **faster**
+> * Space for data is lost within the blocks due to the pointer, the data in a **block is no longer a power of 2**
+> * **Diminished reliability**: if one block is corrupted/lost, access to the rest of the file is lost.
+
+![Linked list file storage](/lectures/osc/assets/b8.png)
+
+##### File Allocation Tables
+
+* Store the linked-list pointers in a **separate index table** called a **file allocation table** in memory.
+
+![FAT](/lectures/osc/assets/b9.png)
+
+> **Advantages**
+>
+> * **Block size remains power of 2** - no more space is lost to the pointer
+> * **Index table** can be kept in memory allowing fast non-sequential access
+>
+> **Disadvantages**
+>
+> * The size of the file allocation table grows with the number of blocks, and hence the size of the disk
+> * For a 200GB disk, with 1KB block size, 200 million entries are required, assuming that each entry at the table occupies 4 bytes, this required 800MB of main memory.
+
+#### I-Nodes
+
+Each file has a small data structure (on disk) called an **I-node** (index-node) that contains it's attributes and block pointers
+
+> In contrast to FAT, an I-node is **only loaded when a file is open**
+>
+> If every I-node consists of $n$ bytes, and at most $k$ files can be open at any point in time, at most $n\times k$ bytes of main memory are required.
+
+I-nodes are composed of **direct block pointers** (usually 10) **indirect block pointers** or a combination thereof.
+
+![I-nodes](/lectures/osc/assets/c1.png)