Amit’s Notes¶

Examples¶

Code 301 Redirection

An example of this is when requesting a certain snapshot from the debian archives. Let’s request for a date (January 02, 2012 22:05:11) 20120102T220511Z:

$ http --headers get http://snapshot.debian.org/archive/debian/20120102T220511Z/pool/main/b/bash/
HTTP/1.1 301 Moved Permanently
Accept-Ranges: bytes
Age: 0
Cache-Control: public, max-age=600
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 224
Content-Type: text/html; charset=UTF-8
Date: Wed, 01 Oct 2014 18:36:27 GMT
Expires: Wed, 01 Oct 2014 18:46:26 GMT
Location: http://snapshot.debian.org/archive/debian/20120102T214803Z/pool/main/b/bash/
Server: Apache
Vary: Accept-Encoding
Via: 1.1 varnish
X-Varnish: 1485917301

Notice that we get back a 301 code that stands for redirection. We then get redirected to http://snapshot.debian.org/archive/debian/20120102T214803Z/pool/main/b/bash/.

Code 302 Found

Indicates resource resides temporarily under a different URI (10.3.3 302 Found).

$ http get amits-notes.readthedocs.org
  HTTP/1.1 302 FOUND
  Connection: keep-alive
  Content-Language: en
  Content-Length: 0
  Content-Type: text/html; charset=utf-8
  Date: Tue, 14 Oct 2014 18:37:30 GMT
  Location: http://amits-notes.readthedocs.org/en/latest/
  Server: nginx/1.4.6 (Ubuntu)
  Vary: Accept-Language, Cookie
  X-Deity: chimera-lts
  X-Fallback: True

GET¶

Fetch a resource. Example in python:

def get():
# Simple GET of index.html
headers = { 'User-Agent': 'http_client/0.1',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate' }
http_conn = http.client.HTTPConnection("localhost")
http_conn.set_debuglevel(1)
http_conn.request("GET", "/", headers=headers)

## Response
resp = http_conn.getresponse()
print()
print("Status:", resp.status, resp.reason)

## Cleanup
http_conn.close()

Difference Between POST and PUT¶

1. POST is used for creating, PUT is used for updating (and creating). It’s also worthwhile to note that PUT should be idempotent whereas POST is not.

   • Idempotent means that same request over and over has same result. Thus, if you are doing PUT and connection dies, you can safely do a PUT again.

2. Also, according to HTTP/1.1 spec:

   The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line

   The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI.”

3. Thus, POST can be used to create. PUT can be used to create or udpate.

4. Difference is in terms of API calls. You usually do a POST to an API endpoint (or a URL that already exists).

POST https://www.googleapis.com/dns/v1beta1/projects/project/managedZones

{
    parameters*
}

5. With PUT you actually create a valid path under the URL:

PUT /questions/<new_question> HTTP/1.1
Host: wahteverblahblah.com

6. Thus, you use PUT to create the resource and then use that URL for POST.
7. Note that with POST, server decides new URL path, with PUT user decides.

keepalive in Linux¶

Default is two hours before starting to send keepalive packets:

# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75

# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

To add support to your application use setsockopt() and configure the socket connection for keepalive.

Can also use libkeepalive with LD_PRELOAD to add support to any C application.

Basic Auth¶

This is the simplest form of authentication since it doesn’t require cookies, session identifier or login pages. It uses standard HTTP Authorization header to send login credentials. Thus, no handshakes need to be done.

Typically used over https since encoding is done in base64 (passwords sent as plain text). Passwords can be easily decoded.

On Server, status code 401 is sent back and the following header is used:

WWW-Authenticate: Basic realm="Restricted"

On Client, the Authorization header is used with the following format:

Authorization: Basic base64("username:password")

Example in python:

def get_auth():
# GET with authorization of index.html
authstring = base64.b64encode(("%s:%s" % ("amit","amit")).encode())
authheader = "Basic %s" % (authstring.decode())
print("Authorization: %s" % authheader)

headers = { 'User-Agent': 'http_client/0.1',
            'Accept': '*/*',
            'Authorization': authheader,
            'Accept-Encoding': 'gzip, deflate' }
http_conn = http.client.HTTPConnection("localhost")
http_conn.set_debuglevel(1)
http_conn.request("GET", "/", headers=headers)

## Response
resp = http_conn.getresponse()
print()
print("Status:", resp.status, resp.reason)

## Cleanup
http_conn.close()

Digest¶

Basically uses MD5 of password and nonce value to prevent replay attacks. Now, pretty much replaced by HMAC (keyed-hash message authentication code).

A basic digest authentication session goes as follows:

1. HTTP client performs a request (GET, POST, PUT, etc)

2. HTTP server responds with a 401 error not authorized. In the response, a WWW-Authenticate header is sent that contains:

   • Digest algorithm - Usually MD5.
   • realm - The access realm. A string identifying the realm of the server.
   • qop - Stands for quality of protection (e.g. auth)
   • nonce - Server generated hash, issued only once per 401 response. Server should also have a timeout for the nonce values.

3. Client then receives the 401 status error and parses the header so it knows how to authenticate itself. It responds with the usual header and adds an Authorization header containing:

   • Digest username
   • realm
   • nonce - Sends the server generated value back.
   • uri - Sends the path to the resource it is requesting.
   • algorithm - The algorithm the client used to compute the hashes.
   • qop
   • nc - hexadecimal counter for number of requests.
   • cnonce - client generated nonce, always is generated per request.
   • response - Computed hash of md5(HA1:nonce:nc:cnonce:qop:HA2).
     • HA1 = md5(username:realm:password)
     • HA2 = md5(<request method.:uri)

   Notice how the client does not send the password in plain text.

4. Server computes hash and compares to client’s hash and if it matches sends back OK with content. Note that rspauth sent back by server is a mutual authentication proving to client it knows its secret.

5. Note that each client needs to know the password and the password needs to be shared securely before hand.

Example HTTP Capture:

C:
GET /files/ HTTP/1.1
Host: localhost
User-Agent: http_client/0.1
Accept-Encoding: gzip, deflate
Accept: */*

S:
HTTP/1.1 401 Unauthorized
Server: nginx/1.6.1
Date: Sat, 06 Sep 2014 02:09:24 GMT
Content-Type: text/html
Content-Length: 194
Connection: keep-alive
WWW-Authenticate: Digest algorithm="MD5", qop="auth", realm="Access Restricted", nonce="2a27b9b6540a6cd4"

C:
GET /files/ HTTP/1.1
Host: localhost
User-Agent: http_client/0.1
Accept-Encoding: gzip, deflate
Accept: */*
Authorization: Digest username="amit", realm="Access Restricted", nonce="2a27b9b6540a6cd4", uri="/files/",
response="421974c0c2805413b0d4187b9b143ecb", algorithm="MD5", qop="auth", nc=00000001, cnonce="e08190d5"

S:
HTTP/1.1 200 OK
Server: nginx/1.6.1
Date: Sat, 06 Sep 2014 02:09:24 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Authentication-Info: qop="auth", rspauth="33fea6914ddcc2a25b03aaef5d6b478b", cnonce="e08190d5", nc=00000001..
Content-Encoding: gzip

Example Python Code:

def get_auth_digest():
    resp = get()

    # Get dictionary of headers
    headers = resp.getheader('WWW-Authenticate')
    h_list = [h.strip(' ') for h in headers.split(',')]
    #h_tuple = re.findall("(?P<name>.*?)=(?P<value>.*?)(?:,\s)", headers)
    h_tuple = [tuple(h.split('=')) for h in h_list]
    f = lambda x: x.strip('"')
    h = {k:f(v) for k,v in h_tuple}
    print(h)

    # HA1 = md5(username:realm:password)
    ha1_str = "%s:%s:%s" % ("amit",h['realm'],"amit")
    ha1 = hashlib.md5(ha1_str.encode()).hexdigest()
    print("ha1:",ha1)

    # HA2 = md5(GET:uri) i.e. md5(GET:/files/)
    ha2_str = "%s:%s" % ('GET',path)
    ha2 = hashlib.md5(ha2_str.encode()).hexdigest()
    print("ha2:",ha2)

    # Generate cnonce
    cnonce = hashlib.sha1(str(random.random()).encode()).hexdigest()[:8]
    print("cnonce:",cnonce)

    # Generate response = md5(HA1:nonce:00000001:cnonce:qop:HA2)
    resp_str = "%s:%s:%s:%s:%s:%s" % (ha1,h['nonce'],"00000001",cnonce,h['qop'],ha2)
    resp_hash = hashlib.md5(resp_str.encode()).hexdigest()
    print("resp_hash:",resp_hash)

    # Do another get
    authheader = 'Digest username="%s", realm="%s", nonce="%s", ' \
                 'uri="%s", response="%s", algorithm="%s", qop="%s", nc=00000001, ' \
                 'cnonce="%s"' \
                 % ("amit", h['realm'], h['nonce'], path, resp_hash, h['Digest algorithm'], h['qop'], cnonce)
    print(authheader)
    headers = { 'User-Agent': 'http_client/0.1',
                'Accept': '*/*',
                'Accept-Encoding': 'gzip, deflate',
                'Authorization': authheader
              }
    get(headers)

Cookie Based¶

Cookies are designed to maintain state. Thus, cookie based authentication inherits this stateful principle. Cookie authentication are the most common method used by web servers to know if the user is still logged in or not. The browser keeps sending back the same cookie to the server in every request.

Browser uses Set-Cookie header to ask client to store the cookie. The client uses Cookie header to send back the cookie to the server so the server knows which client it is talking to.

Cookies are incompatible with REST style/architecture since REST is stateless. According to REST style, cookies maintain site-wide state while REST styles maintains application state. In REST, cookie functionality can be achieved using anonymous authentication and client-side state. REST also defines an alternative to cookies when implementing shopping carts. According to REST:

Likewise, the use of cookies to identify a user-specific “shopping basket” within a server-side database could be more efficiently implemented by defining the semantics of shopping items within the hypermedia data formats, allowing the user agent to select and store those items within their own client-side shopping basket, complete with a URI to be used for check-out when the client is ready to purchase.

Cookies have certain rules and attributes:

1. Name/value pair can’t contain spaces or ; =. Usually only ASCII characters. The ; is used as a delimiter.
2. The Secure attribute means this cookie is only used in encrypted communications.
3. The HttpOnly attribute means this cookie can only be used by http/https requests and not by JavaScript, etc. This prevents cross site scripting.

Other notes:

1. Not good practice to store username/password in cookies, even if it is hashed/salted, etc. Can be stolen and eventually cracked.
2. Cookie based authentication basically involves using the cookie the server sent to the client back to the server for every request.

Certificate Based¶

Idea is to separate those who verify password (the server will have a copy or a hash of the password) and those who define the user identity. Thus, certificate authority (CA) issues a private certificate to a user, and guarantees that it can communicate using this key with the public key issued to the other business party.

Note that the downside becomes apparent when large number of clients or users need to authenticate to the server. Thus, CA needs to issue certificate for each user. These certificates needs to be verified and if one user is compromised the certificate of that user can be used to authenticate to the server unless the certificate is revoked.

For the reasons stated above, client authentication is rarely used with TLS. A common technique is to use TLS to authenticate the server to the client and to establish a private channel, and for the client to authenticate to the server using some other means - for example, a username and password using HTTP basic or digest authentication.

The above image depicts certificate-based authentication. The client asks the user to enter a password which unlocks the database holding the private key. The client then uses this private key to sign a random data and sends a certificate to the server. Thus, the password is never sent.

The Red Hat Portal discusses this in great detail.

Trusting a Web Site¶

1. Web browsers know how to trust HTTPS websites based on certificate authorities that come pre-installed in their software.
2. Certificate authorities, such as Comodo and GlobalSign, are in this way being trusted by web browser creators to provide valid certificates.
3. Must use a browser that correctly implements HTTPS with correct pre-installed certificates.
4. User trusts CA to “vouch” for website they issued certificate to.
5. The user trusts that the protocol’s encryption layer (TLS/SSL) is sufficiently secure against eavesdroppers.

TLS/SSL¶

1. Uses assymetric cryptography:
   • Basically known as public key cryptography.
   • Requires two keys. A private/secret and a public key.
   • Public key is used to encrypt plain text or verify a digital signature. Private key is used to decrypt the plain text or create a digital signature.
2. The assymetric key is used for authentication and encrypting the channel. Then, a symmetric session key is exchanged.
3. The session key is used to encrypt data flowing between the parties. Important property is forward secrecy. This means that the short-term session key cannot be derived from long term assymetric secret key.
4. In OSI model equivalences, TLS/SSL is initialized at layer 5 (session layer) and works at layer 6 (the presentation layer). The session layer has a handshake using an asymmetric cipher in order to establish cipher settings and a shared key for that session; then the presentation layer encrypts the rest of the communication using a symmetric cipher and that session key.
5. TLS is the new name for SSL.
6. SSL got to version 3.0 and TLS is “SSL 3.1”.
7. Current version of TLS is 1.2.
8. SSL (secure socket layer) often refers to the old protocol variant which starts with the handshake right away and therefore requires another port for the encrypted protocol such as 443 instead of 80.
9. TLS (transport layer security) often refers to the new variant which allows to start with an unencrypted traditional protocol and then issuing a command (usually STARTTLS) to initialize the handshake.
10. Differences between SSL and TLS in the protocol level:
    • In the ClientHello message (first message sent by the client, to initiate the handshake), the version is {3,0} for SSLv3, {3,1} for TLSv1.0 and {3,2} for TLSv1.1.
    • The ClientKeyExchange differs.
    • The MAC/HMAC differs (TLS uses HMAC whereas SSL uses an earlier version of HMAC).
    • The key derivation differs.
    • The client can send application data can be sent straight after ending the SSL/TLS Finished message in SSLv3. In TLSv1, it must wait for the server’s Finished message.
    • The list of cipher suites differ (and some of them have been renamed from SSL_* to TLS_*, keeping the same id number).
    • There are also differences regarding the new re-negotiation extension.
11. Use port 443 by default.
12. TLS, which uses long-term public and secret keys to exchange a short term session key to encrypt the data flow between client and server.
13. X.509 certificates are used to guarantee one is talking to the partner with whom one wants to talk.
14. Need to ensure scripts are loaded over HTTPS as well and not HTTP.
15. In case of compromised secret (private) key, certificate can be revoked.
16. Use Perfect Forward Secrecy (PFS) so that short term session key can’t be derived from long term assymetric secret key.

Client                                               Server

ClientHello                  -------->
                                                ServerHello
                                               Certificate*
                                         ServerKeyExchange*
                                        CertificateRequest*
                             <--------      ServerHelloDone
Certificate*
ClientKeyExchange
CertificateVerify*
[ChangeCipherSpec]
Finished                     -------->
                                         [ChangeCipherSpec]
                             <--------             Finished
Application Data             <------->     Application Data

Server Setup¶

1. To prepare a web server to accept HTTPS connections, the administrator must create a public key certificate for the web server.
2. This certificate must be signed by a trusted certificate authority for the web browser to accept it without warning.
3. Web browsers are generally distributed with a list of signing certificates of major certificate authorities so that they can verify certificates signed by them.

Other Uses¶

1. The system can also be used for client authentication in order to limit access to a web server to authorized users. To do this, the site administrator typically creates a certificate for each user, a certificate that is loaded into his/her browser.

Permissions¶

Make sure the permissions of the files in the directory are accessible to the other group. Or change the permissions to the user that nginx runs as (for debian it’s www-data).

Setting up Basic Auth¶

1. Install apache2-utils to get htpasswd
2. Create an .htpasswd file in the web root. Make sure the permissions are 644. Note that the password generated by htpasswd is an apache modified version of MD5.

sudo htpasswd -c /usr/share/nginx/html/.htpasswd amit

3. Update /etc/nginx/sites-available/default in the location / and reload nginx:

# Basic auth
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;

Setting up Digest Auth¶

1. apache2-utils includes htdigest (similar to htpasswd) to generate digest key.
2. Create an .htdigest file in the web root. Make sure the permissions are 644. Note that the realm here is “Access Restricted”.

sudo htdigest -c /usr/share/nginx/html/.htdigest "Access Restricted" amit

3. Need to build with nginx-http-auth-digest module from https://github.com/rains31/nginx-http-auth-digest. In order to do this, download nginx debian sources, copy nginx-http-auth-digest to debian/modules, and finally edit debian/rules to build nginx-http-auth-digest (look at –add-module config option).
4. Update /etc/nginx/sites-available/default in the location / and reload nginx:

# Digest auth
auth_digest "Access Restricted";    # Realm
auth_digest_user_file /usr/share/nginx/html/.htdigest;

HTTPie - Command Line HTTP Client¶

Very useful and feature rich command line http client written in Python (http://github.com/jakubroztocil/httpie).

Useful for debugging HTTP requests. For example:

$ http get http://localhost
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Mon, 01 Sep 2014 18:31:03 GMT
Last-Modified: Tue, 05 Aug 2014 11:18:35 GMT
Server: nginx/1.6.1
Transfer-Encoding: chunked

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Example named.hosts file for the Physics Department¶

; Authoritative Information on physics.groucho.edu.
@  IN  SOA niels.physics.groucho.edu. janet.niels.physics.groucho.edu. {
                  1999090200       ; serial no
                  360000           ; refresh
                  3600             ; retry
                  3600000          ; expire
                  3600             ; default ttl
                }
;
; Name servers
              IN    NS       niels
              IN    NS       gauss.maths.groucho.edu.
gauss.maths.groucho.edu. IN A 149.76.4.23
;
; Theoretical Physics (subnet 12)
niels         IN    A        149.76.12.1
              IN    A        149.76.1.12
name server   IN    CNAME    niels
otto          IN    A        149.76.12.2
quark         IN    A        149.76.12.4
down          IN    A        149.76.12.5
strange       IN    A        149.76.12.6
...
; Collider Lab. (subnet 14)
boson         IN    A        149.76.14.1
muon          IN    A        149.76.14.7
bogon         IN    A        149.76.14.12
...

1. The SOA record signals the Start of Authority, which holds general information and configuration on the zone the server is authoritative for.
2. CNAME always points to another name. This name then has an assiociated A record.
3. Note that all names in the sample file that do not end with a dot should be interpreted relative to the physics.groucho.edu (e.g. boson, muon) domain. The special name (@) used in the SOA record refers to the domain name by itself.
4. The name servers for the groucho.edu domain somehow have to know about the physics zone so that they can point queries to their name servers. This is usually achieved by a pair of records: the NS record that gives the server’s FQDN, and an A record that associates an address with that name. Since these records are what holds the namespace together, they are frequently called glue records.

Query All Records using dig¶

$ dig +nocmd com.google any +multiline +noall +answer

Or

$ dig com.google any

Capturing ARP Traffic¶

When using tcpdump to capture ARP, make sure to dump the hex output (-X) and also decode ethernet header using (-e). Note: Use *-XX* to also show ethernet header dump.

$ sudo tcpdump -nnvvv -e -X arp
  tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
  20:01:28.452956 48:5a:b6:51:57:dd > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.1 tell 192.168.1.23, length 46
      0x0000:  0001 0800 0604 0001 485a b651 57dd c0a8  ........HZ.QW...
      0x0010:  0117 0000 0000 0000 c0a8 0101 0000 0000  ................
      0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
  20:01:28.454472 bc:ee:7b:58:17:b8 > 48:5a:b6:51:57:dd, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.1 is-at bc:ee:7b:58:17:b8, length 28
      0x0000:  0001 0800 0604 0002 bcee 7b58 17b8 c0a8  ..........{X....
      0x0010:  0101 485a b651 57dd c0a8 0117            ..HZ.QW.....

Capturing Traffic on Localhost¶

During development, there is usually a local webserver setup in http://localhost. Custom apps/scripts are tested against this local webserver to make them functionally correct. Thus, it is important to be able to analyze traffic to and from the local webserver. Using the local webserver for traffic analysis helps as there are no external traffic that will confuse the analysis.

To capture localhost traffic:

sudo tcpdump -A -v --number -i lo tcp port http

• -A is used to decode protocol in ASCII.
• -v is used for verbose mode. This allows us to see tcp communication details (flags, sequence numbers, etc).
• –number denomitate the packets
• -i lo use local loopback interface
• tcp port http the filter specifying protocol and port to use for capture.

Use -l for line buffering to see data while capturing it to a file.

sudo tcpdump -l -A -v --number -i lo tcp port http | tee /tmp/capture

Capturing GMail Traffic¶

GMail goes over IMAP but not the standard IMAP port (143), it uses 993:

sudo tcpdump -vvv -X --number -i wlan0 host 192.168.1.24 and tcp port 993

Use -vvv (three is max) to decode max level of the packets. Then use -X to decode in Hex and ASCII.

Dropped Packets by the Kernel¶

tcpdump uses a little buffer in the kernel to store captured packets. If too many new packets arrive before the user process tcpdump can decode them, the kernel drops them to make room for freshly arriving packets.

Use -B to increase the buffer. This is in units of KiB (1024 bytes).

Capturing TCP SYN Packets¶

To capture SYN packets only:

$ sudo tcpdump -nnvvv host 192.168.1.116 and "tcp[tcpflags] & tcp-syn != 0"

To capture TCP keepalive packets 1-byte or 0-byte ACKs. Note that a keepalive probe is a packet with no data and ACK flag turned on:

$ sudo tcpdump -vv "tcp[tcpflags] == tcp-ack and less 1"

Capture Outgoing SSH Traffic¶

$ sudo tcpdump -nn src 192.168.1.116 and tcp port 22

Get Time Delta Between Request/Response¶

Pass the -ttt flag to get the time delta between current line and previous line.

$ sudo tcpdump -nS -ttt port http and host snapshot.debian.org

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

00:00:00.000000 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [S], seq 1140376233, win 29200, options [mss 1460,sackOK,TS val 22265623 ecr 0,nop,wscale 7], length 0
00:00:00.228373 IP 193.62.202.30.80 > 192.168.1.170.34233: Flags [S.], seq 1460190713, ack 1140376234, win 5792, options [mss 1350,sackOK,TS val 74072844 ecr 22265623,nop,wscale 7], length 0
00:00:00.000040 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [.], ack 1460190714, win 229, options [nop,nop,TS val 22265680 ecr 74072844], length 0
00:00:00.000119 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [P.], seq 1140376234:1140376399, ack 1460190714, win 229, options [nop,nop,TS val 22265680 ecr 74072844], length 165
00:00:00.222658 IP 193.62.202.30.80 > 192.168.1.170.34233: Flags [.], ack 1140376399, win 54, options [nop,nop,TS val 74072902 ecr 22265680], length 0
00:00:00.001001 IP 193.62.202.30.80 > 192.168.1.170.34233: Flags [P.], seq 1460190714:1460191405, ack 1140376399, win 54, options [nop,nop,TS val 74072902 ecr 22265680], length 691
00:00:00.000032 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [.], ack 1460191405, win 239, options [nop,nop,TS val 22265736 ecr 74072902], length 0
00:00:00.008210 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [F.], seq 1140376399, ack 1460191405, win 239, options [nop,nop,TS val 22265738 ecr 74072902], length 0
00:00:00.183523 IP 193.62.202.30.80 > 192.168.1.170.34233: Flags [F.], seq 1460191405, ack 1140376400, win 54, options [nop,nop,TS val 74072960 ecr 22265738], length 0
00:00:00.000060 IP 192.168.1.170.34233 > 193.62.202.30.80: Flags [.], ack 1460191406, win 239, options [nop,nop,TS val 22265784 ecr 74072960], length 0

Capturing WiFi Packets¶

First, the wlan0 interface needs to be set to monitor mode:

$ sudo ifconfig wlan0 down
$ sudo iwconfig wlan0 mode Monitor
$ sudo ifconfig wlan0 up

Then, run tcpdump with the following flags:

$ sudo tcpdump -I -i wlan0 -w thermostat.pcap -e -s 0 ether host 00:d0:2d:xx:xx:xx

This captures all packets originating from the Honeywell thermostat for example.

Difference between class A(object): and class A:¶

Subclassing object yields a new-style class (in Python 3, class A: defaults to new style). Some differences:

1. Method Resolution Order (MRO) defined by __mro__ attribute of class, defines how inheritance hierarchies are walked. Before, it was depth first. Now, it is more sane and is based on __mro__.
2. The __new__ constructor is added. This allows class to act as factory method, rather than return new instance of class. Useful for returning particular subclasses, or reusing immutable objects rather than creating new ones without having to change the creation interface.
3. Descriptors. These are the feature behind such things as properties, classmethods, staticmethods etc. Essentially, they provide a way to control what happens when you access or set a particular attribute on a (new style) class.

class D(object):
    pass

class E:
    pass

dir(D)
# ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__' , '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__wea kref__']

dir(E)
# ['__doc__', '__module__']

How are arguments passed - by reference or by value?¶

It is actually call by object, call by sharing, or call by object reference, so the answer is neither.

In Python, everything is an object and all variables hold references to objects. So what’s being passed are objects (references) and can only be changed if they are mutable (lists and dicts). Note that numbers, strings, and tuples are immutable.

Sum/multiply all the elements in a list¶

# basic
s = 0
for x in range(10):
    s += x

# the right way
s = sum(range(10))

# basic
s = 1
for x in range(1, 10):
    s *= x

# the other way
from operator import mul
s = reduce(mul, range(1,10))

This brings up the discussion of functional programming concepts in python. These functions can be used in conjunction with lambda instead of longer for loops.

map(function, seq)

Takes a function and applies it to each item in the sequence. The resulting object is an iterable object. Thus, apply list() to the map object to get a list output. Or loop through it.

a = map(lambda x: x*2, range(0,10))
for i in a:
    print(i)

filter(function, seq)

Filter extracts elements in the sequence that return True. Note that function can be None, and thus it will return items that are True.

a = filter(lambda x: x > 1, range(0,10))
list(a)

reduce(function, seq)

Reduce applies a function of two arguments, cumulatively to the items of a sequence. It returns one value back (the cumulative value).

from functools import reduce
a = reduce(lambda x, y: x * y, [1,2,3,4])
a == 24 # True

Note that that function takes two arguments. The sequence of operations goes as follows (((1*2) * 3) * 4).

Difference between tuples and list¶

Lists are mutable while tuples are not. More importantly, tuples can be hashed (used as keys for dictionaries). Tuples are used if order of elements in a sequence matters (e.g. coordinates, points of a path, etc).

t = ((1,'a'), (2,'b'))
dict(t)
# OUT: {1: 'a', 2: 'b'}

dict((y,x) for x,y in t)
# OUT: {'b': 2, 'a': 1}

{y:x for x,y in t}
# OUT: {'b': 2, 'a': 1}

What are decorators and what is their usage?¶

Decorators allow you to inject or modify code in functions or clases. Basically, a wrapper to an existing function. Thus, allows you to execute a code before or after the original code. For example, logging a function.

from __future__ import print_function

def log(fn):
    def wrapper(*args, **kw):
        res = fn(*args, **kw)
        print("%s(%r) -> %s" % (fn.__name__, args, res))
        return res
    return wrapper

@log
def ispal(word):
    if len(word) < 2:
        return True
    return (word[0] == word[-1]) & ispal(word[1:-1])


ispal("test")
ispal("kayak")

Misusing expressions as defaults for function arguments¶

def foo(bar=[]):
    bar.append("paz")
    return bar

1. Expect to return paz everytime foo() is called. But this is not the case.
2. After calling foo() three times, you will get [“baz”, “baz”, “baz”]
3. This is because, the the default value for a function argument is only evaluated once, at the time that the function is defined.
4. To get around it:

def foo(bar=None):
    if bar == None:
        bar = []
    bar.append("paz")
    return bar

Predictive vs Adaptive¶

1. The usual inspiration for methodologies comes from engineering disciplines such as civil or mechanical engineering. Such disciplines put a lot of emphasis on planning before you build.
2. So what we see here are two fundamentally different activities. Design which is difficult to predict and requires expensive and creative people, and construction which is easier to predict. Once we have the design, we can plan the construction.
3. So the approach for software engineering methodologies looks like this: we want a predictable schedule that can use people with lower skills. To do this we must separate design from construction.
4. This is where design notations such as UML come into play. If we can make all the significant decisions using the UML, we can build a construction plan and then hand these designs off to coders as a construction activity.
5. For Engineering designs, mathematical analysis can show if a design will work. However, for UML it is only through peer-review that design can be checked.
6. For a bridge, cost of design is only about 10% of job, rest is construction. For code it is the opposite, apparently only 15% of project is code and unit tests.
7. Thus, source code is the design document and construction is the compiler/linker. Thus, the construction phase is cheap and and should be automated.
8. Conclusions:
   • In software: construction is so cheap as to be free
   • In software all the effort is design, and thus requires creative and talented people (source code)
   • Creative processes are not easily planned, and so predictability may well be an impossible target.
   • We should be very wary of the traditional engineering metaphor for building software. It’s a different kind of activity and requires a different process.

Putting People First¶

1. Executing an adaptive process is not easy. In particular it requires a very effective team of developers. The team needs to be effective both in the quality of the individuals, and in the way the team blends together.

The Self-Adaptive Process¶

1. However there’s another angle to adaptivity: that of the process changing over time. A project that begins using an adaptive process won’t have the same process a year later. Over time, the team will find what works for them, and alter the process to fit.

2. The first part of self-adaptivity is regular reviews of the process. Usually you do these with every iteration. At the end of each iteration, have a short meeting and ask yourself the following questions (culled from Norm Kerth)

   • What did we do well?
   • What have we learned?
   • What can we do better?
   • What puzzles us?

3. While both published processes and the experience of other projects can act as an inspiration and a baseline, the developers professional responsibility is to adapt the process to the task at hand.

Flavors of Agile Development¶

Should you go agile?¶

1. In today’s environment, the most common methodology is code and fix. Applying more discipline than chaos will almost certainly help, and the agile approach has the advantage that it is much less of a step than using a heavyweight method.
2. Simpler processes are more likely to be followed when you are used to no process at all.
3. The first step is to find suitable projects to try agile methods out with. Since agile methods are so fundamentally people-oriented, it’s essential that you start with a team that wants to try and work in an agile way.
4. So where should you not use an agile method? I think it primarily comes down to the people.

User Stories¶

Notes taken from: http://www.mountaingoatsoftware.com/agile/user-stories

1. User stories are short, simple description of a feature told from the perspective of the person who desires the new capability. This is usually a user or customer of the system. Template:

   As a <type of user>, I want <some goal> so that <some reason>.

2. Often written on index cards, sticky notes and placed in a shoe box, arranged on walls, tables, to facilitate planning and discussion.

3. Focus shifts from writing about features to discussing them.

4. User stories can be written at varying levels of detail. Thus, user stories can be written to cover large amounts of functionality. These are generally known as epics. An example:

   As a user, I can backup my entire drive

5. Epics are generally too large to complete in one agile iteration. It is split into smaller user stories. The above epic can be split into dozens (or hundreds) of user stories:

   As a power user, I can specify files or folders to backup based on file size, date created and date modified.

   As a user, I can indicate folders not to backup so that my backup drive isn’t filled up with things I don’t need saved.

6. Note that details can be added to user stories. These can be accomplished by splitting user stories into smaller user stories. Or by adding conditions of satisfaction.

7. Conditions of satisfaction are like high-level acceptance tests. For the example above:

   • Make sure data is verified during copy.
   • Make sure there is a report generated of the backup.

8. Note that user stories are usually written by product owner. However, during breakdowns of user stories, each team member can write it as well.

9. The important fact is that it doesn’t matter who writes the user stories. It is more important to have everyone involved in the discussion of it.

10. User Stories are usually the main composition of a product backlog. Re-prioritization happens often and user stories can be added/removed throughout the agile development process.

11. Note that Fibonnaci sequence is used to estimate story points. The idea is, the larger the story is, the more uncertainty there is around it and the estimate is less accurate. Thus, total number of points give a number on complexity of project.

    • 0, 1 means user story doesn’t take anytime at all
    • Bigger than 13 means very complex and probably needs to be broken up.

Burndown Charts¶

1. A burn down chart is a graphical representation of work left to do versus time.
2. The outstanding work (or backlog) is often on the vertical axis, with time along the horizontal.
3. It is useful for predicting when all of the work will be completed.

Agile Fluency¶

Reference: http://martinfowler.com/articles/agileFluency.html

This diagram explains stages agile teams go through as they gain more experience. This shows successful team progression. Note that fluency here means how a team develops software when it’s under pressure.

Most teams are at one-star level. Number of teams with more stars are fewer as there are factors such as organizational culture, technical debt of code, etc.

Fluency is more about habit than skills and thus requires a lot of practice.

It’s best to choose the level of fluency you want to achieve and to practice everything needed for that level from the beginning.

In other words, if your goal is to have a three-star team, use a three-star approach from the start. Although your team will still progress one level at a time, practicing all the techniques together will allow them to advance more quickly.

Important for organization to support team star goals and for team members to stick together.

1. Scrum is frequently used by one-star teams as their core goal/metric is business value of customer/stakeholders. The focus is on creating value. The idea is organization can realize its investment quickly (2-6 months) and have greater insight into team progress.

2. Two-star teams deliver on market cadence (shipping as often as market will accept it). Usually use XP combined with Scrum project management. Includes continuous integration, test drive development, test driven development, pair programming, and collective ownership.

   • Consistently and predictably deliver value.
   • Includes metrics used by one-star team to report business value. But core metric is to deliver low-defect product and ability to ship on market cadence.
   • Takes significant investment in time as team needs a lot of skill and practice to consistently deliver these kinds of products.

3. Three-star teams deliver the most value possible for your investment. They understand what the market wants, what your business needs, and how to meet those needs.

   • However, Lean Startup is an example of a method that operates at the three-star level. It’s most applicable to new product development. The ideas from Lean Software Development (no relation to Lean Startup) are also useful. Agile chartering, embedded product management teams, customer discovery, and adaptive planning are all examples of techniques used by three-star teams.
   • More formal business value reports are presented. There is a mutual trust between team and organization. Need to incorporate business experts full time in the team.
   • Takes several years to develop because it takes time to develop this level of trust between organization and team.

4. Four-star teams contribute to enterprise-wide success. Team members understand organizational priorities and business direction. Four-star teams will sacrifice their own needs to support the needs of a product more critical to business success. They work with other teams and with managers to optimize the overall value stream.

   • The teams we know that are striving for, and in some cases reaching, four-star fluency are at the “bleeding edge” of Agile practice. They adapt ideas from advanced management theories and innovative product development methods. Techniques include Agile portfolio management, systems thinking, value stream analysis, whole system planning, intact teams, open book management, and radical self-organization.
   • The core metric for four-star teams is whether the team shows understanding of the overall system and reports how its actions affect the enterprise.
   • To date, we’ve most often seen four-star fluency in single-team startups, where it’s not much different from three-star fluency. It seems to be easiest to approach four-star fluency in organizations where trust is high, communication overhead is low, and business information is widely shared.

Managing Deadlines¶

UNIX, Terminals, and C1¶

1. UNIX is a terminal-based system (which means that it is usually operated by entering commands on a terminal).
2. UNIX used to be controlled by a dumb terminal. It is a physical equipment with a 80x25 screen and a keyboard that connects to the UNIX server by a serial line.
3. If you send text to the serial port of the terminal, it is the terminal that decides which actual character to print on the screen. Similarly, when you press a key on the keyboard, it is the terminal that decides which code to send to the UNIX server.
4. On top of this, this same communication channel needs to be used for sending control sequences, such as for moving the cursor, clearing the screen, and these sequences are all sent in-line intermixed with the character codes. This means that we can’t assign characters to every code point. We need control codes, and these control codes cannot overlap the characters in the character encoding.
5. Note that printable ASCII range is from 32 to 126. Codes 128 and above, however, are a free-for-all. Terminals can use them as control codes, printable characters, or basically anything and it doesn’t matter to the UNIX server.
6. Block of control codes 0 to 31 (known as the C0 block) as well as code 127 in ASCII are already allocated to control codes.
7. For reasons of consistency and interoperability, a recommendation was developed that codes 128 to 159 should be reserved for a second control block (called the C1 block), and only codes 160 to 255 be used for printable characters, and this formed the basis for most pre-Unicode UNIX encodings.
8. However, it is not a hard and fast rule, and UNIX itself does not treat the C1 block any differently from the rest of the upper 128 codes.
9. This makes UNIX basically language agnostic. Let’s say we have a UNIX system and we have two files with names consisting of the character codes 68 196 and 68 228. When we connect a Greek language terminal to the system and list the files, we will see “DΔ” and “Dδ”. When we connect a French terminal to the system, we will see “DÄ” and “Dä”, and when we connect a Thai terminal we will see “Dฤ” and “Dไ”. The fact that the three different terminals show 3 different things doesn’t matter. A Thai user can still open one of the files by typing “vi Dฤ” on their keyboard, the same as the Greek can by typing “vi DΔ”. As far as the OS is concerned, as long as the underlying codes are the same, everything works fine. We might note here that “δ” is the lower case letter for “Δ” in Greek, whereas “ฤ” and “ไ” are completely different unrelated letters in Thai. This doesn’t matter to UNIX because the OS doesn’t try to do anything fancy like case-insensitive file names. The same applies to file content. If you print a file to the screen, the codes gets passed directly to the terminal, and it is the terminal that does the rendering. The OS doesn’t need to get involved.
10. Thus UNIX itself does not need an “encoding” setting. It simply doesn’t care.
11. However, any command or system service that is going to output human-readable error messages needs to know which language to output. Similarly, any add-on programs, particularly those that might perform some kind of collation or text processing, want to know what language they should use. But again, this does not need to be a system-wide setting, and only need apply to a particular user’s session. The system thus uses an environment variable (LOCALE) which specifies both the language and character encoding.

DOS/Windows¶

1. DOS and Windows does not use terminals and thus screen is addressed directly through the video adapter.
2. This means that it does not need any control codes. So DOS/Windows assigns printable characters to all of the ASCII control codes (0 to 31 and 127) and all codes above 128.
3. DOS/Windows is still considered ASCII based since maintains all of ASCII printable characters (32 to 126).
4. This obviously is a major problem as line feeds get messed up in text files. And there is a problem with printing to ASCII serial printers which uses ASCII control block for printer sequences.
5. Another problem with Windows is that it uses case-insensitive file names and thus the OS needs to know the encoding (whether Greek, Thai, etc). Thus, Windows requires a system wide encoding.
6. It should come as no surprise then that Microsoft was one of the big backers of Unicode and produced one of the first Unicode-based OSes.

Programming With 8-Bit Encodings¶

1. In most cases you don’t need to write encoding-aware software. For example, the C compiler does not need encoding-awareness.
2. In regular expression, all characters are byte-based and thus user needs to only input correct expressions for their language (e.g. a Norwegian person wanting to match all upper case letters is going to have to use the regexp /[A-ZÅÆ]/ instead of /[A-Z]/, but the actual regular expression parser does not need to be changed).
3. The basic assumption is that we have the same encoding end-to-end, so tools don’t have to do any conversion, they just spit out whatever input they get in.
4. If you want language-dependent operations, then use <locale.h> and <ctype.h>. You will need to perform setlocale and then do isupper, islower, etc.

Multibyte Encodings¶

1. Multibyte encodings are needed for East Asian languages of China, Japan, and Korea (CJK). ASCII is not enough to hold all possible combinations.
2. Other encodings based on ASCII, use multi-bytes to represent a single character.
3. The basic idea is that all non-ASCII letters are encoded as multi-byte sequences with all bytes in the 128 and higher code range. This means that we can suddenly start naming files in Japanese and Chinese without making any changes to the underlying OS.
4. However, the following will not work char mychar = '字'; since this will appear to the compiler as two characters. Also, regexes to do not work as expected. Finally, outputting to fixed-width terminal or printer will not work as expected since these characters now take up two bytes (thus two spaces) to display one character. The exception here is Shift JIS. Single byte characters in Shift JIS take up single character cell while double-byte characters always take two character cells.
5. However, with Shift JIS, they use the code 92 (backslash) as part of second byte. Thus, printf("十"); actually looks like printf("X\"); to the compiler. There are 42 characters that use backslash in Shift JIS.
6. Prior to Unicode, the C and C++ standard libraries do not have any functions for handling CJK encodings. Note that you can use ``mblen()`` to see how many bytes a character takes.

Pre-Unicode Summary¶

1. Firstly, just about every application runs off the idea of a single encoding end-to-end. On UNIX, in particular, many protocols are encoding agnostic.
2. FTP is a great example. It has no concept of encoding or any way of specifying encodings. Whatever raw character data it receives it sends on through untouched. It’s up to the sys admin to make sure everyone using it sticks to the same encoding.

Unicode¶

1. The very first thing we need to understand is that UTF-8 did not exist and was not envisioned when Unicode first came into use, and Unicode referred basically to what we now call UTF-16.
2. Windows NT was the first major OS based on Unicode, and was coupled with NTFS, the first major filesystem to support Unicode.
3. However, NT still had the concept of a default non-Unicode encoding, the same as DOS and Windows 95.
4. This non-Unicode encoding is called the ANSI code page on Windows. The Win32 API thus comes with two complete sets of API, the ANSI API which accept string arguments of type char* encoded in the default non-Unicode encoding, and the Unicode (or wide char) API which accept string arguments of type wchar_t*.
5. On UNIX, things were slightly different, and this is reflected in the C/C++ standard libraries. On UNIX, the expectation was that people would continue to use non-Unicode encodings in conjunction with the LOCALE environment variable, but programmers would be given the option of an automatic conversion to Unicode mode of file operation, that would allow them to code using Unicode while the underlying OS and file content would continue to use non-Unicode encodings.
6. When we open a file in C (using fopen()), the file does not have an orientation. If we call a non-Unicode file function such as fgets(), the file gets set to non-Unicode orientation, and we simply get passed the data from the file.
7. However, if we call a Unicode file function such as fgetws(), the file is put into Unicode orientation. This does not mean that the file on the disk is treated as containing Unicode. Instead, the C standard library assumes that the file is encoded in the non-Unicode encoding as specified by LOCALE and performs implicit automatic conversion of the file content from that non-Unicode encoding into Unicode in the form of wchar_t*.
8. On Windows NT, however, we have a problem. The C standard library still assumes that files are saved in a non-Unicode filesystem. There is no support for Unicode filenames. There is no variant of the fopen() function which accepts a wchar_t* for the filename.
9. UNIX chose 32-bit Unicode standard (ISO based) while Windows went with 16-bit universal encoding (Unicode organization). Thus, eventually we ended up with a 32-bit Unicode standard with UTF-16 as a variable-length 16-bit encoding.
10. Around the sametime UTF-8 was being developed, which was another variable length encoding.
11. Once UTF-8 was released, UNIX had a clear path towards total Unicode compatibility. Simply adopt UTF-8 as your standard encoding and all your problems go away.
12. UTF-8 is the basic standard encoding used in Linux and it’s derivatives. Since UTF-8 can be encoded in a char*, it even works fine with the broken C standard libraries. UTF-8 plugs into the C standard library on what was meant to be the non-Unicode side since it is stored in char* and specified as an encoding using LOCALE. However, Linux/Windows interoperation is worse than ever.

Basic Programming With Unicode¶