Most feature requests suck. They are typically vague, unrealistic or misaligned with actual needs. However, well-written requests can drive meaningful product innovation.
![how-to-write-good-feature-requests](//images.ctfassets.net/vwq10xzbe6iz/3UpOop2KOl0kCX8ownoxCK/4880bcb8e4c6f4af54924a0645b7e048/how-to-write-good-feature-requests.png)

This template guides product owners, internal stakeholders and customers in writing effective feature requests for development teams.

__Key Sections__
1. __Title__: clearly state what the improvement is
2. __Problem__: explain the context, result and impact of the problem
3. __Steps to recreate__: highlight the limits of the current solution
4. __Solution__: propose a solution to the problem

## 1. Title
Clearly state what the feature improvement request is.

__Title format:__
<span style="background:lightblue;border-width:1px;padding:0px 2px">[Product feature]: </span>
<span style="background:lightgreen;border-width:1px;padding:0px 2px">[Action required] </span>
<span style="background:lightyellow;border-width:1px;padding:0px 2px">[Component or outcome]</span>

__Good title:__
> <span style="background:lightblue;border-width:1px;padding:0px 2px">Notifications: </span>
> <span style="background:lightgreen;border-width:1px;padding:0px 2px">Highlight </span>
> <span style="background:lightyellow;border-width:1px;padding:0px 2px">unread @mentions</span>

__Bad title:__
> Missing out on @mentions all the time, needs overhaul for better user engagement

## 2. Problem
Explain the specific situation, the resulting negative impact and why this is problematic for the user or company.

__Problem format:__
<span style="background:lightblue;border-width:1px;padding:0px 2px">When… [specific situation]. </span>
<span style="background:lightyellow;border-width:1px;padding:0px 2px">This happens…[negative impact]. </span>
<span style="background:salmon;border-width:1px;padding:0px 2px">This is a problem because…[for user or company].</span>

__Good problem statement:__
> <span style="background:lightblue;border-width:1px;padding:0px 2px">When users have turned off their email notifications and use the notifications menu to see what they have missed. </span>
> <span style="background:lightyellow;border-width:1px;padding:0px 2px">The mentions go unnoticed due to lack of visual emphasis and become hard to spot amongst other activity updates. </span>
> <span style="background:salmon;border-width:1px;padding:0px 2px">This leads to reduced team collaboration and the mentioners to go back to email communication instead.</span>

__Bad problem statement:__
> People are not seeing @mentions in our platform, which is really annoying. It's just not good enough and makes things difficult for everyone.

## 3. Steps to recreate
Step-by-step guide to recreate the situation and highlight limits of the existing solution. Begin each step with an action verb and keep it concise. Include screenshots and screen recordings.

__Good example:__
> 1. Navigate to user preferences and disable email notifications
> 2. Use another account to @mention the first user and add multiple other non-mention comments
> 3. Access the notifications menu of the first user and check for updates
> 4. Attempt to identify @mentions amidst other activity updates. Note the difficulty in distinguishing the @mention from other activities 

## 4. Solution
Propose a solution to the problem including any potential rabbit holes with it. Be open to other solutions you may not have thought of.

__Proposed solution:__
> __Solution__
> 
> Make it easier for users to see any unread @mentions 
> - Sort any unread @mentions to the top of the notification list
> - Add a prefix to the notification with the following copy “Mentioned you on: [x]”
> 
> __Rabbit holes__
> - Priority sorting @mentions will lose chronological order and may add more confusion
> - We may have to reserve prefixes for future notification messages
> 
> __Other solutions__
> - May be worth exploring other visual methods of highlighting @mentions without losing chronological order


Product feature request template with examples. Writing tips for product owners to clearly communicate feature improvements to development teams.

how-to-write-good-feature-requests

How to write good feature requests

Password hashing is a crucial aspect of cybersecurity and is used by developers to avoid storing actual passwords in a database, which is at risk of hacks and leaks.
![password_security](//images.ctfassets.net/vwq10xzbe6iz/3Dzelf37uaf5qUxbNgY36S/ac23c1700cca34c6a025b3af4e7652e6/password_security.png)
This article demonstrates how hackers can exploit hashing vulnerabilities and why it's essential for developers to use unique salts when hashing passwords.

1. [How does password hashing work?](#how)
2. [Dangers of fixed salts](#dangers)
3. [Secure password hashing](#secure)

### <a name="how"></a>1. How does password hashing work?
To illustrate how password hashing works, we will use bcrypt, a popular Node.js password hashing algorithm.

#### Hashing passwords

In simple terms, hashing works as follows when a user first registers a password:

> bcrypt(password + salt) => salt + hashed_password

Where:
- bcrypt is our hashing function
- password is our plain text password, for example "chicken"
- salt is a unique key
- hashed_password is a converted set of characters after passing through bcrypt(password + salt)

The salt + hashed_password combination is what is stored in the database against the unique user ID or email, not the original password.

The function bcrypt(password + salt) will always generate the same result, being salt + hashed_password.

The function is designed to be one-way, meaning if you have the salt and hashed_password, you cannot reverse engineer the original password.

#### Verifying passwords

In simple terms, verifying that the submitted password matches the hash stored in the database works as follows:

1. Receive the user email and password combination, for example, jo@email.com and "chicken"
2. Look up the hash stored against jo@email.com `salt+hashed_password`
3. Extract the salt from the hash (remember the hash is `salt+hashed_password`)
4. Run the password and salt through the bcrypt function to return the `salt+hashed_password` combination
5. Verify that the `hashed_password` matches the hash stored in the database

The code below illustrates this. The next section talks about the dangers of using the same salt when hashing passwords.

```javascript
const bcrypt = require('bcrypt');

// Our plain text password
const password = "chicken"

// Our random salt, a unique key generated by bcrypt.genSaltSync() 
const salt = '$2b$10$QyI8c1WJnKjBn7pGg1cn8e';

// The function returns a hash
const hash = bcrypt.hashSync(password, salt)
// hash: $2b$10$QyI8c1WJnKjBn7pGg1cn8eRs1DsEeTppMu.RodMnUP6bzQJdvYWIe

// hash has two components: salt+hashed_password
// $2b$10$QyI8c1WJnKjBn7pGg1cn8e + Rs1DsEeTppMu.RodMnUP6bzQJdvYWIe

// Compare password
// extract the salt from the hash above
const saltFromHash = '$2b$10$QyI8c1WJnKjBn7pGg1cn8e'
// Give the password and salt from hash to bcrypt to generate the hash
const compare = bcrypt.hashSync("chicken", saltFromHash)

// Compare the result against the stored hash
if (compare === hash) {
  return true
} else {
  return false
}
```

### <a name="dangers"></a>2. Dangers of fixed salts
To illustrate the dangers of using fixed salts, let's take a look at a simple example in Node.js with bcrypt.

We follow the same example above by hashing the password "chicken." Imagine our developer makes a mistake and uses the same salt to hash our passwords. Now, each time a user registers the password "chicken," we will generate the same hash.

```javascript
const bcrypt = require('bcrypt');

// Users who share the same plain text password
const password = "chicken"

// ======= DANGER EXAMPLE ==========
// Use an example fixed salt to hash our password, generated by bcrypt.genSaltSync(10) and reused
const fixedSalt = '$2b$10$QyI8c1WJnKjBn7pGg1cn8e';

// Hash the password using the fixed salt
const dangerUser1 = bcrypt.hashSync(password, fixedSalt)
const dangerUser2 = bcrypt.hashSync(password, fixedSalt)
const dangerUser3 = bcrypt.hashSync(password, fixedSalt)

// DANGER RESULT: Users that have the same passwords have the same hashed passwords
// dangerUser1: $2b$10$QyI8c1WJnKjBn7pGg1cn8eRs1DsEeTppMu.RodMnUP6bzQJdvYWIe
// dangerUser2: $2b$10$QyI8c1WJnKjBn7pGg1cn8eRs1DsEeTppMu.RodMnUP6bzQJdvYWIe
// dangerUser3: $2b$10$QyI8c1WJnKjBn7pGg1cn8eRs1DsEeTppMu.RodMnUP6bzQJdvYWIe
```

This becomes a major security vulnerability. If a hacker gets a list of all hashes and knows that one belongs to "chicken," they can simply do a quick search to access other users' accounts.

### <a name="secure"></a>3. Secure password hashing
To protect password security, developers should generate random salts each time they hash passwords.

In the example below, we modify the code to generate a random salt each time we hash the same password. Notice this time we generate a unique hash for each user. This is possible because we are using a random salt each time we hash a newly registered password.

Now if hackers crack one password hash, they will not be able to search for others.

```javascript
const bcrypt = require('bcrypt');

// Users who share the same plain text password
const password = "chicken"

// ======= SECURE EXAMPLE ==========
// Use a random salt to hash our password every time, generated by bcrypt.genSaltSync(10) but not reused
const secureUser1 = bcrypt.hashSync(password, bcrypt.genSaltSync(10))
const secureUser2 = bcrypt.hashSync(password, bcrypt.genSaltSync(10))
const secureUser3 = bcrypt.hashSync(password, bcrypt.genSaltSync(10))

// SECURE RESULT: Users with the same passwords have unique hash
// secureUser1: $2b$10$8WVMo207.PukGs.2MFInvOuJxDIMBNXmAUWkedOuZHtCXnui5k8Ou
// secureUser2: $2b$10$AtmhuYfG8dtctiX7ctES5.Omr2Ew/qN05/qLK8KOe4LlF7r6uGBay
// secureUser3: $2b$10$jB54SBsF6YGfIFmZbv.qauXyLCK2d8SR9R.wGBiOSl4SwXVWBTJ96
```

Don't forget the salt 🧂

1) How does password hashing work?; 2) Dangers of fixed salts; 3) Secure password hashing. Code examples using bcrypt and Node.js

password_security

Password Security 101 - Understanding How Hashing Works

A REST API is the most common standard used between Clients (a person or application) who want to access information from the web from Servers (application or database) who have access to that information.

![rest-api](//images.ctfassets.net/vwq10xzbe6iz/5sBH4Agl614xM7exeLsTo7/9e84dce01735f155911e611c42c9793f/rest-api.png)

An Application Programming Interface (API) is a way for two computers to talk to each other over the web. For example, a delivery app can use the Google Maps API to support location tracking instead of building one from scratch. Or a recipe website can use an AI model to suggest alternative recipes based on the same ingredients using the ChatGPT REST API.

An API that follows the REST standard is called RESTful. Here are good practices when designing and using a RESTful API using the example of a survey app.

1. [Basic format](#basic)
2. [HTTP verbs](#http)
3. [URL endpoints](#url)
4. [Status codes](#status)

## <a name="basic"></a>1. Basic format 

The format of a RESTful API has three main components: 1) URL endpoint; 2) HTTP verb; and 3) Body.

__URL endpoint:__
Is a URL link that represents the resources we want to access. Resources can be text, images, documents or any data entry. For example, `example.com/surveys` allows us to view or create survey templates and `example.com/surveys/123/responses` allows us to do the same for all responses of survey 123.

__HTTP verb:__
Tells the server what we want to do with the URL endpoint resource. For example, a `POST` request means we want to create a new survey template and a `GET` request means we want to view an existing survey template.

__Body message:__
Is an optional custom payload which contains a message with the properties and values we want to use to create or update a given resource. 

For example, we want to create a new response for the survey 123 which we captured which includes a positive NPS score of 9, a short feedback message and the unique id of the respondent who gave the feedback. 

```json
// Request:
POST example.com/surveys/123/responses

// Body payload:
{
  survey_id: 123, 
  nps_score: 9, 
  feedback: "love the service", 
  respondent_id: 42
}
```

The server receives the HTTP request and returns a response that includes a HTTP status code and a response usually in JSON format. Here is an example request and response to a REST API server asking to read all the responses for survey 123:

__Request:__
```curl
GET http://example.com/surveys/123/responses
```

__Response:__
```json
// HTTP status code: 200
{
  survey_id: 123,
  survey_title: "nps onboarding survey",
  responses: [
  {
    response_id: 42,
    nps_score: 9,
    feedback: "love the service",
    respondent_id: 42
  }
  ...
 ]
}
```

## <a name="http"></a>2. HTTP verbs
There are 5 basic verb commands when making a HTTP request:
- __GET__: make a read only request to view either a single or list of multiple resources
- __POST__: create a new resource based on the payload given in the body of the request
- __DELETE__: destroy the given resource based on the id provided 
- __PUT__: update the entire fields of the resource based on the given body of the request or create a new one if not already exist
- __PATCH__: update only the fields of the resource if it exists

Most applications and resources will support all these commands. This is commonly referred to as a CRUD application:

| Acronym     | HTTP verb     |
| ---------- | ---------- |
| Create       | POST       |
| Read       | GET       |
| Update       | PUT & PATCH       |
| Delete       | DELETE       |

## <a name="url"></a>3. URL endpoints
A URL endpoint in a RESTful API represents any given object, data or service the API can access. For example, `example.com/surveys` represents the data for all survey templates and `example.com/surveys/123/responses` the data for all responses of a given survey.

URL endpoints should be grouped as pluralised nouns around the business data and objects and not verbs. For example, `example.com/surveys` and not `example.com/getAllSurveys`.

URL endpoints should include the unique identifier as a path after the pluralised noun when viewing, updating or deleting a single item. For example, `example.com/surveys/123`.

Organise URL collections in a logical hierarchy based on relationships. For example, users are most likely to access responses to a given survey template. This would be represented as `example.com/surveys/123/responses`. 

Here is a summary of good practice URL endpoints and how they interact with HTTP verbs:

| URL endpoint resource     | GET     | POST     | PUT     | DELETE     |
| ---------- | ---------- | ---------- | ---------- | ---------- |
| __/surveys__       | Retrieve all surveys       | Create a new survey       | Bulk update surveys (not advised)       | Remove all surveys (not advised)       |
| __/surveys/123__       | Retrieve the details for survey 123       | Error       | Update the details of survey 123 if it exists       | Remove survey 123       |
| __/surveys/123/responses__       | Retrieve all responses for survey 123       | Create a new response for survey 123       | Bulk update responses for survey 123 (not advised)       | Remove all responses for survey 123 (not advised)       |
| __/responses/42__       | Retrieve the details for response 42       | Error       | Update the details of response 42 if it exists       | Remove response 42       |

## <a name="status"></a>4. Status codes
Upon receiving a HTTP request in the basic RESTful format the server will return a HTTP status code along with any optional JSON payloads. Here are the most common HTTP status codes and their meaning:

| Status code     | Meaning     |
| ---------- | ---------- |
| 200 OK       | Request was successful.       |
| 301 Moved Permanently       | For SEO purposes when a page has been moved and all link equity should be passed through.        |
| 401 Unauthorized       | Server requires authentication.       |
| 403 Forbidden       | Client authenticated but does not have permissions to view resource.       |
| 404 Not Found       | Page not found because no search results or may be out of stock.       |
| 500 Internal Server Error       | Server side error. Usually due to bugs and exceptions thrown on the server side code.      |
| 503 Server Unavailable       | Server side error. Usually due to a platform hosting, overload and maintenance issue.       |


How web applications talk to servers: 1. Basic format; 2. HTTP verbs; 3. URL endpoints; 4. Status codes.

rest-api

REST APIs Explained - 4 Components

Unit testing is a method to check that units of code work as expected. It helps teams ship product improvements without breaking existing functionality.
![unit testing](//images.ctfassets.net/vwq10xzbe6iz/6oRDUzlFPphTwBXN2kBF8f/3e833fd1119c5498e514f11a9885bdb4/unit_testing.png)

This article introduces the basics of unit testing in Node.js.

### Sections
1. [Setup unit testing framework](#setup)
2. [Writing unit testing assertion style](#writing)
3. [Testing for validations and throwing errors](#errors)
4. [Handling asynchronous code](#async)
5. [Using dependency injection to stub](#stub)

Code for all examples can be [found here](https://github.com/howardmann/blog_testing).

## <a name="setup"></a>1. Setup unit testing framework
We have been asked to build an API that fetches nutritional data for a given fruit. The 3rd party API already exists but we also need to display calories to kilojoules.

First we must setup our unit testing structure and framework. We will create two functions to achieve this:
1. Calorie to kilojoule converter
2. Fruit data fetcher

Test files should be kept close to the folders and functions being tested. Mirror the naming of your functions with a `.spec.js` extension.

```shell
├── convertCalToKj
│   └── index.js
│   └── convertCalToKj.spec.js
└── fetchFruit
│   └── index.js
│   └── fetchFruit.spec.js
```
We will install [`jest`](https://jestjs.io/) for our testing framework  but you could use other frameworks such as `mocha` and `chai`. We will also install `axios` in advance for our http requester which we will later stub.

```shell
npm init -y
npm i --save-dev jest
npm i --save axios
echo node_modules  >> .gitignore
```

Our package.json should look like below. Add to our scripts `"test: "jest"` and our `jest` config to auto run all files with our `.spec.js` extension.
```javascript
{
  "name": "blog_testing",
  "version": "1.0.0",
  "type": "module",
  "description": "Unit testing basics example",
  "main": "util.js",
  "scripts": {
    "test": "jest" // shortcut to run our test library with npm test
  },
  "author": "Howie Mann",
  "license": "ISC",
  "dependencies": {
    "axios": "^1.1.3" // used for our data fetcher
  },
  "devDependencies": {
    "jest": "^29.2.2" // our testing framework
  },
  "jest": {
    "testMatch": ["**/*.spec.js"], // match all .spec.js files
    "verbose": true // display all individual tests
  }
}
```
## <a name="writing"></a>2. Writing unit testing assertion style
First spec out the functionality we are expecting to test. Wrap our tests in `describe` and `test` blocks. `describe` encapsulates the related functions we are testing and `test` runs the tests.

```javascript
// convertCalToKj/convertCalToKj.spec.js

// Import the function we expect to test
let convertCalToKj = require('./index')

describe('.convertCalToKj', () => {
  // test happy path
  test('should convert calories to kilojoules', () => {
  })

  // test edge cases
  test('should work for number strings', () => {
  })

  // test for errors
  test('should throw error if not given a string', () => {
  })
})
```

Write unit tests following this assertion style:
1. **Input**: the input parameter we pass to the function we are testing
2. **Actual**: the actual result of passing the input to our function `function(input)`
3. **Expected**: the expected result from passing the input to our function

Then compare the actual vs expected resut to check if they match.

Let's apply this assertion style to our first two specs.
```javascript
// convertCalToKj/convertCalToKj.spec.js

// Import the function we expect to test
let convertCalToKj = require('./index')

describe('.convertCalToKj', () => {
  // test happy path
  test('should convert calories to kilojoules', () => {
    // Follow basic `input` `actual` and `expected` style
    // 1. input = input parameters being passed to function being tested
    // 2. actual = actual output of function being passed input
    // 3. expected = expected result of the test case

    let input = 52
    let actual = convertCalToKj(input)
    let expected = 217.568

    // Basic assertion syntax
    // Assertions for strings, numbers use `.toBe()` whilst assertions for checking values of objects use `.toEqual()` or `toStrictEqual()` for deep equality    
    expect(actual).toBe(expected)
  })

  // test edge cases
  test('should work for number strings', () => {
    let input = '52'
    let actual = convertCalToKj(input)
    let expected = 217.568

    expect(actual).toBe(expected)
  })
})
```
We can now make our code work before running the tests to pass by running `npm test` in our terminal.

```javascript
// convertCalToKj/index.js
let convertCalToKj = (cal) => {
  let calorie = parseInt(cal)

  if (isNaN(calorie)) {
    throw new Error('not a number')
  }
  return calorie * 4.184 
}

module.exports = convertCalToKj
```

## <a name="errors"></a>3. Testing for validations and throwing errors
We will write validations to ensure that only numbers or number strings are being passed. 

To test validations, we will deliberately pass inputs that we expect will throw errors. The `jest` framework requires the code we are error testing be wrapped in an anonymous function inside expect `expect(() => function(input))` with the assertion library using the method `toThrow()`.

Write a unit test for when a non number string input is passed to `convertCalToKj()` that an error is thrown with a message `not a number`.

```javascript
let convertCalToKj = require('./index')

describe('.convertCalToKj', () => {
  // Test unhappy path
  test('should throw error if not given a number or number string', () => {
    let input = "banana"
    // When testing to throw errors wrap our `actual` in an arrow function and call it at run-time
    let actual = () => convertCalToKj(input)
    let expected = "not a number"

    // Call the function when inside the expect assertion and use `.toThrow()` to assert an Error has been thrown
    expect((input) => actual()).toThrow(expected)
  })
})
```
Test for multiple bad inputs by looping through and testing each. This approach can also be used for testing multiple happy inputs.
```javascript
let convertCalToKj = require('./index')

describe('.convertCalToKj', () => {
  // Test all unhappy paths
  test('should throw error if not given a number string', () => {
    // Test non number inputs
    let inputArr = ['banana', undefined, true, false, [], {}]
    let actual = (input) => convertCalToKj(input)
    let expected = "not a number"

    // Use forEach loop to test for multiple error assertions
    inputArr.forEach(input => {
      expect(() => actual(input)).toThrow(expected)
    })
  })
})
```

## <a name="async"></a>4. Handling asynchronous code
We will first write an integration test that relies on an external 3rd party library to fetch data. We will test that `fetchFruit(fruit)` will return nutritional information for a given fruit.

We will import `axios` as our request library to dependency inject into `fetchFruit` this will allow us to stub this out in our later tests. Remember to use `async` `await` when testing asynchronous code.

Follow the same assertion style to test that the actual response Object matches the expected Object.
```javascript
// fetchFruit/fetchFruit.spec.js
// Import our function that we are testing
let {fetchFruit} = require('./index')
// Import our request library used for integration test
let axios = require('axios')

describe('.fetchFruit', () => {
  // full integration test using async await
  test('should fetch nutrition information for a given fruit', async() => {
    let input = 'apple'
    // axios library returns response in data property
    let resp = await fetchFruit(input, axios)
    let actual = resp.data
    // Expected API response
    let expected = {
      "genus": "Malus",
      "name": "Apple",
      "id": 6,
      "family": "Rosaceae",
      "order": "Rosales",
      "nutritions": {
        "carbohydrates": 11.4,
        "protein": 0.3,
        "fat": 0.4,
        "calories": 52,
        "sugar": 10.3
      }
    }

    // assertions for objects use .toEqual()
    expect(actual).toEqual(expected)
  })
})
```

Now write our code to satisfy the above spec and pass the tests. We dependency inject our http request helper using default params which will default to using the `axios` library if none is given.
```javascript
// fetchFruit/index.js
let axios = require('axios')

// request=axios default params  allow us to stub for testing
let fetchFruit = (fruit, request=axios) => {
  // 3rd party API library
  let api = "https://www.fruityvice.com/api/fruit/"
  let url = api + fruit

  return request.get(url)
    .catch(err => {
      throw new Error(`fetchFruit: ${err}`)
    })
}
```
## <a name="stub"></a>5. Using dependency injection to stub
We use dependency injection to pass any third party libraries like our http request library `axios` as arguments to our functions.

This allows us to stub them out when writing testing to return an expected fixture we can use to test other functions which we will do next.

To create our stub write a `fakeAxiosRequest` Object which has a similar API method `.get` and returns the expected data as a Promise. Then write a unit test to ensure the stub is working.

```javascript
// 
let {fetchFruit} = require('./index')
let axios = require('axios')

// our expected API fetch result we will test across our suites 
const FETCH_RESULT = {
  "genus": "Malus",
  "name": "Apple",
  "id": 6,
  "family": "Rosaceae",
  "order": "Rosales",
  "nutritions": {
    "carbohydrates": 11.4,
    "protein": 0.3,
    "fat": 0.4,
    "calories": 52,
    "sugar": 10.3
  }
}

describe('.fetchFruit', () => {
  // Stubbing responses using dependency injection
  test('should fake fetch a fruit object', async() => {
    // fake Axios request function with a get method that resolved our expected fruit object
    let fakeAxiosRequest = {
      get: () => {
        return Promise.resolve({data: FETCH_RESULT})
      }
    }

    // Check our stub works
    // Note: we create and test this fake fetch stub in order to test multiple other functions. There is low value in testing a stub standalone 
    let input = 'apple'
    let resp = await fetchFruit(input, fakeAxiosRequest)
    let actual = resp.data
    let expected = FETCH_RESULT

    expect(actual).toEqual(expected)
  })
})
```
Writing stubs with fake requests has the benefit of allowing us to combine tests with other utilities.

Let's put it all together and create a function `fetchFruitWithKj` that fetches nutritional information for a given fruit and also converts and displays the calories in kilojoules.
```javascript
// fetchFruit/index.js
let axios = require('axios')
let convertCalToKJ = require('../convertCalToKj')

let fetchFruit = (fruit, request=axios) => {
  let api = "https://www.fruityvice.com/api/fruit/"
  let url = api + fruit

  return request.get(url)
    .catch(err => {
      throw new Error(`fetchFruit: ${err}`)
    })
}

let fetchFruitWithKj = async (fruit, request=axios) => {
  let resp = await fetchFruit(fruit, request)
  let output = resp.data  
  let calories = output.nutritions.calories
  let kj = convertCalToKJ(calories)    
  // creae copy of output to not mutate
  let newOutput = JSON.parse(JSON.stringify(output))
  newOutput['nutritions']['kilojoules'] = kj
  return newOutput
}

module.exports = {
  fetchFruit,
  fetchFruitWithKj
}
```

We can dependency inject our fakeAxiosRequest stub and test that `fetchFruitWithKj("apple", fakeAxiosRequest)` returns the nutritional information with calories converted to kilojoules.

```javascript
let {fetchFruit, fetchFruitWithKj} = require('./index')

// our expected API fetch result we will test across our suites 
const FETCH_RESULT = {
  "genus": "Malus",
  "name": "Apple",
  "id": 6,
  "family": "Rosaceae",
  "order": "Rosales",
  "nutritions": {
    "carbohydrates": 11.4,
    "protein": 0.3,
    "fat": 0.4,
    "calories": 52,
    "sugar": 10.3
  }
}

describe('.fetchFruitWithKj', () => {
  // Using our stub we can test functions combined with 3rd party API requests
  test('should fake fetch a fruit object with converted kj value', async() => {
    // our fake Axios stub
    let fakeAxiosRequest = {
      get: () => {
        return Promise.resolve({data: FETCH_RESULT})
      }
    }

    // Make copy of FETCH_RESULT object and add nested kj value
    const FETCH_RESULT_KJ = JSON.parse(JSON.stringify(FETCH_RESULT))
    FETCH_RESULT_KJ["nutritions"]["kilojoules"] = 217.568

    let input = 'apple'
    let actual = await fetchFruitWithKj(input, fakeAxiosRequest)    
    let expected = FETCH_RESULT_KJ

    expect(actual).toStrictEqual(expected)
  })
})
```

Code for all examples can be [found here](https://github.com/howardmann/blog_testing)

### References
- Blog by James Sinclair - [TDD Should be Fun](https://jrsinclair.com/articles/2016/tdd-should-be-fun/)
- Video by Dev Mastery - [The Ultimate Coding Workflow](https://www.youtube.com/watch?v=nK3LP-pn_08)


Unit testing in Node.js with examples. How teams ship product  improvements without breaking code. 1) Setup testing framework with Jest; 2) Writing assertion style; 3) Validation and error handling; 4) Handling asyncronous code; 5) Dependency injection to stub.


unit testing

Unit Testing in NodeJS - Basics with Examples

A practical tutorial of using React Hooks, Custom Hooks and Context to add a new feature *"keyboard shortcuts"* to the classic Todo App teaching exercise.

### New Feature
- **Product Brief:** Keyboard shortcut keys to change filter display of todolist results ("a" -> Show All, "d" -> Show Done, "n" -> Show Not Doing)
- **How?** Use React Custom Hooks to listen for key presses and dispatch changes to state

### Gotchas
- **Product Brief:** Disable shortcut keys when user is typing inside input text fields (e.g. when creating or editing todo names)
- **How?** Use React Context to track global state of whenever a user has focus in an input text field

[See live demo](https://mannhowietodoapp2.surge.sh/) and [previous blog](https://mannhowie.com/react-and-redux-a-visual-explanation) for a visual explanation of React.

## 1. What are Hooks
Hooks are reusuable functions for your React app when you want to share logic that fetches and updates state, listens for changes in user and window events or handles authentication. The two built-in hooks that most custom hooks will build atop are `useState` and `useEffect`.

### useState
useState built-in hook is used to read and write values to component state. It takes an initial default value and returns an array with two elements. The first is the current value of the state and the second is a state setter function that updates the current value. 

```javascript
// useState example of a simple counter component
import React from 'react'

export default function Counter() {
  let initialState = 0
  let [count, setCount] = React.useState(initialState);

  let handleIncrement = () => setCount(count + 1)

  let handleReset = () => setCount(0);

  return (
    <div>
      <h1>Counter</h1>
      <p>Count: {count}</p>
      <button onClick={handleIncrement}>Increment</button>
      <button onClick={handleReset}>Reset</button>
    </div>
  );
}
```

### useEffect
useEffect built-in hook manages the component lifecycle and side-effects of components. It is used to fetch and update data, listen for changes to state, register and deregister event listeners and clean up components.

It takes two arguments, the first a callback function and the second an array of state values it will listen for changes before executing the callback function.

```javascript
// useEffect example to fetch and persist between UI state and localStorage
import React from 'react'

export default function Counter() {
  // initial value last saved localstorage if exists else 0
  let initialState = JSON.parse(localStorage.getItem('count') || 0)

  let [count, setCount] = React.useState(initialState);

  let handleIncrement = () => setCount(count + 1)

  let handleReset = () => setCount(0);

  // useEffect listen for changes to count and persist state change to localStorage
  React.useEffect(() => {
    localStorage.setItem('count', JSON.stringify(count))
  },[count])

  return (
    <div>
      <h1>Counter</h1>
      <p>Count: {count}</p>
      <button onClick={handleIncrement}>Increment</button>
      <button onClick={handleReset}>Reset</button>
    </div>
  );
}
```

## 2. Our First Custom Hook
To implement our new shortcut key feature we will create two custom hooks:
1. `useKeyPress`: to register and deregister window event listeners for `keydown` and `keyup` of target keys
2. `useFilterKeyPress`: will use our useKeyPress custom hook to register specific keys and dispatch filter events to update our filter view

### useKeyPress
Our first custom hook useKeyPress takes a single argument `targetKey` (e.g. keyboard letter "a") and returns an updated useState current value (a boolean) whenever a window keydown or keyup event is triggered.

We use the built-in useState hook to keep track of whether the targetKey has been pressed `keyPressed` which returns a boolean.

Our simple handlers check whether the pressed key matches the targetKey and returns either true or false.

We use the built-in useEffect hook to to register our window event listeners for the keydown and keyup events and register our handler callbacks. We also tidy up our event listeners by returning a function which removes the event listeners upon component tear down.

```javascript
// hooks/useKeyPress.js
import  React  from 'react';

export const useKeyPress = (targetKey) => {  
  const [keyPressed, setPressedKey] = React.useState(false)

  let handleKeyDown = (e) => {    
    if (e.key === targetKey) {
      setPressedKey(true)
    }
  }

  let handleKeyUp = (e) => {
    if (e.key === targetKey) {
      setPressedKey(false)
    }
  }

  React.useEffect(() => {
    window.addEventListener('keydown', handleKeyDown)
    window.addEventListener('keyup', handleKeyUp)

    return () => {
      window.removeEventListener('keydown', handleKeyDown)
      window.removeEventListener('keyup', handleKeyUp)
    }
  }, [])

  return keyPressed
}
```
### useFilterKeyPress
Our second custom hook useFilterKeyPress will use our previous custom hook `useKeyPress` to listen for key presses and will then dispatch events to update our parent component state. It takes a dispatch function as an argument from the parent app to dispatch changes to our parent reducer.

We first register our shortcut key presses ("a", "d", "n"). This will return a state boolean for each value and will update whenever a keydown or keyup event is triggered.

We use the useEffect hook to dispatch the relevant filter actions based on which key is pressed. It takes an array of values `[pressA, pressD, pressN]` which will trigger the functions inside the hook whenever those values change.

```javascript
// hooks/useFilterKeyPress.js

import React from 'react'
import {useKeyPress} from './useKeyPress'

export const useFilterKeyPress = (dispatch) => {
  const pressA = useKeyPress("a")
  const pressD = useKeyPress('d')
  const pressN = useKeyPress('n')

  // useEffect hook detect changes when input not in focus in keys above, if true dispatch filter change
  React.useEffect(() => {
    if (pressA) {
      dispatch({type: 'SHOW_ALL'})
    }
    if (pressD) {
      dispatch({type: 'SHOW_DONE'})
    }
    if (pressN) {
      dispatch({type:'SHOW_NOT_DONE'})
    }
  },[pressA, pressD, pressN])
}
```

## 3. React Context API
The React Context API allows us to share global state amongst parent and children components without passing them through intermediaries (known as prop drilling).

In our product brief we are made aware of a gotcha with our shortcut keys. We need to disable the shortcut keys if a user is entering text inside an input text field. This is to prevent the user from unintentionally triggering the filter updates when simply entering text.

As our application scales there is likely to be future input text fields which may be nested deeply as children components.

This is a great use case of the React Context API for us to keep track of whenever a user has `focus` inside an input text field.

### Create our new context
First we create our context object which we will import across parent and children components. We will use this context as a way to track global state of whether an input text field is in focus or not.
```javascript
//context/contextInput.js
import React from 'react'

const ContextInput = React.createContext()

export default ContextInput
```

### Update useFilterKeyPress
Next we must update our custom hook `useFilterKeyPress` to not trigger the filter actions if an input text field is in focus.

We create a new boolean state `isInputFocused` and a setter `setIsInputFocused` which will be based on our `ContextInput` to track whether an input text field is in focus.

We udate our useEffect code to ensure the actions are only dispatched if input is not in focus. And we return the value of `[isInputFocused, setIsInputFocused]` to allow the state to be read and updated. 

```javascript
// hooks/useFilterKeyPress.js

import React from 'react'
import {useKeyPress} from './useKeyPress'

export const useFilterKeyPress = (dispatch) => {
  const pressA = useKeyPress("a")
  const pressD = useKeyPress('d')
  const pressN = useKeyPress('n')
  // global state to detect if text input is in focus
  // within input we will set as true if input onFocus and false when input onBlur (e.g. toggles when input out of focus)
  const [isInputFocused, setIsInputFocused] = React.useState(false)

  // useEffect hook detect changes when input not in focus in keys above, if true dispatch filter change
  React.useEffect(() => {
    if (!isInputFocused) {
      if (pressA) {
        dispatch({type: 'SHOW_ALL'})
      }
      if (pressD) {
        dispatch({type: 'SHOW_DONE'})
      }
      if (pressN) {
        dispatch({type:'SHOW_NOT_DONE'})
      }
    }
  },[pressA, pressD, pressN, isInputFocused])

  return [isInputFocused, setIsInputFocused]
}
```
### Use context and custom hook in our main App
We first use our custom hook `useFilterKeyPress` which returns a value and setter to determine whether any input field is in focus. We pass it our dispatch handler which will allow the hook to update the App's parent state to change filter display options.
```javascript
  import {useFilterKeyPress} from '../hooks/useFilterKeyPress'
  // Custom hook to filter shortcut keypress
  const [isInputFocused, setIsInputFocused] = useFilterKeyPress(dispatch)
```
We then use our `ContextInput` and apply its Provider model which wraps it as a parent component and gives all children component within it access to its context values. We pass it a value property being our state values and setters `isInputFocused` and `setIsInputFocused` which will allow children components to update the state.

The two children components that will use it now include the `<NewTodoForm>` and the `<EditTodoForm>` which is a deeply nested child of the `<TodoList>`. This is the primary benefit of using React Context API, we no longer have to pass handler functions to multiple intermediaries which exposes us to code duplication and error.

```javascript
import {useFilterKeyPress} from '../hooks/useFilterKeyPress'
import ContextInput from '../context/contextInput'

const [isInputFocused, setIsInputFocused] = useFilterKeyPress(dispatch)
return (
    <div>
      <h1>TODOAPP</h1>

      {/* Using React Context for sharing global isInputFocused variable to detect if any text input is focus */}
      <ContextInput.Provider value={[isInputFocused, setIsInputFocused]}>
        <NewTodoForm handleNewTodo={handleNewTodo} />

        <Filter filter={filter} handleFilter={handleFilter}/>

        <TodoList 
          todos={todos}
          editing={editing}
          handleMarkDone={handleMarkDone}
          handleMarkEdit={handleMarkEdit}
          handleDeleteTodo={handleDeleteTodo}
          handleEditTodo={handleEditTodo}
          handleMarkEditUndo={handleMarkEditUndo}
          filter={filter}
        />

      </ContextInput.Provider>
    </div>
```

Final App.js altogether
```javascript
//src/components/App.js

import React from 'react';
import {useLocalStorage} from '../hooks/useLocalStorage'
import {useFilterKeyPress} from '../hooks/useFilterKeyPress'
import ContextInput from '../context/contextInput'

import reducer from '../reducers/index'
import NewTodoForm from './NewTodoForm'
import TodoList from './TodoList'
import Filter from './Filter'

function App() {
  let initialState = reducer(undefined, {})

  // custom hook useLocalStorage
  const [state, setState] = useLocalStorage(initialState, 'state')

  // reducer implementation of dispatch
  const dispatch = (action) => {
    setState(prevState => reducer(prevState, action))
  }

  // State handlers
  let handleMarkDone = (id) => {
    dispatch({type: 'MARK_DONE_TODO', id})
  }
  let handleMarkEdit = (id) => {
    dispatch({type: 'MARK_EDIT_TODO', id})
  }
  let handleMarkEditUndo = (id) => {
    dispatch({type: 'MARK_EDIT_UNDO_TODO', id})
  }
  let handleNewTodo = (content) => {
    dispatch({type: 'ADD_TODO', content})
    dispatch({type: 'SHOW_ALL'})
  }
  let handleEditTodo = ({id, content, done}) => {
    dispatch({type: 'EDIT_TODO', id, content, done})
  }
  let handleDeleteTodo = (id) => {
    dispatch({type: 'DELETE_TODO', id})
  }
  let handleFilter = (e) => {
    let filter = e.target.value 
    dispatch({type: filter})
  }

  // State values from reducers
  let {todo, visibility} = state
  let {todos, editing} = todo
  let {filter} = visibility

  // Custom hook to filter shortcut keypress
  const [isInputFocused, setIsInputFocused] = useFilterKeyPress(dispatch)

  return (
    <div>
      <h1>TODOAPP</h1>

      {/* Using React Context for sharing global isInputFocused variable to detect if any text input is focus */}
      <ContextInput.Provider value={[isInputFocused, setIsInputFocused]}>
        <NewTodoForm handleNewTodo={handleNewTodo} />

        <Filter filter={filter} handleFilter={handleFilter}/>

        <TodoList 
          todos={todos}
          editing={editing}
          handleMarkDone={handleMarkDone}
          handleMarkEdit={handleMarkEdit}
          handleDeleteTodo={handleDeleteTodo}
          handleEditTodo={handleEditTodo}
          handleMarkEditUndo={handleMarkEditUndo}
          filter={filter}
        />

      </ContextInput.Provider>
    </div>
  );

}

export default App;
```

### Use and update context
Finally with our children components we can now access our `ContextInput` and update its state based on whether an input field is in `focus` or not.

To do this we import the context and use React's context hook `useContext` which takes the `ContextInput` object as an argument and returns the values of the parent Provider, in this case our setter function `setIsInputFocused`. In our example we don't require the current value as we are only interested in updating the value.

Then we add two event callback functions on our input text field `onFocus` and `onBlur` to update our `IsInputFocused`value. We set it as true when `onFocus` and  false when `onBlur` to reset it.

As this value is now shared inside the global `ContextInput` object, the parent App component can now access the latest value of `IsInputFocused` and not trigger any filter actions if this value is true.

```javascript
// src/components/NewTodoForm.js
import React from 'react'
import ContextInput from '../context/contextInput'

const NewTodoForm = (props) => {
  const [content, setContent] = React.useState('')
  const [,setIsInputFocused] = React.useContext(ContextInput)

  let handleChange = (e) => {
    setContent(e.target.value)
  }
  let handleSubmit = (e) => {
    e.preventDefault()
    if (content.trim() === '') return
    props.handleNewTodo(content)
    setContent('')
  }

  return (
    <form onSubmit={handleSubmit} style={{border: '1px solid rebeccapurple'}}>
      <p><i style={{color: 'rebeccapurple'}}>NewTodoForm</i></p>
      <input type="text" name="content" value={content} onChange={handleChange} placeholder="Add new todo" onBlur={() => setIsInputFocused(false)} onFocus={() => setIsInputFocused(true)}/>
    </form>
  )
}

export default NewTodoForm
```

Similarly we follow the same pattern for the `EditTodoForm` which is nested multiple levels deep within `TodoList` component without needing to prop drill across multiple components.

```javascript
// src/components/EditTodoForm
import React from 'react'
import ContextInput from '../context/contextInput'

const EditTodoForm = (props) => {
  const [content, setContent] = React.useState(props.content)
  const [done, setDone] = React.useState(props.done)
  // ContextAPI
  const [,setIsInputFocused] = React.useContext(ContextInput)

  let handleChange = (e) => {
    setContent(e.target.value)
  }
  let handleChecked = (e) => {
    setDone(!done)
  }
  let handleSubmit = (e) => {
    e.preventDefault()

    props.handleEditTodo({
      id: props.id,
      content: content,
      done: done
    })
  }
  return (
    <div style={{border: '1px solid blue'}}>
      <i style={{color: 'blue'}}>EditTodoForm</i>
      <form onSubmit={handleSubmit}>
        <input type="text" name="content" value={content} onChange={handleChange} onBlur={() => setIsInputFocused(false)} onFocus={() => setIsInputFocused(true)} />
        <input type="checkbox" name="done" checked={done} onChange={handleChecked}/>
        <input type="submit" value="Update"/>
      </form>
      <button onClick={() => props.handleMarkEditUndo(props.id)}>Undo Edit</button>
    </div>

  )
}

export default EditTodoForm
```

[Click here to see the full github repo](https://github.com/howardmann/react-projects/tree/master/todoapp)

Practical tutorial of using React Hooks, Custom Hooks and Context API when adding new features to your web app. Adding keyboard shortcut features to the classic Todo App teaching exercise.


1. TodoApp

Learn React - Hooks and Context

![fininte state machine](//images.ctfassets.net/vwq10xzbe6iz/6JNh1Rpi6a1VUnTux03TzW/c69b8b54a1be48ec820cec8e721163af/fininte_state_machine.png)

Finite State Machines allow cross-functional teams to visually design complex logic before committing expensive resources to design and code implementation.

It is a powerful tool for product teams and business owners to communicate on customer requirements in order to identify rabbit holes and poor UX early in the process.

They have been widely used in electrical engineering and in the logic design of electronic devices for decades. 

To illustrate the power of Finite State Machines, imagine we have been briefed by the business to build a customer feedback form widget.

## Phase 1 - Simple Feedback Widget
The first phase of our brief is for a simple feedback widget that will do the following:
- Ask for positive or negative feedback
- If positive display a happy success message
- If negative display an apology
- Allow user to cancel the feedback form

We can visualise this simple application using Finite State Machines below:
![phase1](//images.ctfassets.net/vwq10xzbe6iz/3aRVidpZK6c6qevtvEWLfB/9a09666917aeeddc3170c9a4ef80cfcf/phase1.png)

Our application has 4 states:
1. __START__ - our initial state where a user can click to begin the survey
2. __RESPONSE__ - where a user is asked for a positive or negative response and where they can cancel to go back to START
3. __POSITIVE CONFIRM__: an end state upon a user clicking POSITIVE
4. __NEGATIVE CONFIRM__: an end state upon a user clicking NEGATIVE

Our 4 states are also characterised by 4 events and transitions which are conditional upon the current state the application is in.

Our application also triggers actions such as saving the rating and updating the message upon certain events and entering different states.

Once the logic and user flow has been agreed we can easily convert this to design and code stage. In the examples below we use the open source library [XState](https://xstate.js.org/docs/) by [David Khourshid](https://github.com/davidkpiano) to codify our Finite State Machine and use ReactJS to render our UI.

<iframe src="https://codesandbox.io/embed/feedbackphase1-stkhm?fontsize=14&hidenavigation=1&theme=dark&view=preview"
     style="width:100%; height:500px; border:0; border-radius: 4px; overflow:hidden;"
     title="feedbackPhase1"
     allow="accelerometer; ambient-light-sensor; camera; encrypted-media; geolocation; gyroscope; hid; microphone; midi; payment; usb; vr; xr-spatial-tracking"
     sandbox="allow-forms allow-modals allow-popups allow-presentation allow-same-origin allow-scripts"
   ></iframe>

## Phase 2 - Negative Feedback Reason
The business has now come back asking for us to capture the main reasons for negative feedback.

Our application now must:
- Ask the user to select the main reason for negative feedback (Service, Product, Price)
- Also allow a user to select POSITIVE again if they accidentally selected NEGATIVE

With Finite State Machines we can easily modify and expand our logic without breaking any previous logic:
![phase2](//images.ctfassets.net/vwq10xzbe6iz/38VI5AsXioynO833DlaAzM/9f8c7edb547374d6044b6bff26400c85/phase2.png)

Our application has a new __NEGATIVE FEEDBACK__ state which allows a user to select from three of the main reasons for the negative response (SERVICE, PRODUCT or PRICE). Upon selecting an event the reason is captured and the state transitions to the NEGATIVE CONFIRM end state and displays an apology message.

We have now also added a new POSITIVE event which a user can click on to transition directly to the POSITIVE CONFIRM end state.

Our team can now confidently update our application with the new business logic:

<iframe src="https://codesandbox.io/embed/feedbackphase2-lg6sy?fontsize=14&hidenavigation=1&theme=dark&view=preview"
     style="width:100%; height:500px; border:0; border-radius: 4px; overflow:hidden;"
     title="feedbackPhase2"
     allow="accelerometer; ambient-light-sensor; camera; encrypted-media; geolocation; gyroscope; hid; microphone; midi; payment; usb; vr; xr-spatial-tracking"
     sandbox="allow-forms allow-modals allow-popups allow-presentation allow-same-origin allow-scripts"
   ></iframe>

## Phase 3 - Advanced Feedback
The business comes back wanting to direct users to contact sales for pricing discussions if they select PRICE as a negative response. In addition they want to capture any other reasons for negative feedback.

Our application must now grow in complexity to handle the following:
- If user selects PRICE as a negative response, send to different end state with pricing discussion CTA
- New OTHER option for a user to select with ability to leave comment

Once again we modify our Finite State Machines to expand our logic
![phase3](//images.ctfassets.net/vwq10xzbe6iz/6wb9cvXQflR8QIBGul2SOU/b8e775fd2c86f028a3c4652602219969/phase3.png)

Our application has now added two new states:
1. __PRICE CONFIRM__: end state when a user selects PRICE as a negative response. We will display a contact number for pricing discussions. Creating a new end state provides us with flexibility if the business may want to display discount offers or ask the user for more contact information in the future
2. __COMMENT__: when a user selects OTHER as a new reason for their negative response. This state includes three events where a user can UPDATE COMMENT to capture the reason, SUBMIT COMMENT to transition to end state and CANCEL to go back and select a different reason

Once again, our Finite State Machine is able to handle increasing complexity without breaking previous logic. Our designer and developer team can now confidently work on implementing the new logic:

<iframe src="https://codesandbox.io/embed/feedbackphase3-8y5dx?fontsize=14&hidenavigation=1&theme=dark&view=preview"
     style="width:100%; height:500px; border:0; border-radius: 4px; overflow:hidden;"
     title="feedbackPhase3"
     allow="accelerometer; ambient-light-sensor; camera; encrypted-media; geolocation; gyroscope; hid; microphone; midi; payment; usb; vr; xr-spatial-tracking"
     sandbox="allow-forms allow-modals allow-popups allow-presentation allow-same-origin allow-scripts"
   ></iframe>

## Future Complexity
Our application will naturally grow in complexity as business and user requirements evolve. 

Examples may include:
- Ask for feedback for positive response
- Invite user to add users after a positive response
- Ask user to provide the name of the poor service representative
- Introduce a passive response in addition to positive and negative
- etc.

With Finite State Machines we can confidently continue modelling this advanced complexity into our design and visually see how it interacts with previous functionality.


How Finite State Machines allow cross-functional teams to visually design complex logic before committing expensive resources to design and code implementation.

fininte state machine

Finite State Machines - A Common Tool for Product Owners, Designers & Developers

This walkthrough will show you how to deploy a containerized Node.js app to Amazon Web Services and also assign a secure custom domain. We will be applying the following technologies and tools:
- Application: Node.js Express
- Container: Docker and Github
- Cloud Compute: AWS EC2 Ubuntu
- Domain: AWS ElasticIP, AWS Route53 and Namecheap Domains
- HTTPS: Nginx and Let's Encrypt

Click here for a [link to the github repo](https://github.com/howardmann/express_ec2)

## Summary Steps
1. [Create Node App with Dockerfile and deploy to Github](#1)
2. [Create EC2 Instance](#2)
3. [SSH into EC2 Instance](#3)
4. [Setup and Install EC2 Dependencies](#4)
5. [Create Route 53](#5)
6. [Purchase Namecheap Domain](#6)
7. [Setup Nginx and HTTPS](#7)

## <a name="1"></a>1. Create Node App with Dockerfile and deploy to Github
Create a simple Node Express app with the following folder structure
```bash
src
  L index.js
Dockerfile
package.json
```
Create a simple hello world web app.
```javascript
// src/index.js
let express = require('express')
const PORT = process.env.PORT || 3000

let app = express()

app.get('/', (req, res) => {
  res.send('hi')
})

// Custom 404 route not found handler
app.use((req, res) => {
  res.status(404).send('404 not found')
})

app.listen(PORT, () => {
  console.log(`Listening on PORT ${PORT}`);
})
```
Create Dockerfile which will install specific version of node, install dependencies, expose relevant port and run start command.
```Dockerfile
# // Dockerfile

# Select node version and set working directory
FROM node:8-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Install app dependencies
COPY package.json /usr/src/app
RUN npm install

# Bundle app source
COPY . /usr/src/app

# Expose publc port and run npm command
EXPOSE 3000
CMD ["npm", "start"]
```
Test locally by building the Docker image (check Docker desktop app is running).
```
docker build . -t <YOUR_APP_NAME>
```
Test locally by running the app. Expose port 3000 (we will later use Nginx reverse proxy to send HTTPS port 80 requests to localhost:3000)
```
docker run -p 3000:3000 <YOUR_APP_NAME>
```
Commit and deploy to a Github repo where we will later clone from our EC2 Instance.
```bash
git add .
git commit -m "initial commit"
git remote add origin "<YOUR_GITHUB_REPO_URL>"
git push origin master
```

## <a name="2"></a>2. Create EC2 Instance
Create a new AWS EC2 Instance and set a fixed public IP.
- Login to AWS Management Console
- Select EC2 and select Launch Instance
- Choose latest Ubuntu Server
- Setup security groups including HTTP port 80 and HTTPS
![security](//images.ctfassets.net/vwq10xzbe6iz/5dRssqwYvnxOMyFVIOLnX/ba405930771458831715c7e1bfacfceb/security.png)
- Create instance and create name of private key and download (you will need this to ssh login to your cloud instance)
- Create Elastic IP and associate to your instance. This will keep a fixed public IP address for your instance
![elasticIP](//images.ctfassets.net/vwq10xzbe6iz/5AVVA2E0UzUqhGcQfDmWBa/f2c18d8996125fe2534695aac49ed83d/elasticIP.png)

## <a name="3"></a>3. SSH into EC2 Instance
Login to our newly created EC2 Instance via SSH.
- Find and select your running EC2 instance and click Connect then follow the instructions
![ssh](//images.ctfassets.net/vwq10xzbe6iz/226QhCsKBmdu81LQdB2mqF/ad12eddc2891009eb7e04b1732b9b660/ssh.png)
- Download your private key and run the following command to allow read ownership `chmod 400 <YOUR_PRIVATE_KEY_NAME>.pem`
- If you forget your private key you will not be able to easily SSH back in
- To asssist workflow create a folder named SSH in your local directory and include in .gitignore. You can keep your private key here and a textline of the SSH script. Then come back to this folder whenever you need to SSH in.

## <a name="4"></a>4. Setup and Install EC2 Dependencies
Clone github repo, install Docker, build image and run new image.
- SSH into instance (follow instructions by clicking on "Connect" in the instance). See AWS console to find exact ssh command, ensure you are in the root directory with your private key
```
ssh -i "<YOUR_PRIVATE_KEY_LOCATION_FILE.pem> <YOUR_USER>@<YOUR_AWS_PUBLIC_DNS>
```
- Once login successful, clone your github repo
```
git clone <YOUR_GITHUB_REPO_URL>
```
- Install Docker
```
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
```
- Run Docker without sudo in ubuntu
```
cat /etc/passwd | grep ubuntu
sudo usermod -aG docker ubuntu
```
- Exit EC2 Instance and SSH back in and check you can run docker without sudo
```
docker ps
```
- Rebuild the Docker image (similar to what we did when testing locally)
```
docker build . -t <YOUR_APP_NAME>
```
- Run the Docker image in detatched mode `-d` which will let it continue to run in the background. Set restart as always so it will run when the EC2 instance is restarted.
```
docker run --restart always -d -p 3000:3000 -t <YOUR_APP_NAME>
```
- If you exposed port 3000 in the EC2 instance you should now be able to see the application if you visit the public DNS
- To get the container id
```
docker ps
```
- To see container logs
```
docker logs <YOUR_CONTAINER_ID>
```
- To stop the container
```
docker stop <YOUR_CONTAINER_ID>
```

## <a name="5"></a>5. Create Route 53
Setup AWS Route53 for DNS forwarding from our custom domain to relevant EC2 Instances.
- Search for Route53 Service in AWS Management Console
- Get started and create a hosted zone
- Create the hosted zone and enter the domain name you wil purchase from namecheap
- Once clicked you will see the `NS` records. Copy a record of these
![ns](//images.ctfassets.net/vwq10xzbe6iz/6LJAjqKCGYpuSuZlf3WjbB/fffb15c71174129a88101511f1835974/ns.png)

## <a name="6"></a>6. Purchase Namecheap Domain
Purchase custom domain and point to AWS Route 53.
- Purchase domain from namecheap and then set the Custom DNS to each of the `NS` records from Route53
![namecheap-dns](//images.ctfassets.net/vwq10xzbe6iz/1JwQpYnZCGr0qHnfNqq18G/4e156a31c53eddf89e57491a67c74c89/namecheap-dns.png)
- Connect your Route 53 to Namecheap. Copy the public "IPv4 Public IP" of your EC2 Instance. Hit create record set button and create 2 records below. One set name as empty while latter prefi with *. Copy into value the IPv4 Public IP address into reach
![recordset](//images.ctfassets.net/vwq10xzbe6iz/5HDmGeJ5OnLA7DRSZfF0Qe/adf6dac31f91435eecebf221a509d3fa/recordset.png)

## <a name="7"></a>7. Setup Nginx and HTTPS
Nginx reverse proxy port 80 to our Docker instance and setup secure HTTPS.
- SSH into EC2 Instance and setup nginx webserver
```
sudo apt-get install nginx
```
- Install certbot and build certificate
```
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get install python-certbot-nginx
```
- Make the certificate with the given domain names you have purchased. It may prompt for email
```
sudo certbot --nginx -d example.com -d www.example.com
```
- Remove default config file
```
sudo rm /etc/nginx/sites-enabled/default
```
- Create new file 
```
cd /etc/nginx/sites-available/
sudo touch node
sudo vim /etc/nginx/sites-available/node
```
- Paste the following code in the file and replace example.com with your purchased domain and 3000 with your Node port
```
server {
  listen 80;
  listen [::]:80;
  server_name example.com www.example.com;
  return 301 https://$server_name$request_uri;
}

server {
  listen 443 ssl http2 default_server;
  listen [::]:443 ssl http2 default_server;
  server_name example.com www.example.com;

  location / {
    proxy_pass http://localhost:3000;
  }

  ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
  ssl_session_cache shared:SSL:5m;
  ssl_session_timeout 1h;
  add_header Strict-Transport-Security “max-age=15768000” always;
}
```
- Symlink config file to sites-enabled 
```
sudo ln -s /etc/nginx/sites-available/node /etc/nginx/sites-enabled/node
```
- Restart nginx and apply
```
sudo service nginx restart
```
- If nginx error due to port 80 already in use. Find, kill and restart nginx
```
sudo fuser -k 80/tcp
sudo service nginx start
```

That's it. 

You should now be able to visit your custom domain and be pointed to your Dockerized Node App on HTTPS.


Walkthrough to deploy an Express.js app to AWS EC2 with HTTPS and custom domain using Express, Docker, AWS EC2, AWS Route53, Namecheap, Nnginx and Let's Encrypt. 

How to Deploy a Node App to AWS EC2

> [Click here for github repo of todoapp](https://github.com/howardmann/react-projects/tree/master/todoapp) and [here for live demo of app](http://mannhowietodoapp.surge.sh/)

A visual framework to use for applying React & Redux using the classic Todo App teaching exercise. 
## 1. Overview of TodoApp
While simple, our Todo App covers all the core frontend features of: CRUD, filtering and live reload.
![1. TodoApp](//images.ctfassets.net/vwq10xzbe6iz/1lAWM6TneRs77W6nNoELbK/4b6d1ff28385d2e3e1019160b0cf82da/1._TodoApp.png)

## 2. Break Into Components
Our root App component will contain all the state and logic and will pass this down to its child components.
![2. Components](//images.ctfassets.net/vwq10xzbe6iz/56Zz924eNEdkXu56HSX6Aj/2e6a7456392eaf86bb6e34560a624677/2._Components.png)
Separate each child component by a given responsibility:
1. `<APP/>` Stores our state, actions and renders all immediate children
2. `<NEW_TODO_FORM/>` Accepts text input and triggers "ADD" todo action
3. `<FILTER/>` Changes what items we want to display ("SHOW_ALL" or "SHOW_DONE")

> *Optional: Do not separate out `<NEW_TODO_FORM/>` and `<FILTER/>` into children components if logic is simple and instead keep within `<APP/>`*

4. `<TODOLIST/>` maps over filtered todo items and renders either `<TODO_ITEM/>` or `<EDIT_TODO_FORM/>`
5. `<TODO_ITEM/>` renders the item and includes key action handlers such as mark as done, edit and delete
6. `<EDIT_TODO_FORM/>` renders the input as an input form with its own internal state. Note: we create this as a separate component as we require the feature to undo the edit and only update the state if "Update" button is clicked

## 3. Data Down Actions Up (DDAU)
React follows the principle of Data Down Actions Up (DDAU):
![3. DDAU](//images.ctfassets.net/vwq10xzbe6iz/6LpqSeupvNnkf6xbZahPwr/c5fcafdcebed3460fbcc418ccfe519e2/3._DDAU.png)
1. All/ Majority of application state is stored in a root parent component as a POJO. Also, action callbacks to change state are stored in the parent component
2. Parent then passes down the relevant state and action callbacks to its children as `props`
3. Children components render the `props` and listen for UI events to trigger action callbacks
4. Parent components then execute the callback functions to change state
5. React then does the heavy-lifting to update the UI only for the state that has changed


## 4. Create our State (aka: the source of truth)
All dynamically rendered content should be kept in the state:
![4. State](//images.ctfassets.net/vwq10xzbe6iz/3qe9e2C2w5xq5eCbI8b25C/5b397126cbf13ba0cd118e11fb782060/4._State.png)
All items should be kept in an items array as a single source of truth. 

To filter data, create a helper function to render the items array based on what filter type is in the state - ***Do not duplicate state data***.

Decide whether a `<TODO_ITEM/>` vs. `<EDIT_TODO_FORM>`is rendered by creating an array of items that are marked as edit `isEdit: [4...]`.

## 5. Identify Action Handlers
Identify key UI locations where we will trigger changes to state:
![5. Actions](//images.ctfassets.net/vwq10xzbe6iz/57rSFI8h4I07xjsV3IHqzB/f55ba6cca39dfa591ebe13d1a9f4e182/5._Actions.png)

## 6. Separate out Business Logic
Create a reducer which will be the black box handling our business logic. This is the basic principle of Redux:
![6.Reducer](//images.ctfassets.net/vwq10xzbe6iz/43yPgsyUVgpOfNGuLAsOPF/c1f81dac738207400f6235ebe9548d91/6.Reducer.png)
The reducer accepts three paramaters:
1. `prevState`: the entire state that this reducer is responsible for handling
2. `action.type`: simple text instruction for our reducer to execute relevant function e.g. `"ADD_TODO"`
3. `action.payload`: payload options our action.type and function requires to modify state

A new state is then created without mutating previous state. This business logic is easy to unit test and independent of framework. We then require the reducer to update the state of our components.

*Fin*
***

#### Further reading materials with code:
- [React tutorial](https://reactjs.org/docs/hello-world.html)
- [Redux tutorial](https://redux.js.org/basics/basic-tutorial)
- [You don't need redux](https://medium.com/@dan_abramov/you-might-not-need-redux-be46360cf367)
- [Code your own combineReducer](https://blog.jakoblind.no/code-your-own-combinereducers/)




How to structure your next web project using React and Redux principles. A visual framework template you can use. 

2. Components

Learn React - A Visual Explanation

Uncle Bob's famous [Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) is a way to write resilient software.

Resilient software is divided into layers, underpinned by business logic and is independent of technologies. It should be:
1. **Independent of Frameworks**. Libraries and frameworks should be treated as tools and not dependencies.
2. **Testable**. Can be tested without external dependencies.
3. **Independent of UI**. You can easily switch CLI for Web or RasberryPi.
4. **Independent of Database**. Switch out SQL for MongoDB.
5. **Independent of any external agency**. Business rules don't know anything about outside world.

![The Clean Architecture diagram](https://blog.cleancoder.com/uncle-bob/images/2012-08-13-the-clean-architecture/CleanArchitecture.jpg)

In practice, choice of technology should be the last decision you make or code you write (e.g. database, platform, framework).

By following clean architecture, you can write software today that can be easily switched out for different technologies in the future.

## Practical Example of Clean Architecture in Node.js
This is an example of a simple CRUD application with layered software and separation of business logic vs technology. 

It is a simple API for creating `students` and `teachers` and includes validation, persistence and UI. It includes examples using different interfaces (CLI and Web), databases (in memory, MongoDB), and libraries (validation).

[Click here for github repo.](https://github.com/howardmann/clean-node)

*Note: this application is different to the Clean Architecture diagram above but attempts to achieve the same outcome.*

### Software layer overview
The application is separated into three layers. Inner layers cannot depend on outer layers and outer layers should only depend one layer in:

- **Inner Layer**
  - **Models**. Handles the creation, validation and reading of our entities (students and teachers). Note that this should be custom logic and not include the DB implementation of models (e.g. in Mongoose DB ORM, their models should be encapsulated in the DB layer below). Our model schemas live here. 
  - **DB**. Our choice of DB (in memory, MongoDB, SQL), this is independent of the model. Note that in Clean Architecture this is considered an outer layer framework, but for practical applications I find it easier to place it inner and have the data-access layer depend on it rather than injecting it in.
- **Middle Layer**
  - **Data-Access**. Handles transfer between the DB (like an ORM). Depends on the model to validate and create the entity in DB. The key is to have a consistent & custom API that all outer layers communicate with. Testing here will ensure that replacing or using multiple DBs doesn't break anything further upstream. 
- **Outer Layer**
  - **Drivers**. Represents the UI or interface (Web or CLI). It communicates only with the data-access layer.

### Example application structure
```bash
============= INNER LAYER =====================================================================
models                    // create new entity by validating payload and returning new read only object 
  L student
    L index.js            // dependency inject schema/ validation library
    L index.test.js       // tests makeStudent()
    L student.js          // simple model takes info, validates and returns read only object
    L student-schema.js   // student validation schema
  L teacher
    L index.js
    L teacher.js
    L teacher-schema.js
  L validator             // wrapper around JOI validation library
    L index.js            // consistent API if ever to switch out a new validation library
    L index.test.js       // tests for validation schema for all models

db                        // db connection and adapter
  L memory                // in memory JSON
    L students.js         
    L teachers.js         
  L mongodb               // mongodb alternative
    L connection.js       // connection library
    L seeds               // seed library
      L students-seeds.js // async seed students db
    L models
      L student.js        // models specific to mongodb. this is different to our business logic models which handle tests and validation

============= MIDDLE LAYER =====================================================================
data-access               // think of it as our internal ORM (logic for our use-cases lies here)
  L students-db           
    L index.js            // other controllers and drivers rely on this API (findStudent, listStudents, addStudent)
    L memory              // in memory
      L index.js          // expose the memory implementation of findStudent, listStudents, addStudents
      L serializer.js     // serializes to DB specific properties (e.g. serial > id, year > grade)
    L mongodb             // mongodb ORM
      L index.js          // uses mongoose implementation of findStudent, listStudents, dropAll etc.
      L serializer.js     // serializes _id to id
    L postgresql          // TODO: Illustrative
    L airtable            // TODO: Illustrative
  L teachers-db           // per students-db above

============= OUTER LAYER =====================================================================
drivers
  L cli                   // cli driver depends on data-access students-db
  L webserver             // express web-server
    L routes
      L index.js          // routes paths
      L students.js       // requires our data-access students-db
      L teachers.js       // per above
    L server.js
```

### Practical approach
In practice, we can defer technology decisions by writing our application in the following order:
1. Start with models, schema and validation. Write test spec.
2. Defer DB decision by using in-memory JSON store
3. Create a data-access API that depends on model and in-memory DB. Write test spec.
4. First tech decision: UI interface (e.g. web or CLI). Depend on above data-access layer. Write integration test spec. 
5. Second tech decision: DB choice (MongoDB, SQL). Replace in-memory DB store. Write data-access methods specific to the DB choice. Ensure previously written test specs pass.

### Models
Our model is a simple factory function that validates a payload (dependency injected validator) and returns a new object with getters.

```javascript
// models/student/student.js
let buildMakeStudent = function(studentValidator) {
  return ({
    name,
    age,
    grade,
    prefect = false
  } = {}) => {
    let {error} = studentValidator({name, age, grade, prefect})
    if (error) throw new Error(error)

    return {
      getName: () => name,
      getAge: () => age,
      getGrade: () => grade,
      isPrefect: () => prefect
    }
  }
}

module.exports = buildMakeStudent
```
While the model schema is dependent on a validation library and breaks the Clean rules, I find it easier to understand the model by having the schema inside the model. (Note: see the repo for an example of how the validator could be separated into its own software entity).

```javascript
// models/student/student-schema.js
let Joi = require('joi')

module.exports = Joi.object().keys({
  name: Joi.string().required().error(() => 'must have name as string'),
  age: Joi.number().error(() => 'age must be a number'),
  grade: Joi.number().error(() => 'grade must be a number'),
  prefect: Joi.boolean().error(() => 'prefect must be a boolean')
})
```

We then dependency inject the validation library into the model. Note we use a wrapper for the validation library to make it easier to switch out libraries in future.

```javascript
// models/student/index.js
let buildMakeStudent = require('./student')
let studentSchema = require('./student-schema')
let studentValidator = require('../validator/')(studentSchema)

let makeStudent = buildMakeStudent(studentValidator)

module.exports = makeStudent
```

Write two sets of unit tests for our model. First testing the validation library for valid vs invalid payloads and relevant error messages. And second testing the creation and read only of the model entity.

```javascript
// models/validator/index.test.js
studentValidator
  [] validates name:string:required, grade:number, age:number, prefect:boolean 
  [] returns error messages if invalid

teacherValidator
  [] validates name:string:required, subject:string, tenure:boolean
  [] returns error messages if invalid
```

```javascript
// models/student/index.test.js
makeStudent
  [] throws error if invalid payload
  [] must have name
  [] can have grade
  [] can have age
  [] sets prefect to false by default
```
### Data-Access
The data-access layer is one of the most important. We should feel confident to easily replace DBs.

Here we need three key components
1. API and test spec
2. Serializers to adapt custom DB to our model schema
3. DB implementation of the API and passing of test

Start with how we think the outer layers will communicate with the data-access layer and what they should expect to get.

```javascript
// data-access/students-db/index.test.js
// example test spec
studentsDb
  - listStudents() lists students
  - findStudent() find single student by id
  - findStudentsBy() finds all students by property
  - addStudent() inserts a student
  - throws error if student payload invalid
  - deleteStudent() deletes a student
  - dropAll() drops students database
```

Write a serializer which adapts the custom DB schema with our model schema.

```javascript
// data-access/students-db/mongodb/serializer.js
// e.g. mongodb stores property id as _id
const _serializeSingle = (student) => {
  return {
    'id': student._id,
    'grade': student.grade,
    'name': student.name,
    'age': student.age,
    'prefect': student.prefect
  };
};

const serializer = (data) => {
  if (!data) {
    return null
  }
  if (Array.isArray(data)) {
    return data.map(_serializeSingle)
  }
  return _serializeSingle(data)
}

module.exports = serializer
```
Write custom DB implementation of the test spec API. Here is an example of an in-memory implementation (which you should start with) and a MongoDB version. Note: the advantage of writing a test spec is you can focus on simply writing code that just has to work.
#### In-memory data-access:
```javascript
// data-access/students-db/memory/index.js
let STUDENTS = require('../../../db/memory/students') // DB
let makeStudent = require('../../../models/student/index') // model
let serialize = require('./serializer') // serializer custom to db

let listStudents = () => {
  return Promise.resolve(serialize(STUDENTS))
}

let findStudent = (prop, val) => {
  if (prop === 'id') { prop = 'serial' }
  let student = STUDENTS.find(student => student[prop] == val)
  return Promise.resolve(serialize(student))
}

let findStudentsBy = (prop, val) => {
  let student = STUDENTS.filter(student => student[prop] == val)
  return Promise.resolve(serialize(student))
}


let addStudent = (studentInfo) => {
  let student = makeStudent(studentInfo)
  let newStudent = {
    serial: STUDENTS.length + 1,
    year: student.getGrade(),
    name: student.getName(),
    age: student.getAge(),
    prefect: student.isPrefect()
  }
  STUDENTS.push(newStudent)
  return findStudent('serial', newStudent.serial)
}

let deleteStudent = (id) => {
  return findStudent({id})
    .then(student => {
      if (student.id == id) {
        STUDENTS = STUDENTS.filter(student => student.serial != id)
        return {
          id,
          status: 'success'
        }
      }
      return {
        status: 'fail'
      }
    })
}

let dropAll = () => {
  STUDENTS = [];
  return STUDENTS;
}

module.exports = {
  listStudents,
  findStudent,
  findStudentsBy,
  addStudent,
  deleteStudent,
  dropAll
}
```
#### MongoDB data-access:
Note for our MongoDB data-access implementation we depend on the DB from the inner layer. The DB includes MongoDB specific implementation of the model and DB connection.

We write a dropAll function for the purposes of writing a test spec that can be used across any DB choice. E.g. beforeEach test we would drop the DB and seed items for testing.
```javascript
// data-access/students-db/mongod/index.js
let Student = require('../../../db/mongodb/models/student')
let makeStudent = require('../../../models/student/index') // model
let serialize = require('./serializer') // serializer custom to db

let listStudents = () => {
  return Student.find({})
    .then(serialize)
}

let findStudent = (prop, val) => {
  if (prop === 'id') {
    prop = '_id'
  }
  return Student.find({[prop]: val})
    .then(resp => {
      return serialize(resp[0])
    })
}

let findStudentsBy = (prop, val) => {
  return Student.find({[prop]: val})
    .then(serialize)
}

let addStudent = (studentInfo) => {
  let student = makeStudent(studentInfo)
  let newStudent = {
    name: student.getName(),
    grade: student.getGrade(),
    age: student.getAge(),
    prefect: student.isPrefect()
  }
  return Student.create(newStudent)
    .then(serialize)
}

let deleteStudent = (id) => {
  return Student.findByIdAndDelete(id)
    .then(resp => {
      return {
        id: resp._id.toString(),
        status: 'success'
      }
    })
    .catch(err => {
      return {
        status: 'fail'
      }
    })
}

let dropAll = () => {
  return Student.remove()
}

module.exports = {
  listStudents,
  findStudent,
  findStudentsBy,
  addStudent,
  deleteStudent,
  dropAll
}
```
### Drivers
Finally we write our drivers whose only dependency is our data-access layer. The drivers should not communicate directly with the model or db.

Any changes to the inner layer will cascade up and as a result there should be less testing done on the outer layers. E.g. if we were to test changes to the model schema in the driver layer we would need to rewrite our tests here as well.

```bash
drivers
  L cli
    L index.js
  L webserver         // express webserver
    L routes
      L index.js
      L students.js   // depends on students data-access layer
      L teachers.js   // depends on teachers data-access layer
    L server.js       // depends on routes
```

Writing Clean Architecture in Node.js. REST API Tutorial using MongoDB, Express and other best practices. How to write software that lasts and can be independent of technology choice.

Clean

Clean Architecture Node.js - Build a REST API

How to validate requests and handle errors in Express.js using [Joi](https://github.com/hapijs/joi) validation library. [Click here for a link to the repo](https://github.com/howardmann/validation).

1. [Gentle introduction to Joi validation](#1-joi)
2. [General route error handling in Express.js](#2-route)
3. [Three examples of handling request validations](#3-validation)
4. [Flash form validation errors](#4-flash)

## <a name="1-joi"></a>  1. Gentle introduction to Joi validation
Joi is an expressive validator library for JavaScript objects. It helps ensure inputs are checked before progressing (e.g. to a DB).

You first create a schema and then validate the payload against the schema. If invalid it will throw the relevant error message.
```javascript
let Joi = require('joi')

// Create schema
let createProductSchema = Joi.object().keys({
  description: Joi.string().required(),
  price: Joi.number().min(0.01).required()
})

let payload = {
  description: 'red book',
  price: 0.0001
}

Joi.validate(payload, createProductSchema)
  .then(validated => {
    console.log(validated);
  })
  .catch(err => {
    console.log(err);
    // [ { message: '"price" must be a number',
    // path: [ 'price' ],
    // type: 'number.base',
    // context: { value: null, key: 'price', label: 'price' } } ]
  })
```

We can DRY our code and refactor the schemas and validators into separate files
```javascript
// validation/schemas/productSchema.js
let Joi = require('joi')

// We can reuse schemas
let priceSchema = Joi.number().min(0.01)

// Schema for creating a product, all fields are required
let createProductSchema = Joi.object().keys({
  description: Joi.string().required(),
  price: priceSchema.required()
})

// Schema for editing a product, all fields are optional and we can have a custom error message
let editProductSchema = Joi.object().keys({
  description: Joi.string(),
  price: priceSchema.error(() => 'when editing price must be number greater than 0.01')
})

module.exports = { 
  createProductSchema, 
  editProductSchema
}
```

```javascript
//validation/createValidator.js
let Joi = require('joi')

// DRY our code with higher order function
const createValidator = (schema) => 
  (payload) => {
    return Joi.validate(payload, schema, {
      // shows all error messages instead of first error message
      abortEarly: false
    })
  }

module.exports = createValidator
```

## <a name="2-route"></a> 2. General route error handling in Express.js
We need to create two error handlers in our express application to handle all server errors and 404 route errors.

This way if we throw an error in any of our other routes the error handler will catch and display the relevant error message.

```javascript
let express = require('express')
let bodyParser = require('body-parser') // handle req.body
let app = express()
app.use(bodyParser.json())

// Require routes
app.use(require('./routes'))

// Custom server error handler
app.use((err, req, res, next) => {
  if (err) {
    console.error(err.message)
    if (!err.statusCode) {err.statusCode = 500} // Set 500 server code error if statuscode not set
    return res.status(err.statusCode).send({
      statusCode: err.statusCode,
      message: err.message
    })
  }

  next()
})

// Custom 404 route not found handler
app.use((req, res) => {
  res.status(404).send('404 not found')
})

const PORT = 3000
app.listen(PORT, () => {
  console.log(`Listening on PORT ${PORT}`);
})
```

## <a name="3-validation"></a> 3. Three examples of handling request validations
We can validate incoming Express payload requests using Joi in 3 ways:
1. We handle the payload within the route and catch error if its invalid
2. Create a custom middleware ([see next section](###Generic-Middleware)) to handle the validation and throw error if invalid
3. Use the [celebrate](https://github.com/arb/celebrate) library which does same as option 2 but with more features

```javascript
let express = require('express');
let router = express.Router();


// Validation
let createValidator = require('../validation/createValidator')
let { createProductSchema } = require('../validation/schemas/productSchema')
let validate = require('../validation/validateMiddleware')
let {celebrate} = require('celebrate')


// 1. Validation using Joi within route
let validateProductCreate = createValidator(createProductSchema)

router.post('/products', (req, res, next) => {
  validateProductCreate(req.body)
    .then(validatedProduct => {
      res.send({status: 'success', validatedProduct})
    })
    .catch(next)
})

// 2. Validation using Joi custom middleware
router.post('/products2', validate(createProductSchema), (req, res, next) => {
  let payload = req.body
  res.send({status: 'success', payload})
})

// 3. Validation using celebrate middleware library
router.post('/products3', celebrate({body: createProductSchema}), (req, res, next) => {
  let payload = req.body
  res.send({status: 'success', payload})
})

module.exports = router;
```
### Generic Middleware
For option 2 we can create a generic middleware to handle validations.
```javascript
// validation/validateMiddleware
let createValidator = require('./createValidator')

let validateMiddleware = (schema) =>
  (req, res, next) => {
    let payload = req.body
    let validate = createValidator(schema)

    // proceed next if validated otherwise catch error and pass onto express error handler
    validate(payload)
      .then(validated => {
        req.body = validated
        next()
      })
      .catch(next)
  }

module.exports = validateMiddleware
```

For option 3 we must also add the celebrate error handler in our server file above our custom error handlers. This gives us more verbose error handling messaging.
```javascript
//server.js
let {errors} = require('celebrate') // handle celebrate joi errors

let app = express()

// Require routes
app.use(require('./routes'))

// Celebrate library error handler
app.use(errors())

// Custom server error handler
app.use((err, req, res, next) => {
})

// Custom 404 route not found handler
app.use((req, res) => {
})

const PORT = 3000
app.listen(PORT, () => {
})
```

## <a name="4-flash"></a> 4. Flash form validation errors
The methods above handle API validation errors. However, we may also want to display form validation errors in the browser as a flash message:

![form validation](https://raw.githubusercontent.com/howardmann/validation/master/img/validation.png)

In this example we will have a signup form that requires three fields (email, name and password). It will validate and flash relevant validation errors.

We will achieve this in stages:
1. Setup express to  display flash messages
2. Create user schema and HTML form
3. Setup routes to handle validations
4. DRY using middleware

### 1. Display flash messages
To display flash messages we need two libraries:
- `connect-flash` for flashing messages; and
- `express-session` for storing sessions server-side in memory

We write a custom middleware to store the session message as a global variable in views in order to access it in any view template.

```javascript
// server.js
let session = require('express-session')
let flash = require('connect-flash')

// session and flash views
app.use(session({
  secret: 'ilikecats',
  cookie: {},
  resave: false,
  saveUninitialized: true
}));
app.use(flash())

// store three flash variables as global variables in views
app.use((req, res, next) => {
  res.locals.messageSuccess = req.flash('messageSuccess')
  res.locals.messageFailure = req.flash('messageFailure')
  res.locals.validationFailure = req.flash('validationFailure')
  next();
})
```

Then in our views layout we insert placeholders to be able to render these flash messages if they exist. For the `validationFailure` it will be passed an array of messages so we want to loop through and display as a bullet points.
```hbs


<div style="background-color: red">
  {{messageFailure}}
</div>

<div style="background-color: green">
  {{messageSuccess}}
</div>

<div style="background-color: red">
  {{#if validationFailure}}
  <p>Validation Errors</p>
  {{/if}}
  <ul>
    {{#each validationFailure}}
    <li>
      {{this}}
    </li>
    {{/each}}
  </ul>
</div>

{{{body}}}
```
### 2. User schema and form
As with our product schema above we will create a user schema to handle validation using `Joi`.

```javascript
//validation/schemas/userSchema
let Joi = require('joi')

module.exports = Joi.object().keys({
  email: Joi.string().email(),
  name: Joi.string(),
  password: Joi.string().min(7).alphanum()
})
```

Now create a simple HTML form in our view template
```handlebars

<form action="/signup" method="POST">
  <input type="email" placeholder="enter email" name="email">
  <input type="text" placeholder="enter name" name="name">
  <input type="password" placeholder="enter password" name="password">
  <input type="submit">
</form>
```

### 3. Setup signup routes
Use the validation helpers we used before along with the schema we created above. Create two routes: 1) for displaying the signup form and 2) for handling the form submission.

Validate the payload using Joi and if validation errors flash validation errors and redirect bac to form.
```javascript
//routes
let express = require('express');
let router = express.Router();
let createValidator = require('../validation/createValidator')
let {createUserSchema} = require('../validation/schemas/userSchema')
let validateUser = createValidator(createUserSchema)

// display signup form
router.get('/signup', (req, res, next)) => {
  res.render('signup')
}

// handle submit signup form 
router.post('/signup', (req, res, next)) => {
  let payload = req.body
  
  // Joi validation
  validateUser(payload)
    .then(data => {
      // If success; flash success message and redirect to form
      console.log(data)
      req.flash('messageSuccess', 'Success valid input')
      res.redirect('/signup')
    })
    .catch(err => {
      // If validation error flash array of messages and redirect
      let errorMessages = err.details.map(el => el.message)
      console.log(errorMessages)
      req.flash('validationFailure', errorMessages)
      res.redirect('/signup')
    })
}

module.exports = router
```

### 4. DRY with middleware
If we have many forms that require validating (e.g. for both edit and create) we can refactor our validation into a custom middleware.

Create a validateForm middleware function that takes a Joi schema as an argument and redirects back with validationFailure flash message if error.

```javascript
//validation/middlewares/validateForm 
let createValidator = require('../createValidator.js')

module.exports = (schema) =>
  (req, res, next) => {
    let payload = req.body
    let validate = createValidator(schema)

    validate(payload)
      .then(validated => {
        req.body = validated
        next()
      })
      .catch(err => {
        let errorMessages = err.details.map(el => el.message)
        console.log(errorMessages)
        req.flash('validationFailure', errorMessages)
        res.redirect('back')
      })
  }
```

We can now use it as reusable middleware for any route.
```javascript
//routes/index.js
let validateForm = require('../validation/middlewares/validateForm')

// middleware handles redirect back with validationFailure flash message
// route then only has to deal with success outcome
router.post('/signup', validateForm(createUserSchema), (req, res, next) => {
  let payload = req.body
  // handle payload etc. in db
  console.log(payload);
  req.flash('messageSuccess', 'woohoo')
  res.redirect('/signup')
})

```

Learn how to validate requests and handle errors using Node Express.js with Joi validation library. Validate API requests and display HTML form errors using custom middleware.

Express

Express Validation and Error Handling

Tutorial and demo app using passport.js for basic principles of authentication and authorization.

[Click here for a live demo of the app.](https://authdemo.now.sh)
You can use the example admin credentials (email: howie@email.com password: chicken)

[Click here for the github repo](https://github.com/howardmann/authentication)

### Overview of demo app:
As a public user, I can:
- view the homepage
- visit the signup page and create a new account
- visit the login page

As a logged in user, I can:
- view my private profile page
- view special pages only visible to logged in users
- logout

As a logged in user with admin privileges, I can do all of the above plus:
- view all users
- view admin pages

Here is an overview of pages and permissions.
```shell
PAGES
page              permission
/                 public homepage
/special          restricted to signed up and logged in users
/users/profile    restricted to profile of logged in user
/admin            restricted to loggedin admin users
/users            restricted to loggedin admin users

AUTH
method    path             description
GET       /login           render login view form page
POST      /login           authenticate email and password and redirects to profile page if success and login page again if fail
GET       /signup          render signup view form page
POST      /signup          create user and redirect to profile page if success and signup page with flash message if fail
GET       /logout          clears session and redirects to login page


```

Dependency descriptions
```javascript
  "dependencies": {
    "bcrypt": "^3.0.6", // for hashing passwords 
    "body-parser": "^1.19.0", // to capture form post data in req.body
    "connect-flash": "^0.1.1", // flash messages 
    "dotenv": "^8.0.0", // environment variables
    "express": "^4.17.1", // http web server
    "express-handlebars": "^3.1.0", // template engine
    "express-session": "^1.16.1", // track sessions
    "mongoose": "^5.5.12", // mongodb ORM
    "passport": "^0.4.0", // authentication
    "passport-local": "^1.0.0" // passport strategy for local DB auth (e.g. username and password)
  }

```

This is a summary of the relevant directories
```shell
auth // where all the authorization and authentication logic lives
  L middleware
    L flashMessageInViews.js  // enables all views to access {{messageSuccess}} and {{messageFailure}}
    L userInViews.js // enables all views to check if user session exists, e.g. enables us to flash login or logout link
  L authorization.js // middleware to gate routes (e.g. loginRequired, adminRequired, signupRequired)
  L index.js // entry point
  L logInOut.js // login and out pages and passport authentication logic
  L register.js // signup page and create user passport logic
  L serializer.js // serializes session user_id via passport
routes // where routes live
  L index.js // calls auth methods
  L pages.js // other pages
  L users.js // users pages
views
  L auth // templates for login and signup page
    L login.hbs
    L signup.hbs
  L pages
  L users
  L layouts
repositories // applying clean programming principles (separation of concerns) to make it easier to switch out DBs
  L userRepository.js // model wrapper for user CRUD operations
  L memory // user in memory (start off for dev purposes)
  L mongodb // user in mongodb db (for persistence)
model
  L monbodb // mongodb model and schema
db
  L mongodb // mongodb connection and seeds
  L memory // simple array for dev purposes
server.js // express webserver and dependencies 
```
### Tutorial
We will approach authentication and authorization via the following steps:
1. Register user
2. Login and logout authentication
3. Flash messages
4. Authorization
5. Using it in routes

## 1. Register User
Passport.js is a library that makes it easier to handle session serialization, redirects and other aspects of authentication.

We will be using the `passport-local` strategy which is your vanilla username and password login strategy.

In the code below we we first start with the functionality to sign up a user. The local-signup strategy is middleware that our route will pass through. 

The first options that are passed tell passportJs that the email and password fields we are using differ from the default username field (in our case we use the name `email` in our form for username). The final field `passReqToCallback` lets us access the req object in the callback (which we will need to find other properties in the req.body e.g. the phone number).

The callback queries our DB to check if the user already exists, if it does then we return `done(null, false)` which will trigger the `failureRedirect`. If it doesn't then we create the user, hash the password using bcrypt and return `done(null, {id: newUser.id})`. By not passing false to the second param we trigger the `successRedirect` path. Here we pass the user_id to be serialized only into the session accessible via `req.user`, instead of passing the entire user object and passwordHash. You can pass anything you want, but we are passing only the id for security reasons.


```javascript
// auth/register.js
let register = module.exports = {}

// Dependecies
let bcrypt = require('bcrypt');
let passport = require('passport');
let LocalStrategy = require('passport-local').Strategy;
let User = require('../repositories/userRepository.js')

// =======USER SIGN UP AND HASH PASSWORD STRATEGY========
passport.use('local-signup', new LocalStrategy({
    usernameField: 'email', // map username to custom field, we call it email in our form
    passwordField: 'password',
    passReqToCallback: true // lets you access other params in req.body
  },
  async (req, email, password, done) => {
    // Return false if user already exists - failureRedirect
    let user = await User.findBy('email', email)
    if (user) { return done(null, false) }

    // Create new user and return the user - successRedirect
    let newUser = await User.create({
      email,
      passwordHash: bcrypt.hashSync(password, 10), // hash the password early
      phone: req.body.phone
    })

    // save the user_id to the req.user property
    return done(null, {id: newUser.id})
  }
))
```

Once we have our strategy we create two methods, 1 to render the signup form and the second to pass the forms payload to the passport signup strategy and handle the appropriate redirects.

```javascript
// auth/register.js

// GET route to render signup page
register.signupPage = (req, res, next) => {
  res.render('auth/signup')
}

// POST route to signup user and redirect 
register.signup = passport.authenticate('local-signup', {
  successRedirect: '/users/profile',
  failureRedirect: '/signup',
  failureFlash: {
    type: 'messageFailure',
    message: 'Email already taken.'
  },
  successFlash: {
    type: 'messageSuccess',
    message: 'Successfully signed up.'
  }
})

```
Later in our `routes/index.js` we will pass these two paths to:
```javascript
// routes/index.js
router
  .get('/signup', auth.signupPage)
  .post('/signup', auth.signup)

```

Visiting GET '/signup' will render the HTML signup form.

```html
// views/auth/signup
<h2>Signup</h2>

<form action="/signup" method="POST">
  <input type="email" placeholder="enter email" name="email" required>
  <br>
  <input type="password" placeholder="enter password" name="password" required>
  <br>
  <input type="tel" placeholder="enter phone" name="phone">
  <br>
  <input type="submit" value="Login">
</form>
```
Submitting the payload will then trigger the passport signup strategy and redirect accordingly. Passport.js enables us to set the success and failure flash messages in the passport.authenticate method. This is then accessible via `req.flash()`.

## 2. Login and Logout Authentication
Login and logout authentication follows the similar approach as registering new users.

We create a new `local-login` passport strategy which checks the DB if the user exists and if the submitted password matches the passwordHash in the DB (using bcrypt). If it does, we call done with the serialized user_id saved as a session in `req.user` and then trigger the `successRedirect` path. If not we call `done(null, false)` which triggers the `failureRedirect` path.

Similarly we create a loginPage path which renders the HTML login form and a method to handle the POST login request.

A logout path is straight forward, we call req.logout() which clears the session, we flash a message and redirect to a URL.

```javascript
//auth/logInOut.js
let logInOut = module.exports = {}

// Dependencies
let bcrypt = require('bcrypt');
let passport = require('passport');
let LocalStrategy = require('passport-local').Strategy;
let User = require('../repositories/userRepository.js')

// ======USER LOGIN AUTHENTICATION STRATEGY=======
passport.use('local-login', new LocalStrategy({
    // Fields to accept
    usernameField: 'email', // default is username, override to accept email
    passwordField: 'password',
    passReqToCallback: true // allows us to access req in the call back
  }, async (req, email, password, done) => {
    // Check if user and password is valid
    let user = await User.findBy('email', email)
    let passwordValid = user && bcrypt.compareSync(password, user.passwordHash)

    // If password valid call done and serialize user.id to req.user property
    if (passwordValid) {
      return done(null, {
        id: user.id
      })
    }
    // If invalid call done with false and flash message
    return done(null, false, {
      message: 'Invalid email and/or password'
    });
}))


// GET route to render login form view
logInOut.loginPage = (req, res, next) => {
  res.render('auth/login')
}

// POST route to handle req.body payload. Pass to passport login strategy with relevant redirects
logInOut.login = passport.authenticate('local-login', {
  successRedirect: '/',
  failureRedirect: '/login',
  failureFlash: {
    type: 'messageFailure',
    message: 'Invalid email and/ or password.'
  },
  successFlash: {
    type: 'messageSuccess',
    message: 'Successfully logged in.'
  }
})

// GET logout route, flash message and redirect
logInOut.logout = (req, res, next) => {
  req.logout();
  req.flash('messageSuccess', 'Successfully logged out')
  res.redirect('/login')
}

```

## 3. Flash messages
In order to display the flash messages we have been flashing above we need some middleware. This is a simple middleware that stores the success and failure flash messages in the gloal res.locals property. This makes it accessible in all our views via {{messageSuccess}} and {{messageFailure}}.

```javascript
//auth/middleware/flashMessageInviews.js
// global route middleware 
// success and failure flash message in views
module.exports = (req, res, next) => {
  res.locals.messageSuccess = req.flash('messageSuccess')
  res.locals.messageFailure = req.flash('messageFailure')
  next();
}
```

In addition in order to know if someone has logged in or not we use another middleware to check if the user is authenticated. This lets us be able to display either login or logout links depending if someone is logged in or out.

```javascript
//auth/middleware/userInViews.js
// global route middleware 
// user boolean accessible in views if authenticated
module.exports = (req, res, next) => {
  res.locals.user = req.isAuthenticated()
  next()
}
```

To use these we need to call it in our main server.js file.
```javascript
//server.js
// Custom middleware authentication and flash message view middleware
app.use(userInViews)
app.use(flashMessageInViews)

// Routes
app.use(require('./routes/index'))
const userInViews = require('./auth/middleware/userInViews.js')
const flashMessageInViews = require('./auth/middleware/flashMessageInViews.js')
```

Now in our view layout template we can display our flash messages and appropriate login or logout links.

```handlebars
//views/layouts/main.hbs
  <h1>Web App</h1>
  <div style="background-color: red">    
    {{messageFailure}}
  </div>
  <div style="background-color: green">    
    {{messageSuccess}}
  </div>
  
  <nav>
    <p>
      {{#if user}}
        <a href="/logout">Logout</a>
      {{else}}
        <a href="/login">Login</a>
      {{/if}}
      
      <a href="/signup">Signup</a>
    </p>
```

## 4. Authorization
Authorization middleware lets us gate certain routes depending on a users authentication and permission level. In addition it can then redirect to specific pages with appropriate flash messages.

In our example we use three forms of authorization:
- loginRequired: needs to be logged in otherwise redirect to login page
- adminRequired: needs to be an admin: true user otherwise redirects to profile page
- signupRequired: same as loginRequired except redirects to signup page

```javascript
//auth/authorization.js
let authorization = module.exports = {}

let User = require('../repositories/userRepository')

// Higher order function to keep DRY
let authenticate = ({type, message, redirectPath}) => {
  return (req, res, next) => {
    let isAuthenticated = req.isAuthenticated()
    if (!isAuthenticated) {
      req.flash(type, message)
      return res.redirect(redirectPath)
    }
    next()
  }
}

// ===AUTHORIZATION MDDLEWARE
authorization.loginRequired = authenticate({
  type: 'messageFailure',
  message: 'Must be logged in',
  redirectPath: '/login'
})

authorization.signupRequired = authenticate({
  type: 'messageFailure',
  message: 'Must be signed up',
  redirectPath: '/signup'
})


authorization.adminRequired = async (req, res, next) => {
  let id = req.user && req.user.id
  let user = await User.findBy('id', id)
  let isAdmin = user && user.admin

  if (!isAdmin) {
    req.flash('messageFailure', 'Admin only')
    return res.redirect('/users/profile')
  }

  next();
}
```

## 5. Using it in routes
Now we have our logic and flash views ready we can integrate to our routes/index file. This should be self explanatory.

```javascript
const express = require('express')
let router = express.Router()

// === AUTH ENDPOINTS ===
const auth = require('../auth')
router
  .get('/login', auth.loginPage)
  .post('/login', auth.login)
  .get('/signup', auth.signupPage)
  .post('/signup', auth.signup)
  .get('/logout', auth.logout)

// === BOILERPLATE === 
// pages endpoints
let pages = require('./pages')

router
  .get('/', pages.home)
  .get('/admin', auth.loginRequired, auth.adminRequired, pages.admin)
  .get('/special', auth.signupRequired, pages.special)
  
// users endpoints
let users = require('./users')
router
  .get('/users/profile', auth.loginRequired, users.show)
  .get('/users', auth.loginRequired, auth.adminRequired, users.index)

module.exports = router
```

## How to use this demo app
This demo app can be used as boilerplate authentication for other apps.

To reuse follow these steps:
1. Ensure you have the minimum dependencies installed (see above)
2. Copy the entire auth folder
3. Create an auth folder in your views to render the appropriate login and signup page (customise depending on form fields)
4. Modify the userRepository in order to create and read user details
6. Update your routes/index file to include the auth routes and use the appropriate auth authorization middleware for your specific routes
7. Update your server.js file to include the relevant authentication dependencies (order matters)

Learn how to handle web authentication and authorization in Node using Passport.js. Create signup and login page, different types of users and handle authentication for specific pages.

passport.js

Node.js Authentication Tutorial with Passport

SQL cheat sheet using starcraft relational data model based on races, planets and heroes.

## Table and associations
- Database: Starcraft
- Tables: `Races`, `Planets`, `Planets_Races` (join table), `Heroes`
- Associations:
  - `Races` many-to-many `Planets` 
    - 1 planet can be controlled by many races and similarly 1 race can control many planets)
  - `Races` one-to-many `Heroes`
    - 1 race has many heroes but 1 hero can only belong to a single race (no biracial)
  - `Planets` one-to-many `Heroes`
    - 1 planet can give birth to many heroes but 1 hero can only be born in a single homeworld 

![relational db](https://raw.githubusercontent.com/howardmann/sql_basics/master/draw.png)

## Postgresql commands
Create database. Ensure postgresql application is running
```bash
# run postgresql in terminal
psql
# create database
CREATE DATABASE {database_name};
# connect to database
\c {database_name}

# other useful terminal commands
\c {database_name}# connect to [database] 
\l # list all databases
\dt #list all tables in current database
\d {table_name} #list schema of table
\q # quit
```
## Create Tables

```sql
CREATE TABLE Races (
    id serial PRIMARY KEY,
    name varchar(225),
    description varchar(225)
);

CREATE TABLE Planets (
    id serial PRIMARY KEY,
    name varchar(225),
    moons int
);

CREATE TABLE Heroes (
    id serial PRIMARY KEY,
    planet_id int REFERENCES planets ON DELETE SET NULL,
    race_id int REFERENCES races ON DELETE SET NULL,
    alias varchar(225),
    level int
);

CREATE TABLE Planets_Races (
    planet_id int REFERENCES planets ON DELETE SET NULL,
    race_id int REFERENCES races ON DELETE SET NULL
);
```

## Drop tables
```sql
DROP TABLE Heroes CASCADE;
DROP TABLE Races CASCADE;
DROP TABLE Planets CASCADE;
DROP TABLE Planets_Races CASCADE;
```

## Seed Database
Seed `races`
```sql
-- Delete existing data
DELETE FROM Races;
-- RESET SERIAL PRIMARY KEY
ALTER SEQUENCE races_id_seq RESTART WITH 1;
-- SEED NEW DATA
INSERT INTO Races (id, name, description) VALUES
(1, 'Zerg', 'Eats humans'),
(2, 'Terran', 'Marine hoorah'),
(3, 'Protoss', 'For Aiur'),
(4, 'Xel Naga', '...');
```

Seed `planets`
```sql
-- Delete existing data
DELETE FROM Planets;
-- RESET SERIAL PRIMARY KEY
ALTER SEQUENCE planets_id_seq RESTART WITH 1;
-- SEED NEW DATA
INSERT INTO Planets (id, name, moons) VALUES
(1, 'Char', 0),
(2, 'Earth', 1),
(3, 'Aiur', 6),
(4, 'Korhal', 3);

```

Associate `planets` to `races`. Zerg controls Char (1,1) and shares control of Earth (2,1). Terran controls Korhal (4,2) and shares control of Earth (2,2). Protoss control Aiur (3,3).
```sql
-- Delete existing data
DELETE FROM Planets_Races;
-- SEED NEW DATA
INSERT INTO Planets_Races (planet_id, race_id) VALUES
(1, 1),
(2, 1),
(4, 2),
(2, 2),
(3, 3);
```

Seed `heroes` and associate them with planets and races
```sql
-- Delete existing data
DELETE FROM heroes;
-- RESET SERIAL PRIMARY KEY
ALTER SEQUENCE heroes_id_seq RESTART WITH 1;
-- SEED NEW DATA
INSERT INTO Heroes (id, planet_id, race_id, alias, level) VALUES
(1, 2, 2, 'Jim Raynor', 5),
(2, 1, 1, 'Queen of Blades', 20),
(3, 3, 3, 'Zeratul', 15),
(4, 4, 2, 'Arcturus Mensk', 5);
```

## QUERIES
Basic Queries
```sql
-- Find all planets
SELECT * FROM Planets;
-- id |  name  | moons
-- ----+--------+-------
--  1 | Char   |     0
--  2 | Earth  |     1
--  3 | Aiur   |     6
--  4 | Korhal |     3

-- Find all races only display id and name
SELECT id, name FROM Races;
-- id |   name
-- ----+----------
--  1 | Zerg
--  2 | Terran
--  3 | Protoss
--  4 | Xel Naga

-- Find all heroes
SELECT * FROM Heroes;
-- id | planet_id | race_id |      alias      | level
-- ----+-----------+---------+-----------------+-------
--  1 |         2 |       2 | Jim Raynor      |     5
--  2 |         1 |       1 | Queen of Blades |    20
--  3 |         3 |       3 | Zeratul         |    15
--  4 |         4 |       2 | Arcturus Mensk  |     5

-- Find all Heroes level 10 and above only displaying alias as "Boss Hero"
SELECT alias as "Boss Hero" , level FROM Heroes WHERE level > 10;
--    Boss Hero    | level
-- -----------------+-------
-- Queen of Blades |    20
-- Zeratul         |    15
```
## IMPORT & EXPORT
IMPORT csv file into table
```sql
\copy <table_name> FROM '<file_path>' CSV HEADER
```
IMPORT CSV file into table, only specific columns
```sql
\copy <table_name>(<column_1>,<column_1>,<column_1>) FROM '<file_path>' CSV HEADER
```
EXPORT table into csv file
```sql
\copy <table_name> TO '<file_path>' CSV
```
EXPORT table into csv file, only specific columns
```sql
\copy <table_name>(<column_1>,<column_1>,<column_1>) TO '<file_path>' CSV
```

## JOINS
![sql_joins](//images.ctfassets.net/vwq10xzbe6iz/5KN4Uq6wxnbS82uHjJ1cEL/186a86cce3f10ec4fbf1487c51da820f/sql_joins.png)

INNER JOIN query: Race `has_many` Heroes
```sql
-- Find all heroes belonging to race:Terran
SELECT * 
FROM Heroes
INNER JOIN Races
ON Heroes.race_id = Races.id
WHERE Races.name = 'Terran';
-- id | planet_id | race_id |     alias      | level | id |  name  |  description
-- ----+-----------+---------+----------------+-------+----+--------+---------------
--  1 |         2 |       2 | Jim Raynor     |     5 |  2 | Terran | Marine hoorah
--  4 |         4 |       2 | Arcturus Mensk |     5 |  2 | Terran | Marine hoorah
```

LEFT JOIN query: Planets `has_many` Races
```sql
-- Find all planets including their races (if any)
SELECT planets.name as planet_name, races.name as race_name
FROM planets
LEFT JOIN planets_races
ON planets_races.planet_id = planets.id
LEFT JOIN races
ON races.id = planets_races.race_id
ORDER BY planet_name;
--  planet_name | race_name 
-- -------------+-----------
--  Aiur        | Protoss
--  Braxis      | Terran
--  Char        | Zerg
--  Earth       | Terran
--  Earth       | Zerg
--  Kaldir      | Zerg
--  Korhal      | Terran
--  Mar Sara    | Terran
--  Moria       | Terran
--  New Folsom  | Terran
--  Phaeton     | Zerg
--  Shiloh      | Terran
--  Tarsonis    | 
--  Umoja       | Terran
--  Xt39323     | 
-- (15 rows)
``` 

FULL OUTER JOIN query: Planets `has_many` Races
```sql
-- Find all planets and races including combinations
SELECT planets.name as planet_name, races.name as race_name
FROM planets
FULL OUTER JOIN planets_races
ON planets_races.planet_id = planets.id
FULL OUTER JOIN races
ON races.id = planets_races.race_id
ORDER BY planet_name;
--  planet_name | race_name 
-- -------------+-----------
--  Aiur        | Protoss
--  Braxis      | Terran
--  Char        | Zerg
--  Earth       | Terran
--  Earth       | Zerg
--  Kaldir      | Zerg
--  Korhal      | Terran
--  Mar Sara    | Terran
--  Moria       | Terran
--  New Folsom  | Terran
--  Phaeton     | Zerg
--  Shiloh      | Terran
--  Tarsonis    | 
--  Umoja       | Terran
--  Xt39323     | 
--              | Xel Naga
-- (16 rows)
``` 

INNER JOIN query: Planets `has_many` Races
```sql
-- Find only planets that have races
SELECT planets.name as planet_name, races.name as race_name
FROM planets
INNER JOIN planets_races
ON planets_races.planet_id = planets.id
INNER JOIN races
ON races.id = planets_races.race_id
ORDER BY planet_name;
-- planet_name | race_name 
-- -------------+-----------
--  Aiur        | Protoss
--  Braxis      | Terran
--  Char        | Zerg
--  Earth       | Zerg
--  Earth       | Terran
--  Kaldir      | Zerg
--  Korhal      | Terran
--  Mar Sara    | Terran
--  Moria       | Terran
--  New Folsom  | Terran
--  Phaeton     | Zerg
--  Shiloh      | Terran
--  Umoja       | Terran
-- (13 rows)
```

INNER JOIN query: Race `has_and_belongs_to_many` Planets
```sql
-- Find all planets controlled by race: Zerg
SELECT *
FROM Planets
INNER JOIN Planets_Races
ON Planets.id = Planets_Races.planet_id
INNER JOIN Races
ON Planets_Races.race_id = Races.id
WHERE Races.name = 'Zerg';
-- id | name  | moons | planet_id | race_id | id | name | description
-- ----+-------+-------+-----------+---------+----+------+-------------
--  1 | Char  |     0 |         1 |       1 |  1 | Zerg | Eats humans
--   2 | Earth |     1 |         2 |       1 |  1 | Zerg | Eats humans

-- Find all races that control planet: Earth 
SELECT Races.id, Races.name as race_alias
FROM Races
INNER JOIN Planets_Races
ON Races.id = Planets_Races.race_id
INNER JOIN Planets
ON Planets_Races.planet_id = Planets.id
WHERE Planets.name = 'Earth';
-- id | race_alias
-- ----+------------
--  1 | Zerg
--  2 | Terran

```

## AGGREGATE FUNCTIONS
COUNT function: Heroes
```sql
-- Count how many heroes there are with alias'
SELECT count(Heroes.alias) from Heroes;
-- 4
```

INNER JOIN & GROUP BY query: Race `has_many` Heroes
```sql
-- Count all heroes by race order by hero count descending
SELECT Races.name, count(Heroes.alias) as "hero_count"
FROM Races
FULL OUTER JOIN Heroes
ON Races.id = Heroes.race_id
GROUP BY Races.name
ORDER BY hero_count DESC;

   name   | hero_count
----------+------------
 Terran   |          2
 Protoss  |          1
 Zerg     |          1
 Xel Naga |          0
```

ARRAY_AGG function
```sql
-- Expand query above and include comma separated string list of hero alias names
SELECT Races.name, count(Heroes.alias) as "hero_count",
ARRAY_TO_STRING(ARRAY_AGG(Heroes.alias), ', ') as "hero_list"
FROM Races
FULL OUTER JOIN Heroes
ON Races.id = Heroes.race_id
GROUP BY Races.name
ORDER BY hero_count DESC;

   name   | hero_count |      hero_list
----------+------------+----------------------------
 Terran   |          2 | Jim Raynor, Arcturus Mensk
 Protoss  |          1 | Zeratul
 Zerg     |          1 | Queen of Blades
 Xel Naga |          0 |
```

LEFT OUTER JOIN:
Find all unique planets and include their race count and list of race names
```sql
SELECT planets.name, count(races.name) as race_count, ARRAY_TO_STRING(ARRAY_AGG(races.name), ', ') as race_list
FROM planets
LEFT OUTER JOIN planets_races
ON planets.id = planets_races.planet_id
LEFT OUTER JOIN races
ON races.id = planets_races.race_id
GROUP BY planets.name
ORDER BY race_count DESC;
--     name    | race_count |  race_list   
-- ------------+------------+--------------
--  Earth      |          2 | Zerg, Terran
--  Umoja      |          1 | Terran
--  Moria      |          1 | Terran
--  Mar Sara   |          1 | Terran
--  Shiloh     |          1 | Terran
--  Aiur       |          1 | Protoss
--  New Folsom |          1 | Terran
--  Phaeton    |          1 | Zerg
--  Kaldir     |          1 | Zerg
--  Korhal     |          1 | Terran
--  Braxis     |          1 | Terran
--  Char       |          1 | Zerg
--  Tarsonis   |          0 | 
--  Xt39323    |          0 | 
```


SQL basic boilerplate using starcraft relational data model based on races, planets and heroes

sql_joins

SQL Basics

Retail today is dominated by online shopping and Amazon, leaving most retailers struggling to survive. Yet, Costco is thriving despite having the smallest e-commerce channel. How? By focusing on what sucks, then working hard to avoid it.
![Top 5 US retailers](//images.ctfassets.net/vwq10xzbe6iz/2D9n9SqEYYIEqStScm44Tk/98ed3662f472cbe91991580f68768041/Top_5_US_retailers.png)

__Sections__
1. [What makes a business suck?](#what)
2. [How Costco avoids failure](#how)
3. [Johnny Carson on how to guarantee a life of misery](#johnny)

## <a name="what"></a>1. What makes a business suck?
Imagine you inherit a grocery store. Your spouse warns you not to take the job. She lists the things that will make your lives miserable, including:
- Unreliable staff;
- Store theft;
- Unsold inventory; and
- Copycat competitors.

You ignore her and tell her about the amazing growth opportunities from e-commerce, new store expansion, fit outs and expanding into new product lines.

Fast forward and the bad things she warned you about become worse:
- Staff turnover grows and customer satisfaction drops.
- Bad customers steal from you, take advantage of returns policies and force you to hike prices for everyone.
- Unsold stock piles up, forcing you to discount and lose money.
- Copycat competitors open next door and sell items cheaper with better service.

__Your business and life sucks.__
## <a name="how"></a>2. How Costco avoids failure
Costco takes a unique and oftentimes counterintuitive approach to their business strategy. Focus on what sucks, then work hard to avoid it.

#### 1. How to avoid unreliable staff?
__Pay staff more and treat them well__

Costco spends 37% more on their employees than Walmart, offers better healthcare and promotes from within. By hiring good people, giving them good jobs and paying them good wages, Costco achieves 3x more revenue per employee than Walmart and a 90% employee retention rate.

> If you hire good people, give them good jobs, and pay them good wages, generally something good is going to happen.

Jim Sinegal, Costco Co-Founder

#### 2. How to avoid store theft?
__Choose better customers__

Costco charges a $US60 annual membership fee. They deliberately choose to have fewer but higher quality shoppers visit their stores. Not surprisingly, Costco shoppers spend more, steal less and treat staff better. Costco’s inventory shrink (loss from theft, damage and fraud) is estimated to be around 0.1% of annual sales compared to a 13x higher industry average of 1.6%.

#### 3. How to avoid unsold inventory? 
__Stock fewer categories__

Costco stocks fewer product categories, but offers bigger discounts. Costco limits markups of its goods to 15% above cost, attracting volume and further negotiating power with suppliers to further lower costs and pass on savings to customers. The average revenue per Costco store is 3.7x higher than Walmart in the US, with annual sales of $US299m vs $US79m per store.

#### 4. How to avoid competitor copycats? 
__Pay more for better locations__

Thanks to advice from Charlie Munger, Costco began paying more to acquire land and store locations to avoid copycat competitors setting up next door.

> We used to buy the cheapest land possible…That was fine until you have a competitor in the same market. Location does matter. Charlie said ‘Why don’t you pay twice as much and get a better site.’ From a long-term ROIC perspective, it has worked very well.

Richard Galanti, Costco former CFO
## <a name="johnny"></a>3. Johnny Carson on how to guarantee a life of misery
Johnny Carson, legendary host of the Tonight Show in a Harvard graduation speech as recalled by Charlie Munger said:

> Carson couldn’t tell the graduating class how to be happy, but he could tell them from personal experience how to guarantee misery.
> 
> These included:
> - Ingesting chemicals in an effort to alter mood or perception;
> - Envy; and
> - Resentment.

Charlie noted:
> I can still recall Carson’s absolute conviction as he told how he had tried these things on occasion after occasion and had become miserable every time.

So how do you avoid failure? 

__Focus on what sucks, then work hard to avoid it.__


Costco's retail strategy: 1) Pay staff more and treat them well; 2) Choose better customers; 3) Stock fewer categories; 4) Pay more for better locations.

Top 5 US retailers

How Costco avoids failure? Focus on what sucks

When it comes to returning capital to shareholders, dividends are the clear winner over share buybacks. While buybacks often serve short-term executive goals, dividends provide a more transparent and consistent return to shareholders.

A share buyback, where a company acquires and cancels its own shares, increases the ownership percentage of remaining shareholders. However, these buybacks are frequently motivated by the desire to enhance executive KPIs, such as share price and earnings per share, rather than long-term shareholder value.

__Sections__
1. [Share buyback vs dividends: Impact on shareholder value](#value)
2. [After-tax returns: Share sales vs dividends](#tax)
3. [Incentive alignment: Executives and shareholders](#incentive)

## <a name="value"></a>1. Share buyback vs dividends: Impact on shareholder value
The common but incorrect argument favoring buybacks over dividends is that they boost earnings per share and the valuation of the company.

The overall equity valuation of a company is unaffected by its choice to execute a share buyback or distribute dividends. Let’s explore this with an example of a company with $100 profit, a PE ratio of 10x, and decision to distribute the profit as dividend or buyback:
![shareholder impact](//images.ctfassets.net/vwq10xzbe6iz/43uHuzBUBSwZiBJbVc0olT/10cb91f8175ee19c8e7e0fbbb705aae0/shareholder_impact.png)
1. __Retaining Cash__: The company's equity valuation remains at $1,000, with a total of 1,000 shares outstanding, each priced at $1.00.
2. __Paying Dividends__: When a dividend of $0.10 per share is paid out, the equity valuation decreases by the dividend amount, and the share price adjusts to $0.90. The overall value to the shareholder, however, stays at $1.00 per share, combining a $0.90 share value with a $0.10 cash dividend.
3. __Conducting a Share Buyback__: The equity valuation decreases by the buyback amount of $100. After purchasing shares at $1.00 each, the share count reduces to 900. The remaining shareholders' equity value per share stays constant at $1.00.

__Market timing in buybacks__

As share buybacks are a human decision, they tend to occur when market sentiment is buoyant and there is surplus cash on hand. This can hurt shareholders, when companies repurchase overvalued stock and deplete funds needed for future business growth.

The argument that companies should buyback shares when it is trading below intrinsic value assumes management has perfect knowledge of the intrinsic value of the company. If this were true they should immediately retire, mortgage their house and go all in on the stock.

## <a name="tax"></a>2. After-tax returns: Share sales vs dividends
Another common argument favoring share buybacks is their tax efficiency. Dividends are taxed at the recipient's marginal rate, whereas profits from share sales, including those from buybacks, are often subject to the lower long-term capital gains tax.

The table below compares the after tax return of receiving $100 in ordinary cash dividends vs selling $100 worth of shares held for longer than 12 months. In countries like the US and UK there can be a material after tax benefit in realising value from capital gains vs cash dividends.

![tax](//images.ctfassets.net/vwq10xzbe6iz/1Xqix5YJDTMMuEPMOKmr2h/7e64173821644ecf8969772516425a97/tax.png)

__Tax effective alternatives__

However, it's important to recognize that this tax benefit is not unique to buybacks but extends to all capital gains. A shareholder is still able to tax effectively sell a portion of their shareholding at anytime if they believe the shares are overvalued.

In places like Australia, dividend franking credits are used to avoid double taxation of company profits. As a result, Australia has amongst the highest dividend payout ratios in the world. 

If tax efficiency were the sole rationale, companies might prefer pro-rata off-market buybacks, allowing all shareholders to participate instead of large institutions and hedge funds which dominate the volume of shares traded. Yet, the prevalent practice is on-market buybacks aimed at boosting share prices.

## <a name="incentive"></a>3. Incentive alignment: Executives and shareholders
Charlie Munger’s famous quote, “Show me the incentive and I’ll show you the outcome” highlights the importance of aligning executive and shareholder interests.

Warren Buffet proposes several principles for better alignment:
- Favoring cash bonuses based on business performance rather than stock options dependent on share price.
- Encouraging earnings on additional capital that surpass defined hurdle rates, like Return on Invested Capital.
- Focusing on revenue and profitability metrics, such as revenue, EBIT and free cash flow targets.
- Setting goals based on operational milestones that reflect long-term performance.

Share buybacks when used judiciously and not solely in order to promote executive incentives can still be an effective way of enhancing shareholder value. As Warren Buffet famously said 
> Picking the right CEO is 10 times more important than the compensation.


Why dividends are better than share buybacks. Understand impact on shareholder value. After tax returns. Incentive alignment of executives and shareholders.

shareholder impact

Share Buybacks vs Dividends: Who wins?

Don't ask "what" or "why" when users suggest features. Instead, ask "when did you last face this problem?”. This leads to quicker, better solutions.

![ask_when](//images.ctfassets.net/vwq10xzbe6iz/6IyPtJI4JfRmHsFoI1ODAk/03c6368f3bada5a73dbd04f552de1871/ask_when.png)

Here are two examples that illustrate the benefits of asking "when":
1. [Blocking contractors from deleting tickets](#permissions)
2. [Designing a custom dashboard for energy visualization](#dashboard)

## <a name="permissions"></a>1. Blocking Contractors from Deleting Tickets
See how "what", "why", and "when" can lead to different solutions when handling ticket deletions by contractor.

| Question     | Answer     | Solution     | Build cost     |
| ---------- | ---------- | ---------- | ---------- |
| What feature do you want?       | I don’t want contractors to be able to delete tickets       | Contractor role and permissions. Ticket assignee permissions. Admin role to set contractor permissions.       | $$$       |
| Why do you want this feature?       | I need access to all my tickets       | Archive folder before permanent delete. Make archive folder visible to admin users.       | $$       |
| When did you last face this problem?       | On the last project an external contractor on our team deleted a ticket by accident, thinking it would only be removed from his view       | Update delete confirm modal with a warning displaying the names of other users who will lose access to this ticket (social warning)       | $       |

## <a name="dashboard"></a>2. Designing a custom dashboard for energy visualization
Next, see how asking different questions shapes quicker, better solutions for an energy dashboard.

| Question     | Answer     | Solution     | Build cost     |
| ---------- | ---------- | ---------- | ---------- |
| What feature do you want?       | I want a custom dashboard to visualise my building's energy consumption       | Data integration. Widget customization. Chart types. Preset templates. Drag and drop functionality. Export capability. External sharing.       | $$$$$       |
| Why do you want this feature?       | I need to report on monthly energy deviations       | Data table view. A4 report generator. PDF export. Word export. Custom date range. Company logo. Custom formatting.       | $$$       |
| When did you last face this problem?       | At our last building tenant committee meeting I did not have the utility bills in time and couldn't include the monthly energy consumption figures in the excel sheet prior to the meeting       | CSV export of monthly consumption figures.       | $       |

### References
- Ryan Singer, and Chris Spiek (Hosts). (2017). Demand Thinking. [Episode 1: Feature requests aren’t demand](https://demandthinking.com/episodes/2017/7/23/episode-1-think-you-might-be-building-the-wrong-thing-youve-misinterpreted-demand).


Explore examples of revealing struggles from feature requests: 1) manage user permission; 2) custom visualization dashboard

ask_when

How to respond to product feature requests? Ask “When was the last time”

Here are the 6 key financial ratios that can offer insights into a business’s operational performance. This guide uses practical examples from well-known companies - Best Buy, Netflix, Meta and Apple. These ratios are:

1. [Gross Margin](#gross-margin)
2. [Operating Margin](#operating-margin)
3. [Free Cash Flow Conversion](#fcf-conversion)
4. [Free Cash Flow Margin](#fcf-margin)
5. [Cash Conversion Cycle](#ccc)
6. [Leverage Ratio](#leverage-ratio)

![financial-ratios](//images.ctfassets.net/vwq10xzbe6iz/5QJuRsYRmu8hUMFsmQxoAm/51781a90e0fd1d0ea40ab5a6f8a2fbea/financial_ratios.png)

### <a name="gross-margin"></a>1. Gross Margin
> Gross Margin = Gross Profit / Revenue

Gross margin is a profitability indicator that shows what portion of revenue remains after deducting the cost of revenues. It's calculated by dividing gross profit by the total revenue for a given period. 

Best Buy, a big-box electronics retailer operates with low gross margins (21%). Its cost of revenues include the cost of products purchased (e.g. from Apple), shipping expenses and in-store staff costs.

Meta has high gross margins (78%) as it primarily generates ad revenue on its social media platforms (Facebook, Instagram, WhatsApp) and captures the majority of that revenue. This compares to YouTube which has lower gross margins as it splits its ad revenue 55% to content creators.

### <a name="operating-margin"></a>2. Operating Margin
> Operating Margin = Operating Profit / Revenue

Operating margin, also known as operating profit margin, serves as a key measure of profitability. This figure represents the percentage of total revenue that remains after accounting for all operating expenses - these include costs beyond just those of producing goods or services, but do not encompass financing or tax expenses. It's determined by dividing operating profit (alternatively called operating income or EBIT) by the total revenue for a specified period.

Comparing Apple and Meta, we see that despite a lower gross margin (43% vs. 78%), Apple maintains a higher operating margin (30% vs. 20%). This is due to Apple's efficient operations, strong brand, and strategic marketing spending. On the other hand, Meta spends more on research and development (30% of revenue vs. Apple's 7%) on its platforms and new VR technology, and incurs significant data management, privacy, and security costs.

### <a name="fcf-conversion"></a>3. Free Cash Flow Conversion
> FCF Conversion = Free Cash Flow / Operating Profit (EBIT)

Free Cash Flow Conversion is a critical financial metric showing how efficiently a company can turn its operating profit (earnings before interest and taxes or EBIT) into free cash flow. Free cash flow represents the cash a company generates, which is available to all stakeholders, including shareholders, debt holders, and others (net operating cash flows minus capital expenditures). This metric is calculated by dividing free cash flow by the operating profit for a specified period.

Comparing Apple and Netflix, Apple is more efficient at converting its profits into available cash with a free cash flow conversion rate of 93%, compared to Netflix's 29%. 

Under Tim Cook, Apple dramatically improved its cash flow by pushing inventory risks to suppliers such as Foxconn. Cook introduced “Just-in-Time” inventory management, meaning Apple only orders components as needed, reducing storage and related costs. Despite being a hardware supplier and retailer, Apple has less inventory on hand than Best Buy which generates less than 10% of Apple’s revenues. 

Netflix's low cash flow conversion is due to its heavy upfront investment in content. It spends a lot upfront on producing and acquiring shows and movies, which it then gradually expenses over years. This means the cash is spent before it fully impacts their profits, leading to lower cash flow.

### <a name="fcf-margin"></a>4. Free Cash Flow Margin
> FCF Margin = Free Cash Flow / Revenue

This ratio indicates how much of the company's revenues are converted into free cash flow. A higher percentage means the company is more efficient at converting sales into cash. It is calculated by dividing free cash flow by the revenue over the specified period.

Apple leads in this metric as well with 28%. This means that for every $100 in revenue Apple generates, $28 is available to shareholders to be paid out as dividends, share repurchases or further reinvestment. This compares to Netflix with 5% and Best Buy at 2%.

### <a name="ccc"></a>5. Cash Conversion Cycle
> Cash Conversion Cycle = Days Inventory Outstanding + Days Sales Outstanding - Days Payable Outstanding

The Cash Conversion Cycle (CCC) measures the length of time a business takes to pay for inventory from its suppliers to when it collects cash from its customers. Calculated as the days inventory, plus the days sales (receivables collection period) minus the days payable.

- Days Inventory Outstanding = Inventory Closing / Cost of Revenues x 365 days
- Days Sales Outstanding = Receivables Closing / Revenues x 365 days
- Days Payable Outstanding = Payables Closing / Cost of Revenues x 365 days

When we compare Best Buy and Apple, we see quite a contrast. Best Buy has a CCC of 4 days while Apple has a CCC of negative 71 days.

Apple's negative CCC indicates that it collects payments from its customers much quicker than it needs to pay its suppliers. This situation is a result of Apple's efficient supply chain, powerful negotiation position with suppliers, and fast inventory turnover due to high consumer demand for its products.

On the other hand, Best Buy's positive CCC reflects its retail business model, where it must purchase inventory upfront, hold it in stores until a customer purchases it, and then wait for the transaction to clear. Best Buy's shorter cash conversion cycle still reflects efficient operations for a retail company, but its nature of business doesn't allow the luxury of a negative CCC like Apple.

### <a name="leverage-ratio"></a>6. Leverage Ratio

> Leverage Ratio = Net Debt / Operating Profit (EBIT)

The leverage ratio is an important metric that assesses a company's financial health by measuring the extent to which it uses borrowed money (debt) to finance its operations. In this context, we're specifically looking at net debt (total debt minus cash and cash equivalents) divided by operating profit, also known as earnings before interest and taxes (EBIT).

Apple's leverage ratio of -0.6x suggests a strong financial stance, with more cash than total debt, largely due to efficient cash management and substantial profits. In comparison, Netflix, with a 1.5x leverage ratio, carries more debt than cash, attributed to substantial upfront investments in content, often financed through debt.

A more conservative measure would be to look at the leverage ratio against Free Cash Flow. Given Netflix’s poor FCF conversion this would imply a leverage ratio (Net Debt/ FCF) of 5.1x. This means that it would take more than 5 years for Netflix to repay its debts from operating free cash flows leaving nothing for shareholders or reinvestment.


1. Gross Margin; 2. Operating Margin; 3. FCF Conversion; 4. FCF Margin; 5. Cash Conversion Cycle; 6. Leverage Ratio. Understand their significance with real-world examples from Best Buy, Netflix, Meta, and Apple

financial-ratios

Understanding 6 Key Financial Ratios - Best Buy vs Netflix vs Meta vs Apple

Cognitive biases can negatively impact your ability to think critically and make informed decisions. Once you are aware of them, it will help you identify and navigate them.

Here is an explanation of the 7 main types of cognitive bias, some examples and how you can navigate them:
1. [Survivorship Bias](#survivorship)
2. [Confirmation Bias](#confirmation)
3. [Framing Bias](#framing)
4. [Groupthink](#groupthink)
5. [IKEA Effect](#ikea-effect)
6. [Availability Bias](#availability)
7. [Commitment Bias](#commitment)

## <a name="survivorship"></a>1. Survivorship Bias
> Focus on the traits of the successful whilst ignoring the failures.

![survivorship-bias](//images.ctfassets.net/vwq10xzbe6iz/t9fd2snGayd6dZM2Dn9Js/5e615d3df42fa017582a58b9f06a0e57/survivorship.JPG)
This involves limiting our analysis to instances that have passed a success hurdle and not analysing those that didn’t pass the hurdle. This commonly occurs when we adopt key learnings from successful case studies yet remain blind to the unsuccessful instances who may have also followed the same learnings. 

### Examples
The famous [WWII case study](https://www.researchgate.net/publication/254286514_Abraham_Wald's_Work_on_Aircraft_Survivability) involves mathematician Abraham Wald who was tasked with identifying where to place more armour on fighter planes. The officers data of bullet holes in returning planes supported placing more amour on the fuselage vs the engine. However, Wald’s insight was to ask where the bullet holes weren’t - i.e. the engine. The surviving planes had fewer hits to the engine because those that were hit didn’t return.

A modern example would be to look at a customer acquisition funnel with a 10% activation rate. Common analysis would be to take key learnings from the behaviour and personas of this group to further lift activation. However, the greater insight would be to analyse the other 90% and ask why aren’t they activating? The latter is more challenging and often involves less access to data, but the payoff is greater.

### How to Navigate?
Ask questions about what other data and groups are missing. Identify the success hurdle and ask why others have not passed it.

## <a name="confirmation"></a>2. Confirmation Bias
> Focus on data to promote our existing beliefs.

![confirmation-bias](//images.ctfassets.net/vwq10xzbe6iz/3Fwt3vluPTrXkA41Yfj6Dx/585fec98fc5647b58a14f4d568b973c7/confirmation.JPG)
This involves seeking out information that validates our existing views and prior commitments. We also discount other information that may invalidate our views. Confirming evidence makes us feel good and people do what makes them feel good. The counter approach is the scientific method, which attempts to ask questions first before gathering evidence to either support or reject a hypothesis. The former is easier and faster, the latter is harder and time consuming.

### Examples
The CEO has a new idea to launch a new product and instructs a team to conduct market research and competitor analysis on the feasibility of this product. The market research team is limited in scope and risks being urged to validate the success of the CEO’s new product idea. A better task for the team would be to investigate what the best product would be to solve a particular problem.

### How to Navigate? 
Try and identify if there is a pre-existing belief (yours or otherwise). Reflect on information you have rejected. Ask others with a different viewpoint.

## <a name="framing"></a>3. Framing Bias
> Solving a problem based on the context it is presented.

![framing-bias](//images.ctfassets.net/vwq10xzbe6iz/1RJWOsPkQawJKKxYk8oIQM/6ee3c51d2e488ae5ecd40e1abd233596/framing.JPG)
The specific context a problem is framed/ presented can influence the options and choices available.

### Examples
The problem of low company profitability can be either framed as a need to cut costs or to increase revenue. The way the problem is framed will influence the options available (i.e. fire and restructure or enter new markets).

### How to Navigate? 
Ask what is the ultimate problem we are trying to solve. For example, ask what the outcome would be if we solved this problem. See if there are other solutions to achieving the same outcome.

## <a name="groupthink"></a>4. Groupthink

> The desire for group harmony leads to poor decision making.

![groupthink](//images.ctfassets.net/vwq10xzbe6iz/3BvESluOEJuRmYVODq44U1/12de08589e23649ad287ff3abba0912b/groupthink.JPG)
People tend to avoid conflict, particularly in the presence of a majority and people of authority. People assume others have made informed decisions and fear the risk of looking un-knowledgable in front of others. 

### Examples
The term groupthink arose from the [case study](https://www.bassberry.com/wp-content/uploads/Effective_SEC_Rulemaking_Letters2.pdf) of the failed US Cuba Bay of Pigs invasion. In 1961, the CIA presented to President Kennedy a Cuba invasion plan formulated under the previous administration. Kennedy accepted it and early objections were dismissed and eventually no objections or alternatives arose. Kennedy learnt from this and a year later during the Cuban Missile Crisis intentionally absented himself from initial meetings to encourage free discussion. Two teams were created to evaluate the options before presenting to Kennedy. 

### How to navigate? 
Be clear on the objective and values. Encourage other views to solve the objective.

## <a name="ikea-effect"></a>5. IKEA Effect
> Overvaluing things we place successful effort into

![ikea-effect](//images.ctfassets.net/vwq10xzbe6iz/1o8egJQ2qvuZqLQjsPpOSf/5c7e44c7d77fc37f809a05fe47b6d357/ikea-effect.png) 
The IKEA effect is a finding that people tend to ascribe a higher valuation towards self-assembled products compared to objectively similar products which they did not assemble ([see article for study](https://mannhowie.com/ikea-effect)).
### Examples
A product team has spent significant time developing a high fidelity prototype based on the sales team's feedback. It has the latest best practice design patterns. They have not shipped an early version yet. The IKEA Effect makes the team less likely to cull or dramatically change the product in response to real world usage. They are at risk of devoting resources to failing projects. 
### How to navigate? 
Treat your ideas like cattle, not like pets. Spike rough prototypes, talk to customers and run growth experiments.

## <a name="availability"></a>6. Availability Bias
> Overestimate the likelihood or frequency of an event when it is easier to recall

![availability-bias](//images.ctfassets.net/vwq10xzbe6iz/3xmF85xOP7EIdJWIa0tJ9D/e54863460e43ac8d37a49683182e7fd0/availability-bias.png)
People tend to assess the relative importance of issues by the ease of which they are retrieved from memory. Factors that influence memory include: recency, vivid emotion, and imagery.
### Examples
Tversky and Kahneman's famous 1973 study had participants listen to a list of 39 names being read and were then asked to recall whether the list contained more men or women. The list either had fewer names of famous women and more names of less famous men (e.g. Elizabeth Taylor vs William Fulbright) or a list with fewer names of famous men and a list with more names of less famous women (e.g. Richard Nixon vs Lana Turner).

Over 80% of the participants incorrectly judged the list with famous names to be more frequent in terms of gender. 

### How to navigate?
Look for different examples: seek out and compare counter examples of what is easier to remember. For example older vs recent instances, positive vs negative experiences, multiple instances of any memorable imagery.

## <a name="commitment"></a>7. Commitment Bias
> Following actions consistent with prior beliefs, words and actions

![commitment-bias](//images.ctfassets.net/vwq10xzbe6iz/64jQHFYz9cGi3m4paOJYvj/346f4a84c3224ec7e8b9601900999f1b/commitment-bias.png)
Following through with commitments help confirm self-image and avoid cognitive dissonance. They are most effective when they are active, public, effortful and viewed as internally motivated.
### Examples
At the height of the Cabbage Patch Kids craze in the 1980s the company was forced to halt its pre-christmas advertising after it was accused of over advertising and under-supplying toys. The strategy aimed to get parents to commit to their children of buying the toy in the lead up to Christmas. Then hoping parents would go to the store but left purchasing a different toy and then returning post Christmas to honour the original purchase commitment.
### How to navigate? 
Remind yourself it is ok to change your mind. Beware sales tactics that employ commitment bias to make prospects agree to a little purchase before a big purchase.

What are the 7 common types of bias: 1) Survivorship Bias; 2) Confirmation Bias; 3) Framing Bias; 4) Group Think; 5) IKEA Effect; 6) Availability Bias; 7) Commitment Bias.

survivorship

7 Types of Bias - Examples & How to Navigate

The Kano Model explains the tradeoff between product investment and customer satisfaction across three types of features:
1. [Basic Expectations](#basic-expectations)
2. [Performance Payoff](#performance-payoff)
3. [Excitement Generators](#excitement-generators)

![kano-model](//images.ctfassets.net/vwq10xzbe6iz/4vLc84rCsiPmeYdWwgsTBz/be84dfb7ec872fd4272d360d7bdd07aa/kano_model.png)

Framing features using the Kano Model’s three categories can help product teams prioritise and communicate the value of their investments.

## <a name="basic-expectations"></a> 1. Basic Expectations
Basic expectations are offerings that users expect you to meet. They are not a source of satisfaction but can cause frustration if not met. You can only screw up basic expectations. Examples include instant hot water in hotels, rechargeable EV batteries, reliable e-commerce delivery and file saving in documents.

Investment in these features are expensive and at best achieve a neutral customer satisfaction. Users don’t tell others about them if they are met but will complain to others if not met. Over-investment can cause diminishing returns. Good enough is usually enough.

Too many unmet basic expectations create openings for competitors. Smart startups will focus on nailing less features. This lowers the investment hurdle to achieve basic expectations and creates opportunity to invest in Performance Payoff and Excitement Generators.

## <a name="performance-payoff"></a> 2. Performance Payoff
Performance payoff features are not absolutely necessary but increase the enjoyment of the offering and can make it easier to achieve tasks. These features have a linear relationship between investment and customer satisfaction. Examples include room service in hotels, long-range EV batteries, next-day delivery and advanced formulas in spreadsheets and tables.

Over-investment in performance payoff increases the risk of feature rot, product complexity and creating new unmet basic expectations.

## <a name="excitement-generators"></a>3. Excitement Generators
Excitement generator are sources of unexpected delight and “wow factors”. They are not expected and do not cause frustration when not present. Examples include complementary whiskey samples in hotels, remote EV climate control, bonus subscriptions for Prime and sheets.new (try it).

Word of mouth travels with excitement generators. 

Excitement generators fail if basic expectations are unmet and performance payoffs are subpar. They are a luxury investment and typically only a few will become “showstopper” features. 

Over time excitement generators will become basic expectations which further raises the investment hurdle for competitors. 

## Summary
The Kano Model's three major product feature categories with examples:

| Customer Satisfaction     | Basic Expectations     | Performance Payoff     | Excitement Generators     |
| :---------- | :---------- | :---------- | :---------- |
| Expected by users?       | Yes       | No       | No       |
| Frustrated if missing?       | Yes       | No       | No       |
| Delighted if met?       | No       | Yes       | Yes       |
| Word of mouth if met?       | No       | Yes       | Yes       |
| Investment vs Payoff       | Diminishing       | Linear       | Exponential       |
| __Examples__       |        |        |        |
| Hotel       | Instant hot water       | Room service       | Whiskey samples       |
| Tesla       | Rechargeable battery       | Fast charging       | Remote climate control       |
| Amazon Prime      | Reliable delivery       | Next day delivery       | Prime Video, Audible, Twitch       |
| Google Sheets       | File saving       | Formulas       | sheets.new       |

### References
- Understanding the Kano Model ([Article](https://articles.uie.com/understanding-the-kano-model-a-tool-for-sophisticated-designers/)) - Jared Spool 
- Building a Winning Strategy Using the Kano Model ([Video](https://www.youtube.com/watch?v=Hr1rN3jibIk)) - Jared Spool

How to prioritise product roadmap for customer satisfaction: 1) Basic expectations; 2) Performance payoff; 3) Excitement generators. 

kano-model

Kano Model - 3 Types of Product Features with Examples

The person responsible for employee training and development is their direct manager. It is not a shared responsibility of the organization and should not be outsourced.
![employee-training](//images.ctfassets.net/vwq10xzbe6iz/7HzNVyRSlNcsPrmDJUxw8u/df11dc787065c48fd5fd6a32197991f3/employee-training.png)

This article argues why training is the manager’s job and how they can engage with others to take action to improve it:
1. [Why training is the manager’s job](#why-training)
2. [Benefits of manager training](#benefits-of)
3. [How managers can engage others to help train subordinates](#how-managers)

## <a name="why-training"></a>1. Why training is the manager’s job
Formel Intel CEO Andy Grove argues that as the manager is responsible for the output of the team, then if an employee is not doing their job it is the manager’s job to find out why and take action to improve it.

In his book High Output Management, Andy makes the syllogism that if you accept that:
1. Training, along with motivation, is the way to improve the performance of your subordinates; and
2. What you teach must be closely tied to what you practise, and that training needs to be continuous rather than a one-time event
3. Therefore it is clear that the who of training is you, the manager.

This means that the manager needs to be responsible and actively involved in the training of their subordinates rather than leaving it to HR or external trainers.

## 2. <a name="benefits-of"></a>Benefits of manager training
The two main benefits of a manager who trains is they are able to 1) better identify skills gaps; and 2) send the strongest positive culture:

### 1. Better identify skills gap
Managers are responsible for setting the team’s output (the what) and are closer to the alignment with the company’s objectives (the why). This is needed to both identify and prioritise the relevant skills and knowledge gaps which may often change.

### 2. Send the strongest positive culture
People look to managers as the “big dog’s” in the company, and culture often emulates their behaviour for better or for worse. A manager who invests their own time in training sends the message that continuous improvement and taking ownership trumps talent and finger pointing.

## 3. <a name="how-managers"></a>How managers can engage others to help train subordinates
Here is a summary of how managers may approach employee training with their subordinates and how to engage people in other divisions to assist.

| Person     | Manager Do’s ✅     | Manager Don'ts ❌     |
| ---------- | ---------- | ---------- |
| Manager (you)       |  <ul><li>Identify competency gaps and communicate why they are important to business objectives</li><li>Stretch employees circles of competence</li><li>Focus training and feedback on less vs more. Training should be compounding not one-off</li><li>Set time to continuously review work</li></ul> | <ul><li>Micromanage tasks</li><li>Blame employees for gaps</li><li>Criticise without giving examples of what good looks like</li><li>Provide too much information in a short period of time</li></ul>|
| Employee (subordinate)       | <ul><li>Review and QA their work to identify competency gaps</li><li>Listen to employee feedback to identify motivation gaps</li></ul>| <ul><li>Ask employee to propose external training seminars they are interested in (this puts burden on them)</li><li>Accept employee proposed training that misaligns with team output</li></ul>|
| HR      | <ul><li>Work with HR to align competency training with career and remuneration progression</li><li>Set better recruitment and hiring criteria for competency and coachability</li></ul> | <ul><li>Ask HR to propose learning and development plan</li><li>Ask HR to outsource to external training consultants</li></ul>|
| Subject Matter Expert       | <ul><li>Ask internal Subject Matter Experts (SMEs) with skills outside your circle of competence to assist with internal training and reading materials</li><li>Record training sessions with SMEs who are time poor</li><li>Regular catch-ups with SMEs to review employee participation</li></ul>| <ul><li>Ask SMEs to be responsible for training</li><li>Ask too much of SMEs that it may take away from their team’s output</li><li>Ask employee to engage directly with SME to level up</li></ul>|

## References
- Andy Grove (1995). ‘Why Training is the Boss’s Job (Chapter 16)’ In [High Output Management](https://www.amazon.com.au/High-Output-Management-Andrew-Grove/dp/0679762884)


Training and development is the manager’s job and should not be left to HR or outsourced. 1) Why training is the manager's job; 2) Benefits of manager training; 3) How managers can engage others to help train subordinates.

employee-training

Employee Training: Who is the Person Responsible?

Here are five rules of thumb to follow when designing user friendly data tables for a web app:
1. [First column readable](#first-column)
2. [Default sort by most common query](#default-sort)
3. [Place filters close to headers](#place-filters)
4. [View more fields using quick view](#view-more)
5. [Use pagination for more records](#use-pagination)

## <a name="first-column"></a>1. First column readable
Use human readable text for the first column. Do not use ids meant for record keeping. It should easily convey meaning about the record. It's ok if you have duplicate names.

![readable first column](//images.ctfassets.net/vwq10xzbe6iz/4kCSVKLBq2bitzrSZqmIQe/a42a5f8d0a2de31051a1ba0f662760b1/readable_first_column.png)

## <a name="default-sort"></a>2. Default sort by most common query
Decide what most users should first query when using your data table. Make this your default sort order when navigating to the table. Avoid sorting alphabetically by the first column.

![default sort by common query](//images.ctfassets.net/vwq10xzbe6iz/UK5tiCHmZK1qZXGhsJPzC/88e0b3dcfe8352e9e517b9296bdd3d08/default_sort_by_common_query.png)

__Bonus:__ 
Develop your data table to support dynamic urls for custom sort and filter options. Use this to design email campaigns for different tasks (e.g. follow up on overdue items, update your assigned items, shout out recently completed items).

## <a name="place-filters"></a>3. Place filters close to headers
Filters with good mapping should be placed close to the columns and headers that they control. Take advantage of spreadsheet conventions where column headers control sorting and filtering. Avoid advanced filter options placed outside your data table which can be hard to find and interrupt flow.

![filters close to headers](//images.ctfassets.net/vwq10xzbe6iz/7qD3btAg9jsuEvO6cP4La1/ae459bf2f3c38753749ad5ee9e5ff202/filters_close_to_headers.png)

## <a name="view-more"></a>4. View more fields using quick view

Show more fields and controls of a single record by using a quick view (side-panel or modal). Place your edit and delete controls inside the quick view. Avoid collapsible accordions. They are inconsistent with table headers and cause user cleanup issues.

![view more quick view](//images.ctfassets.net/vwq10xzbe6iz/3Jo30qeyHzCNh3LwUKRN5W/ff9b753cae1199caeab198fba74081b4/view_more_quick_view.png)

## <a name="use-pagination"></a>5. Use pagination for more records
Decide on the max rows each user should see then add pagination to indicate how many more records remain. Avoid “load more” buttons and infinite scroll. This causes issues of relocating records and false sense of completed search.

![use pagination](//images.ctfassets.net/vwq10xzbe6iz/6JMex6tpn6SA7IZS1fg6mU/a2a87b6e84712ef694db83dd5a034ed9/use_pagination.png)

## Conclusion
Data tables should help users find, compare and update data. Follow design patterns that help your users best achieve these goals.  

### Reference
- Data Tables: Four Major User Tasks ([Article](https://www.nngroup.com/articles/data-tables/)) Nielsen Norman Group 
- Infinite Scrolling: When to Use it, When to Avoid it ([Article](https://www.nngroup.com/articles/infinite-scrolling-tips/)) Nielsen Norman Group


Data tables in web apps should help users find, compare and update data. 5 best practices: 1) First column readable; 2) Default sort; 3) Filters close to headers; 4) Quick view more fields; 5) Pagination

readable first column

Data Table UX: 5 Rules of Thumb

Here are five rules of thumb you can apply to common UI design challenges when building web applications:

1. [Use defaults](#use-defaults)
2. [Obvious settings](#obvious-settings)
3. [Show progress](#show-progress)
4. [Avoid interruptions](#avoid-interruptions)
5. [Flat hierarchy](#flat-hierarchy)

## <a name="use-defaults"></a>1. Use defaults
Provide default values to guide users on the most common choices or behaviour you want to encourage.

![use defaults](//images.ctfassets.net/vwq10xzbe6iz/58lgklls1TQqgvFstGDF0B/938dffbe618bb3d424cf8c3acac015c4/use_defaults.png)

Defaults reduce the number of choices and decision fatigue users must face. They provide context as to the next decision required. Attempt to reduce the number of inputs you require by collapsing optional values. 

## <a name="obvious-settings"></a>2. Obvious settings
Settings should be obvious and unambiguous. Use radio buttons or checkboxes to indicate whether a setting is active. Where possible opt for binary settings instead of multiple states (e.g. flash mode on/ off vs flash mode auto, on or off).

![obvious settings mode](//images.ctfassets.net/vwq10xzbe6iz/7n6LxMmrztHcv9hRjouZbx/82419567536619908d199d5c20f16ebd/obvious_settings_mode.png)

Clearly label preference settings with positive active labels. Avoid negative labels. Checkboxes should enable things, not disable them.

![obvious settings preferences](//images.ctfassets.net/vwq10xzbe6iz/4HvT2YSTTMnzQFeQ5Wvh8x/c3551e5ea801c581972cfff1ede51a9c/obvious_settings_preferences.png)

## <a name="show-progress"></a>3. Show progress
Feedback should be immediate to communicate to users that the action has been received. When feedback cannot be immediate, use loading spinners for short waits (<10 seconds) and progress loaders for long waits (>10 seconds).

![show progress](//images.ctfassets.net/vwq10xzbe6iz/dMSFEcZYTeCzT6K5bx3LL/ce2eb70fb73b68448425fe946840141b/show_progress.png)

## <a name="avoid-interruptions"></a>4. Avoid interruptions
Avoid interrupting users' workflow by making decisions to allow slips with ways to undo. Allow slips with the use of less intrusive toasts vs intrusive warning confirm modals.

![avoid interruptions](//images.ctfassets.net/vwq10xzbe6iz/51iV8AagtUtyl8UeASDp8W/a41fb967b86ee1455f6d21a701419849/avoid_interruptions.png)

## <a name="flat-hierarchy"></a>5. Flat hierarchy
Use flat instead of deep hierarchies. Content is more discoverable if it is not deeply nested inside hierarchies. People are good at naturally grouping categories based on labels without relying on deep nesting.

![flat hierarchy](//images.ctfassets.net/vwq10xzbe6iz/4AoNBBWhKfLtA0ouDCSYjC/9a4ca81d0feb42cbadff92b4e566e7d4/flat_hierarchy.png)

### References
- The Power of Defaults ([Article](https://www.nngroup.com/articles/the-power-of-defaults/)) - Nielsen Norman Group
- Flat vs Deep Website Hierarchies ([Article](https://www.nngroup.com/articles/flat-vs-deep-hierarchy/)) - Nielsen Norman Group
- Designed for Use: Create Usable Interfaces for Applications and the Web (2016) ([Book](https://pragprog.com/titles/lmuse2/designed-for-use-second-edition/)) - Lukas Mathis


UI design patterns for web apps: 1) Use defaults; 2) Obvious settings; 3) Show progress; 4) Don’t interrupt; 5) Flat hierarchy


show progress

User Interface Design: 5 Rules of Thumb

The Cognitive Reflection Test (CRT) is a simple three question test that measures people’s tendency to answer questions with the first idea that comes to their mind without checking it.

The first and most popular question is the “bat and ball problem” which more than 50% of students at Harvard, MIT and Princeton get wrong. 

![cognitive-reflection-test](//images.ctfassets.net/vwq10xzbe6iz/55mZtOXLz6Z3uVaaFzEFKy/cce62be0aa1919ebdcf07e80fa73f713/cognitive-reflection-test.png)

### Sections
1. [The Bat and Ball Problem](#bat-and-ball)
2. [How to Improve Your CRT Score](#improve-crt)
3. [Examples of Reacting vs Checking](#react-vs-check)

## <a name="bat-and-ball"></a>1. The Bat and Ball Problem
__Problem:__
- A bat and a ball cost $1.10. 
- The bat costs $1.00 more than the ball. 
- How much does the ball cost?

__First Reaction:__
- 10 cents

Nearly all people have a first reaction of 10 cents which is wrong. The correct answer is 5 cents. 

In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton gave the incorrect intuitive response and did not actively check their response. A failure to check is instructive because it requires such minimal mental effort.

Anyone who would pause and check would recognise that if the ball is 10 cents the bat would need to cost $1.10 and the sum would be $1.20, which is above the target $1.10. 

Daniel Kahneman describes this cognitive bias as the distinction between a person’s two processes of thinking:
- __System 1__: executed quickly with little conscious deliberation; and
- __System 2__: executed slowly and more reflective

System 1 processes occur instantly and effortlessly for example knowing the answer to 5 x 5 or indicating left when turning the car left. 

System 2 processes require more effort, concentration and calculation for example the answer to 25 x 7 or correcting for car slides on icy roads. 

## <a name="improve-crt"></a>2. How to Improve Your CRT Score
Here are three ways you can improve your CRT score and mitigate against biases of System 1 thinking:
1. __Acknowledge__: that you will always have an initial “gut” reaction to an event or problem and jump to solution mode. This is unavoidable and it is ok. Note your initial reaction and solution;
2. __Pause and check__: your first idea before sharing your feedback. Check who the decision would impact and how much effort the solution would require; and
3. __Ask questions__: to find out who really has the problem and where the problem comes from. Check again whether your first idea would answer these questions.

## <a name="react-vs-check"></a>3. Examples of Reacting vs Checking
Here are three examples of how product, customer success and marketing teams may react with their first idea and how they could pause and check.

| Example     | React First Idea     | Pause & Check     |
| ---------- | ---------- | ---------- |
| __Product__ <br> Watching an end-user struggle to upload a file       | Propose feature solution of drag and drop file upload       | Before solution mode: <ul><li>check funnel data of what % of users fail upload</li><li>are those not uploading files blocking the flow of other users</li><li>are there non-UX factors causing this (e.g. file size, internet speed)</li></ul> |
| __Customer Success__ <br> Unhappy customer feedback that solution was not a fit with their internal system and they may now churn       | Blame sales team for selling to clients with poor system fits and create a new process to screen out incompatible customers       | Before solution mode: <ul><li>check how many existing customers are at risk of incompatibility. Are they using workarounds</li><li>could the cause of the incompatibility be known in advance of the sale</li><li>does the customer still see value in the solution</li></ul> |
|__Marketing__ <br> Prospect complains angrily they are getting too many marketing emails | Disable this week’s new marketing campaigns | Before solution mode:<ul><li>confirm how many other recent complaints have been received</li><li>check if prospect is an existing paying customer or opportunity</li><li>if not, check if prospect fits the ideal sales qualified target</li></ul>|

To learn more about how to pause and check, read about the 4 simple questions you can ask to help identify and solve the right problems before jumping to solution mode ([How to solve the right problems](https://mannhowie.com/problem-solve)). 

## References
- Shane Frederick (2005). ["Cognitive Reflection and Decision Making"](https://www.aeaweb.org/articles?id=10.1257/089533005775196732). Journal of Economic Perspectives.
- Daniel Kahneman (2011). [Thinking Fast and Slow](https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555). Chapter 3, The Lazy Controller


Why 50% of Harvard, MIT and Princeton students get the bat and ball problem wrong. 1) The Bat and Ball Problem; 2) How to Improve Your CRT Score; 3) Examples of Reacting vs Checking in Product, Customer Success and Marketing.


cognitive-reflection-test

Cognitive Reflection Test - Examples of Reacting vs Checking

Google acquired YouTube for US$1.6bn in 2006. In 2022, YouTube could be worth over US$180bn achieving a +110x return. Analysis of YouTube revenue and usage statistics and financial metrics.

yt valuation comparison

How Much is YouTube Worth? Over US$180bn in 2022

The IKEA effect describes how people come to overvalue things in which they have successfully put effort into. This cognitive bias can pose risks to organisations in the form of sunk cost effects and “not invented here” syndrome.

![ikea-effect](//images.ctfassets.net/vwq10xzbe6iz/1o8egJQ2qvuZqLQjsPpOSf/5c7e44c7d77fc37f809a05fe47b6d357/ikea-effect.png)

This article explains when and why the IKEA effect emerges and introduces four ways you can defend against it.
1. [What is the IKEA effect?](#what-is-the-ikea-effect)
2. [Pitfalls of the IKEA effect](#pitfalls-of-the-ikea-effect)
3. [Four ways to avoid the IKEA effect](#ways-to-avoid-the-ikea-effect)

### <a name="what-is-the-ikea-effect"></a> 1. What is the IKEA effect?
The IKEA effect is a finding that people tend to ascribe a higher valuation towards self-assembled products compared to objectively similar products which they did not assemble.

A study by Norton, Mochon and Ariely ([2011 paper](https://www.hbs.edu/ris/Publication%20Files/11-091.pdf)) found participants who self-assembled an IKEA storage box (Builders) valued the purchase price 63% higher than those who inspected the same pre-assembled box (Non-Builders). Similarly, the Builders also rated liking the box 55% more than Non-Builders. 

__IKEA Effect Experiment A - Perceived value of IKEA storage box by Builders vs Non-Builders__

|   Measure   | Builders (Self-assemble)     | Non-Builders (Inspect pre-assembled)     | % Difference     |
| ---------- | ---------- | ---------- | ---------- |
| Willingness To Pay (0-100 cents scale)       | 78       | 48       | 63%       |
| Liking of Box (1-7 scale)       | 3.8       | 2.5       | 52%       |

*Participants: N = 50*

In another experiment one group of participants were asked to create origami (Builders) and another asked to inspect their creation (Non-Builders). Similar to the first experiment, the Builders valued their creation 360% higher than the Non-Builders who ascribed close to no value to the origami (“nearly worthless crumpled paper”).

__IKEA Effect Experiment B - Perceived value of origami creation by Builders vs Non-Builders__

|  Measure    | Builders (Self-create)     | Non-Builders (Inspect creation)     | % Difference     |
| ---------- | ---------- | ---------- | ---------- |
| Willingness To Pay (0-100 cents scale)       | 23       | 5       | 360%   
*Participants: N = 106*

### <a name="pitfalls-of-the-ikea-effect"></a> 2. Pitfalls of the IKEA effect
The IKEA effect explains broadly why people may come to fall in love with their own ideas. With the effect strongest when people expend a greater amount of their own effort and perceive their effort as successful.

The IKEA effect has negative implications for organisations more broadly as a contributor to two key organisational pitfalls:
- __Sunk cost effects__: people continue to devote resources to failing projects in which they have previously invested
- __“Not invented here” syndrome__: people refuse to use perfectly good ideas developed elsewhere in favour of their own, often inferior internally-developed ideas

Here are some examples of how the pitfalls could occur:

| Team     | Effort     | Pitfall ⚠️     |
| ---------- | ---------- | ---------- |
| Fund Manager       | Creating a company financial model with industry and market share forecasts       | Perceiving company to be more valuable than it may be even when forecasts are not met       |
| Sales Executive       | Developing a sales slide deck on a unique selling proposition       | Selling the solution instead of listening for customer struggles       |
| Product Designer       | Building high fidelity prototype of MVP product       | Over Indexing visual design feedback and distracting from focus on user goals       |
| Growth Marketer       | Developing Account Based Marketing strategy based on LinkedIn research       | Avoid comparing cost of acquisition vs other more effective initiatives like SEO and direct prospecting       |

### <a name="ways-to-avoid-the-ikea-effect"></a> 3. Four ways to avoid the IKEA effect
Here are four ways you can avoid and defend against the IKEA effect.

#### 1. Acknowledge your unconscious bias
In the IKEA effect study, participants were found to be unaware of their unconscious bias. 92% of the participants when asked said they would pay more for pre-assembled products compared to self-assembly. This was in direct contradiction of their actions in the study. 

Acknowledging you are prone to the cognitive bias of the IKEA effect can be the most effective way to mitigate against it. When completing an effort on a solution or idea, admit to yourself that you probably love it more than other solutions, because you worked on it.

#### 2. Spike rough prototypes
The IKEA effect study found that the effect was strongest only when the effort was successful. Failure to complete tasks had corresponding negative psychological consequences. People ruminate more on tasks they failed to complete than on those they successfully completed, leading to negative affect and regret.

Spike rough prototypes and MVPs to test ideas early. Embrace sharing ideas that you are embarrassed with so it will be easier to cull.

#### 3. Talk to customers
Framing your solutions and efforts towards the customer struggle can help mitigate your bias against any one solution.

Talking to customers is the best way to achieve this. Learn how to conduct user interviews to surface customer's struggling moments and understand why they don't love their current solutions ([example user interview questions](https://mannhowie.com/user-interviews)).  

#### 4. Run growth experiments
Growth experiments can help mitigate against [confirmation bias](https://mannhowie.com/cognitive-bias#confirmation) by setting the discipline to state the problem statement, hypothesis and success criteria upfront before expending effort on a solution. Learn how to run your own growth experiment in this tutorial ([How to run a growth experiment](https://mannhowie.com/growth-experiment)).

The Build, Measure and Learn framework popularised by Eric Ries in his book [The Lean Startup](http://theleanstartup.com/book) explains how to apply a learning and feedback loop to measure the success of products and solutions.

__Sources__
- Norton, M. I., Mochon, D., & Ariely, D. (2012). [The IKEA effect: When labor leads to love.](https://www.hbs.edu/ris/Publication%20Files/11-091.pdf)
- Ries, Eric. [The Lean Startup. (2011)](http://theleanstartup.com/book)


When and why the IKEA effect causes people to overvalue their own ideas. Examples of organisational pitfalls of the IKEA effect. 4 ways you can defend against the IKEA effect: 1) Acknowledge your unconscious bias; 2) Spike rough prototypes; 3) Talk to customers; 4) Run growth experiments

ikea-effect

The IKEA Effect - Why people fall in love with their own ideas

The average is the most commonly used metric we use to understand data and infer insights - it is also the most dangerous.
![beware_average](//images.ctfassets.net/vwq10xzbe6iz/12kJw6jBqfvjjSCHyMxUNm/2dbf7edf379236641095dffe09c670fb/beware_average.png)

# The Pitfall of the Average
The US Air Force began life in the late 1940s with a high rate of aircraft accidents and pilot fatalities (excluding combat losses) with 23.6 aircraft destroyed per 100,000 flying hours. This dramatically reduced to 4.3 by the end of the 1960s driven largely by the work of one man, Lieutenant Gilbert S. Daniels who saw the pitfall of using “the average” to design solutions.

> The tendency to think in terms of the “average man” is a pitfall…it is virtually impossible to find an “average man” in the Air Force population.
[Gilbert S Daniels (1952 US Air Force Report)](https://apps.dtic.mil/sti/citations/AD0010203)

US Army engineers had standardised the design of pilot cockpits to conform to the average dimensions of a 1926 pilot. This was revisited in the 1950s by Lt. Daniels who measured the bodily dimensions of more than 4,000 pilots and found that zero pilots matched the average dimensions.

The US Air Force responded and demanded that its manufacturers design its cockpits to be adjustable to individual dimensions vs the standardised average. As a consequence, both pilot performance and accident/ mishap rates dramatically improved.
![us_airforce](//images.ctfassets.net/vwq10xzbe6iz/6mtEbxaIXNU60Svdu9uGsO/3c61d041d241f4040e7269db2552b8eb/us_airforce.png)

# A Modern Example
A modern example of the pitfall of the average would be to look at user activity. Imagine we build software that helps users perform inspections and we are tasked with improving user engagement.

Looking at the data we find that the average count of monthly inspections performed by our active users is 32 inspections per month:

![table_inspections](//images.ctfassets.net/vwq10xzbe6iz/3SfYJ8CwfGsBPFptTVL3jB/44bd12fd8ec35a7fc0c8efe608a47ed3/table_inspections.png)

Knowing this, we can deduce that our users on average perform 32 inspections per month or approximately one inspection a day. We begin building solutions such as daily notifications to help remind our users to perform their daily inspection.

__Stop.__

Reducing a large data set into a single number can often mask useful information and give us the wrong answer. Instead we can look at the distribution of our data and graph the same data as a histogram to reveal how often each different value occurs in a dataset.
![histogram_inspections](//images.ctfassets.net/vwq10xzbe6iz/1UixJwtgp0xcrWIvV6I6J2/ce429dc02a478f3c49abb41f28b113bc/histogram_inspections.png)

Now looking at the distribution of our data we can see that none of our users perform the average number of 32 inspections per month. Instead we see that 60% of our users perform 10 or fewer inspections per month and a large group (+30% of users) perform 70 or more inspections per month.

The distribution of our data could suggest that the usage of our software differs widely by different cohorts of users. One cohort could be using it to perform monthly scheduled safety checks vs another cohort using it to perform factory line QA checks. 

Next time when using the average, pause before making any conclusions and ask: "How many instances of the data actually reflect the average?". 

__References:__
- [THE “AVERAGE MAN”? - Gilbert S Daniels (1952 US Air Force Report)](https://apps.dtic.mil/sti/citations/AD0010203)
- [Trends in U.S. Air Force Aircraft Mishap Rates (1950-2018) - Thomas Light, Thomas Hamilton, Spencer Pfeifer](https://www.rand.org/content/dam/rand/pubs/research_reports/RRA200/RRA257-1/RAND_RRA257-1.pdf) 
- [The End of Average - Todd Rose (2015)](https://www.penguin.com.au/books/the-end-of-average-9780141980034) 


The average is a myth. Understand the pitfalls of the average and how it can conceal useful information. How the US Air Force threw out the average and saved lives. Use distributions to better understand your data.

beware_average

How the US Air Force Ditched the "Average" and Saved Lives

Product, marketing and sales teams can learn to run growth experiments to validate new strategies in the real world before incurring significant resource investment.
![guess_vs_experiment](//images.ctfassets.net/vwq10xzbe6iz/40uy9hD2VDsloevHEODX5m/7231e7f18856fa8c00e0f9eaef7347ec/guess_vs_experiment.png)

[Confirmation bias](https://mannhowie.com/cognitive-bias#confirmation) is a psychological tendency where we seek information that validates our existing beliefs as well as avoiding contradictory information. This bias often causes teams to waste resources on initiatives that validate existing views but are ineffective in the real world.

This article explains the 4 key steps of a growth experiment and a practical example of how you can run your own growth experiment quickly and cost effectively:
1. [Hypothesis](#hypothesis)
2. [Trigger](#trigger)
3. [Action](#action)
4. [Measure](#measure)

Imagine we are in the product marketing team of a B2B software startup that provides a tool for capturing customer complaints called Capturo.

## <a name="hypothesis"></a>1. Hypothesis
Set a hypothesis early. Start by identifying a customer struggle and business outcome of what you are trying to achieve. Form your own explanation as to what is causing the customer struggle and come up with a solution as to what could ease that struggle. [User interviews](https://mannhowie.com/user-interviews) are an effective tool at this stage if you have the time.

It is important at this stage to avoid the pitfall of falling in love with your explanation and solution. Focusing on the struggle will allow you to run a new growth experiment if the current one fails.

### __Example Hypothesis__
__Struggle:__ Our new users are observed to create a lower number of customer complaint forms on the platform than our existing users which is leading to lower product engagement and complaints captured.

__Explanation:__ We suspect two core reasons: 1) New users are unfamiliar with the types of questions to ask to gather complaints; 2) New users lack the time or know-how to digitise their existing questions into our platform.

__Solution:__ Our proposed solution will address the first explanation of new users not knowing what questions to ask. We will research and propose the most frequently asked questions to address customer complaints by service type (e.g. contact centre, restaurant, consultation).

__Hypothesis:__
> “Teaching new users the most commonly asked customer complaint questions relevant to their business will help them create more forms on the platform and gather more customer complaints”

## <a name="trigger"></a>2. Trigger
Establish a trigger point to communicate with your customer around their struggle and your proposed solution. The most effective mediums are email and calling customers directly. In-app banners, notifications and user flows are expensive if you do not already have the infrastructure in place. As this is a growth experiment we do not need to worry about solutions that don’t scale yet. 

The three most important components of a trigger:
1. __Audience:__ identify your target audience. Start with bigger groups, segmenting at this stage will be time consuming and take longer to gather results
2. __Message:__ keep consistent with your original hypothesis. There is no need to A/B test on a small scale but avoid message variation overkill
3. __Timing:__ identify the best time to contact your users to get their attention. Be sparing for when you trigger again due to lack of response

### __Example Trigger__
__Audience:__ our growth experiment will be to email new users who have created two forms or less in the platform from their first three months of joining

__Message:__ “Real examples of customer survey questions [Dental Practices] are using in Capturo” or “Top 6 customer survey questions [Dental Practices] ask in Capturo”

__Timing:__ Send when admin users are most active on the platform (for example mid-week mornings)

## <a name="action"></a>3. Action
Establish an action that you can measure based on the trigger. This can include opening a link to an article, registering interest via a form, logging in to the platform to perform an activity, emailing or calling back to respond.

Without creating an action it will be difficult to measure the success of your growth experiment.

### __Example Action__
Create a webpage link in the email which takes a user to the top customer survey questions.
E.g email link.
“View top 6 customer survey question examples”

## <a name="measure"></a>4. Measure
Measure and track the performance of the trigger and action stages. Start simple and use a spreadsheet to track key results. Free or cheap email tracking plug-ins are an effective way of tracking email open and click-through rates.

Once you feel you have gathered enough insight (can take a few weeks) assess the results and determine if your hypothesis was correct. Within the same experiment you can go back and modify the trigger and action to gather more data if needed.

### Example Measure
Create a spreadsheet with columns to track the following:
User contact email and id
- Email message sent
- Date and time sent
- Date and time opened email (use email tracking tools)
- Date and time opened link (use email tracking tools)
- Already sent (Y/N)
- Created a new template (Y/N) (check manually each week)

After measuring the performance assess engagement metrics for example:
- Total users contacted
- Average contact count per user
- % email open rate
- % click through rate
- % users creating new template

__Happy case__
If the % click through rate and % users creating a new template is good we have quickly validated that knowing the right questions to ask can help boost form creation. We now have an improved business case for investing more resources in-app to build onboarding tours in the form creation stage.

__Sad case__
If the % open rate is poor after adjusting for message and timing then the solution of our hypothesis is likely incorrect. We can go back to our original hypothesis and form a solution for our second explanation (e.g. already know what questions to ask but don’t have the time to create a new form). 


4 steps to a successful growth experiment: 1) Hypothesis; 2) Trigger; 3) Action; 4) Measure. Practical example for how teams can run growth experiments quickly and cheaply.

guess_vs_experiment

How to Run a Growth Experiment - Learn the 4 Steps

Here are three core formatting best practices you can apply for financial models and data analyses in Excel and Google Sheets.

[Formatting cheat sheet with examples (free download)](https://docs.google.com/spreadsheets/d/1lPK9rY2XF0C5kbo1EtUGiCEvozOfaUPxt9zilfthTZc/edit#gid=0)

## 1. Color Coding Cells
- <span style="color:black;border:solid 1px;border-width:1px;padding:0px 2px">Black</span> = formula, calculations and links to other cells
- <span style="color:blue;border:solid 1px;border-width:1px;padding: 0px 2px">Blue</span> = hard coded input cells and historical data
- <span style="color:blue;border:solid 1px;border-width:1px;padding: 0px 2px;background:yellow">Blue with yellow background</span> = key input assumptions
- <span style="color:green;border:solid 1px;border-width:1px;padding: 0px 2px">Green</span> = link to other spreadsheets in the same file

__Example - Big Tech Company Metrics__
![COLORCODE](//images.ctfassets.net/vwq10xzbe6iz/1B5GvkyNo7NBDedwEllnDM/1a7fa72588abd089f5d470992a41baef/COLORCODE.png)

## 2. Financial Tables
- __FY20XX__ = Use financial year end for your column headers, include the year end and metric in the table header
- __Revenue__ = bold important headline metrics (Revenue, EBITDA, EBIT, NPAT)
- Cost of Revenue = indent sub metrics used to calculate headline metrics (e.g. COGS)
- Financial formatting = use brackets for negative values and avoid more than 1 dp
- Left to right = older to newer FY dates ordered left to right

__Example - Alphabet Income Statement__
![TABLE](//images.ctfassets.net/vwq10xzbe6iz/1HaM9TUGNg0UkczOVAIDgl/98d7bb87c0cda476c90562f9a85056f5/TABLE.png)

### 3. Key Metrics
- __FY20XX__ = Align your key metrics under the same columns as the financial table
- *Revenue growth yoy* = italic text for key metrics
- *23%* = italic text including the unit metric (e.g. %, x)
- <span style="color:gray;border:solid 1px;border-width:1px;padding: 0px 2px;background:gray">Grey</span> = grey out cells if computed metric not able to be calculated

__Example - Alphabet Income Statement Key Metrics__
![METRIC](//images.ctfassets.net/vwq10xzbe6iz/jKJYZgcCeyTjKK6kjKaR8/e98f783e10d0b4fb8fc2326d4de988ce/METRIC.png)

[Formatting cheat sheet with examples (free download)](https://docs.google.com/spreadsheets/d/1lPK9rY2XF0C5kbo1EtUGiCEvozOfaUPxt9zilfthTZc/edit#gid=0)

Investment Banking Excel and Google Sheet formatting best practices for building spreadsheets and financial models: 1) Color Coding Cells; 2) Financial Tables; 3) Key Metrics

COLORCODE

Financial Model Formatting - 3 Core Best Practices

Why do most startups fail to find product market fit? 
![beware_solutions_seeking_problems](//images.ctfassets.net/vwq10xzbe6iz/2rGuWsynYvglJNeJeWcQoz/b11da185061ef24f957efe1f9683d4f8/beware_solutions_seeking_problems.png)

Because our [confirmation bias](https://mannhowie.com/cognitive-bias) creates tunnel vision towards our existing solutions. Instead we should frame our focus on solving the user’s struggle.

Here are 5 questions you can ask in user interviews to validate if your startup has found product market fit.

Imagine we are a startup building solutions for mortgage brokers.

## 1. What progress are you trying to make in X?
__*Example:*__
__Q.__ What progress are you trying to make in finalising loan applications?

__A.__ I’m looking for my team to close more loans and shorten the bank approval times.

## 2. What is your biggest struggle when trying to make progress in X?
__*Example:*__
__Q.__ What is your biggest struggle in getting bank approvals completed on time?

__A.__ Document gathering with our customers. A single important lost or incomplete document can slow the whole process down and cause frustration amongst both lenders and borrowers, both of whom are our customers.

## 3. Tell me about the last time you encountered this struggle?
*__Example__*
__Q.__ Tell me about the last time you encountered this struggle? (create a timeline of the event as if you were in their shoes)

__A.__ Last month one of our new starters had a backlog of 10 applications with one of our newer lenders. An applicant customer did not submit a key piece of information that the new lender required but this only came to light a couple of weeks after the applicant had already submitted all the documents we asked. The lender was frustrated that we missed the information (as they said they had asked us for this before) and the applicant was livid as they had already terminated their rental lease.

## 4. What solutions have you previously tried?
__Example:__
__Q.__ What solutions have you previously tried to avoid or manage this struggle?

__A.__ We’ve used shared cloud storage solutions like Google Drive and Dropbox, onboarding checklists to make sure we don’t miss things and our CRM tool to keep track of emails.

## 5. What didn’t you love about those solutions?
__Example:__
__Q.__ What don’t you love about those solutions you’ve previously tried?

__A.__ Some of the tools create more work for our team in managing the document workflow and version control, we still have to review all the documents to make sure they are complete. As a result we still rely heavily on email to follow-up and confirm next steps with both lenders and customers but then it’s easy for us to miss things and drown in emails.

## Bad User Interview Questions
User interviews to find product market fit should be framed around the user’s struggle and not solution ideas. 

Bad user interview questions include:
- What ideal solution would you like to solve your struggle? ❌
- Do you have any feedback on our solution? ❌
- Would you use this solution if we offered it? ❌
- What do you think we could do to improve your process? ❌
- Do you know anyone else in your company who would use our solution? ❌

## User Interview Insights
Following the insights from our 5 user interview questions we are now better able to frame the struggle for our mortgage broker manager as:

> “Communicating between lenders and customers the missing information needed to complete a loan application”

Solutions we can now explore to solve the problem could include:
- Assignees for onboarding docs (whose court is it in now) 💡
- Automatic email reminders of incomplete docs 💡
- Auto-logging emails to capture a digital chat history 💡
- Identify which docs have not been updated in a while 💡

Framing the challenge around the user struggle allows us to continue iterating on solutions to help users make progress instead of being fully invested into one solution. 

⚠️ Beware solutions seeking problems.


5 user interview questions to validate product market fit: 1) What progress are you trying to make?; 2) What is the hardest thing about making progress?; 3) Tell me about the last time you encountered the struggle; 4) What solutions have you previously tried? 5) What didn’t you love about those solutions?

beware_solutions_seeking_problems

5 Questions to Ask to Find Product Market Fit - User Interviews

Inversion is a mental model of framing a problem from an opposing objective in order to solve the original problem.
![inversion](//images.ctfassets.net/vwq10xzbe6iz/61qV3DFcVLXMoW1R899hWB/db5a16bc87fc56ff29624d857bbed49f/invert_image.png)
It is a favourite trick of famous value investor Charlie Munger. He gives an [example of how the inversion process](https://youtu.be/_uacTl8Lv_U) helped him in his early career become a better meteorologist:

> “When I was a meteorologist in World War II, they told me how to draw weather maps and predict the weather. What I was actually doing was clearing pilots to take flights. 
> 
> I just reversed the problem. I inverted, __I said:__ __suppose I want to kill a lot of pilots. What would be the easy way to do it__? And I soon concluded that the only easy way to do it was to get the planes into icing conditions they couldn’t handle or to get the pilot into a place where he’d run out of fuel before he could safely land. 
> 
> So I made up my mind I was going to stay miles away from killing pilots by either icing or getting them sucked into conditions where they couldn’t land. I think that helped me be a better meteorologist in World War II. I just reversed the problem.
> 
> Algebra works the same way. Human beings should do the same thing in their ordinary walks of life. Just constantly invert. You don’t think about what you want. You think about what you want to avoid. When you think about what you want to avoid, you also think about what you want. And you just go back and forth all the time.”

This article looks at how others have used the inversion process to innovate big ideas and new solutions in both life and growing start-ups.

## Productivity
In solving the problem of decluttering one’s life, Marie Kondo frames the challenge of effective tidying __not as what we want to get rid of__, but instead __choosing what we want to keep__.

The big idea of the [KonMari Method](https://konmari.com/what-is-konmari-method/) is to keep things that spark joy: “Does this spark joy? If it does, keep it. If not, dispose of it.” 

By reframing the problem she was able to develop her tidying techniques as “involving only two essential actions: discarding and deciding where to store things.”

## Product Innovation
In solving the problem of product innovation, Bob Moesta and the late Clay Christenson frame the question __not as what new features do customers want__, but instead understand __what struggles they experience with their current solution__. 

The big idea behind their innovative [Jobs To Be Done (JTBD) framework](http://jobstobedone.org/) is understanding the struggles a customer faces in making progress in their lives and innovating around those struggles.

By reframing the problem in understanding the struggle, Bob Moesta innovated his JTBD framework to look at “four forces of progress” where progress only occurs when:

> (__Push__ of the current struggle + __Pull__ of the new solution) > (__Habit__ of the old solution + __Anxiety__ of the new solution).

## Project Management
In solving the problem of delivering meaningful work on time and on spec, Ryan Singer from Basecamp frames the challenge __not as how long will it take to execute__, but instead __how much time do we want to spend__. 

The big idea of [Shape Up](https://basecamp.com/shapeup) is to focus less on estimates and more on appetite in order to narrow down problems and design outlines of solutions that fit within constraints of appetite.

By reframing the problem based on appetite, Basecamp’s Shape Up techniques focus on setting bounded solutions using fat marker sketches and addressing risks and rabbit holes early on. All the techniques are designed to set guardrails based on appetite to de-risk projects from derailing or growing in scope.


How Charlie Munger used the inversion process to save the lives of pilots during WWII. How inversion helps frame innovative solutions in productivity, product design, and project management.

inversion

Invert, Always Invert - Find out how to kill pilots

Alibaba operates three of the largest digital platforms in China: e-commerce, cloud and payments. We estimate Alibaba Group (BABA:NYSE 9988:HKG) to be valued at up to US$900 billion based on a Sum Of The Parts (SOTP) valuation of its various operating segments:

#### Alibaba Group SOTP Valuation (2021)
![alibaba sotp](//images.ctfassets.net/vwq10xzbe6iz/5JX9uAgNOkj85xN3eJWyA8/16003aacd7bc9972fb8da8e3a55152a6/alibaba_sotp.png)

This article analyses Alibaba’s two main operating segments that represent up to 88% of the group valuation (Taobao/ Tmall and Alibaba Cloud) and shows how you can formulate your own valuation and thesis. To learn the fundamentals of valuing any technology company refer to [this article with a case study of how to value YouTube](https://mannhowie.com/youtube-valuation). 

__Key Sections:__
1. [China E-Commerce Market Share](#industry)
2. [Taobao/ Tmall Valuation](#taobao)
3. [Alibaba Cloud Valuation](#cloud)

## <a name="industry"></a>1. China E-Commerce Market Share
China dominates global online spending. Over 57% of total global online spend happens in China with an estimated +1 billion online shoppers spending over US$2.8 trillion annually. Online spending in China alone is more than 70% more than the next 10 countries combined.

#### Global E-Commerce Spend by Country (2021 Forecast) & China E-Commerce Market Share by Company (2020 Actual)
![global ecommerce](//images.ctfassets.net/vwq10xzbe6iz/5RpP7NWgAWsBH0A4nt048d/f2e7b2c21f59148f2c90ad8d99d618e0/global_ecommerce.png)

China is likely to continue to dominate global online spending the next decade and beyond. Relative to the United States, China has 5x the online shopping population and is still only at 20% of US GDP and Income per capita. Across e-commerce there are two economies: China and the Rest of the World.

Within China, e-commerce is dominated by three players: Alibaba, JD.com and Pinduoduo. Alibaba’s annual Gross Merchandise Value (GMV) across its two platforms Taobao and Tmall in China alone is more than all annual online spend across the US, UK and Japan combined. 

#### China E-Commerce Major Players
![china ecommerce table](//images.ctfassets.net/vwq10xzbe6iz/2vOtxucZPBnNJvlZH9KklU/c8ed8fe3cda225faf94169a6df37080a/china_ecommerce_table.png)
Across the three major Chinese players Alibaba maintains its dominant position. However newcomer pinduoduo has grown considerable share in a very short period focusing on a new trend in e-commerce in China, lower basket value social commerce.

## <a name="taobao"></a>2. Taobao / Tmall Valuation
Alibaba’s largest and most profitable segment is its e-commerce marketplace platform in China known as Taobao and Tmall. We value this segment at RMB 4 trillion (US$ 624 billion) representing 80% of the entire valuation of Alibaba.

### What is Taobao and Tmall?
With over 811 million active annual customers, Taobao is effectively the internet platform for e-commerce in China. Nearly all merchants and businesses that sell to Chinese consumers have an online store on Taobao. 

There are two main platforms within the one Taobao mobile and web app: Taobao and Tmall with a total GMV split 50/50:

- __Taobao__ is a marketplace platform where any merchant, person or small business can set up an online store and sell to the Chinese population.

- __Tmall__ is a B2C platform reserved for brand companies where they can set up their own dedicated stores. Most international brands selling into China will transact through their Tmall stores vs their own online shops.

#### Taobao vs Tmall
![taobao vs tmall](//images.ctfassets.net/vwq10xzbe6iz/4QhVDL6MlaqeWzoag1v54z/9ee6f6d521faa305e273bfd0f4407907/taobao_vs_tmall.png)

Alibaba’s e-commerce business model is marketplace-based and facilitates transactions between buyers and sellers. Unlike Amazon and JD.com, Alibaba does not purchase and resell inventory nor does it own or operate its own fulfilment and distribution centres. It is the sellers responsibility for organising delivery to the buyer either direct or via Alibaba’s network of 3rd party logistics providers (via its global Cainiao logistics business).

### How does Taobao Tmall generate revenue?
Ads and commissions:

- __Ads (70% of e-commerce revenue)__: merchants buy keywords and display ads on Taobao and Tmall to attract traffic and purchases to their stores. Ad monetization represents 2-3% of GMV transacted through the platform. Alibaba’s high margin ad revenue model is similar to Google’s cash cow Search business.

- __Commissions (30% of e-commerce revenue)__: merchants pay commissions on transactions through the Tmall network as part of hosting their stores on the largest e-commerce platform in the world. Commission monetization represents ~1% of GMV.

### What is the growth thesis for Alibaba China E-Commerce?
The e-commerce market in China is now highly mature with over 50% of retail sales done online and Alibaba maintaining the dominant share. Having been the dominant e-commerce platform in China for the last decade we expect overall growth to slow. 

Having said that, we still expect Alibaba to remain the dominant e-commerce platform in China and expect Alibaba’s annual GMV to grow another 50% over the next decade and ad monetization to increase from ~3% to +3.5%.

We can represent these assumptions as key operating metrics to arrive at a forecast annual revenue by the end of 2030 of RMB 521bn (US$82 bn), up from ~RMB 300bn (US$47bn) ending 2020.

#### Alibaba China Retail Marketplace Revenue Model (RMB)
![alibaba retail revenue model](//images.ctfassets.net/vwq10xzbe6iz/7AgIVJxBlV9K6uenKzyOX9/af5ab81611f9c3a6cba49f4a7b23d585/alibaba_retail_revenue_model.png)

### China Retail Marketplace DCF Valuation
We arrive at our RMB 4 trillion ($USD 624 bn) valuation for Alibaba’s China e-commerce business applying the following key assumptions:
- __EBITA margins__: remain stable at 72% given the high margin operating model of its ad ad and commission businesses. We expect Alibaba’s other segments to operate at break-even largely to support its high margin e-commerce segment (e.g. Cainiao logistics) 
- __Tax rate__: long term effective tax rate of 20% (China based)
- __Stock Based Compensation__: while non-cash we treat this as an expense as it’s largely a substitute for compensating staff. We expect SBC to remain at 25% of total revenue going forward to attract key talent
- __Reinvestment (Net Capex)__: assume a capital-light sales to capital ratio of 1.5 (in line with Alibaba historical and compares to 1.9 for Alphabet) meaning for every one dollar of incremental annual revenue growth, Alibaba will reinvest 66% back into the business for future growth
- __Discount rate__: assume 8% discount rate, low by historical standards but higher now given subdued global interest rates
- __Long term growth rate__: 3.5% above global GDP long term GDP growth of 2-3% as China is still expected to grow faster than the rest of the world for the next decade

#### Alibaba China Retail Marketplace DCF Model (RMB millions)
![alibaba retail dcf](//images.ctfassets.net/vwq10xzbe6iz/3em6HkDrD1MyZwUDTEdslh/e15b1815727f85c2aadc60bb3221838d/alibaba_retail_dcf.png)

## <a name="cloud"></a>3. Alibaba Cloud Valuation
Alibaba’s second most valuable operating segment is its cloud computing business Alibaba Cloud. We conservatively value this segment at RMB 390 billion (~US$60 billion) on the basis it continues to dominate the Chinese market but will face stiff competition globally particularly given political concerns in western economies around hosting of data by Chinese domiciled companies.

### What is Alibaba Cloud?
Alibaba Cloud is the 4th largest cloud computing provider globally and the largest in the Asia Pacific with +40% share in China. It sells cloud-based compute and storage infrastructure services to businesses looking to host their applications and data primarily in China (although it operates globally).

#### Global Public Cloud Infrastructure Services Spend 2020

![global cloud spend](//images.ctfassets.net/vwq10xzbe6iz/77Vqh2ZPUOpbpsZ2EmAFer/5f135f4d953b45be2e4716c4d5588999/global_cloud_spend.png)

Alibaba Cloud is similar to Amazon Web Services offering similar competing offers often at even lower prices when hosted in China vs AWS. It has a clear advantage over the other hyperscale cloud providers within China given its significant existing presence in the Chinese market and wider local market support.

### Alibaba Cloud DCF Valuation
Our RMB 390 billion (US$ 60 billion) DCF valuation for Alibaba Cloud is based on the following key assumptions:
- __Revenue__: Achieve revenues similar to that of current AWS levels by 2030, largely driven by growth in the local Chinese and Asian market. Alibaba is following in the similar footsteps as AWS (~7 years behind) as both the local market leader and without competition from global hyperscale providers. However growth is expected to slow as we expect international growth to be more difficult
- __EBITA Margin__: Achieve similar EBITA margins of +30% as AWS as it reaches scale and continues to reinvest for growth
- __Tax and SBC__: Effective tax rate of 20% and lower stock based compensation expense as % of revenues vs Alibaba China Retail as lower proportion of costs represented by people
- __Reinvestment Rate__: Sales to capital ratio of 1.5 continuing to reinvest 66% of net revenue growth back into the business each year
- __Discount and growth rate__: 8% discount rate and 3.5% LT growth rate in line with Alibaba China Retail

#### Alibaba Cloud DCF Valuation (RMB millions)
![alibaba cloud dcf](//images.ctfassets.net/vwq10xzbe6iz/1yolWWSzSGMD3dC1BOLU7j/d8fa2dedd86c7bde6d2fa70393c5700c/alibaba_cloud_dcf.png)

Alibaba operates three of the largest digital platforms in China: e-commerce, cloud and payments. We estimate Alibaba (BABA:NYSE 9988:HKG) to be valued at up to US$900 billion based on a Sum Of The Parts (SOTP) valuation of its various operating segments.

global ecommerce

How Much is Alibaba Worth? US$900 billion in 2021

A Director is a person generally appointed by a company’s shareholders to attend, participate and vote in Board meetings and oversee [the role and function of a Board](https://mannhowie.com/board-role).

## 5 Basic Fiduciary Duties of a Director
> A Director has a ‘fiduciary relationship’ with a company which is a duty of fidelity and trust to always act in the best interests of the company as a whole and not oneself or a third party. 

In performing their role, a Director is subject to five key duties and obligations under corporation law, and the company’s constitution:

1. ✅Good Faith & Best Interests
2. ✅Act for Proper Purpose
3. ✅Avoid Conflicts of Interest
4. ✅No Misuse of Position or Information
5. ✅Exercise Care & Diligence
  
## Acting with Loyalty and Good Faith
The first four fiduciary duties relate to acting with loyalty and good faith. Below is a summary with breach examples:

| 	Fiduciary Duty	 | 	Description	|	Breach Example	|
| ------------------ | --------------------------	| ------------------------------------ 	|
| __1. Good Faith & Best Interests__ | Decision making on behalf of the best interests of the company as a whole | ❌ Vote influenced by an appointee or major shareholder |
| __2. Act for Proper Purpose__ | Doing things for their intended purpose in the interests of the company as a whole rather than a collateral benefit | ❌ Issuing shares to dilute a shareholder vs raising proceeds for a valid purpose |
| __3. Avoid Conflicts of Interest__ | Must not place oneself in actual or potential possibility of conflict | ❌Not disclosing material interests in a takeover company or competitor |
| __4. No Misuse of Position or Information__ | Must not misuse position to gain an advantage for oneself or someone else or to cause detriment to the company | ❌Insider trading. Setting up a competing company. Phoenix activities |

## Exercising Care and Diligence
Simply acting with loyalty and good faith is not a sufficient liability defence. A Director must also act with skill, care and diligence in overseeing strategy, risk and management of a company. 

At a minimum a Director should exercise care and diligence across these four areas:
1. ✅Familiarity with business
2. ✅Read Board papers
3. ✅Make enquiries
4. ✅Board approvals

| 	Care & Diligence	 | 	Description	|	Breach Example	|
| ------------------ | --------------------------	| ------------------------------------ 	|
| __Familiar with business__ | Become and stay on top of the business and financials | ❌Not understanding how business operates. Insufficient accounting knowledge. |
| __Read Board papers__ | Request and read all Board papers in advance, carefully review Board minutes | ❌Relying on opinions of other Directors. Disputing minutes in hindsight. |
| __Make enquiries__ | Question management and seek external advice on issues that have significant potential to harm company | ❌Relying on management reports. Lack of reported risks |
| __Board approval__ | Important strategic decisions, contracts and announcements should be brought to Board  | ❌No risk appetite framework. Not reviewing material market announcements |

## Statutory Duties & Liabilities
In addition a Director faces statutory duties and liabilities dependent on the legal jurisdiction:

| Statutory Duty/ Liability | Breach Example |
| --------- | ---------- |
| __Insolvent Trading__ | ❌Incurring further debts of any kind if a company is unable to pay all its debts as and when they become due and payable (including employee pay) |
| __Continuous Disclosure__ | ❌For listed companies not immediately disclosing a material contract loss |  
| __Tax Laws__ | ❌Un-remitted employee withholding tax or retirement contributions | 
| __Competition & Consumer Law__ | ❌Cartel behaviour. Truth in advertising |
| __Work Health & Safety Law__ | ❌Breach of work health and safety regulation (face imprisonment for reckless breaches) |
| __Anti-Discrimination Law__ | ❌Discrimination in relation to age, disability, union, race, sex etc |
| __Environmental Law__ | ❌Federal and state environment protection legislation |

## What are the Consequences of Breach of Directors’ Duties and Laws?
#### __Criminal sanctions__  👮
Imprisonment dependent on the severity of the outcome and if directors found to have acted dishonestly or recklessly in failing to comply with duties. Not protected under insurance and indemnification

#### __Civil sanctions/ Disqualification__ ⚖️
Liable for civil penalties including fines, class action lawsuits and disqualification as a director for long periods of time 

#### __Commercial/ Brand damage__ 📰
Reputational damage can damage the company as a whole well beyond the penalties imposed on its directors 

## How can Directors Best Protect Themselves Against Liability?
A Director should at a minimum follow these seven steps:

- ✅Always act in __good faith__ and best interests of the company as a whole and for proper purpose
- ✅Avoid/ disclose __conflicts of interest__
- ✅Exercise __care and diligence__ by reading Board papers, making proper enquiries and staying up to date with the business and industry
- ✅Engage __external advice__ where required but always make independent assessments
- ✅Avoid __insolvent trading__
- ✅Be familiar with the __laws of your industry__
- ✅Obtain __insurance and indemnification__



Five basic fiduciary duties of a Board member with breach examples. What are the consequences of a breach of Directors' duties and laws? How can Directors best protect themselves against liability?

Howie Mann
Startup Strategy
Subscribe
Articles Search About Subscribe

Topics

Popular Articles

Articles > engineering (12)