Algorithm Analysis

1. Complexity of Algorithms

There are many ways to accomplish a given (same) task.

In computer programming, you can write code in many ways to solve the same problem, where some are efficient while others are not.

Example: Printing the elements in an array

void print1(int[] ar, int size)
{
  int i;
  for (i = 0; i < size; i++)
    System.out.println(ar[i]);
}

void print2(int[] ar, int size)
{
  int i, j;
  for (i = 0; i < size; i++) {
    for (j = 0; j < i; j++) 
    {}
    System.out.println(ar[j]);
  }
}

To find out which ones are more efficient, we can of course run the code to empirically test them.
But we can also theoretically/mathematically analyze the code without running the code.

The complexity of an algorithm refers to the amount of time and/or space it requires to execute.
The analysis of an algorithm is the process by which we find an estimate for its complexity.
The time needed to execute an algorithm is a function of the size of the input (n). Some typical n's are:
- A numeric value (e.g. n in n!)
- Number of elements in a list/array -- size of the list/array
The intuition is that, as the size of the input becomes larger, the algorithms would take more time -- because there are more elements to process.
Here, we are interested in the rate of growth (asymptotic growth) -- the rate of the increase in the running time with respect to the input size.
Asymptotic growth is categorized into several classes (complexity classes). For example, in the increasing order of , including:
- Constant (1) -- not literally 1 but some constant
- Logarithmic (log₂n, written as lgn)
- Linear (n)
- nlogn (= n * lgn)
- Quadratic (n²)
- Cubic (n³)
- Exponential (e.g. 2ⁿ, mⁿ where m is some constant)

Slower the growth (i.e., lower complexity order), the more efficient the algorithm is, or the algorithms "scales up" better -- Even when the data becomes large, the running time does not degrade so much.

n	T(n) = 1	T(n) = lgn	T(n) = n	T(n) = n²	T(n) = n³
1	1	1	1	1	1
10	1	3	10	100	1000
100	1
1000	1

2. Running Time Calculations

Preliminaries: Some known sums:
1. Arithmetic Series:
2. Geometric Series:
  Example:
  
  2ⁿ⁺¹ - 1
  1 + 2¹ + 2² + .. + 2ⁿ = ---------- = 2ⁿ⁺¹ - 1
  2 - 1
When we analyze an algorithm, we count how many times a particular (important) line is executed.
Example:
```
1. for i := 1 to n do
2.   for j := 1 to n do
3.     x := x + 1        // count this line
```
The number of times the line 3 is executed when the input size was n:

T(n) = n + n + ... + n = n * n = n^{2

|<-- n of them -->|}

More examples:

for i := 1 to 2n do
  x := x + 1        // count this line

T(n) =

for i := 1 to 2n do
  for j := 1 to n do
    x := x + 1        // count this line

T(n) =

for i := 1 to n do
  for j := 1 to i do
    x := x + 1        // count this line

T(n) =

3. Logarithmic Growth

Example 1:

1. i := n
2. while i >= 1 do
3.   begin 
4.   x := x + 1    // count this line
5.   i := i / 2    // i becomes half
6.   end

iteration value of i
(at the top of loop) number of times
line 4 is executed

1 n 1

2 n/2¹ ¹

3 n/2² ¹

k n/2^k-1 = 1 1

k+1 n/2^k = 0 0
(loop not executed)

We are interested in what k is (because that's the number of times the line 4 is executed). In other words,

T(n) = 1 + 1 + 1 + .. + 1 = (k * 1)
|<-- k of them--->|

To derive k, we look at the relation at the last iteration (kth):

So, T(n) = log₂n + 1 (where log₂n is alternatively written as lgn).

Example 2:

1. i := n
2. while i >= 1 do
3.   begin 
4.   for j := 1 to n do
5.     x := x + 1      // count this line
5.   i := i / 2        // i becomes half
6.   end

iteration	value of i (at the top of loop)	number of times line 4 is executed
1
2
3
k
k+1

So, T(n) = = (k * ___ )
|<---- k of them ---->|

We know from example 1 that k = lgn + 1. Therefore, we get

T(n) = (k * ____) =

4. Big-Oh (O) Notation

T(n) is often expressed as a function that has several terms. Each term also may have constant coefficients. For example,

T(n) = 60n² + 5n + 1
But we are mostly interested in an approximation -- the order of the function (or order of growth). So we only look at the dominant term (60n² in the above) and drop its coefficient (60) to obtain the complexity/growth rate (n²).

There are three notations for expressing asymptotic growth:
1. Big-Oh -- e.g. O(n²) -- for upper bound
2. Omega -- e.g. W(n²) -- for lower bound
3. Theta -- e.g. Q(n²) -- for tight bound

In this course, we only cover the Big-Oh. For the example above, we have:

T(n) = 60n² + 5n + 1 = O(n²)

As a note, the formal definition of Big-Oh is as follows.
A function f(n) is of order at most g(n), noted f(n) = O(g(n)), if there exists c, n₀> 0 such that for all n >= n₀, we have that f(n) <= c*g(n).
Example:
60n² + 5n + 1 = O(n²) ... NOTE: f(n) = 60n² + 5n + 1, and g(n) = n²

Proof:

EXERCISES: For each of the algorithm segments, derive the exact number of times a particular line is executed, and state the complexity of the algorithm segment.

1. for i := 1 to 2n do
2.   x := x + 1          // count this line

Number of times line 2 is executed:

The complexity is O(n).

1. for i := 1 to n do
2.   for j := 1 to n do
3.      x := x + 1       // count this line

Number of times line 3 is executed:
The complexity is O(n²).

1. for i := 1 to 2n do
2.   for j := 1 to n do
3.      x := x + 1       // count this line

Number of times line 3 is executed:
The complexity is ____________.

1. for i := 1 to n do
2.   for j := 1 to 2n do
3.      x := x + 1       // count this line

Number of times line 3 is executed:
The complexity is ____________.

1. for i := 1 to n do
2.   for j := 1 to i do
3.      x := x + 1       // count this line

Number of times line 3 is executed:
The complexity is ____________.

1. for i := 1 to n do
2.   x := x + 1          // count this line
3. for j := 1 to n do
4.   for k := 1 to j do
5.     y := y + 1        // count this line too

Number of times the lines 2 and 5 are executed (total):
The complexity is ____________.

1. i := n
2. while i >= 1 do
3.   begin
4.   for j :=1 to n do
5.     x := x + 1        // count this line
6.   i := floor(i/2)
7.   end

Number of times line 5 is executed
The complexity is ____________.

5. Search algorithms

Linear search in an unsorted (or any) sequence

Problem: finding an element in an array of size n
Input: key, array A = [x₁, x₂,�, x_n], n
Output: the index of key in A, (0 if key is not in the list)

Program find_key( key, A, n )
1. index := 1
2. while index <= n
3.   begin
4.   if key = A[index] then   // count this comparison
5.     return index
6.   index := index + 1
7.   end
8. return 0
end find_key

Complexity

Worst-case	T(n) = _____ = O(___)
Best-case	T(n) = _____ = O(___)
Average-case	T(n) = _____ = O(___)

Binary Search -- for a sorted list (in the ascending order)
Example: Search for 16
- The algorithm:
Problem: find an element in a sorted list (whose index is 0 through n-1)
Input: key, L a list and n
Output: The index of key in L, -1 otherwise
```
procedure BinarySearch(key, L, n)
1.  low := 0 
2.  high := n-1
3.  while (low <= high) do
4.    begin
5.    mid = floor((low + high)/2)
6.    if (key = L[mid]) then        // count this comparison
7.      return mid
8.    else if (key < L[mid]) then 
9.      high := mid - 1
10.   else 
11.     low := mid + 1
12.   end
13. return -1    // key not found (fall through)
end BinarySearch
```
- Complexity
  1. Worst -case
    The worse case occurs when key is not present is the list. How many runs through the while loop will we do?
    - Initially, we have n element to consider.
    - At the end of the first loop, we are down to n/2.
    - After the second loop, we have n/4.
    - Next, n/8, etc.
    - In general, at the end of the kth loop, we have n/2^k elements left to consider.
    How far can we keep dividing (what is k?)? Again, according to the algorithm we exit the loop when low > high. This means, in particular, that when we are down to 1 element and process it, we stop the loop. So,
    
    So we get T(n) = lgn, thus O(lgn).
  2. Best-case
    The best case happens when key is present at the middle of the array, namely L[n/2]. In this case, the loop executes only once, regardless of the size of array (i.e., n). So, we get T(n) = 1, thus O(1).
  3. Average-case