415 lecture note #5 

[Ch 3] Algorithms (3)

3.5 Complexity of Algorithms

0. Overview

• The complexity of an algorithm refers to the amount of time and/or space it requires to execute.
• The analysis of an algorithm is the process by which we find an estimate for its complexity.

• The time needed to execute an algorithm is a function of the size of the input (n).  Some typical n's are:
  • A numeric value (e.g. n in n!)
  • Number of elements in a list/array -- size of the list/array
• Various cases for complexity
  1. Best-case -- minimum time required to process the input of size n
  2. Worst-case -- maximum time
  3. Average-case -- average time

  Example:

  Problem: finding an element in an array of size n
  Input: key, array A = [x₁, x₂,…, x_n], n
  Output: the index of key in A, (0 if key is not in the list)

  Program find_key( key, A, n )
  1. index := 1
  2. while index <= n
  3.   begin
  4.   if key = A[index]     // equality test
  5.     then return index   // key found, exit the program
  6.   index := index + 1    // increment the index by 1
  7.   end
  8. return 0
  end find_key

  How fast can the key be found?

  • In the best case, key is at A[1] and we only go through the loop once.
  • In the worse case, key is at A[n] (or not even in the list) and we must go through the loop n times.
  • On average, assuming that key can be anywhere in the list with equal probability, we would expect to go through the loop about n/2 times.

• So, the running time of depends most often on n.  As n gets bigger, the running time grows with various rate (asymptotic growth).

  Example:  times it takes to find the key in the algorithm above

  Best case	Worst case	Average case
  n times needed 1   10    100   1000	n times needed 1   10    100   1000	n times needed 1   10    100   1000

1. Number of times a line is executed

• When we analyze an algorithm, we count how many times a particular (important) line is executed.

  Example:

  1. for i := 1 to n do
  2.   for j := 1 to n do
  3.     x := x + 1        // count this line

  The number of times the line 3 is executed when the input size was n:

  T(n) = n + n + ... + n  = n * n = n^{2
                |<-- n of them -->|}

• More examples:

  1. for i := 1 to 2n do
       x := x + 1        // count this line

     T(n) =
     
      

  2. for i := 1 to 2n do
       for j := 1 to n do
         x := x + 1        // count this line

     T(n) =
     
      

  3. for i := 1 to n do
       for j := 1 to i do
         x := x + 1        // count this line

     T(n) =
     
      

2. Logarithmic Growth

• Example 1:

  1. i := n
  2. while i >= 1 do
  3.   begin 
  4.   x := x + 1          // count this line
  5.   i := i / 2          // i becomes half
  6.   end

We are interested in what k is (because that's the number of times the line 4 is executed).  In other words,

T(n) = 1 + 1 + 1 + .. + 1  = (k * 1)
          |<-- k of them--->|

To derive k, we look at the relation at the last iteration (kth):

So, T(n) = log₂n + 1   (where log₂n is alternatively written as lgn).

• Example 2:

  1. i := n
  2. while i >= 1 do
  3.   begin 
  4.   for j := 1 to n do
  5.     x := x + 1          // count this line
  5.   i := i / 2            // i becomes half
  6.   end

  iteration	value of  i (at the top of loop)	number of times line 4 is executed
  1
  2
  3
  k
  k+1

  So, T(n) =                                    = (k * ___ )
                   |<---- k of them ---->|

  We know from example 1 that k  = lgn + 1.  Therefore, we get

  T(n) = (k * ____) =

   

3. Various Asymptotic Growth Functions

• Common growth functions (in an ascending order)

  T(n)	Name
  1	constant
  lgn	logarithmic
  n	linear
  nlgn	n log n
  n2	quadratic
  n3	cubic
  nm	(general) polynomial (m is a fixed, non-negative integer; ; e.g. n4, n5)
  mn	exponential (m >= 2; ; e.g. 2n, 3n)
  n!	factorial

   

• As the input size n becomes large, some growth functions grow very fast, while others grow slowly.

  n	T(n) = lgn	T(n) = n	T(n) = n2
  1
  10
  100
  1000

4. Big-Oh (O), Omega (W) and Theta (Q) Notations

• T(n) is often expressed as a function that has several terms.  Each term also may have constant coefficients.  For example,

  T(n) = 60n² + 5n + 1

• But we are mostly interested in an approximation -- order of the function (or order of growth).  So we only look at the dominant term (60n² in the above) and throw away the coefficient (60).  Then we get, for the above,

  T(n) = Q(n²)

NOTE:  Any function of the form n^k is called a polynomial of degree k.

 

• There are three notations for expressing asymptotic growth:
  1. Big-Oh -- e.g. O(n²) -- for upper bound
  2. Omega -- e.g. W(n²) -- for lower bound
  3. Theta -- e.g. Q(n²) --  for tight bound

NOTE:  Throughout the rest of this document, n is a positive integer (i.e., n >= 1).

(1) Big-Oh

• A function f(n) is of order at most g(n), noted f(n) = O(g(n)), if there exists c, n₀ > 0 such that for all n >= n₀, we have that f(n) <= c*g(n).

  

   

• Examples: 
  1. 60n² + 5n + 1 = O(n²)   ...  NOTE: f(n) = 60n² + 5n + 1, and g(n) = n²

     Proof:

     

      

  2. 

     Proof:

      

      

      

      

      

      

      

      

      

  3. n + nlgn = O(nlgn)

     Proof:

      

      

      

      

      

      

      

      

  4. n + nlgn = O(n²)

     Proof:

     
     
     
     
     
     
     
     
     
     
     

(2) Omega

• A function f(n) is of order at least g(n), noted f(n) = W(g(n)), if there exists c, n₀ > 0 such that for all n >= n₀, we have that f(n) >= c*g(n).

  

   

• Examples:
  1. 60n² + 5n + 1 = W(n²)

     Proof:

     

  2. 60n² + 5n + 1 = W(n)

     Proof:

     

      

  3. 

     Proof:

(3) Theta

• A function f(n) is of exact order g(n), noted f(n) = Q(g(n)), if f(n) = O(g(n)) and f(n) = W(g(n)).
• Examples:
  1. 60n² + 5n + 1 = Q(n²)

     Proof:

     From (1) a. and (2) a. above.

  2. 3n² + 2n lgn = Q(n²)

     Proof: