Skip to content

Latest commit

 

History

History
81 lines (49 loc) · 1.2 KB

lecture_13.md

File metadata and controls

81 lines (49 loc) · 1.2 KB

Lecture 13

Big data; algorithms

Airflow

Share your experience.

  • What was interesting?
  • What was hard?

ETL vs. CI

Lab 13

Big-O notation

Exercise

  1. Write out ten two-digit numbers.
  2. When I say "go", add them up.
  3. Record the time.

For this exercise, don’t care if the answer's right or not.

  1. Form groups of three or four.
  2. Someone be the reducer, others will be the workers.
  3. Write out ten two-digit numbers, split up among the workers (5-5 or 3-3-4).
  4. When I say "go":
    1. Workers:
      1. Add your numbers.
      2. Tell the reducer your sum.
    2. Reducer:
      1. Add them up.
      2. Record the time.

Takeaways?

Back to Big-O

Draw chart of times on chalkboard, individual and team

How long would we expect each to take for 100 (10x) numbers?

Parallelization

  • Threads
  • Distributed workloads
  • MapReduce

Sync vs. async

What's a real-world example?

Project Parts 7 and 8