Optimization for Data Science · Lecture 7
Memory and time as dimension grows
Float32: 4 bytes per number. Arithmetic time at a hypothetical sustained 100 GFLOP/s (1011 operations per second). These are scaling estimates, not measured runtimes.
| Dimension d | dd entries / d operations | d2d² entries / d² operations | d3d³ entries / d³ operations | Choleskyd² entries / ≈ d³/3 operations |
|---|---|---|---|---|
| 1,000 (103) | 4 kB/10 ns | 4 MB/10 μs | 4 GB/10 ms | 4 MB/3.3 ms |
| 1,000,000 (106) | 4 MB/10 μs | 4 TB/10 s | 4 EB/116 days | 4 TB/38.6 days |
| 1,000,000,000 (109) | 4 GB/10 ms | 4 EB/116 days | 4 × 1027 B/317 million years | 4 EB/106 million years |
kB = 103 bytes; MB = 106; GB = 109; TB = 1012; EB = 1018. One year = 365.25 days. Times are rounded.
The first three columns pair a hypothetical count of stored numbers with the same count of operations. They are scaling models, not the memory requirements of a particular algorithm. The Cholesky column counts one full matrix reused in place, before extra workspace; its arithmetic cost is approximately d³/3.
Time = operation count / 1011. Actual runtime also depends on memory bandwidth, overhead, and implementation. Large cases presume enough memory to hold the matrix; the tiny times omit overhead. At 1 TFLOP/s, divide the arithmetic times by 10.
Cholesky uses quadratic memory and cubic arithmetic work.