Floating Point¶

L08 Floating Point | CS61C Course Notes

Fixed Point¶

To represent numbers with fractional parts in 32 bits (e.g. 4.25, 5.17), start with a strawman: Fixed Point (定点数) — the binary point’s position is fixed.

Metrics: Range and Step Size¶

Two metrics for any number system:

For integer systems, step size is always 1:

With fractions, infinitely many reals lie between consecutive values, so step size becomes a key measure of precision.

Binary Point¶

Left of a decimal point: \(10^0, 10^1, \ldots\); right: \(10^{-1}, 10^{-2}, \ldots\).
A binary point works the same way: left \(2^0, 2^1, \ldots\); right \(2^{-1}, 2^{-2}, \ldots\).

Fixed point = a fixed-width bit string with a pre-agreed binary-point position.

Example: 6-bit unsigned fixed point, binary point 4 bits from the right (2 integer bits + 4 fractional bits):

  1 0 . 1 0 1 0
  ↑ ↑   ↑ ↑ ↑ ↑
  2¹ 2⁰  2⁻¹ … 2⁻⁴

\(2.625\) encodes as 101010:

Range & Step Size¶

Arithmetic¶

• Addition: align binary points, then reuse an integer adder (e.g. \(1.5 + 0.5 = 2\)).
• Multiplication: must shift the binary point by the number of fractional digits (e.g. \(1.5 \times 0.5 = 0.75\)) — more awkward than addition.

Use Cases & Limits¶

Fixed point is still useful when the value range is known and fast arithmetic matters (e.g. some graphics rendering).

Once the binary point is fixed, you are stuck: the 6-bit example cannot represent numbers much larger than \(3.94\), nor non-zero values between \(0\) and \(1/16\).

Scientific apps often need all of:

• Very large: seconds in a millennium \(\approx 3.16\times 10^{10}\)

• Very small: Bohr radius \(\approx 5.29\times 10^{-11}\,\mathrm{m}\)

• Mixed integer + fraction: e.g. \(2.625\)

Covering all three with fixed point needs ≥ 92 bits (~34 integer + ~58 fractional), far beyond a typical 32-bit integer width.

It means that we need a representation that can move the binary point with the magnitude — Floating Point (浮点数), inspired by scientific notation.

Normalized Numbers¶

Floating point does not fix one binary-point location for all numbers. For each value, it stores what the significant bits are and which power of two they sit at — the same idea as scientific notation.

Inspirations about Scientific Notation¶

Take decimal \(0.1640625\). Its binary is roughly \(\dots000000.001010100000\dots\): long runs of zeros, with only a short “energetic” stretch of bits (10101). Scientific notation keeps that energy and records the scale separately.

Base-10 scientific notation for the same value: \(1.640625 \times 10^{-1}\).

Normalized form: exactly one non-zero digit left of the point. For a given significant-figure budget, that form is unique (e.g. \(1.0\times 10^{-9}\), not \(0.1\times 10^{-8}\) or \(10\times 10^{-10}\)).

Binary Normalized Form¶

Same idea in binary: \(0.1640625 = 1.0101_{\text{two}} \times 2^{-3}\).

Two consequences for a 32-bit float design:

1. Split the bits into at least mantissa and exponent fields.

2. In binary, the only non-zero digit is 1, so a normalized mantissa is always 1.xyz... — the leading 1 is redundant.

IEEE 754 Single-Precision Floating Point and Normalized Numbers¶

IEEE 754 single precision is C’s float (32 bits). UC Berkeley’s William Kahan led the standard (Turing Award, 1989) so machines would agree on floating-point results.

Two useful notions:

Layout (MSB = bit 31, LSB = bit 0):

Normalized numbers (when \(\text{exponent} \in [1, 254]\), i.e. not all-0 / all-1) decode as:

with \(\text{bias} = 127\) for single precision.

How each field is interpreted (none of the three fields use two’s complement):

Design Principles¶

The three fields are shaped to squeeze more accurate normalized values into 32 bits, not to look like integer encodings.

Double-Precision¶

C’s double is IEEE 754 double precision (64 bits): 1 sign + 11-bit exponent (bias \(1023\)) + 52-bit significand. Larger significand → better accuracy; range roughly \(2.0\times 10^{-308}\) to \(2.0\times 10^{308}\).

Special Numbers¶

Normalized floats are only part of IEEE 754. Single precision dispatches on the biased exponent:

These specials exist so float arithmetic can handle overflow / underflow more gracefully than integer wrap-around.

Overflow and Underflow¶

Because exponents \(0\) and \(255\) are reserved, the normalized single-precision magnitude range is about:

(and the negative mirror). Extremes:

With fractions, both ends matter:

IEEE 754’s response: overflow → \(\pm\infty\) / NaN; underflow → approach zero via denorms (gradual underflow), not an immediate jump to zero.

Zero¶

Like sign-magnitude, there are two zeros (useful for directed limits and \(\pm\infty\)):

Zero is not normalized — there is no leading 1 in \(0.0\). Hardware treats biased exponent 00000000 as “no implicit 1”; if the significand is also zero → \(\pm 0\); if non-zero → denorms.

Infinity¶

Infinity is kept distinct from other errors. Example: dividing by \(\pm 0\) yields \(\pm\infty\), so comparisons like \(x/0 > y\) stay meaningful.

Not a Number (NaN)¶

Invalid ops (e.g. \(\sqrt{-4}\), \(0/0\)) should bubble up rather than crash or wrap like integer overflow.

>>> import math
>>> x = math.sqrt(-1)   # nan
>>> y = x + 3           # nan
>>> z = y * 10          # nan
>>> w = z / z           # nan  (not 1!)
>>> math.sin(x)         # nan

Denormalized Numbers¶

Gap around zero¶

Overflow → \(\infty\) feels natural (one step past the largest finite). Underflow → jump straight to \(0\) is harsher: between \(0\) and the smallest normalized \(2^{-126}\) there is a large gap, while neighboring normalized values near that scale differ by only \(2^{-149}\):

The implicit leading 1 forces that jump: you cannot represent values with mantissa \(< 1\) at exponent \(-126\) in the normalized encoding.

Gradual underflow¶

Denorms fill that gap so underflow loses precision gradually instead of snapping to zero:

That choice keeps step size \(2^{-149}\) uniform from the denorm range through the smallest normalized floats → gradual underflow.

Floating Point Operations¶

Addition¶

Float addition is harder than integer addition: you cannot add significands until the exponents match.

Typical pipeline:

1. Denormalize (shift) so both operands share the same exponent

2. Add (or subtract, if signs differ) the significands

3. Keep that matched exponent

4. Renormalize the sum (may bump the exponent), then round into the significand field

Because floats only approximate reals — and step size grows with magnitude — addition is not always associative.

Rounding Modes¶

Hardware keeps a few extra bits of precision during arithmetic, then rounds into the stored significand. Four common modes:

Unbiased / round-to-nearest-even is the usual default: on a exact halfway case, round so the result’s LSB is even. Over many ties, ups and downs cancel → less systematic bias.

Casting and Converting¶

Rounding also bites when converting between C types:

• int → float: not every integer is exact in float (only 23 + 1 = 24 bits of significand precision). e.g. \(2^{24}+1\) snaps to the nearest even float.

• float → int: C truncates toward zero — (int)1.5 becomes 1.

Round-trips are therefore not identity. These need not always print "true":

/* Code A: large ints may not survive float */
int i = /* ... */;
if (i == (int)((float) i)) {
    printf("true\n");
}

/* Code B: fractional floats lose the fraction */
float f = /* ... */;
if (f == (float)((int) f)) {
    printf("true\n");
}

Odds and Ends¶

Precision vs. Accuracy¶

High precision allows high accuracy but does not guarantee it.

Example: float pi = 3.14; still uses the full significand (precise encoding), but \(3.14\) is a poor approximation of \(\pi\) (inaccurate). All fixed-width formats remain imperfect for some values.

Specific Formats¶

Beyond float / double, IEEE 754 also defines:

Domain accelerators often pick different tradeoffs. e.g. Google TPU bfloat16: 16 bits with 8 exponent + 7 significand — same range class as float32, less fraction precision. That favors ML training (gradients vanishing toward zero) over digit-level accuracy.

Other formats: Nvidia TensorFloat-32 (tf32), plus many int4 / int8 / fp16 mixes on TPUs / Tensor Cores.