Despite the tremendous interest in halide perovskite solar cells, the structural reasons that cause the all‐inorganic perovskite CsPbI₃ to be unstable at room temperature remain mysterious, especially since many tolerance‐factor‐based approaches predict CsPbI₃ should be stable as a perovskite. Here single‐crystal X‐ray diffraction and X‐ray pair distribution function (PDF) measurements characterize bulk perovskite CsPbI₃ from 100 to 295 K to elucidate its thermodynamic instability. While Cs occupies a single site from 100 to 150 K, it splits between two sites from 175 to 295 K with the second site having a lower effective coordination number, which, along with other structural parameters, suggests that Cs rattles in its coordination polyhedron. PDF measurements reveal that on the length scale of the unit cell, the PbI octahedra concurrently become greatly distorted, with one of the IPbI angles approaching 82° compared to the ideal 90°. The rattling of Cs, low number of CsI contacts, and high degree of octahedral distortion cause the instability of perovskite‐phase CsPbI_3. These results reveal the limitations of tolerance factors in predicting perovskite stability and provide detailed structural information that suggests methods to engineer stable CsPbI₃‐based solar cells.