Structural variations (SVs) affect more of the cancer genome than any other type of somatic genetic alteration but difficulties in detecting and interpreting them have limited our understanding. Clinical cancer sequencing also increasingly aims to detect SVs, leading to a widespread necessity to interpret their biological and clinical relevance. Recently, analyses of large whole-genome sequencing data sets revealed features that impact rates of SVs across the genome in different cancers. A striking feature has been the extent to which, in both their generation and their influence on the selective fitness of cancer cells, SVs are more specific to individual cancer types than other genetic alterations such as single-nucleotide variants. This Perspective discusses how the folding of the 3D genome, and differences in its folding across cell types, affect observed SV rates in different cancer types as well as how SVs can impact cancer cell fitness.