The large-scale interannual variability of sea ice concentration in the Southern Ocean is largely controlled by atmospheric dynamics. By contrast, based on satellite observations, we show here that the local amplitude of interannual variations of the winter sea ice edge position is mainly modulated by the ocean bottom topography and oceanic processes. The standard deviation of the latitude of the ice edge displays substantial variations as a function of longitude, with prominent sharp peaks covering only a few degrees of longitude close to the main topographic features of the Southern Ocean. There, mesoscale eddy activity and the variability of the Antarctic Circumpolar Current jets are large, influencing both oceanic heat transport and sea ice velocity, thereby leading to large interannual changes in the position of the ice edge. Owing to such bathymetric control, these regions showing high variability in the winter ice edge position have remained relatively stable over recent decades, despite ample changes observed in other characteristics of the sea ice cover during the same period. Eddy-rich global sea ice-ocean models based on NEMO-SI3, both forced by ERA5 surface fluxes or coupled with the atmospheric model IFS, can reproduce the sharp peaks in the variability of the ice edge position, indicating that they adequately simulate the dominant influence of topography on currents and eddy activity. However, this requires a realistic mean ice edge position; otherwise, model biases can displace the ice edge away from regions of strong eddy activity and therefore distort the interannual variability.