Using new data on roll-call voting of U.S. state legislators and public opinion in their districts, we explain how ideological polarization of voters within districts can lead to legislative polarization. So-called “moderate” districts that switch hands between parties are often internally polarized: the ideological distance between Democrats and Republicans within these districts is often greater than the distance between liberal cities and conservative rural districts. We present a theoretical model in which intra-district ideological polarization makes candidates uncertain about the ideological location of the median voter, thereby reducing their incentives to offer moderate policy positions. We then demonstrate that among districts with similar median voter ideologies, the difference in legislative behavior between Democratic and Republican state legislators is greater in more ideologically heterogeneous districts. Our findings suggest that accounting for the subtleties of political geography can help explain the coexistence of a polarized legislature and a moderate mass public.