We investigate the notion of utterance acceptability in context, and through it, the underlying competence notion coherence, understanding it quite directly as a limit on variability in discourse follow-ups. We will assemble a test suite of relevant cases (in English), and collect acceptability judgements. Building on this, we will study which aspects of this notion, if any, neural language models capture. Finally, we will use the results of these studies to investigate whether we can improve the models by providing inductive biases that introduce discourse structural knowledge. A particular focus in these studies is on coherence in dialogue.