Consulting dictionaries during writing requires time and cognitive resources. ColloCaid, a writing assistance prototype freely available online, was designed to minimize the cognitive strain on writers by embedding a collocation database within the writing environment. Usability surveys have shown ColloCaid can indeed help. In this study, we go beyond user perceptions. Using authentic excerpts of student academic writing by 27 advanced L2 English speakers, we analysed (1) the lexical coverage of the tool, (2) the collocation changes prompted by the tool, (3) the reasons behind decisions to revise collocations, (4) the effect of revisions prompted by ColloCaid, and (5) the participants’ perceptions of using the tool to revise authentic writing assignments. Our findings indicate that ColloCaid offered good academic collocation coverage, that the participants tended to accept its collocation prompts with discernment, and that the revisions made resulted in more fluent texts overall.