This makes it easier to test the refresh logic directly and decouples it
from the rest of the annotation display.