Only new top level annotations should be moved to the new group when
switching groups.

Fixes hypothesis/h#3559