This frees us up to remove the other mouse events, which makes this
a bit cleaner to my eyes. It also avoids passing the selection
through the scope, storing no state between events.