It’s just the way that feature was designed, but we agree there should be a way to close them without voting.
We’ve moved this to the Suggestions section for votes in favor of it.
Fortunately, even without that option, the polls are displayed temporarily.