Well, in that case, we shouldn't allow anyone on a Worldcon committee to be eligible for a Hugo Award

Well, yes. This.

At least in my opinion.

I'm not really sure I see how anybody not explicitly involved in counting or handling the ballots is really in a position to have any kind of conflict of interest.