Hugo Awards: Plus 2

[kevin_standlee]

Here's yet another proposal to try and counteract bad actors (I call them "Griefers") trying to disrupt the Hugo Awards by deliberately nominating works that they expect will be disliked by the majority of the membership as a whole, taking advantage of the "first-five-past-the-post" nature of the nominating round. The other proposals I've written up depend on the entire membership participating in a second round of voting, either with 3-Stage Voting (members vote down potential finalists) or Double Nominations (members select finalists from a list of top 15 semi-finalists). This proposal invokes the subjective judgment call of the Worldcon Committee (in practice, of the Hugo Awards Administration Subcommittee), hereafter just "the Committee" or "the Administrators," to add works to the final ballot.

This proposal would authorize the Committee to add up to two additional works to the final ballot. The Committee's selection would be limited to adding not more than two works from among those works that were among the top 15 nominees or that appeared on at least 5% of the nominating ballots cast in that category.

I would not necessarily require the administrators to say that they had exercised their rights when making the finalist announcements. Historically, there have been as many as eight finalists on the ballot due to last-place ties. While historically such longer-than-five-finalist ballots have been explained as due to ties (this happened once in the Retrospective Hugo Awards this year), I could certainly see Administrators not saying why the ballots are longer and making us wait until after the ceremony, when the post-award report would make it clear.

This proposal has the advantage of not requiring an additional round of voting, and does not triple the number of finalists. The biggest disadvantage is that it requires the Administrators to make subjective judgement calls, which historically they are loathe to do, and for good reason.

I can see the prospect of Committees expanding to include members brought on for their literary judgement rather than for their administrative skills. Such members would of course be ineligible for the Hugo Award in the year they were on the Committee, like all of the others.

Although I wrote this as "plus 2," there's no reason it couldn't be 3 or 4 or even 5, although I'm leery about doubling the size of the final ballot.

There is precedent, albeit an extra-legal one, for Administrators augmenting the ballot. In 1989, a single category came into question, due to a large number of nominating ballots being submitted simultaneously, paid for by consecutively-numbered money orders, and in the names of people for whom it is unclear they knew they had memberships. Apparently all of these ballots named a single person in one category and nominated nothing else. (There is no implication that the named person had anything to do with this scheme, and some implication that it was the result of overly enthusiastic fans of that person with more money than sense taking advantage of the system as it then existed.*) Instead of simply disqualifying the ballots, the Committee elected to add a sixth name to the finalists in that category. Shortly thereafter, one of the finalists withdrew.

Just like all of the other proposals out there (including the two up for ratification this year), this is no panacea, and I think there's a real concern that letting the Administrators exercise subjective judgement over whether extra works should be placed on the ballot undermines the popularly-selected nature of the Awards. On the other hand, I do recognize that we currently have a situation where people have not been nominating in good faith, and there are people who I respect but with whom I do not agree who think the Administrators should be intervening dramatically, rejecting individual ballots or finalists. This proposal seems to me to be the least-intrusive way of letting the Committee do something without totally rejecting the nature of an award selected by the members of the World Science Fiction Society.

---

*In those days, you could join at the time you nominated, right up to the end of nominating, just as you currently can do when you vote on the final ballot or on site selection. It was in reaction to this particular case that WSFS adopted the rule that you have to be a member no later than the end of January in order to nominate, rather than up through the end of nominations. As a further reaction to the cut-off date, WSFS extended nominating (but not final ballot) eligibility to include the members of the previous Worldcon, in order to not disenfranchise "regular" members who tend to join every year but who might not have gotten around to joining the current year by the end of January. Adding the members of the subsequent year's Worldcon was a later addition, and has only been part of the system since the 2012 Hugo Awards.

Comments

[rono-60103.livejournal.com]

I'm wondering if there is a simple (or somewhat simple) query that could be run on the normalized nomination data that could be used to detect that fraction of the ballots that contained exactly the shortlist in a given category. This would give the administrators a bit of objective data to work with.

I'm not going to see if I can come up with one for several reasons, including lack of time and not having access to any normalized nomination data at this point in time (I have access to some raw data - but normalizing takes a fair amount of work)

[history-monk.livejournal.com]

I would be very leery of this idea. It plays into the hands of the idiots who claim, regardless of evidence, that the nominations and results are manipulated.

[kevin-standlee.livejournal.com]

It does have pitfalls, for certain. The actions of the Noreascon 3 committee were highly controversial at the time back in 1989 as well; this proposal essentially writes into the rules a sanction for what N3 did. OTOH, if we're going to continue to have Griefers, do we not have to have some mechanism for dealing with them? "Ignoring the trolls" turns out to not be a viable strategy, and it's unclear to me whether either or both of the proposals up for ratification this year are going to do enough to discourage the bad actors to give up and go away.

I prefer solutions that leave the decisions in the hands of the electorate as a whole, and I retain confidence (some suggest misplaced) in the electorate as a whole to reach a conclusion that is broadly approved by the membership overall. The No Awards given out last year were a confirmation of how Instant Runoff Voting doesn't actually choose the most-liked candidate so much as the least-disliked one. I don't think many of us participating were happy that No Award won in those five categories, but were relieved that something worse did not not win.

I'm concerned that calls — possibly outliers, but at least one of them comes from a past Hugo Award winner — for Administrators to simply throw out "slate voters" ballots or ignore nominations for specific works that are "abusive" will continue and will grow in volume. Indeed, it appears to me that many commentators from outside of our community do not understand why the "Hugo Judges" don't just toss the obvious bad actors out, and when you try to explain it to them, either their eyes glaze over or they think we're fools for not taking Quick Decisive Action by the Strong Man, like (say) Goodreads did when they were being trolled by effectively the same gang of Griefers. So I'm trying to think of ways to balance conflicting issues, which is IMO a hallmark of democracy when it works as it should.

[exsmof.livejournal.com]

I am not certain why 6/4 doesn't do much of this already, without the judgement call. You could still pack the ballot, but it required a bit more logistics organizing groups of people to nominate slightly different slates. I would have a problem with the Administrators getting to pick any of the 6-15 (with the constraint that they have to gotten at least 5%) without regard to the actual nomination count. If they picked 8, and 13, then those that placed 6-7, and 9-12 would have a legitimate beef with the ballot. As it would arise after the fact the "I was robbed!" flame wars would get intense.

[kevin-standlee.livejournal.com]

I know how I would approach it as an Administrator. I assume that Griefers cannot possibly "pack" all fifteen places on the ballot without having a majority of the overall voting strength. (If they do, then they deserve to win, frankly; it's meant to be a popularly-voted award, after all.) But it might be possible to pack more than just the top five positions. I also assume that the Administrators, with access to the raw ballot data, can see the patterns in bloc voting that are very difficult to define by an algorithm. (You might call this the "I know it when I see it" assumption.) Therefore, rather than just seating positions 6 and 7 (the 1989 Worldcon simply seated the sixth place nominee), I might end up picking 7 and 10 because in my opinion, the nominees in positions 6, 8, and 9 all got there due to a pattern of bloc voting that suggests bad-faith behavior. Whether this would work in practice, I'm not certain, but it does introduce the element of Administrator Judgement that some people are insisting we must include in the process.

Indeed, there are people looking at the awards from outside and cannot fathom why the "Hugo Judges" haven't already tossed out the Griefers. When you explain all of the reasons why not, either their eyes glaze over or they say that you're crazy. They think you should take Quick Decisive Action by the Strong Man, like Goodreads did when the Griefers tried to game their system. They don't understand that Goodreads is a privately-run company that can make its own rules about its own system, and isn't bound to follow rules imposed upon them by their own users. Similarly, Locus might decide that they're being freeped and decide to ignore what they decide to be freepers when counting the ballots for the Locus Awards, and they wouldn't even have to say they'd done so. They're not responsible to the voters, but to the members of their parent non-profit organization. WSFS, however, is responsible to its members and is supposed to act only within its rules, which are deliberately designed to discourage Strong Men and Quick Action.

[lindadee.livejournal.com]

I was just thinking that this proposal sounded a lot like 4 and 6. At last year's BM, even though the original proposal was 4 and 6, we voted to fill in the blanks to see if we preferred something else. 4 and 6 won.

[kevin-standlee.livejournal.com]

I don't see this as being much like 4/6, unless you really believe that nobody will form counter-slates, or that Griefers will find ways to form up to dominate the top 6.

I don't think Griefers can possibly dominate the top 15 without forming a true majority, in which they win anyway. This proposal gives Administrators sanction to decide which works are being cast by voters who they believe are acting in good faith.

FWIW, it also shares elements with the World Fantasy Awards, where a jury can add works to the final ballot, which is in turn voted upon by the members.

[kevin-standlee.livejournal.com]

Lisa, after reading the above, asked me to post the following on her behalf:

This is an idea I had kicked around with Kevin on several occasions. The idea is not to give the Administrators overt power, as the top 5 would already be decided, but to give a safety net of a small group who could add in a small number of alternatives that didn't make the final five due to "unforeseen complications." In one practice, we already have something not too far from this, as Worldcons are already allowed to add a category for one year. So I don't see it being so absurd to allow each Worldcon to add some additional contenders for the Award, as long they already had a significant level of support. After all, they wouldn't be marked as "added by the Committee." You wouldn't know one way or the other when voting on them.

The proposal above is indeed based on Lisa's discussions with me. I wasn't sure whether she wanted that known when I wrote it up, and I added the information about how the 1989 Worldcon dealt with a small-scale ballot-stuffing situation.

[malkaesther.livejournal.com]

Again I worry - will Hugo Admins be willing to do this? I can see the sense in this proposal. But every time I've seen an online discussion about increasing their powers I've seen pushback from the admins.

[kevin-standlee.livejournal.com]

I would suggest giving them the authority to do this but not requiring it. I know most of the people who have been administrators. They're good people. Let them decide whether the actions are egregious enough to justify intervention. Heck, require that it be re-ratified every year for five consecutive years, taking effect after the first ratification, but requiring it be re-ratified annually until the fifth year, and then withhold consent if an Administrator oversteps.

[malkaesther.livejournal.com]

I'm not worried about how the admins will use the power. I'm worried about the admins quitting if we decide to give them this power. I don't know many Hugo Admins but the few I've met online have all said "no way I don't want that kind of power. No. No. No"

My question is "How do the Admins FEEL about this proposal. Are they willing to take the power and use it."

[kevin-standlee.livejournal.com]

In my opinion, sensible administrators will recruit a panel of people with the literary judgement necessary to decide whether they should exercise their authority or not. Those people would be part of the Hugo Awards Administration Subcommittee, but will not be involved in the administrative site, but the literary-judgement side.

[piet nel (from livejournal.com)]

Why not simply appoint someone independent of the Administrators - say, a panel of three with no other admin connection to the Hugos - to nominate the additional stories? I'm still not really comfortable with the idea, but that at least takes the Administrators off the spot.

[hampus eckerman (from livejournal.com)]

That is a version of what the Bram Stoker Award does. They have a separate jury known to be knowledgeable about the field. That jury will pick out what they think are the most worthy candidates. In the second nomination round, nominations from ordinary members are bundled together with those from the jury and then the members as a whole votes for which of these that they think should go on to next round.

But then it is a part of the ordinary voting process, not something to be used exclusively when there is trouble. And that is why I think that wouldn't work. Should we always choose this panel, even if they wouldn't have to do anything at all during ordinary years? That would politicize the process even more when they did something.

[kevin-standlee.livejournal.com]

This is why I think it should always be the Administrators' job (even if they add some extra people and give them a fancy special name). If you adopt an anti-troll rule, you always have to have it there even if you don't use it, because if you take it away, the trolls will almost certainly come back.

For the past few years, we've been dealing with Griefers by assuming that if we ignore them and No Award them, they'll get tired of it and go away. They've so far shown that they are stupider than we thought and they want to keep coming back for the lulz. We can't ignore them; we have to find something that deals with them, and allows them to not completely dominate the process.

Some want to find a way to arbitrarily exclude them, which I cannot accept. As unpleasant as I consider their wrecking tactics, they have the right to be members like anyone else (no matter who pays for their memberships, incidentally, although there is no evidence that they're being bankrolled). What I don't think they have the moral right to do (although right now they have the legal right to do) is dominate 90% of the shortlist with 20% of the voting power. This is patently unfair, and we need to find a way to give the other 80% of the electorate a reasonable representation on the shortlist.

[kevin-standlee.livejournal.com]

It doesn't matter if you appoint other people or not — they'd still have to be classed as Hugo Administrators, because any sensible rule would require that a Separate Panel of Hugo Judges were also ineligible for the Award that year. If it makes people feel better, we could create another group and give them a different name, but they would have to have exactly the same level of access to the data as the Administrators do, so functionally they would be Administrators themselves.

I think that in practice the existing Administrators (which is the Hugo Award Administration Subcommittee of the current World Science Fiction Convention, remember; "Administrators" is a shorthand term) would want to add additional members to their Subcommittee, and give them the specific task of determining who, if any, of the Top 15 (also shorthand; see full version above) should be added. Again, if giving those additional people a different name makes it feel better to some, we could do so, but they'd fall under exactly the same rules as the existing Administrators.

[redbird]

Having a separate subcommittee for this purpose, rather than adding to the powers/duties of the people doing the current Hugo administration work, might take some stress off the people doing the currently-defined job. They'd still get some "you should" and "why didn't they?" and "of course you did" from people who didn't understand the process (or wanted to pretend they hadn't), but if you know you can't do something, you're less likely to lie awake worrying about whether to do it.

[kevin-standlee.livejournal.com]

Sure, have a separate subcommittee. Call them whatever you want to call them. But functionally they'd still be Hugo Award Administrators, because they would have to have exactly the same level of access to the data and the same restrictions on their own personal eligibility.

I don't really care what you call them. They have the same authority and the same functionality, so if it makes people feel better to call some of the members of the Committee "Administrators" and other members of the Committee "Judges," that doesn't bother me. I'm interested in function more than form here.

[redbird]

It's not just about what to call them. What I was wondering—since this would require amending the constitution anyhow—is whether it would be a good idea to have two small groups of people, both of whom have the data access and restrictions on their eligibility, but only some of whom could add items to the ballot.

Functionally, a Hugo administrator isn't just someone who has access to the data and is ineligible for the Hugos this year. Functionally, they are people with specific powers and responsibilities. In order to carry those out responsibilities they have access to the data, and in order to carry them out fairly they are ineligible for the Hugos that year.

As I see it, a problem with the "Plus 2" proposal is that it puts a small group of people in a position where they are likely to be blamed for what ends up in the final Hugo ballot. I agree that calling people "judges" rather than "administrators" or "clerks" or "secretaries" isn't likely to make much difference. It's possible that separating out the tasks, so the less subjective ones are done by one group and the potentially more subjective ones by another, would help there. (On the other hand, that would work only to the extent that people understood that lines had been drawn, and agreed on whether "when was this published?" and "how do we reach so-and-so?" are a different kind of question from "did slating affect the ballot and if so, what should we add?")

[kevin-standlee.livejournal.com]

I'm not sure you realize this, but as written the entire Worldcon Committee is ineligible for the Hugo Award in their year unless they irrevocably delegate the authority to run the Hugo Awards that year to a subcommittee. In practice, they always do so.

I think that people are so used to what Worldcon Committee always do in practice that they assume that it's hard-coded into the WSFS Constitution, which it is not.

Having to explicitly declare a separate group of Hugo Judges is actually a more substantial change to the Constitution than simply expanding the existing authority of the existing Committee. Adding a new authority is a sentence or two. Hiving things off makes for a much more substantial revision.

[hampus eckerman (from livejournal.com)]

My problem with this idea is that it would keep the nominations of the bad faith actors on the ballot. Only in this year we have one nomination with death threats, one with gross homophobia and one with dinosaur erotica. I don't just want extra nominations in addition to these, they should never have been finalists in the first place. If we are going to make changes to the voting process, in some way moving No Award before the finalists are announced has to be first priority.

[kevin-standlee.livejournal.com]

That's a reasonable position, and that's what 3SV and DN/AV (Double Nominations with Approval Voting) do in different ways.

All three of the proposals I've written up have positive and negative aspects to them. None of them are perfect. There is no possible "perfect" solution, in that nothing is going to satisfy everyone. The question I have is whether there is any proposal that has sufficient support to make it through two WSFS Business Meetings and work in practice. Unlike EPH and 4/6, these three proposals are mostly incompatible with each other and thus anyone wanting to propose them to the WSFS Business Meeting is going to need to consider the practical politics of proposing solutions to a fractious group of fans who are known to obsess over outliers and insist that if a solution isn't perfect, it must not be allowed to pass.

[(Anonymous)]

I am nervous about this because, as Hampus et al have observed, it gives the Griefers an excuse (however threadbare) to claim they were blocked by some shadowy cabal rather than being rejected by the mass of voters. I also think that the widely-learned judgment to add items, and even the ability to find such judgment, is is a different skill set from any part of convention running. (Some people who do Program might think they know enough to do this; I don't.) I am also favoring the argument that the tripe put up by the Griefers should not have whatever honor comes from being a Hugo finalist/shorlistee/....

Note that N3 responded to blatant fraud: last-minute memberships allegedly for people in the midwest, all paid for at a single New York post office. As you note, there's no indication that the Griefers have done even a better-covered-up form of this, let alone something this blatant.
Also: IIRC there were 3 categories affected; most of the noise may have been on the most obvious one.

/CHip

[kevin-standlee.livejournal.com]

That's why any sensible Worldcon committee would empanel people with that judgement as part of their Hugo Award Administration Subcommittee.

The counter to the Griefers is that "Your nominations weren't blocked, and you're still on the ballot, but you have to compete with works nominated by the rest of the members. If you're as good as you think you are, surely the majority of the members will recognize your brilliance."

Personally, I favor solutions that rely upon the entire membership participating; however, I do recognize that there are people who distrust democracy and thing that a Panel of Experts will do a better job. Allowing such a Panel of Experts (added to the existing Hugo Administration Subcommittee created by the current Worldcon Committee) is a way to create a hybrid that retains mostly the input of the membership.

Again, discussion of these three proposals that I've written up highlights that nothing is going to be perfect, and there are issues with all of them. We as members of WSFS need to decide how to balance things, and not get so tangled up in single issues that we end up doing nothing at all. I'm pretty sure that "doing nothing" is the worst choice right now.

[hampus eckerman (from livejournal.com)]

As I see it - this one is not enough. It might work in addition 3SV or DN if there would be such a strange case that trolls managed to fill up the whole long list of 15 items, but otherwise not enough.

I fully agree with you that it would be the absolute highest degree of self harm to not vote for something that made the system more troll safe. Even if Beale would disappear, someone else might come in 5-10 years and we had to start from the beginning. And we need to be pro-active. Not just wait and see. After a while, people will get tired of reading and voting if griefers will take the nomination slots anyhow.

3SV or DN are those that I think would work best of these proposals because both of them have the power to remove all griefers before the finals. That would mean less people hurt in all ways.

[kevin-standlee.livejournal.com]

I rather expect that if all three of the proposals I have written up were simultaneously proposed, none of them would pass, because every proposal would have a majority opposed to it for one reason or another, while a majority (probably a super-majority) would vote yes on the question, "Do we need to Do Something?"

[hampus eckerman (from livejournal.com)]

He, I think the voting would have to be like the Hugo runoffs. Let everyone answer which ones they like the best. Remove the least popular. Repeat until only one remains. Vote on if that can be implemented.

But best if there could be enough support for one proposal so the others will withdraw or never be placed at all.

[baroquepath.livejournal.com]

The problem, as I understand it, is that two hundred members of Gamergate bought 50$ "supporting memberships" to cast troll votes for Hugo nominees.
The existing fan vote was sufficiently diffuse (in some categories) that the two hundred slate votes were able to dominate.

One simple solution would seem to be to increase the requirements for voting.
So we can imagine rules like the following:

• You can't vote unless you have an attending membership for the year in question
  
• You can vote with either a supporting or attending membership, but only if you have physically attended a worldcon in some past year
  
• You can vote on the actual Hugo ballot with either membership, but you can't nominate unless you've attended a worldcon in some past year
  
• You can vote or nominate in certain categories like Best Novel where we expect the fan signal to be pretty strong, but you can't vote for more minor categories like Best Short Story unless you've attended a worldcon

The general idea is:
(1) Gamergate members, although demonstrably crazy enough to pay 50$ to wreck the Hugos, are not crazy enough to physically attend a Worldcon in order to wreck the Hugos.
(2) The Hugo voting process does not rely heavily enough on "supporting memberships" that it would break the whole thing if we restricted their ability to vote or nominate.

I haven't seen anyone float a suggestion that's like this at all. That makes me think there must be some obvious flaw in it, but I'm not sure what the flaw might be. Thoughts?

[baroquepath.livejournal.com]

As to the N+2 thing: I agree with Hampus, in that I want a solution that doesn't put troll content on the ballot in the first place. If we're okay with having some troll content on the ballot so long as it's not all troll content, we can stick with EPH.

The "double nominations with approval voting" thing sounds appealing.

[kevin-standlee.livejournal.com]

Here's my problem with "doesn't put troll content on the ballot in the first place." Those people have the same right as you and I did to buy their memberships and vote them. It's fair for them to have some representation on the ballot, but not to dominate it entirely, particularly when they consistently force through material that >60% of the members dislike so much that they'd rather give no award at all than to any of those choices.

[hampus eckerman (from livejournal.com)]

My problem is more with your wording of fair. I do not think making cartels to game the system is fair, so I have no problems with a voting system that will remove troll items before the finals. I don't think it is fair to other contestants to leave them in place. Better again to have No Award as early as possible in the process.

[kevin-standlee.livejournal.com]

You are quite definitely not the first person to suggest any of those things. Lots of people have floated them in multiple places. And every one of them is problematical in different ways.

Banning supporting members from voting throws the baby out with the bathwater, as you'd be banning people who are certainly part of the ongoing Worldcon community but aren't in a position to attend every year.

As far as requiring that you attended a Worldcon in some past year goes: How could you possibly enforce this? For example, someone with a supporting membership claims that they attended the 1993 Worldcon in San Francisco. How do you prove or disprove that statement? The ConFrancisco committee dissolved many years ago. There's no central database of Every Worldcon Member Who Every Was, or any chance that we'd ever be able to create one. And how would you distinguish between someone who had an attending membership and attended and one who had an attending membership but didn't show up? Remember, every Worldcon is a completely independent entity, with only a tiny bit of legal overlap between them. The three Worldcons centered on the current year have a small amount of requirement to cooperate in order to administer their respective conventions, but otherwise it would be like trying to make DragonCon share its membership list with ComicCon International (San Diego) and Wizard World.

Trying to pick and choose categories in which people are deemed worthy to participate is too much trouble, in my opinion.

You'd almost certainly have an easier time trimming the rights of the previous and subsequent years' members to participate in Hugo Award nominating. Perhaps you might limit nominating to only attending members of the previous/subsequent years' conventions, while continuing to allow all voting members of the current Worldcon to participate. Assuming most Griefers never attend, that would double or triple the cost to continue their campaign, assuming you think that cost is a sufficient barrier.