st challenges Baseline Automated Ranking System (BARS)

[Cyclone Boom 16,851]

Thanks everyone for the suggestions.

Right, some big news...

I've now created an interactive spreadsheet, which hopefully provides a solution to the issues, and makes it easy for us to calculate results.

Baseline Automated Ranking System (BARS)
Features Summary

Ranking

• Automatic ranking of challenge results.
• Priority sorting by the rating, then the number of votes.
• Entries with more votes are ranked higher, in case the rating is identical.
   

Ties

• In the event of identical ties, the rep prize is split between those entries.
  This still keeps to the total Prize Fund (as close as possible when rounded to a whole number).
   
• Ties are clearly highlighted on the rank.
   

Baseline Method

• As a solution for entries with a low sample, the "Baseline Method" is used.
• Consider it a cut line, which is established using the averages of votes & ratings.
• There are two baselines: Base Votes and Base Rating.
   
• Base Votes limits the possible ranking for entries with a small # of votes (compared to others).
  It's calculated using 75% of the mean (average) votes, and is rounded to a whole number.
   
• The Base Rating is calculated the same way, using 75% of the average rating.
  This is used to make the ranking more logical. For example, entries with a low votes sample, but a higher rating will rank above. This is because the priority is still given to the rating.
   
• Entries outside the band are highlighted with a red flag icon.
   

Reputation Prize Fund

• Prize fund is calculated and distributed to the top entries. This is based on the number of entries & the challenge type.
• New "Bonus" rep, which adds a bit of variance to the Prize Fund. This is calculated based on the average rating, and the number of entries.
   
• Calculation to add the author's new reputation total.
• Checkbox to mark when the prize has been awarded.
   

Data

• Validation checks to ensure data is entered accurately.
• Alerts box which show any glaring mistakes with the data. This kind of works like a checklist.
• Options to customise each challenge (e.g. ranks which receive a rep prize, manual adjustment of Prize Fund & multipliers).
  This means prizes could be allocated say to the top 10.
   
• Cell formatting for certain conditions (e.g. top 3, tied ranks, duplicates).
• Each challenge is stored on an individual sheet.
• Misc stats for each challenge.

 

Preview from the "City Life" challenge (click for full size):

How does the ranking work?

The basic concept is quite simple.

For each entry, it first checks whether it meets the 'Baseline' for rating and votes. If so, it looks at the rating value and ranks them accordingly, highest to lowest. Then a tiebreaker is used, which is calculated using the votes tally. This allocates a decimal number, which is unique to the # of votes. The more votes an entry has, the smaller the tiebreaker will be (and vice versa). It's then added to determine the final rank.

If an entry fails the 'Baseline' test, it's placed in a second rank. Here entries are still sorted by the rating, but appear further down the list.

The majority of the calculations are done using hidden sort columns. Really it's a bit more complex than it looks from the outside!

 

Why not just do a simple sort using the ratings?

Well, the main reason is to ensure consistency. It allows results to be decided automatically, and eliminates the need to manually split ties, or decide how many votes is a fair sample. All it requires is data to be entered accurately (in the spreadsheet).

Initially I was considering doing something even more complex. For example, to use weighted ranks based on the rating & votes for each entry. Essentially to award points to each entry, and then sort highest to lowest. But in the end, I think the 'Baseline Method' is more transparent, easier to understand, and keeps most of the sorting by the ratings.

 

Will the data be revealed for each challenge?

Yes, when results are announced, I'll include a link to the spreadsheet. This will be a read-only version uploaded to OneDrive.
 

The voting data for all challenges can be found at the following link:
ST Challenges - Results Data   (Excel spreadsheet shared on OneDrive)

 

Hope this sounds useful!

If all goes well, we aim to bring all results up-to-date this weekend.

 

[matias93 7,535]

Well, this is a beauty show of electoral engeeniering, and surely a great solution to almost every possible problem with future contests by popular vote in the site.

[rivit 3,391]

Well Cyclone,

 given your extensive efforts in coming up with something that should be able to deal with a fair few of the potential oddities of voting and the current backlog, my vote to fly this for the next 10 rounds and then take a checkpoint.

 looks good Ron

[Toothless Stitch 2,901]

Haven't had the chance to write so I'll do now: Thanks CB for your work and effort, I'm glad that you became a "member" of our team (well you're an Admin so nevermind lel). Is it just me or is this way of getting the winner quite awesome?