Results System Discussion [New Method!]

[Cyclone Boom 16,851]

UPDATE 16th October:

New ranking method announced!
https://community.simtropolis.com/forums/topic/69295-baseline-automated-results-system-bars/

Original post:

Thanks all for your concerns.

Again just to clarify, the entries were ordered using the Gallery's default rating sorting, like in this current voting album.

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)

City Life was really the first major challenge under this new system, where previously we just used total votes. It does seem quite volatile, where a single vote can dramatically change the listing order shown. It would imply that having more votes doesn't necessarily mean a higher ranking, but a combination of both factors.

I agree it'd be great to make this more transparent, so everyone knows how the order is determined.

We're very much open to flexibility, such as creating a more logical system (spreadsheet), where there's more weight for higher averages. Should this happen, we'd recalculate the order for this challenge (and if any others that follow), and allocate reputation to any 'new' winners. Also to keep it fair, the current winners would still retain their prizes.

 

[wallacet 336]

Thanks all for your concerns.

Again just to clarify, the entries were ordered using the Gallery's default rating sorting, like in this current voting album.

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)

This was really the first challenge under this new system, where previously we just used total votes. It does seem quite volatile, where a single vote can dramatically change the listing order shown. It would imply that having more votes doesn't necessarily mean a higher ranking, but a combination of both factors.

I agree it'd be great to make this more transparent, so everyone knows how the order is determined.

We're very much open to flexibility, such as creating a more logical system (spreadsheet), where there's more weight for higher averages. Should this happen, we'd recalculate the order for this challenge (and if any others that follow), and allocate reputation to any 'new' winners. Also to keep it fair, the current winners would still retain their prizes.

It seems that, this being a competition where we are supposed to rate EACH image, there shouldn't be a case where one image has more votes than the others. This will totally screw up an images ratings...like you said, if one image has 5 votes and another 10, a bad vote will totally screw over the one with 5 votes. I do not know why someone who is going through the gallery would only vote for certain images... Maybe stress that users who want to vote need to assign ratings for ALL images, in order to keep things constant and stable. I also understand that in a large gallery, people don't feel like scrolling towards then end....I do not know if the first uploaded image shows up in the gallery first or not, but maybe find some way to randomize the order in which the images show up in the gallery? That way it would give the images at the end of the gallery a better chance to not get tanked by one rating.

[feyss 1,981]

Thanks for the answer Cyclone Boom!

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)

I took a look to the current challenges and I don't think it works like this. For instance, in the sport complexes challenges, Tonraq with 10 votes and 4 stars must be showcased first in the album according to this logic. The same for the Cities Skylines challenge: the pictures are not in the good order.

So, it must be something else, but i don't know what.

[MushyMushy 3,617]

I would think that if the number of votes were a part of it that SimCoug's image would be in the rankings, since his image had like 21 votes if I remember correctly.

This is just my opinion, but while I do agree that a picture that has 1 or 2 votes and 5 stars should not win, I think that the number of votes should be irrelevant beyond a certain point. To me, that makes it seem like there's a bit of a popularity contest going on in addition to the voting for which image is actually the best (in each person's opinion). If there was a contest where, for example, Person A had 20 votes and 4.5 stars and Person B had 35 votes and 3.5 stars, and Person B won, I would think that that doesn't really represent a contest for best picture of X. Of course, the best way to solve this is to stress the importance of voting for everything, and not just a few of them.

  Edited August 25, 2015 by MushyMushy  

[Cyclone Boom 16,851]

Thanks all for the input.

Here's the full voting data for this challenge:
 

Entry Name	Album Rank	Rating	# Votes
Baie de Byron: Market Square	1	3.69	13
Hollogne	2	3.95	21
Jericho Suburbs	3	3.71	17
Hakata Ward, Kaijo	4	3.61	18
The American Dream	5	3.50	18
A Rainy Day in a Divided Society	6	3.83	18
Left off of the tourist brochure	7	3.86	21
Stadium Peninsula	8	2.70	10
EStPaul	9	3.22	9
Downtown Rockdale	10	3.33	15
Railroad Avenue, Barnanova	11	3.13	15
Urban Encroachment	12	3.38	13
Martian downtown	13	2.93	15
Castle of the Bay, capital city of the Schwegau	14	3.00	16
Start of the M134; Annuq	15	3.44	16
E2-1000 Series Shinkansen	16	2.67	15
Coastbourg's city-ish life	17	3.45	20
Neo Kontreal	18	3.17	18
Summer night in Manhattan  (Late entry)	19	3.17	6
Prospect Island	20	1.85	13
BIG CITY DOWNTOWN	21	1.93	14
Ecotopia Part V	22	1.58	12

Total votes: 333

Average # of votes: 15.14

Average rating: 3.14

 

Hmm, can anyone see any correlation here?

The ratings shown are based on the mean (total stars / # of votes). But I'm almost wondering if it's also considering the median, which ignores outliers (e.g. 1 star ratings).

E.g. There could be an image rated 3.80, which has 10 votes:

5, 5, 5, 5, 5, 4, 4, 3, 1, 1

The two 1's would drag the mean down. However, if the calculation ignores this, it could be ranking the image higher because of the other values. Or maybe it's using the number of 4 & 5 star ratings -- giving higher weighting based on data we can't see.

Of course, this is only speculation, and I guess it'd be difficult to prove without some serious testing.

 

If the plan is to manually decide the results in future, I like the sound of prioritising the rating, making this more important than # of votes. It'd be ideal to strike a balance between the two. We'd also need to decide how to rank images with the same rating, but a different # of votes.

[RipRap 4,400]

Actually, I think the image ordering is broken. The thing hasn't budged starting when the voting restarted. I remember Olseannes (or whatever it was that ended up being 2nd) being ahead of me by 4 places when it restarted and ended up at the same spot when we closed the contest. 

I think we need to manually count the votes rather than this system. It's complicated and broken.

[SimCoug 5,972]

I agree, the album order appears to be broken or poorly coded.  I'd hate to have to require each contest be 'hand counted' (that would be a lot of extra work for people)... I wonder if there is a way to get this fixed by the site hosting company (although it appears they have bigger fish to fry at the moment). 

[art128 4,618]

Uhm. I don't understand... Why make things extremely complicated with some badass maths formula when you can just use the regular voting system we had before, if it's available on this version of the website?

[Hamish 1,369]

Oh, seems like we need to re-think this. It was working well in testing. We'll re-sort these and give the winners who lost out some reputation and recognition. 

Sorry guys, I thought it was working alright in testing. I'd rather not go back to the old way, the new gallery's are nice to look through. We'll work it out and it will be fairer in the next round. While looking at ratings, for now, assume it's the image with the highest rating which wins.

[timmie 988]

Not that it's very important to me, but when are we going to get the results from the challenge of last week (i.e. Transport Hubs and Interchanges)? Seeing as today is they last day of the new challenge and the results of the previous one still haven't been posted

[Cyclone Boom 16,851]

Not that it's very important to me, but when are we going to get the results from the challenge of last week (i.e. Transport Hubs and Interchanges)? Seeing as today is they last day of the new challenge and the results of the previous one still haven't been posted

Yes apologies for the delay -- it's fallen way behind schedule.

We'll aim to announce the results shortly for all completed challenges, and get everything back synchronised. This will include the reworked results for City Life, based on a manual sort of the ratings. In the future, we'll aim to publish results the day after voting closes (at the latest).

A couple of things need establishing:

1)  How to handle images with a high rating (say 5 stars), but a low # of votes compared to other images?

2)  What to do in the event of ties:
    a) Images with the same rating but a different # of votes -- which should rank higher?
    b) Images with identical votes and rating -- how to decide the winner?

I agree this shouldn't be a popularity contest, far from it. Although not sure it's fair if an image won purely based on the rating number, with a very small sample of votes. Maybe each needs a minimum number to 'qualify', perhaps based on the ave number of votes per image? All this data would be already available from the manual sorting.

As for ties, should both be awarded the same prize, or use a tiebreaker. For example, the earlier submitted image (since it'd be placed at the bottom of the album).

Don't want to complicate this unnecessarily, just make the results as fair & clear as possible.

Any ideas welcome...

[matias93 7,535]

Since this is a type of Range (or Score) Voting System, maybe the soft quorum method could be useful to resolve part of it:

http://rangevoting.org/BetterQuorum.html 

[timmie 988]

@Cyclone Boom, excellent points. It does seem that most people vote for all the pictures in the album. For the Transport Hubs, the number of votes per image ranges between 11 and 14. That's not a lot, so I can imagine that if you set the limit at 10 votes, you could eliminate a lot of pictures in future competitions.

About ties, I don't know if it is possible, but you could use a Eurovision-like system: in the Contest, countries with more 12 points, 10 point, etc. are ranked higher in case of a ties. On the other hand, the number of voters in the Contest is quite big, that's not the case for the competitions here.

  Edited September 20, 2015 by timmie