Jump to content
Cyclone Boom

Results System Discussion [New Method!]

13 posts in this topic Last Reply

Recommended Posts

UPDATE 16th October:

New ranking method announced!
http://community.simtropolis.com/forums/topic/69295-baseline-automated-results-system-bars/


Original post:

Thanks all for your concerns.

Again just to clarify, the entries were ordered using the Gallery's default rating sorting, like in this current voting album.

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)


City Life was really the first major challenge under this new system, where previously we just used total votes. It does seem quite volatile, where a single vote can dramatically change the listing order shown. It would imply that having more votes doesn't necessarily mean a higher ranking, but a combination of both factors.

I agree it'd be great to make this more transparent, so everyone knows how the order is determined. :)

We're very much open to flexibility, such as creating a more logical system (spreadsheet), where there's more weight for higher averages. Should this happen, we'd recalculate the order for this challenge (and if any others that follow), and allocate reputation to any 'new' winners. Also to keep it fair, the current winners would still retain their prizes.

 

Share this post


Link to post
Share on other sites

Thanks all for your concerns.

Again just to clarify, the entries were ordered using the Gallery's default rating sorting, like in this current voting album.

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)


This was really the first challenge under this new system, where previously we just used total votes. It does seem quite volatile, where a single vote can dramatically change the listing order shown. It would imply that having more votes doesn't necessarily mean a higher ranking, but a combination of both factors.

I agree it'd be great to make this more transparent, so everyone knows how the order is determined. :)

We're very much open to flexibility, such as creating a more logical system (spreadsheet), where there's more weight for higher averages. Should this happen, we'd recalculate the order for this challenge (and if any others that follow), and allocate reputation to any 'new' winners. Also to keep it fair, the current winners would still retain their prizes.

It seems that, this being a competition where we are supposed to rate EACH image, there shouldn't be a case where one image has more votes than the others. This will totally screw up an images ratings...like you said, if one image has 5 votes and another 10, a bad vote will totally screw over the one with 5 votes. I do not know why someone who is going through the gallery would only vote for certain images... Maybe stress that users who want to vote need to assign ratings for ALL images, in order to keep things constant and stable. I also understand that in a large gallery, people don't feel like scrolling towards then end....I do not know if the first uploaded image shows up in the gallery first or not, but maybe find some way to randomize the order in which the images show up in the gallery? That way it would give the images at the end of the gallery a better chance to not get tanked by one rating.

Share this post


Link to post
Share on other sites

Thanks for the answer Cyclone Boom!

At the moment, it's not clear what the formula is, or how this exactly works. But it seems to take into consideration the:

a) Number of votes
b) Average rating
c) Submission date (in event of ties, the earlier submission is listed higher)

I took a look to the current challenges and I don't think it works like this. For instance, in the sport complexes challenges, Tonraq with 10 votes and 4 stars must be showcased first in the album according to this logic. The same for the Cities Skylines challenge: the pictures are not in the good order.

So, it must be something else, but i don't know what.

Share this post


Link to post
Share on other sites

I would think that if the number of votes were a part of it that SimCoug's image would be in the rankings, since his image had like 21 votes if I remember correctly.

This is just my opinion, but while I do agree that a picture that has 1 or 2 votes and 5 stars should not win, I think that the number of votes should be irrelevant beyond a certain point. To me, that makes it seem like there's a bit of a popularity contest going on in addition to the voting for which image is actually the best (in each person's opinion). If there was a contest where, for example, Person A had 20 votes and 4.5 stars and Person B had 35 votes and 3.5 stars, and Person B won, I would think that that doesn't really represent a contest for best picture of X. Of course, the best way to solve this is to stress the importance of voting for everything, and not just a few of them.


  Edited by MushyMushy  

Share this post


Link to post
Share on other sites
  • Original Poster
  • Thanks all for the input. :)

    Here's the full voting data for this challenge:
     

      Entry Name

    Album Rank

    Rating

    # Votes

      Baie de Byron: Market Square

    1

    3.69

    13

      Hollogne

    2

    3.95

    21

      Jericho Suburbs

    3

    3.71

    17

      Hakata Ward, Kaijo

    4

    3.61

    18

      The American Dream

    5

    3.50

    18

      A Rainy Day in a Divided Society

    6

    3.83

    18

      Left off of the tourist brochure

    7

    3.86

    21

      Stadium Peninsula

    8

    2.70

    10

      EStPaul

    9

    3.22

    9

      Downtown Rockdale

    10

    3.33

    15

      Railroad Avenue, Barnanova

    11

    3.13

    15

      Urban Encroachment

    12

    3.38

    13

      Martian downtown

    13

    2.93

    15

      Castle of the Bay, capital city of the Schwegau

    14

    3.00

    16

      Start of the M134; Annuq

    15

    3.44

    16

      E2-1000 Series Shinkansen

    16

    2.67

    15

      Coastbourg's city-ish life

    17

    3.45

    20

      Neo Kontreal

    18

    3.17

    18

      Summer night in Manhattan  (Late entry)

    19

    3.17

    6

      Prospect Island

    20

    1.85

    13

      BIG CITY DOWNTOWN

    21

    1.93

    14

      Ecotopia Part V

    22

    1.58

    12


    Total votes: 333

    Average # of votes: 15.14

    Average rating: 3.14

     

    Hmm, can anyone see any correlation here? o.O

    The ratings shown are based on the mean (total stars / # of votes). But I'm almost wondering if it's also considering the median, which ignores outliers (e.g. 1 star ratings).


    E.g. There could be an image rated 3.80, which has 10 votes:

    5, 5, 5, 5, 5, 4, 4, 3, 1, 1

    The two 1's would drag the mean down. However, if the calculation ignores this, it could be ranking the image higher because of the other values. Or maybe it's using the number of 4 & 5 star ratings -- giving higher weighting based on data we can't see.

    Of course, this is only speculation, and I guess it'd be difficult to prove without some serious testing.

     

    If the plan is to manually decide the results in future, I like the sound of prioritising the rating, making this more important than # of votes. It'd be ideal to strike a balance between the two. We'd also need to decide how to rank images with the same rating, but a different # of votes.

    Share this post


    Link to post
    Share on other sites

    Actually, I think the image ordering is broken. The thing hasn't budged starting when the voting restarted. I remember Olseannes (or whatever it was that ended up being 2nd) being ahead of me by 4 places when it restarted and ended up at the same spot when we closed the contest. 

    I think we need to manually count the votes rather than this system. It's complicated and broken. :P

    Share this post


    Link to post
    Share on other sites

    I agree, the album order appears to be broken or poorly coded.  I'd hate to have to require each contest be 'hand counted' (that would be a lot of extra work for people)... I wonder if there is a way to get this fixed by the site hosting company (although it appears they have bigger fish to fry at the moment). 

    Share this post


    Link to post
    Share on other sites

    Uhm. I don't understand... Why make things extremely complicated with some badass maths formula when you can just use the regular voting system we had before, if it's available on this version of the website?

    Share this post


    Link to post
    Share on other sites

    Oh, seems like we need to re-think this. It was working well in testing. We'll re-sort these and give the winners who lost out some reputation and recognition. 

    Sorry guys, I thought it was working alright in testing. I'd rather not go back to the old way, the new gallery's are nice to look through. We'll work it out and it will be fairer in the next round. While looking at ratings, for now, assume it's the image with the highest rating which wins.

    Share this post


    Link to post
    Share on other sites

    Not that it's very important to me, but when are we going to get the results from the challenge of last week (i.e. Transport Hubs and Interchanges)? Seeing as today is they last day of the new challenge and the results of the previous one still haven't been posted :)

    Share this post


    Link to post
    Share on other sites
  • Original Poster
  • Not that it's very important to me, but when are we going to get the results from the challenge of last week (i.e. Transport Hubs and Interchanges)? Seeing as today is they last day of the new challenge and the results of the previous one still haven't been posted :)

    Yes apologies for the delay -- it's fallen way behind schedule.

    We'll aim to announce the results shortly for all completed challenges, and get everything back synchronised. This will include the reworked results for City Life, based on a manual sort of the ratings. In the future, we'll aim to publish results the day after voting closes (at the latest). ;)


    A couple of things need establishing:

    1)  How to handle images with a high rating (say 5 stars), but a low # of votes compared to other images?

    2)  What to do in the event of ties:
        a) Images with the same rating but a different # of votes -- which should rank higher?
        b) Images with identical votes and rating -- how to decide the winner?


    I agree this shouldn't be a popularity contest, far from it. Although not sure it's fair if an image won purely based on the rating number, with a very small sample of votes. Maybe each needs a minimum number to 'qualify', perhaps based on the ave number of votes per image? All this data would be already available from the manual sorting.

    As for ties, should both be awarded the same prize, or use a tiebreaker. For example, the earlier submitted image (since it'd be placed at the bottom of the album).

    Don't want to complicate this unnecessarily, just make the results as fair & clear as possible.

    Any ideas welcome... :)

    Share this post


    Link to post
    Share on other sites

    @Cyclone Boom, excellent points. It does seem that most people vote for all the pictures in the album. For the Transport Hubs, the number of votes per image ranges between 11 and 14. That's not a lot, so I can imagine that if you set the limit at 10 votes, you could eliminate a lot of pictures in future competitions.

    About ties, I don't know if it is possible, but you could use a Eurovision-like system: in the Contest, countries with more 12 points, 10 point, etc. are ranked higher in case of a ties. On the other hand, the number of voters in the Contest is quite big, that's not the case for the competitions here.


      Edited by timmie  

    Share this post


    Link to post
    Share on other sites
    Guest
    This topic is now closed to further replies.

    • Similar Content

      • By RandyE
        How would you rate each of the city-builder games you've actually played.  Listed below I've included the games generally covered on ST.  If there is a city-builder game that you play that is not on the list then add it to your list of games actually played and give it a rating.
        The ratings are from 1 to 10.  There are 9 listed game titles and 1 'other' which you can name or choose not to.  If you have not actually played the game give it a rating of n/a, or simply leave it off your list.
        A rating of 1 would be what you consider the least, a rating of 10 would be what you consider the best.  You can either rate them as a list from 1 to 10, or rate each one individually between 1 and 10.  Either way it should be obvious which way you decided to rate the games.
        You can copy and paste the text below to enter your ratings.  You can also quote the text below and enter your ratings in the quote box.
        +++++++++++++++++++++++++++++++++++++++++++++++++
        1. Cities Skylines  ___
        2. SimCity (2013) ___
        3. SimCity 4 (Rush Hour) ___
        4. SimCity 3000 ___
        5. Cities XXL ___
        6. SimCity Buildit  ___ 
        7. City Life ___
        8. SimCity 2000 ___
        9. SimCity Societies ___
        10. Other ___
        ++++++++++++++++++++++++++++++++++++++++++++
      • By Fantozzi
        There was a time uploads on the STEX were rated simply with 1 -5 stars and without a written comment and nobody did know - why someone only rated something 3 stars.
        Now you need to give a comment. And I thought this comment is to explain the rate you give. F.e. if someone uploads a yellow house and you don't like the color yellow - you give 3 stars and and write: this house is yellow and i don't like it yellow, blue it would be much better (as you could before without telling your motivation).
        So now other users will see - Ah, that's the reason fantozzi rated this 3 stars, but me, I love the colour yellow, so why should I care about fantozzis rating?
        As me, being objective, I can try, but it's almost impossible, as I'm not almighty in my knowledge and capabilities and on judging content I'm not always 'right' in the sense of public opinion or the opinion of everyone and therefore my opinion might be 'strange' to others. That's a result of me being a subject, an individual, I think. I can fail. Even on opinion or judging. So I can't always do right to everybody.
        Now I see subjective ratings on uploads will be deleted.
        So objective criteria is needed - criteria that can be applied on all ratings the same way and exclude subjective judgements like about which colour is better for a building.
        Or I won't rate anymore anything here - as beeing attacked by moderators just for stating my subjective (and maybe wrong - in a public view) opinion, I don't like this.
        So please tell me what I am allowed to say and what not when rating.
      • By korver
        Full size version (~4MB) of the GIF on the first comment.
         
        Please keep in mind that a time lapse animation was one of the official suggested ideas for this challenge on the forum thread.
        An animated GIF displaying my recreation of the Holy Monastery of St. Nicholas in Meteora, Greece along with some of the surrounding scenery.  Wanting to feel closer to heaven, the locals would build their monasteries high in the sky, perched precariously on the edge of sheer granite rock cliffs - truly an amazing sight to behold.
        One of the frames in this picture (fall) was originally featured in my "Greece - Part II" update in my CJ, True Earth, so for this GIF I decided to showcase Meteora in its entirety with Spring, Summer, and Winter frames as well. Each frame showcases extensive custom lotting, intricate MMPing, and custom content creation (various terrain mods created for this picture + a number of new MMPs made/modded heavily)
        -
        The first frame starts off with summer in Meteora. Predictably, this time is the most popular season for tourists to come visit Meteora. The harvest season that begins in late summer is a big attraction as well for tourists.
        The second frame displays some of Meteora's famous fall foliage. Oaks and maples among others will come alive, displaying various shades of red, orange, and yellow. This coupled with the much more comfortable temperatures makes it a great time to come visit.
        The third frame shows off Meteora blanketed under a layer of snow in the midst of a snowfall. Despite Meteora's relatively close proximity to the Ioanian and Aegean Seas, it's just high enough in the mountains (1,000-2,000 feet above sea level) to receive regular snow in the winter. Additionally, the mountains will essentially "block out" gusts from the sea, allowing for temperatures to vary wildly. For those wanting to visit Meteora in the wintertime, be prepared for frigid temperatures and constant frost.
        Before looping, the fourth and final frame shows Meteora in the spring. The cold winters often delay Springtime for a bit, but when it comes it can truly be quite beautiful. Be prepared for fog/rain showers however, a common site during this time of the year.
      • By ST Challenges
        Voting Rules
        Votes may be cast by rating images inside the album. You may rate all entries once at any rating value (1-5). All votes are anonymous. We encourage you not to rate your own entry. Try to vote on all other entries, as this will help ensure the results are evenly distributed.  
        Suggested Voting Criteria
        Here are some ideas to help you evaluate each entry when voting:
        How well does the animation portray the chosen sequence of action? How accurate are the transitions between frames? Does it appear interesting? The effort you think has gone into producing the animation. To what extent does it create visual appeal?  
        The winners will be the top 3 ranked images after a 6 day voting period. Reputation prizes will be awarded based on the number of entries, and the overall average rating. See this topic for more info.
        For this  Extended  challenge, rep prizes will be given for the top 3 placed entries, along with points for the "Overall" and "Weekly & Extended" leaderboards.
         
      • By ST Challenges
        Voting Rules
        Votes may be cast by rating images inside the album. You may rate all entries once at any rating value (1-5). All votes are anonymous. We encourage you not to rate your own entry. Try to vote on all other entries, as this will help ensure the results are evenly distributed.  
        Suggested Voting Criteria
        Here are some ideas to help you evaluate each entry when voting:
        How well the image portrays the mood & atmosphere of the city. The extent of creativity used within the scene, or throughout the image's composition. The amount of attention to detail, adding to the overall appeal. How much effort you think has gone into creating the image. Does the image stand out as being visually interesting?  
        The winners will be the top 3 ranked images after a 10 day voting period. Reputation prizes will be awarded based on the number of entries, and the overall average rating. See this topic for more info.
        For this  Monthly  challenge, rep prizes will be given for the top 5 placed entries, along with points for the "Overall" and "Monthly & Custom Content" leaderboards.
         
    • Recently Browsing   0 members

      No registered users viewing this page.

    ×

    Help Keep Simtropolis Online, Open & Free!

    stexcollection-header.png

    Get the best of the best from the STEX!

    Expand your city with the best from the Simtropolis Echange. Make a donation and get one or all three discs today!

    Make a Donation, Get a Gift!

    We need to continue to raise enough money each month to pay for expenses which includes hardware, bandwidth, software licenses, support licenses and other necessary 3rd party costs.

    By way of a "Thank You" gift, we'd like to send you our STEX Collector's DVD. It's some of the best buildings, lots, maps and mods collected for you over the years. Check out the STEX Collections for more info.

    Each donation helps keep Simtropolis online, open and free!

    Thank you for reading and enjoy the site!

    More About STEX Collections