bugs Simtropolis 9.0.x - Bugs & Issues Report Thread

[Indiana Joe 1,750]

Seems to be working well so far!  The STEX takes a bit longer to load than other pages though.  I'd be curious to see number-of-visitor statistics over the last few days, if there's a time when the server overload suddenly started happening.

Matt

[Indiana Joe 1,750]

I am double posting, take me away if you must.

 

Site's feeling pretty unsteady, getting a lot of 502's/504's.  As someone undertaking a project that involves searching the STEX extensively, I feel the need to suggest/request bringing back some of those restrictions for a bit, and limiting it to members again if that has been lifted.

[CorinaMarie 29,605]

16 minutes ago, Indiana Joe said:

I am double posting, take me away if you must.

No worries. We pretty much abolished that rule since there are many times when double posting is a useful thing.

 

16 minutes ago, Indiana Joe said:

Site's feeling pretty unsteady, getting a lot of 502's/504's.

522s/524s I presume.

 

16 minutes ago, Indiana Joe said:

I feel the need to suggest/request bringing back some of those restrictions for a bit, and limiting it to members again if that has been lifted.

Yeah, that's likely what we'll need to do, but we want to at least keep running the current test to see if it does keep the site from totally crashing and not coming back. So, please bear with us as the only way to tell is to give the server enough rope to see if it does or doesn't hang itself.

Also, as mentioned in my Announcement, I highly recommend making a copy of any post you are about to make prior to clicking the submit button.

[CorinaMarie 29,605]

Announcement updated.

We've again blocked all guests as the site passed the test for not crashing, but was unbearably laggy. Let us know how the response speed is now. (Or rather in 10 minutes from now and afterward to give the site time to cool down from the guest's activity.)

[jeffryfisher 2,176]

On 9/27/2018 at 4:26 PM, CorinaMarie said:

the site passed the test for not crashing, but was unbearably laggy.

I've been brushing up on web site performance tuning and bot management. This "How to Detect..." article looks useful. I did some .htaccess programming back in 2006-09. Let me know if I can help.

[CorinaMarie 29,605]

22 minutes ago, jeffryfisher said:

I did some .htaccess programming back in 2006-09. Let me know if I can help.

I've replied to your PM (moments ago). For the detailed details let's keep that part in the private convo.

[catty-cb 4,394]

My site got hit by a denial-of-service (DoS) attack a few years ago, obviously a big difference between my site (which is/was mainly reference based so few users) and Simtropolis also the DoS attack wasn't aimed at my site I was collateral damage as I'm on a shared server and they were trying to attack another website.

Not sure just how things are set up on Simtropolis but what saved my site was the commercial firewall I had installed

https://www.rsjoomla.com/joomla-extensions/joomla-security.html

It took us about 24 hours to gain access to the site once logged on I switched the firewall from passive watching (I controlled who got banned) over to the firewall controlling the site and auto-banning anything it didn't like the look of ... within about 15 minutes of the switchover it was IP banning a couple of hundred addresses a minute over the next few hours that slowly started to drop and by the next day it was down to half-a-dozen bans a minute and another day later it was back to normal operations ... I never swapped it back ... it does mean I occasionally have to un-ban someone that the firewall didn't like the look of, but its worth the hassle.

As I said I don't know how Simtropolis web software handles these issues but if its possible to add more firewall security to the site and its a question of cost I'd be happy to donate towards it and I'm sure there are other people on the site that would be willing to contribute as well.

-catty

[CorinaMarie 29,605]

@catty-cb Thanks.

I've copied your message directly into the chat channel with Dirk so it will be there when he peeks in.

[ricb 19]

The last two days I've gotten access forbidden messages when navigating to Simtropolis.  Usually getting access late in the day.  I'm on Pacific US time.

[CorinaMarie 29,605]

10 minutes ago, ricb said:

The last two days I've gotten access forbidden messages when navigating to Simtropolis.  Usually getting access late in the day.  I'm on Pacific US time.

As mentioned in the Announcement, Simtropolis has been under heavy web traffic attack which causes the site to become unresponsive after a while. When it goes down completely we change the access code to 403 Forbidden to let the server recover. When you see that particular message it means we should be back in about 20 minutes. On the other hand, if you get a Cloudflare 522 or 524 error then it means one of us staff needs to set it to 403. And, unfortunately, we can't be here all the time so it may have to wait for one of us to get home from work to do that.

@Cyclone Boom discovered a likely cause that we should be able to mitigate. The announcement will be updated when we know for sure.

[Cyclone Boom 16,877]

16 minutes ago, ricb said:

The last two days I've gotten access forbidden messages when navigating to Simtropolis.  Usually getting access late in the day.  I'm on Pacific US time.

As per the site announcement we're currently experiencing outages due to degraded site performance. The 403 Forbidden messages you'll have noticed are triggered when we attempt to revive the site. Basically it needs some time to cool down and then the server is able to recover. Usually this will last for 20-30 minutes at a time, and from where the site will return as it was. Rest assured you've not been banned or anything.

We're continuing to investigate the issues and trying different things to resolve the problems.

[ricb 19]

I'm going to make a small donation.  It is the only way I can help with the tech issues.  Maybe towards security equipment or software.  It will have to be small as I am on a fixed income.  
Simtropolis has been a source of great ways to help feed my SimCity addiction so I am glad to do it.

[redfox85 659]

tin foil hat .... I'm assuming the site is Canadian hosted ... and its interesting to note the sheer number of Canadian companies that have had there websites DDOS and randsomware launched against PoS systems over the past couple weeks. That includes presumably two banks who lost all online service for most of a day. Simtrops high usage just may have got stuck it in the middle.

[CorinaMarie 29,605]

8 hours ago, ricb said:

I'm going to make a small donation.

Thank you! Any amount helps cause it all adds up.

I've updated the Announcement again this morning with the latest news on our progress. I'm deliberately not saying specifically which utilities we've replaced nor what settings we've used as we don't want this information to fall into the wrong hands. It did turn out that an exploit was found outside of ST and used for one of them last month and we believe that was potentially the bulk of our problem here too. @Cyclone Boom discovered that the other day and we've implemented a replacement which does not have that same vulnerability. Prior to this discovery we'd tweaked other server settings which we'll need to revisit since they prolly didn't need altered (tho we didn't know that at the time).

With regard to overall donations, we've barely been covering expenses each month while occasionally having a shortfall. It seemed like the exact wrong time to mention this while the site was so sluggish and going offline multiple times per day. Now that we believe we are close to having it resolved it might be a good time to come up with ideas to hopefully increase the donations.

[Doni98 80]

All my modified mods disappeared! And the Search section do not find mods!

[T Wrecks 10,209]

Just stopping by to express my gratitude to you guys - @CorinaMarie, @Cyclone Boom and of course our benevolent @Dirktator - for going through this nightmare and getting our community back online again. Thank you for your endurance and dedication!

[Tarkus 6,932]

Indeed, I'll second T Wrecks.  I know you all had to be at wit's end with all the sustained technical issues--thank you for your perseverance through all that, and welcome back!

-Tarkus

[Cyclone Boom 16,877]

As of this post, STEX files are currently temporarily hidden to staff only. That explains why you will receive a 2D175/1 error (or similar) when attempting to view a file right now. Rest assured the file attempting to be accessed is intact at the source and just not viewable yet.

After @Dirktator has made additional adjustments on the server side through the night, early indications from bringing the site back online suggest performance has notably improved this morning, even with database background processes still progressing. Therefore very shortly we shall be making the STEX fully open and accessible again. Then all being well it could just be a bit more fine-tuning here and there, before we can finally call it mission accomplished.

Stand by everyone.

[Tyberius06 12,189]

Same here! Thanks all of you to bring Simtorpolis back! 

- Tyberius

[AsimPika3172 1,136]

Thanks for REVIVING this forum! Keep indexing guys!  

[Cyclone Boom 16,877]

Update:

The site has returned and the STEX is now back fully online to members and guests. We're not completely sure how this will go, but early signs of performance are promising, especially with 2 database processes still completing. We'll continue to closely monitor performance and take measures necessary in response.

Thanks all.

[T Wrecks 10,209]

Although it's probably not possible, I'd sure like to know what the problem actually was, and I like the thought of you rubbing it in that inept technician's* face, too.

_________________
*It was posted on the Simtrop Twitter account that a technician was called for help, but they only started the same processes that Cori and CB had already tried, and claimed that the problem was due to the system being generally too slow to handle the load and the software version not being current, when the same system was running perfectly fine just a few days before!

[TekindusT 3,809]

May I also praise the new approach to communication the Admins took this time. ST has gone through several severe outages throughout its history. Given that Dirk and the other techies have their hands full enough by solving the outages, updates on the site status were usually scarce if any. This time, however, Cori has been very active in Twitter, and those with Twitter account could follow almost to the hour the progress on the site and what was going on. That's been really useful (and soul-soothing in a way), appreciate that!!

[Dirktator 2,320]

6 hours ago, T Wrecks said:

Although it's probably not possible, I'd sure like to know what the problem actually was, and I like the thought of you rubbing it in that inept technician's* face, too.

We're still working out the exact details, and I'd still like to give the server a couple days to see if it's solved, but I will detail and summarize what had happened for those who want to know. You can see a bit of a snapshot of what we had gone through via Cori's announcement and twitter updates.

5 hours ago, TekindusT said:

May I also praise the new approach to communication the Admins took this time. ST has gone through several severe outages throughout its history. Given that Dirk and the other techies have their hands full enough by solving the outages, updates on the site status were usually scarce if any. This time, however, Cori has been very active in Twitter, and those with Twitter account could follow almost to the hour the progress on the site and what was going on. That's been really useful (and soul-soothing in a way), appreciate that!!

Not only has she been our tireless communicator, Cori's also been an integral help in the effort to bring things back up by testing and retesting our systems, taking backups offsite, documenting system and server settings, and  doing a tonne of  research for us --- all on top of being a moral support through it all -- I'm glad to have @CorinaMarie and @Cyclone Boom in the trenches for this one!

[jeffryfisher 2,176]

On 10/9/2018 at 9:36 AM, TekindusT said:

This time, however, Cori has been very active in Twitter, and those with Twitter account could follow almost to the hour the progress on the site and what was going on.

Even without being a Twit, I was able to tune in (at the very end, after I went to SC4D and saw a link to the Simtropolis Twitter page).

I hope that Cori, CB and Dirk will add me to their short list of techs willing to help. Though most of my decades of DB and web site experience has been design, I've done my share of trouble-shooting, security, SEO, reliability and performance optimization. That includes having written .htaccess rules (that were eventually hoisted up to Apache main config files). Sadly, my PM to CB volunteering to help was eaten by one of the crashes 

As long as I am unemployed, I can donate much more time than money. That, and I have a passion for unraveling mysteries, exterminating bugs, and solving problems. So if you have any more trouble, please pull me inside the tech loop where I can see logs.

I still suspect that the changes in image storage (following the offsite debacles) is contributing to site stress. Simtropolis probably needs a new strategy for storing and caching large image files (or scrubbing them from pages served to robots).

PS: I second TWrecks' motion to post or point at the deep-in-the-weeds tech talk so I can see where I might join the fight. If there can be a continuing effort to defend against Bad Robot etc, count me in.

[CorinaMarie 29,605]

Since we are back to running quite normal and the re-re-indexing is not taxing the system I'm going to hide the Announcement which has all the various day by day updates. Y'all won't be able to get to it once I make it inactive so I'm tossing a copy of it here in a spoiler box for those who might later be curious.

Spoiler

 

Update: 09 Oct, 2018 @ 19:47 EDT

Having run so well all day today, but with the STEX so slow we've decided to do the re-indexing again. The reason being is while that was running the first time we toggled the STEX completely off with the idea that'd save time and it could be done later. Turns out we cannot run it on individual tables so the whole process has to be initiated again so it will include the STEX. 

We really only expect minor slowdowns (if any) since this is something that has been done in the past each time the board software was updated. It does process the files concurrently, but it does them in batches of a little bit of each one so we doubt there will be any more noticeable affect than has been seen in the past 18 hours with the post re-indexing running.

Update: 09 Oct, 2018 @ 08:28 EDT

The site has returned and the STEX is now back fully online to members and guests. We're not completely sure how this will go, but early signs of performance are promising, especially with 2 database processes still completing. We'll continue to closely monitor performance and take measures necessary in response.

Update: 09 Oct, 2018 @ 05:34 EDT

As of this update, STEX files are currently temporarily hidden to staff only. That explains why you will receive a 2D175/1 error (or similar) when attempting to view a file right now. Rest assured the file attempting to be accessed is intact at the source and just not viewable yet.

After Dirk has made additional adjustments on the server side through the night, early indications from bringing the site back online suggest performance has notably improved this morning, even with database background processes still progressing. Therefore very shortly we shall be making the STEX open and accessible to everyone once again. Then all being well it could just be a bit more fine-tuning here and there, before we can finally call it mission accomplished.

Update: 07 Oct, 2018 @ 18:23 EDT

A multitude of things went wrong, but we believe we are back with 99.99732% of the content in all areas. Due to how we had to perform the recovery this time, it's possible a portion of things such as STEX uploads or attachments on exactly 30 Sept, 2018 might be missing and any posts, PMs, status updates, etc (but not STEX files & attachments) after 09:17 AM EDT on 05 Oct, 2018 will be missing. The site was briefly operational a few times after my full database backup on 05 Oct and posts, PMs, status updates, new member registrations, etc from there and after are lost.

We do apologize for any loss and the inconvenience of this whole mess.

Update: 06 Oct, 2018 @ 09:02 EDT

Things suddenly went very wonky and the site has crashed.

Update: 06 Oct, 2018 @ 09:01 EDT

We were deliberately on and offline for the past hour whilst tweaking a few more things. Here's hoping it all goes well.

Update: 05 Oct, 2018 @ 09:54 EDT

We're back finally. Y'all prolly noticed the 403 Forbidden error for the past several hours. Normally (if any of this can be called normal) we only needed to force that error for 20 - 30 minutes to allow the server to recover. This time we've had to leave it that way for the duration of this recent problem.

Here's what happened: Late yesterday afternoon we attempted to optimize the database table sizes using a configuration wizard with the idea to improve access times.  However, this resulted in the inability to start all. Soft rebooting didn't help so a hard, full system reboot was the only option.  At this point we found several tables got corrupted.  Working from backup copies CB has spent the night piecing everything back together.

We are almost, but not quite, totally certain everything is restored to how it was prior to this catastrophe. We are now back to where we were before this situation and we'll still need to try to balance some of the settings. We are hoping this can be done with limited, random down time.  

As has been my suggestion during this whole kerfluffle, I recommend doing a copy & paste to a text editor on your comp for any lengthy post prior to clicking the submit button so you have a copy should ST crash while attempting to post.

Update: 04 Oct, 2018 @ 06:03 EDT

Overall the performance seems to be vastly improved. Jumping from forum to forum to the STEX to the home page to individual posts seems to be very fast most of the time, but slightly delayed (10 - 40 seconds) about 10% of the time. That's way better than we've had for a while. However, when I checked Dirk's chat channel it turns out we did have another crash about an hour after I went to bed. It's then been up and running all night without another crash and still nicely responsive this morning. I personally believe we now need only do some fine tuning of the maximum concurrent requests and the number of virtual servers settings.

Update: 03 Oct, 2018 @ 21:30 EDT

Tis been nearly an hour with the site behaving better than it has in the past two weeks. While it's too early to tell if we've truly fixed it we are hopeful. So, we've also increased the number of concurrent downloads to 2 for guests (back to where it was initially) and to 10 for members which we feel should be plenty sufficient.

Update: 03 Oct, 2018 @ 20:37 EDT

Zippy zoom fast for the past 10 minutes. We hope this continues and the site remains stable.

Update: 03 Oct, 2018 @ 18:52 EDT

Dirk has made some significant changes to one of the utilities the server uses. We hope this will prevent the high volume web traffic we suspect has been causing the site to lag (and eventually crash).

Update: 01 Oct, 2018 @ 19:49 EDT

Well, down again went ST and so we've decided that the removal of the Search Box didn't help and we are putting it back. We are also bringing the Gallery back too since hiding it hasn't seemed to help either. Bottom line, we will continue to bring the site back manually each time it crashes until a resolution can finally be reached. Grrrr.

Update: 01 Oct, 2018 @ 12:00 EDT

The implemented site access permission rules from yesterday have been amended. These should now fully block overzealous traffic from accessing Simtropolis. We'll monitor progress to see whether this improves site responsiveness.

Update: 01 Oct, 2018 @ 07:27 EDT

In another attempt to diagnose possible causes we've temporarily disabled the local site search function. The idea being to reduce load on the database should that be a contributing factor to the slow performance. For now we recommend using Google and a custom command to filter results from ST.

Like so:

site:simtropolis.com <search term>

Update: 30 Sept, 2018 @ 17:59 EDT

Cyclone Boom has created a much more robust set of instructions for the site rules which were implemented moments ago. For now the STEX will remain accessible and both members and guests are at normal status with regard to permissions.

Update: 30 Sept, 2018 @ 16:11 EDT

So, the site went offline twice this morning (and was brought back) and now twice this afternoon (also brought back). We are currently considering what to try next. We very much do not want to keep blocking guests so we are considering taking the STEX offline.

Update: 29 Sept, 2018 @ 22:44 EDT

We are attempting a new way to thwart the web traffic attack on Simtropolis. Guests are re-enabled with download access to the STEX. Also, we've put the members download speed back to 2,048 kB/s and guests increased to 1,024 kB/s (it was previously 500 kB/s).

Update: 29 Sept, 2018 @ 10:22 EDT

Site went down again. I've reset it and blocked all guests for now.

Update: 29 Sept, 2018 @ 08:48 EDT

The site was down when I logged on this morning. I've reset it and now we are back running. Seems there is more to research.

Update: 28 Sept, 2018 @ 15:26 EDT

With the site responding much better we have removed the 10 second delay for downloads for both guests and members.

Update: 28 Sept, 2018 @ 13:39 EDT

Turns out we were under attack. No details will be provided, but we believe we have that resolved.

Update: 28 Sept, 2018 @ 13:19 EDT

We've tweaked some more and guests are re-enabled with STEX downloading available.

Update: 27 Sept, 2018 @ 21:19 EDT

The last test showed the server became quite laggy, but did not fail as it has before. As we will need to delve into this further we've again blocked all guest activity for now such that the site can be more responsive. Apologies to the guests who now can't even see this message.

Update: 27 Sept, 2018 @ 13:56 EDT

Guest STEX downloading available again. Have at it so we can pressure test the new settings.

Update: 27 Sept, 2018 @ 13:37 EDT

We've been tweaking stuff (or rather Dirk has) and then all of us testing. We had temporarily disallowed guests during this time.

Guest access is re-enabled with some new settings. (Downloads are still member only atm.)

Update: 27 Sept, 2018 @ 10:59 EDT

The site has been going offline every few hours so we regret that we must again temporarily disable guest downloads on the STEX.

Dirk has checked the server's settings and everything shows we should be able to handle all of the load. This indicates there is some other underlying problem that has to be found and corrected. Please bear with us while we continue to research the problem.

Update: 26 Sept, 2018 @ 07:30 EDT

Site is still lagging and occasionally going completely offline with the 522/524 errors. We apologize for the inconvenience.

Update: 25 Sept, 2018 @ 18:57 EDT

With the success of Dirk's server tweaks we have now brought the STEX back online for guests. Their download speed will be unchanged from 500 kB/s, but we are temporarily adding a 10 second delay as well as one download at a time allowed. The 20 downloads per day limit will remain the same as it's always been. We'll see how this test goes and if successful then bring the Gallery back and later remove the additional temporary restrictions we've imposed for downloads.

Update: 25 Sept, 2018 @ 13:20 EDT

Dirktator our Webmaster has adjusted the site's server configuration in attempt to resolve the degraded performance. Should this provide a notable improvement, we will attempt to bring the STEX fully back online later this evening and see how it goes. Here's hoping!

For now the STEX remains online to members only with the same download throttling in place.

Update: 25 Sept, 2018 @ 08:00 EDT

As previously mentioned we are temporarily adding a few limits for member downloads. Transfer speed will be dropped to 1,024 kB/s (previously 2,048 kB/s), one download at a time (previously unlimited simultaneous downloads), and a 10 second delay before the download starts (previously a zero second delay). Again, these are temporary measures we are taking in the hopes that it will keep the site from going totally offline.

Update: 25 Sept, 2018 @ 07:15 EDT

The site went totally unresponsive twice overnight.

Cyclone Boom was able to get us going again. The underlying server issue is still eluding us. We will need to go the other direction in an attempt to balance member access to the STEX while not losing the site altogether. This means the planned reopening of the STEX for guest downloads will be postponed and we may need to place some temporary limits on members such as reducing the file transfer speed, the number of simultaneous downloads, and a short waiting time before the download starts. To be honest, this really sucks. On the other hand, any usable access to the files will be better than none at all.

We appreciate everyone's patience while we struggle with these problems.

Update: 24 Sept, 2018 @ 21:00 EDT

All seems to still be working with quick response times. As mentioned below we will continue this test until tomorrow evening before adding guests back to the settings. Tis now my bedtime so please don't break anything while I sleep.

But, if anything seems wacky, please post in the ST Bugs Thread.

Update: 24 Sept, 2018 @ 19:45 EDT

We've been up for one hour with this STEX test and so far server loads are looking good, page load times seem normal, and avatar hover overs are mostly zippy fast. The plan is to run with this configuration until tomorrow evening to give us plenty of time to see if the site stays responsive before we add the STEX access back for guests. (Rest assured, dear guests, we are not trying to force you to join ST. It's just that we have to figure out what has been causing the site to totally crash.)

Update: 24 Sept, 2018 @ 18:41 EDT

We've brought the STEX back online for members only as our next test.

This test might also overload the site as there is some underlying bug which we have not yet identified. Both Cyclone Boom and I are here to keep an eye on the performance. Should ST get overloaded and go down you will first start seeing long page load times and eventually 522/524 errors from Cloudflare. At that point CB will work his magic and then you'll start seeing a 403 Forbidden error. (This means all access is blocked so the site can re-recover.) And then in about 10-15 minutes we'll bring it back without the STEX again should this bad stuff happen causing the aforementioned site crash.

See also: My initial Bugs Thread post and CB's recent post there saying we are up in test mode for members.

Update: 23 Sept, 2018 @ 22:22 EDT

The site stayed up a wee bit longer, but still succumbed to whatever is causing the trouble. At this point we are taking the STEX and the Gallery offline, but will try to keep the forums going.

Update: 23 Sept, 2018 @ 18:59 EDT

Cyclone Boom has optimized all database tables which is sort of like defragmenting a hard drive. We've brought the site back online and are hoping it helps with the problems. The previous message (below) still applies until we are sure everything is fixed.

Previous message: Simtropolis is experiencing random spikes of heavy server load and excessive lag to the point the whole site becomes unresponsive. We've been taking it offline to let it catch up with the load and then bringing it back. This is the best stop gap measure we have at the moment. If you are creating a new topic or post I suggest making a copy and paste to a text editor before clicking the submit button as we can't be sure it won't go offline at the wrong moment for you.

 

Note: Not everything we suspected was going wonky really was. For instance, the original theory it was a new form of attack was supported by blocking guests and the site then ran well turned out to be a stop gap measure such that the performance degradation was slowed enough to seem like it fixed things. On the other hand, @Cyclone Boom's revised rules have been kept in place because it has reduced random bot access to nearly zero which results in about half as many guests online at any given time.

[korver 27,685]

I'm having trouble posting mosaics at full resolution. Any help would be appreciated.

Previously, in the editor you could just click and drag your picture to any resolution (by clicking the little boxes at the corners and dragging them) - but in the past couple of days these have gone missing.

The only way to resize a picture now it seems is to double left click a picture and a little box comes up where you can manually edit the resolution of a picture - but that doesn't allow for the picture's height to go above 1280 pixels, so you can't post large mosaics at proper resolution anymore.

I've tried with a couple browsers now but still no luck. Would it be possible to restore the ability to drag a picture to the proper resolution in the editor - or simply get rid of the 1280 pixel height restriction in that little box that comes up?

[CorinaMarie 29,605]

1 hour ago, korver said:

Would it be possible to restore the ability to drag a picture to the proper resolution in the editor - or simply get rid of the 1280 pixel height restriction in that little box that comes up?

I hope so. I'll kick this one over to @Cyclone Boom since he's the site techy expert.

[redfox85 659]

Umm sorry I just broke all the lights along the top ... 

[CorinaMarie 29,605]

3 minutes ago, redfox85 said:

Umm sorry I just broke all the lights along the top ... 

Yikes! I hope you didn't get cut or hurt.