simcity 4 Sharing Plugins Folders - Discussion

[Langbardr 54]

I don't play SC4 at the moment, but I think about it pretty much every day, wanting to dive back in. But every time I remember that it takes five minutes to launch my game, I think about how messy my mod folder is, and how painful it is to organize, to look for, and to install them. And it turns off my will to play.

So I'm glad that this thread exists, and I sincerely hope it leads to something concrete. I can't be of any help, but I wanted to show my support and thank every person for their current and future efforts in this matter.

[Ulisse Wolf 4,471]

9 minutes ago, Langbardr said:

I don't play SC4 at the moment, but I think about it pretty much every day, wanting to dive back in. But every time I remember that it takes five minutes to launch my game, I think about how messy my mod folder is, and how painful it is to organize, to look for, and to install them. And it turns off my will to play.

So I'm glad that this thread exists, and I sincerely hope it leads to something concrete. I can't be of any help, but I wanted to show my support and thank every person for their current and future efforts in this matter.

If you have too many mods I suggest you run the datpacking

There is also a tutorial on how to datpack on NAM

[Langbardr 54]

17 hours ago, ulisse said:

If you have too many mods I suggest you run the datpacking

There is also a tutorial on how to datpack on NAM

Yeah I know about datpacking. I really should do it one day. But that's just another step that keep me from playing...

[rsc204 21,869]

56 minutes ago, Langbardr said:

Yeah I know about datpacking. I really should do it one day. But that's just another step that keep me from playing...

Right, but DATPacking is nothing to do with the discussion here and EVERYTHING to do with your hardware and it's capabilities. SC4 can only load data as fast as both your computer and the game itself is capable of doing. All DATPacking does is to lower the number of overall files, but if you've a modern SSD, the difference is hardly noticeable in terms of both loading times and in-game responsiveness. That's not to say there is no benefit, but honestly I don't DATPack much when playing under Windows and I don't feel I'm missing out.

However, the reason DATPacking exists in the first place, is that traditional HDDs need to 'seek' every file. Imagine the difference between doing this for a cassette, where the correct physical location of the tape has to be reached before it can play the track you want, v.s. a CD where it can skip much quicker to a new track. That's a pretty good comparison (for anyone old enough to remember tapes and CDs at least :/ ). If you don't have an SSD, nothing discussed here is likely to be of much help to you, no one to my knowledge has talked about providing pre-packed Plugins and frankly that would be a terrible idea IMO.

[Langbardr 54]

4 hours ago, rsc204 said:

If you don't have an SSD, nothing discussed here is likely to be of much help to you, no one to my knowledge has talked about providing pre-packed Plugins and frankly that would be a terrible idea IMO.

Well, I was just mentioning that, beside managing custom content, long launch time was part of why I refrain to play. I did not mean that the matter discussed here could provide a solution to it.

But I'm saddened to hear that datpacking can not do much better than a SSD, as I already use one.

[Matrim Cauthon 344]

3 minutes ago, Langbardr said:

Well, I was just mentioning that, beside managing custom content, long launch time was part of why I refrain to play. I did not mean that the matter discussed here could provide a solution to it.

But I'm saddened to hear that datpacking can not do much better than a SSD, as I already use one.

Anecdotally, it can still make a significant difference. I have a decently fast SSD setup (not a crazy NVME 2gbs or anything, but 400 MBs in realworld applications is typical), and it makes a pretty big difference for me. Especially notable in loading City Tiles after the initial one and reductions in simulation jitter. It's definitely worth a try, and Datpacking is pretty easy. Just make sure you keep a unpacked copy of your mods somewhere incase you want to change something down the road.

[rsc204 21,869]

9 hours ago, Langbardr said:

long launch time was part of why I refrain to play.

Well that's on you personally then, if that's a turn off, nothing the community does or doesn't do can solve that problem. Either use less mods or accept you've got to wait.

Even with a full NAM controller, I am at most 2m before the game loads and yes that IMHO is too long also. However, bear in mind I'm a developer and need to test stuff constantly, meaning I am jumping in and out of the game at times a lot and those load times really start to get under your skin. If I was just playing, meh, it's only a few minutes and frankly in the pre-SSD world that was pretty normal for all games. Geez, even the XBox 360 and PS4 suffer from horrendous load times, Red Dead Redemption 2 for example really sucked balls, however the game was worth the wait.

9 hours ago, Matrim Cauthon said:

Anecdotally, it can still make a significant difference.

Depends on your definition of significant, have you actually ever got a stopwatch out and timed it? I have, the difference is negligible, marginally faster loading, but not enough to really make it worthwhile. I have to DATPack if I use the Mac version and contrary to the 'it's straightforward' argument, I find it anything but, it's another layer of complexity and pain in the arse I'd much rather do without, exactly why I never bothered under Windows. Again though I am not a typical player, I have multiple Plugins suites active at any time and mess so much that I can't have everything packed away, without causing me a huge additional workload and complicating my modding endeavours. But for normal folk, if you wait until your Plugins are mostly complete, then only add to them occasionally, I would imagine it's a lot more tolerable.

9 hours ago, Matrim Cauthon said:

Especially notable in loading City Tiles after the initial one

I'll run some tests on that, but bear in mind the NAM Controller is loading into RAM when you load the first city, but not thereafter, so could that be making it seem better than it really is?

9 hours ago, Matrim Cauthon said:

reductions in simulation jitter.

Again it's really minor, SSDs don't have seek times, file access is in nanoseconds, so it doesn't matter how many files they need to access, it all happens in relative terms the same. Yes, I too have noticed how for example having all your MMPs in a single DAT, reduces the delay when opening the menu that would otherwise be present. But unless you've a really large Plugins suite, the difference is very minor overall. Case in point, the delay only happened when testing the MMPs of ErnestMaxis, which is over 700 new entries, but yes when things get to that extreme, DATPacking does give a little boost.

Yes, if you have thousands of model files knocking around, that will take longer to refresh than loading the data from a single file, but since megapacks are already fairly commonplace, that shouldn't really be the case for most users. Again, in the real world and I've tested/timed it too, the difference between a sensible 3GB Plugins suite packed and unpacked on an SSD just isn't a big deal. I've never had a crazy large suite like some, so yes I can imagine when you hit 10/16GB, things are very different. That said, I've yet to meet anyone with so many Plugins, where their mods are well organised and anything other than a dump of everything they could download/install, which I can't help feel might distort how much of that is directly related to DATPacking.

9 hours ago, Matrim Cauthon said:

ot a crazy NVME 2gbs or anything, but 400 MBs in realworld applications is typical

If you actually can get 400 MB/s file transfers, under Windows, I would be genuinely shocked. I have a 1,800 MB/s NVMe drive and the simple fact is, nothing is optimised in the OS/Apps I use to take advantage of it. Honestly, it's no faster than the older SATA3 drives, of which I've had 4 or 5, which run at around 500 MB/s. Even between SSDs, such transfer rates are unsustainable and only come in very short bursts, as soon as you try moving many GBs at once, the median transfer rate doesn't look anything like as impressive.

SC4 specifically can not take advantage of either faster SSDs nor RAMDrives, it's been empirically proven again and again, they don't result in better performance. So I am left to conclude that the legacy code is introducing bottlenecks that without a rewrite of the code, will always limit performance. It's the same with your CPU and GPU, SC4 simply can't use a fraction of the power of modern components.

[Flann 1,709]

I'm my experience datpacking seems to make the game run a bit smoother, but this may just be placebo effect.  One area where is definitely improves the experience is switching in and out of underground view to construct subways.  If you have severe lag doing this, in my experience datpacking resolves it.

[rsc204 21,869]

Don't get me wrong, I'm not saying DATPAcking with an SSD is a complete waste of time or anything. But when compared to the night and day difference when running on a traditional HDD, literally the reason this system came about, the benefits are greatly reduced. In context of the current discussion, I just wanted to make it clear that even if the community were somehow able to pre-pack such Plugins for users, it wouldn't dramatically affect the loading times. In other words, loading times once you are on an SSD are pretty much what they are and not a product of any lack of optimisation, rather limitations of old code.

[Matrim Cauthon 344]

9 hours ago, rsc204 said:

 

Depends on your definition of significant, have you actually ever got a stopwatch out and timed it? I have, the difference is negligible, marginally faster loading, but not enough to really make it worthwhile. I have to DATPack if I use the Mac version and contrary to the 'it's straightforward' argument, I find it anything but, it's another layer of complexity and pain in the arse I'd much rather do without, exactly why I never bothered under Windows. Again though I am not a typical player, I have multiple Plugins suites active at any time and mess so much that I can't have everything packed away, without causing me a huge additional workload and complicating my modding endeavours. But for normal folk, if you wait until your Plugins are mostly complete, then only add to them occasionally, I would imagine it's a lot more tolerable.

Yeah, when I did the original datpacks about 2 years ago, it cut off a lot of time and honestly was what got me back into playing. I was really suffering from tiles change waits.

But yeah, it you're constantly making alterations to your mod load out, it's a huge extra work load. I'm a little in between, but careful pre-packing folder management keeps the work load overall low, for me at least.

My current region has had the RES mods completely reworked three times (my fault, I messed with occupancy a lot time ago and had to restore things to a more sane level, and had a bunch of MN stuff when I use DN) and handling the Datpacking was honestly the smallest element of that.

 

10 hours ago, rsc204 said:

I'll run some tests on that, but bear in mind the NAM Controller is loading into RAM when you load the first city, but not thereafter, so could that be making it seem better than it really is?

Yeah, that's what I was trying to convey with the "after the initial one" bit. That first tile load always takes a while for the NAM code, so I measured off of tile loads after that.

 

10 hours ago, rsc204 said:

Again it's really minor, SSDs don't have seek times, file access is in nanoseconds, so it doesn't matter how many files they need to access, it all happens in relative terms the same. Yes, I too have noticed how for example having all your MMPs in a single DAT, reduces the delay when opening the menu that would otherwise be present. But unless you've a really large Plugins suite, the difference is very minor overall. Case in point, the delay only happened when testing the MMPs of ErnestMaxis, which is over 700 new entries, but yes when things get to that extreme, DATPacking does give a little boost.

Yes, if you have thousands of model files knocking around, that will take longer to refresh than loading the data from a single file, but since megapacks are already fairly commonplace, that shouldn't really be the case for most users. Again, in the real world and I've tested/timed it too, the difference between a sensible 3GB Plugins suite packed and unpacked on an SSD just isn't a big deal. I've never had a crazy large suite like some, so yes I can imagine when you hit 10/16GB, things are very different. That said, I've yet to meet anyone with so many Plugins, where their mods are well organised and anything other than a dump of everything they could download/install, which I can't help feel might distort how much of that is directly related to DATPacking.

My experience for this was after I added a significant Freight trucks automata mod, which saw a lot of simulation jitter (though honestly it wasn't the only addition at the time, so it might not be that, but I was closely watching automata performance at the time) and Datpacking resolved that.

But I think that may largely also play into what you're mentioning in the last bit. There are inherent limitation in how SC4 is parsing data that is causing the majority of the bottlenecks IMO and something about datpacking improves that to a limited extent.

I think that fileseek times exacerbate that issue rather than cause it, explaining why the problem doesn't disappear on a SSD.

 

11 hours ago, rsc204 said:

If you actually can get 400 MB/s file transfers, under Windows, I would be genuinely shocked

To be fair, I'm more speaking about burst transfer than sustained(measured via task manager, but 200-300+ MB/s file transfers for larger files is pretty typical for me. I have a chipset RAID-0 of Two SSD's, so that might be giving me a bit of an edge here. That does drop for large numbers of smaller files, the windows overheard is real.

[Matrim Cauthon 344]

Update - did a quick test, swapping my packed mods for their unpacked source.

Both tests loaded a tile and then exited to attached region, all tests run with focus. I did not measure game boot times.

Packed run 1 - 7 seconds to load tile

Packed run 2 - 6 seconds to load tile

 

Unpacked run 1 - 1 minutes 40 seconds to load tile

Unpacked run 2 - 1 minute 42 seconds to load tile.

Initial tile load had less of a difference, but still notable. 

Packed initial load - 1 minute 50 seconds

Unpacked initial load 3 minutes 22 seconds.

Edit: My plugins folder is 3.25 GB - 1.9 of which is packed(half of which are dependencies). NAM is unpacked currently.

Edit2: Because my load times honestly sound unbelievable, here is video proof https://streamable.com/nu2ttx

[rsc204 21,869]

8 hours ago, Matrim Cauthon said:

Update - did a quick test, swapping my packed mods for their unpacked source.

My results are rather different:

3.04GB of Plugins, covering 4,242 files and 585 folders, on an Intel 660p NVMe drive. System is an i7 4770 / 32GB DDR3 1600 / Z97 Express Chipset.

Initial Load (to Region View screen):

UnPacked   58.21
Packed        59

Complete load of first large tile w/100k sims / 48k jobs (combined startup time):

Unpacked   36.54 (1:34.75)
Packed        23 (1:22)

2nd subsequent Load of same city:

UnPacked    22.94
Packed        10

3rd Load:

UnPacked    25.5
Packed        10

Times are approx and measured using a stopwatch on a phone, but all will be within a few seconds of reality. I could extract accurate timing info from the video I made, but I've frankly no interest in doing so, the slightly less perfect figures above give the general overview I think.

Note the DATPacked results were based on a 'Double DATPack'. So I started by DATPacking every folder in the root of Plugins. Once those were converted to DATs, I put them into a new folder and DATPacked that, which is as optimised as it's possible to make things.

Now 10-12 seconds faster startup overall is nothing to be sniffed at I suppose, I was genuinely surprised to see how much quicker the subsequent loads occurred with Packed Plugins, I wasn't really expecting such a difference.

I never have livestreamed anything and whilst I did take a real-time capture with OBS, it's a very boring 17 minutes of me trying to find something to say, whilst getting distracted by my cat's appearance and a moment of genuine surprise when the datpacked game loaded the secondary save much quicker. Not to mention I've a hideous cold and there is sniffling, coughing and all manner of other fun things you probably don't want to hear. It's even embarrassing how completely unprepared I was to make the video and I catch my own self out many times, like trying to remember which NAM version I've got (42) and thinking I had the RHW 'full' controller, when I clearly can't have because it wouldn't have loaded anything like as fast. If people really want to see it, I'll upload it to YouTube, but this is as far from entertainment as you are ever going to get!

Still, an improvement of 12 seconds is not nothing, but it's hardly life changing either. How it is taking your game so long to do the secondary loads in the first place I can't explain. But the info on your number of files/folders might be useful by way of comparison. My working theory is that your Plugins folder unpacked is probably either messy or stuffed full of many folders and subfolders, which are sub optimal for loading. Mine has been painstakingly put together over 6-7 years and is as elegant as I know how to do things. Bear in mind, I am from an era where 8 character folder names were your limit and many nested folders were a bad idea.

  Edited February 26, 2022 by rsc204  
Edited to correct that it's a 660p SSD, not a 600p

[Matrim Cauthon 344]

2 hours ago, rsc204 said:

How it is taking your game so long to do the secondary loads in the first place I can't explain. But the info on your number of files/folders might be useful by way of comparison. My working theory is that your Plugins folder unpacked is probably either messy or stuffed full of many folders and subfolders, which are sub optimal for loading.

This is very likely the cause. My packed results are likely only better because I'm loading a smaller city.

My file and folder count is significantly higher, and while I have done a fair bit of optimizing I have largely maintained the structure provided by the mods, primarily organizing into category and consolidating any content from the same creator. 

The files I pack weigh in at 11,094 files and 1,774 folders. All images and txt files removed. Running a i7-3820 / 32GB DDR3 / X79

The files that remain unpacked between tests(including NAM) count 1,135 files and 313 folders

 

I think your meticulousness benefits you immensely here.

[TheMurderousCricket 5,121]

Really, reeeealy late to the show, but just to answer the very first question in this discussion...

What I'd like to get in case someone shares their plugin folder... No... What I hoped for in the past was reclaiming the content that Wolfze had produced over the years. Unfortunately, I registered here too late to ever interact with him or download any of his stuff. And I've seen it - it was terrific.

I am the type of Simtropian who searches for growables and ploppables that diversify the cityscape so this is what I would look for (and offer) if I were to engage in the plugin sharing. On the other hand I wouldn't want to get a cease and desist order from a once-member-gone-rouge, so I know that this particular dream will not come true. 

[Benshannaboy 23]

Is there any update on the ModPacc?! I wanna get back into this game, but don't want to redownload all my mods again.

[BartonThinks 3,416]

10 hours ago, Benshannaboy said:

Is there any update on the ModPacc?! I wanna get back into this game, but don't want to redownload all my mods again.

I know in my case that things stalled out, partly because I wanted to get the LEX folks on board with a trial run and they were busy with NAM projects at the time I asked, partly because I replaced my computer and it's taken time to get SC4 back up and running, and mostly because my work life no longer allows for as much time modding SC4 (though SC4 might be to blame for that a bit -- a lot of the soft skills I picked up through modding the game have proven really useful at work).

As far as ModPacc Zero -- @CorinaMarie and @Cyclone Boom have both mentioned that they also put their project on hiatus due to a combination of workload issues and trouble finding users to help test different mods.

Thankfully, Project ZIP by @Tyberius06 has been making a LOT of progress recently. That's making it a lot easier to download and install a number of older content.

[memo 1,479]

Something that has been mentioned several times in this thread is the idea of using a package manager for distribution of custom content (@matias93 @Handyman @T Wrecks @Tarkus). As a returning player myself, creating a new plugins folder from scratch is off-putting and a package manager for SC4 content would be immensely useful here.

For anyone unfamiliar with the concept, think of a sophisticated installer that allows you to select what you want to install and then downloads and installs exactly what you need, including all dependencies, into the correct locations in your plugins folder. Package managers are the standard way to install software on Linux distributions and are a joy to use, as it only takes one person to figure out how to install something, so that everyone else can benefit from it. You always have the option to install software without a package manager as well and customize it to your liking, but often there is no need for that.

Setting up such a system is much more achievable than you might think, as long as we use a package manager that already exists. A good candidate for that might be Conda, as it is cross-platform, comes with a graphical front-end and has a large user base.

I've set up this GitHub repository which guides you through the installation and illustrates how this can work for SC4 custom content by installing some example packages. This is already functional. I encourage you to go through the process or at least read the steps in the README as this has a lot of potential. The initial setup might look a bit daunting and the graphical front`end has some quirks, but that seems tolerable compared to some of the quirks we have gotten used to over the years (dependencies, puzzle pieces, crowded menus, etc). (Side note: I have not tested this on Windows yet, but in principle it should work or can be fixed easily.)

Such a repository could host metadata for a selection of custom content (such as MODPACCs and their dependencies). The metadata contains the information that is needed to install plugin files in an automated fashion. As Conda is a fully fledged package manager, it is possible to automate even the selection of dark-night/standard-night model variants or CAM/no-CAM variants.

What is missing to make use of this right now is a way to distribute the actual custom content in the form of Conda packages. A Conda package consists of the actual data, like a props pack .dat file, as well as metadata. There are mainly two options going forward:

1. Host the Conda packages alongside the original zip files directly on the STEX and LEX. In the same vein as Project ZIP (many many thanks to the herculean efforts by @Tyberius06), this merely constitutes a repackaging of existing content which seems to be an accepted practice in our community. Alternatively, seek explicit permission from authors to rehost the files somewhere else.
2. Instead of including the .dat files directly, just include download links to the files on the archives. Conda places those download links into the plugins folder and another program must be used to automatically bulk download all the files. Such a program would have to be written -- maybe LEXDownloaderX could be repurposed for this. This approach subverts many features of Conda because it installs files Conda does not know about, so such a setup would be a bit brittle, but might be a short term client-side solution which is even site-agnostic.

The first option would be most preferable. It would require a collaboration of many people, especially the admins of both sites, but the potential outcome seems well worth it for the longevity of our community (the NAM Team has ideas for 10 years to come).

TL;DR Have a look at this step-by-step guide https://github.com/memo33/sc4pac-example
 

[matias93 7,535]

That's fantastic!! It's exactly what I was thinking about... 5 years ago??? wow

I gave it a try, and while it works flawlessly on Windows 10, I cannot say the same about Windows 7, which is the obsolete installation on which I have SC4 running. I don't remember if the results were made public, but I seem to remember that the NAM Team survey showed that it was about 10% of the users that play on such old operating systems, so I guess anything we do should be parallel to the existing way to directly download the files.

Thinking on circumvent the issue, I also tried to install Chocolatey on Windows 7, to no avail, but it did work as well on Windows 10, so if Conda proves difficult, we might have a second platform option to develop this. It's biggest downside is that it requires PowerShell to run, so it's tied to Windows. Still, its community packages page might be a good reference of how to offer content this way: just copy and paste a simple command from the file description, or, if the site allows, click on a button to make a list that can run at once when copied.

[catty-cb 4,413]

23 hours ago, memo said:

For anyone unfamiliar with the concept, think of a sophisticated installer that allows you to select what you want to install and then downloads and installs exactly what you need, including all dependencies, into the correct locations in your plugins folder. Package managers are the standard way to install software on Linux distributions and are a joy to use, as it only takes one person to figure out how to install something, so that everyone else can benefit from it. You always have the option to install software without a package manager as well and customize it to your liking, but often there is no need for that.

They are brilliant,  I use the Synaptic Package Manager on Linux Mint

Start it up

Pick a program

It works out if it needs to install anything else

Does a check to see if anything related to the program and is already installed on your computer needs to be updated

Goes and downloads the files from the file archive

and then installs them on your computer

I love it, and its just as simple if you change your mind to uninstall a program

[TogaMasterJohn 711]

As a user and a fan of Linux (I primarily use Linux Mint on my main machine and GalliumOS on my modified Chromebook aka "Tuxbook"), I still use Windows 7 on my main machine for what I can't run under Linux via Wine or Proton, and for what I can't do under Linux in one way or another. I have no interest whatsoever in upgrading to 10 or 11. I can run SC4 on either Windows 7 or Linux (SC4 on Linux can be interesting to setup and run; I won't get into that here).

Anyways...
I love the Synaptic Package Manner. As catty-cb said and demonstrated above, it is incredibly easy to use. Package managers can be quite helpful and quite awesome.

I think it would be awesome to see a class-platform package manager for SC4 (or a Windows program that works just fine under Wine). I wonder if it would be possible to develop one for the STEX and the LEX, or have it be based around them. Select the plugin(s) you wish to download, and have it download any necessary dependencies. Of course, checking to see if you already have said files might be interesting.

I can of two possible options off the top of my head as of now (besides having a download history of sorts, like the LEX has):

1) You could have a master folder for all of the plugins you have downloaded in a separate folder (the user could select where the folder should go), so that way the package manager could check and see if you already have those files. This could be a good idea, especially if the user wishes to DAT-pack files. {Of course, DAT-packing is generally recommended if you have a mechanical hard disk drive and not a solid state drive OR if you are playing the Mac port of the game}

• What happens if the master folder gets deleted by accident or the drive it is on fails? Perhaps the package manager could periodically (or each time on startup and/or shutdown) check and make note of the files that is in the master folder.
  • Would these notes (or a note file) be stored on the user's computer? If so, a reminder to back up the note file somewhere from time to time could be useful to some users.

2) You could have the package manager check the plugins folder itself, if the user does not wish to have a master folder to contain all of their downloaded plugins. 

• However, having a master folder shouldn't be too much of an issue for most users, unless the user does not have a lot of space to work with. 

How would the package manager manage the downloaded plugins? Either it could extract (if necessary) said files and then place them in the user's plugins folder, or the user could manually do the work themselves after everything has been noted and saved. There are a few possibilities here alone.

How would the package manager deal with DAT-packed files? Well, this is where a download history feature could come in handy (or something like option 1). I can't remember if JDatPacker creates log files for what all gets DAT-packed in a file (I haven't used it in forever). If so, maybe the package manager could make note of what the user already has, assuming the log files can be read and/or if they exist.

• If such a logging feature in JDatPacker or another DAT-packer does not exist, maybe one could be modified or even developed for this purpose... perhaps a DAT-packer could be built into the package manager.
• Perhaps a backup feature for the Plugins (and even the Regions) folder could be a handy feature as well - make a copy of the folder and store it somewhere on the user's machine, on a flash drive, etc.

Of course, this is all just an idea... and there is a lot of 'how?' questions. This really is just the tip of the barrel. 

While I have taken classes that were related to programming logic and I have had some experience developing things, I do not have a lot of experience with big programming or scripting languages (except for JavaScript; even then I am by no means a pro). I often tend to think like a programmer. I often tend to draw out GUIs, write code, and/or design flowcharts all in my head. It's complicated for me to explain how I process such things 

I really think such a solution could be useful to a lot of players, as well as our community.

[memo 1,479]

 

@matias93 Thanks for testing this on Windows. The problems on Windows 7 can be resolved I am sure, as Conda is officially supported on Windows Vista and newer.

Chocolatey is also an interesting option. From the looks of it, Chocolatey would make it easier to implement something along the lines of option 2 from my previous post, since Chocolatey allows you to distribute installation scripts (which could download files from the STEX/LEX), while Conda only extracts prepackaged files (like a zip archive). Both approaches have their pros and cons. The one thing I would miss most with Chocolatey is cross-platform compatibility. As a non-Windows user, I am heavily biased in this regard though. It might be possible to find a way to use Chocolatey with Docker on other platforms, but I expect that to be a challenge of its own.

 

 

@catty-cb Thank you. – Simutrans looks very interesting, by the way. I'm looking forward to giving it a try.

 

 

@TogaMasterJohn Thanks for sharing your ideas. What you describe might be closer to a downloader like LEXDownloaderX. It can list dependencies and keep track of what you downloaded, but would not be able to automate the installation process all the way.

Unlike a downloader, a package management system can never be a general purpose front-end for all the content that is available on the STEX and LEX. There is just too much content. The metadata needed for that cannot be generated automatically, so it is more realistic to provide the metadata only for a selection of high quality content which is collected and maintained in a central location as a collaborative effort by the community.

Before writing our own package manager, we should first explore all the available possibilities. While writing a custom application might give us exactly what we want, it would not only be a huge amount of work, but also require lots of maintenance on our part, and it is not clear that the end result would be better than what we already have. I think it would be more realistic to write a custom front-end for Conda or another package manager that presents SC4 content in a nice and accessible way. That way, a large part of the programming and maintenance work would be externalized. But I would not worry about pretty front-ends right now, since with Conda we could already have something that is basically functional right now, which would allow us to focus more on the actual content to include.
 

[catty-cb 4,413]

12 hours ago, memo said:

@catty-cb Thank you. – Simutrans looks very interesting, by the way. I'm looking forward to giving it a try.

Been exploring since yesterday and the Simutrans version that you can download from the Synaptic Package Manager is version 120.4.1 and the Steam version is 123.0.1 and that's the same version you can download from the Simutrans website

https://www.simutrans.com/en/

So I've updated to the latest version (installed the windows version as the linux version seemed a bit fiddly) and have to say the graphics is a lot better in the 123.0.1 version

[Cyclone Boom 17,032]

Just popping in here in response to the ideas @memo explained for a package manager. While the idea was discussed in the early days of the topic here years ago, to see this practical implementation gives scope for a possible way forward. Of course, the biggest thing is the effort needed to update file entries to offer this method, although if we pull together with volunteering, then most anything is possible. I very much support (and I know Cori does also) the idea behind this approach. Figuring out a scripted installation for what the framework would handle (using the instructional metadata files). Then it'd become click and install for everyone.

In terms of:

On 8/4/2022 at 1:11 PM, memo said:

Host the Conda packages alongside the original zip files directly on the STEX and LEX. In the same vein as Project ZIP (many many thanks to the herculean efforts by @Tyberius06), this merely constitutes a repackaging of existing content which seems to be an accepted practice in our community.

Speaking what applies to the STEX here, I believe this would be perfectly reasonable in terms of adding a repackaged version on file entries. It'd be available as an alternative choice under the Download button too, and so this way authors would still receive downloads and also associating everything on the same entries. The original files can then remain there separately for anyone who prefers manual installations. A key thing then is people can also receive reputation and comments on their files.

If there would be some downloader tool possible alongside to automate the process, that might be useful too. Just I feel that'd be best not as the only option.

 

On 8/4/2022 at 1:11 PM, memo said:

The first option would be most preferable. It would require a collaboration of many people, especially the admins of both sites, but the potential outcome seems well worth it for the longevity of our community (the NAM Team has ideas for 10 years to come).

Yes, I very much agree.

For an effort to update files with such repackaged versions, a team of volunteers can be given STEX editing access to help, along with any site staff who wish to assist too. At this point there's seems like plenty to figure out including best compatibility between legacy systems, but I do like this concept. Absolutely the functional side is the most important thing to begin with (which appears to be there already with Conda, etc), and then always if a refined GUI is possible it'd be even more ideal.

In the realm of accessibility, all this kind of fits in with the ongoing new STEX categories as well.

[matias93 7,535]

Then, a priority now would be to agree on a common metadata and naming scheme, to avoid file duplication issues between different sites, and instead use them as mirrors. Keep in mind that, to avoid the need to use secondary IDs, we should strive to generate unique names for each file, while also keeping them as simple and human-readable as possible.

It would be useful if we could get a small random sample of the STEX database (not the files themselves, only the metadata) as an excel file, to experiment with metadata fields in a way that, if not completely automated, at least allows for a very simple data entry task when applying it to the full exchange.

[catty-cb 4,413]

54 minutes ago, matias93 said:

It would be useful if we could get a small random sample of the STEX database (not the files themselves, only the metadata) as an excel file

Go take a look in the omnibus as I was indexing all the SC4 exchanges including the STEX, I eventually stopped but I had most of the sites indexed and when I stopped and shutdown my website i uploaded all the indexes to the the omnibus.

I still have all the original excel spreadsheets if that's easier.

Edit: here is one of them, I split them up by the alphabet

 

  Edited August 7, 2022 by catty-cb  
added a bit

[nos.17 5,265]

Happy to see some activity in here again, and I suppose that means I should share my visions and ideas towards a "Mod Manager" of sort; it's something I've been thinking and planning about for nearly a year mostly independently, but more recently with two other people.

 

My vision

A while ago I had the pleasure of reading through this discussion in its entirety, and while quite long winded at times, I found some excellent kernels in there that sparked some ideas, and I began wondering how and what I could build off of. While most of the rest of the thread was constructive as far as the "soft" concepts and ideals for the system, I pulled out a handful of posts which I think were trending the right way and were the most constructive as far as physical needs and requirements. Reading through it all, I couldn’t help but to see strong parallels between things brought up earlier in this thread and this Prop + Texture Catalog concept which I have built and slowly expanded a few times.

I’m envisioning some kind of system that
a) keeps track of all content uploaded to STEX/LEX,
b) keeps track of content of items in a player’s Plugin folder, and
c) is able to bridge the two (downloading content, managing dependencies, etc.)

Regardless of how the end system gets implemented (some sort of package manager solution sounds great to me), the system ultimately still needs to know what additional things are required for a given upload. For that, there needs to be something major in place first - some sort of database of information where things are managed and linked. For this database, I think we’d need at least 3 groups of tables. One for the uploads themselves and metadata about them (name, URL, date uploaded, version, date modified, maybe even tags/categories, possibly dependency breakdowns, definitely incompatibilities, and/or other metadata as appropriate). A second for all of the TGIs in each upload (kind of like a much-expanded Prop & Texture Catalog but for all other exemplars too including buildings and other required files). The third is a list of the TGIs currently in the player’s Plugin folder. Exactly whatever the data structure ends up looking like is not important for now, and I am focusing for now on just massing an index of everything I can.

Some in the Sharing Plugins Discussion have offered the opinion that such a task would be herculean; I don't necessarily agree with that. It certainly will take a lot of time and investigative work, but from what I have tested quickly so far it seems doable. I think that should maintain even as the scope scales up. I'm making some initial assumptions about the degree of detail this theoretical dependency manager would go to (include to the individual building/prop/texture/LUA/etc exemplars for lots and ignore the FSH/BMP files which are technically dependencies for models).

 

My implementation so far

I loved the idea of MODPACCs discussed, and figured bundling all of my B62 content would make a great start. First, I started a C# library to read and parse DBPF files. This was mostly a personal challenge to learn and accomplish, and I believe it will prove invaluable to me as I work towards achieving many of my ideas, from this plugin sharing to reversible datpacking to improved PIMX. This code library allows me to look into a DBPF file to actually read its contents - it won't matter if someone has datpacked or renamed or done anything else - we still should be able to know what a specific file is and what it contains. Additionally, I will need to transition to something actually suited to handle this huge amount of data - my JavaScript filtering in the Prop + Texture Catalog which while pretty neat and something I am proud of will definitely not be up to the task. It’s served its purpose. With assumptions I mentioned above in terms of scope and detail, I'm wildly guessing the end result will be somewhere on the order of tens of millions of records. A proper data management implementation should eat that up just fine considering that we'd only be adding a few thousand more records each month at the most, assuming roughly the current rate of uploads.

(Timeout here to send a huge thank you to @memo - your jDBPFX library was invaluable in helping me getting started on my own library - it was a tall task learning C# and how to analyze DBPF at the same time and you were a huge help for the latter).

Anyways, using csDBPF, I created a barebones program to use that library to parse through the B62 content and create CSV tables as placeholders for data tables in an actual database I described above.

• The first table examined all of the LotConfiguration Exemplars and listed all of the IIDs for rep 13 of the LotConfigPropertyLotObjectData properties – this is essentially a list of IIDs of all buildings, props, textures, flora used on the lot. This is the list of all LotConfigPropertyLotObjectData used in the B62 Remastered lots.
• The second table or set of tables is a list of TGIs and some identifying information about the file they come from, and associated tables detailing metadata about each upload. Here all TGIs uploaded ever are represented. After referencing the first table against the second table, I can have the exact list of dependencies for a given item. If this list is complete, then by definition I also know which packs the IIDs are located in and where to download them. (Tangentially, I would also be fascinated to see a heat map type visual of which dependency packs are most utilized and even which props are most referenced too - tons of minimally useful yet fascinating knowledge is hidden here).
• A third table could contain all of the TGIs you have in your plugins folder. This can be cross referenced between the first two tables to determine what you already have in plugins, and what needs to be downloaded.

I've put together a quick prototype using all of the uploads in my B62 Remastered series. It's located here in Power BI, but the gist is that i can know the required dependency packages for any chosen lot. It's also embedded below for you to play with.

Despite my csDBPF library being in its infancy, I am pretty much already able to accomplish putting together table one (IIDs used for a given lot or set of lots) and table three (all TGIs in a plugin folder) listed above. Table two (the list of all TGIs uploaded) is where things get interesting...

 

Indexing all known uploads

Looking from the LEX side of things for a moment, @CasperVg has built a fantastic system with API access to LEX file data (example [LEX login required]). This makes it fantastically easy to index and cache information on each upload. I'd be trivial to iteratively look at each item and store relevant information in a database for referencing.

The STEX database stores a lot of this same information, but because there's no custom API, it's a little bit more difficult to get to everything. Critically it also does not list the most important thing either: dependencies. On a related side note:

On 8/6/2022 at 10:48 PM, matias93 said:

It would be useful if we could get a small random sample of the STEX database (not the files themselves, only the metadata) as an excel file, to experiment with metadata fields in a way that, if not completely automated, at least allows for a very simple data entry task when applying it to the full exchange.

A long time ago, CB kindly shared a sample of it with me. Here's a link to what it looks like. Basically there's two main tables: one for each file listing containing the listing information, and one for "attachments" to each listing including screenshots and the uploaded file(s) to download.

 

Circling back to figuring out what is required for each upload, this is where I am currently stuck right now. It's possible to iterate over the STEX database and programmatically download, extract and analyze each file to get a rough idea of what IIDs are used (and thus what dependencies are required), there's no way of knowing if that item is the original prop/texture/building or if it is an override and actually refers to some other upload. Note that this also does not cover other dependencies which may be required like queries, sounds, LUA scripts, or other kinds of resources, though I'm sure that can be figured out in time.

All of this might look something like @catty-cb has done with her Catalog, but would be more comprehensive and would include the LEX and possibly other non English language sites as well. This section is still kind of up in the air while I work to improve and simplify and expand my csDBPF library and further build out a draft of the database schema expressed in the Power BI report linked above. I've pretty much taken a break from both of those for the summer, but towards the fall I'll be diving head first again into those projects and ramping up collaboration with others on this and related projects.

 

Conclusion

The way I see it there's really three aspects to a sort of "mod manager" that can manage downloads and dependencies: the part the user interacts with, the storage of the files, and the data that is the foundation for the system. Regardless of what the first two parts turn out to look like, I've chosen to get started on the data component because it seems reasonable to accomplish and make significant progress without having to come to some sort of consensus as a community.

That was quite a lot, but it basically summarizes most of what I have been thinking and slowly working towards for nearly a year. That's what has been floating around in my head, and I'm super excited to hear what yall might think about it, and curious about any other potential issues or shortcomings I might have missed.

[catty-cb 4,413]

WOW  @nos.17 that's impressive, need to come back later and have a proper read of your post.

The catalog on the omnibus was actually part of a database I had on my website, when it came time to shut down my site I had to download the different tables into separate spreadsheets which is why there is also a catalog of TGI numbers in the omnibus with the name of the item and also .... I think they might be in my club there are some zips containing eighteen thousand pictures of plugins each saved with the TGI number.

Feel free to use any of this data for your own project

[memo 1,479]

On 8/6/2022 at 8:30 PM, Cyclone Boom said:

Speaking what applies to the STEX here, I believe this would be perfectly reasonable in terms of adding a repackaged version on file entries. It'd be available as an alternative choice under the Download button too, and so this way authors would still receive downloads and also associating everything on the same entries. The original files can then remain there separately for anyone who prefers manual installations. A key thing then is people can also receive reputation and comments on their files.

Thanks for being so open to the idea.

In terms of the way Conda would work, it looks like Conda expects all the package files in a single directory on the server (example), which could be uploaded in some form via ftp. It seems to me that this would not directly be compatible with the way the STEX operates, unless some form of redirections could be installed.

If this is not possible or not practical, then a client-side solution in form of some downloader application, like @nos.17 suggested as well, might be more flexible, as it would require less admin involvement and instead would be more open to becoming a collaborative process. Though, such a program doesn't yet exist.

I've also explored the capabilities of some other package managers (npm, yarn), but as far as I can see, they lack some useful functionality that Conda provides: the possibility to install packages into a well-organized plugins folder hierarchy, and the option for package variants (such as dark-night/standard-night).

 

@nos.17 Very interesting development. What you have so far could definitely be useful, especially to make sure that the dependencies of a particular upload are complete. My suggestion for the future would be to try to keep the different goals and tasks separate from each other, as best as possible. That way, if one aspect turns out to be difficult to solve, it might be easier to find a workaround without impacting the rest of the goals too much.

[Flann 1,709]

Much has been said about sharing plugins folders, but I'm here to direct everyone's attention to a new project by @memo which may end up solving this problem for good.  It is called sc4pac.  This is a package manager for SimCity 4 plugins that handles the installation of mods and dependencies for us.  The project is still getting off the ground but the functionality is there and it is already being put to use.  Read more here: 

Why does this matter?  Well, if we can come together as a community to contribute the necessary package metadata, we might very soon be able to share plugins folders not as raw data, but rather in the form of a package configuration that anyone with sc4pac could install.  It would solve many of the headaches we anticipate would come with other sorts of plugin shares.  I encourage anyone who's interested in helping make mod installation a simpler, more pleasant experience to check out that thread and see if you might be able to help this vision come to life.