[sc4pac] Let's write our own package manager

[memo 1,440]

Tired of tracking down dependencies? Let me introduce - sc4pac, a package manager/mod manager for SC4 plugins:

sc4pac website: https://memo33.github.io/sc4pac/

What is sc4pac?

What is a package manager anyway? Package managers are programs that automate the process of installing software (see Wikipedia). You simply select a package to install. No need to worry about dependencies – they will be installed for you.

Concretely, the main tasks sc4pac handles are:

• downloading and installation of plugin files
• managing dependencies (These are installed automatically for you.)
• ensuring correct load order
• ensuring compatibility between choices (Do you use a DarkNite mod? Great. DarkNite BATs will be chosen automatically where available.)
• updating installed plugins when newer versions become available
• reproducibility of your plugins folder (This also makes it possible to share your plugins folder, in the form of a configuration file.)
• verifying file integrity to prevent malicious tampering

Download: The sc4pac GUI is available for download on the STEX:

 

Originally, sc4pac came just with a command-line interface. Here's how it looks like in action:

How does it work?

Sc4pac gets its information from curated metadata stored at https://github.com/memo33/sc4pac and then downloads the actual files off of Simtropolis or SC4E. Feel free to send a PR adding more package metadata.

Who is this for?

Anyone starting to build a new plugins folder from scratch and who does not know where to start. Anyone wishing to have a more organized plugins folder. Anyone overwhelmed by the amount of available plugins and the complexity of managing dependencies by hand.

If you already have a large and tidy and heavily-customized plugins folder, sc4pac may be of limited use to you.

What can be installed so far?

There is a selection of package highlights. This list will grow in the future, as more packages are added. It's a manual process that takes some time. In total, there are more than 200 550 3000 packages already.

Personally, I mostly aim for plugins that seamlessly integrate with the original game aesthetics. I think this is an opportunity for us to collectively compile a base set of high-quality plugins suitable for many newcomers and returnees.

TL;DR: Check out sc4pac.

  Edited February 9, 2025 by memo  
Added STEX release

[Flann 1,709]

This is really excellent work.  In my opinion, this is the future of plugin management for the community.  I am very excited for the possibility of sharing a plugin configuration which should make it quite easy to spin up a new region or set up a multiplayer server.  I also think this might allow us to achieve some decoupling of the plugin organization from the community sites and maybe the possibility of additional mirrors of the content for increased redundancy.

I think what we need most now is some help in creating the metadata for the most essential plugins so that this tool becomes a viable option for putting together a relatively full-featured plugins collection for the user.

[TheMurderousCricket 5,049]

I think I could use a hand with organizing some pieces of my content.

I often have a feeling that some of the plugins I have are mostly useless as they contain hundreds of items and just two minuscule objects are actually needed as dependencies for something... Is sc4pac able to show what files are redundant?

[memo 1,440]

On 10/24/2023 at 11:20 AM, TheMurderousCricket said:

I often have a feeling that some of the plugins I have are mostly useless as they contain hundreds of items and just two minuscule objects are actually needed as dependencies for something... Is sc4pac able to show what files are redundant?

Not exactly. Identifying redundancies is more of a job for Data Node or a hypothetical advanced datpacker. Sc4pac does not know anything about the files you've already manually put in your plugins folder. However, sc4pac helps you avoid redundancies to some extent by installing only the files you need, based on the available metadata. Even if you uninstall something, then superfluous dependencies are removed as well.

[Aesonus 19]

This is good work, indeed. This is very much like other package managers like npm and composer. My only reservation is that some (Windows) users might not be comfortable with using the command line. I might be willing to write a GUI for this, assuming there is enough support for such a thing.

Anyway, this is great and definitely something that is needed given the complicated mess of dependencies that exist.

 

Edit:

Apologies, I did not read the original post initially.

Quote

There are some ideas for a mod-manager-like graphical front-end as well, but there is still a lot of opportunity for collaboration if you have an interest in front-end development.

I would be interested in that, just not in Java. I hate coding in Java . I would probably write the GUI in Python and use the command line tool under the hood.

  Edited October 25, 2023 by Aesonus  

[memo 1,440]

19 hours ago, Aesonus said:

I would be interested in that, just not in Java. I hate coding in Java . I would probably write the GUI in Python and use the command line tool under the hood.

That would be awesome. You'd have my support. The CLI is probably going to need some adjustments for non-interactive use, such as more structured, parseable output. I haven't worked out the details, but do let me know what you need and I'll make the changes.

[Vitoraju 675]

Wow! Great news. Thanks for sharing with the community! 

[Shadow Assassin 336]

On 10/26/2023 at 10:30 AM, Aesonus said:

This is good work, indeed. This is very much like other package managers like npm and composer. My only reservation is that some (Windows) users might not be comfortable with using the command line. I might be willing to write a GUI for this, assuming there is enough support for such a thing.

Anyway, this is great and definitely something that is needed given the complicated mess of dependencies that exist.

 

Edit:

Apologies, I did not read the original post initially.

I would be interested in that, just not in Java. I hate coding in Java . I would probably write the GUI in Python and use the command line tool under the hood.

lol I mentioned the same thing about Java apps to @memo. I've been looking into possible approaches for the GUI, would be very happy to hear any input into it!

[Aesonus 19]

4 hours ago, Shadow Assassin said:

lol I mentioned the same thing about Java apps to @memo. I've been looking into possible approaches for the GUI, would be very happy to hear any input into it!

I think that using the tkinter library in python for the GUI would be simple and also has the benefit of being cross platform.

As for parsing command line output, I don't see any issues with making that happen, provided the output is consistently formatted. Of course if there was an issue with it, I would submit an issue on github.

I have not used Mod Manager before, so I don't know what the interface looks like. Maybe someone could take some screen caps and show me what it looks like.

[Ulisse Wolf 4,408]

14 minutes ago, Aesonus said:

I think that using the tkinter library in python for the GUI would be simple and also has the benefit of being cross platform.

As for parsing command line output, I don't see any issues with making that happen, provided the output is consistently formatted. Of course if there was an issue with it, I would submit an issue on github.

I have not used Mod Manager before, so I don't know what the interface looks like. Maybe someone could take some screen caps and show me what it looks like.

You can take a cue from the Cities Skylines mod manager 

https://steamcommunity.com/sharedfiles/filedetails/?id=2881031511

 

[Flann 1,709]

I've been meaning to learn more about GUIs and tkinter for Python, so I'd be interested in contributing as well.

[rsc204 21,808]

What is needed to get at the Metadata required? As in how can others help to fill in the gaps here?

In an ideal world, it would be time to split packages into individual contents. But I can also understand the much larger burden of such an approach. yet I feel obligated to at least mention it since that would be the holy grail IMHO. Especially that with some smart tools, there is scope to automate much of this. Manual intervention only really needed when IDs conflict.

Is this going to be more like an automated dependency tracker, i.e. the metadata knows which files (deps) a given package needs? Does this allow for common dependencies to be kept, where necessary, when uninstalling a package using a dep that other content also requires?

But otherwise a great project and whatever way this ends up the prospect of pre-packaged Plugins is something I feel is hugely important to keeping many interested players from being overwhelmed by the work needed to set it up.

[Flann 1,709]

1 hour ago, rsc204 said:

What is needed to get at the Metadata required? As in how can others help to fill in the gaps here?

This file is a commented example of the metadata specification.  The most important parts are those which define the assets (files on the exchanges) and the packages (the installable collections built from the assets).  For an asset, it is required to provide:

• assetId (a unique identifier)
• version number
• timestamp
• the url of the download

For packages, it is required to provide:

• group (e.g. team name, author name, or package owner)
• name
• version
• installation subfolder
• a list of dependencies
• a list of assets and which files to install from them if installing a subset

Packages also have additional optional metadata which would give additional context to the package, including:

• summary
• warning
• conflicts
• description
• author
• images
• website

2 hours ago, rsc204 said:

In an ideal world, it would be time to split packages into individual contents.

It's definitely designed with splitting assets in mind.  For example, the large LEX common dependencies pack is already split by its component packs.  We can also split apart growables from ploppables assuming the uploader has them in separate files.

2 hours ago, rsc204 said:

Is this going to be more like an automated dependency tracker, i.e. the metadata knows which files (deps) a given package needs? Does this allow for common dependencies to be kept, where necessary, when uninstalling a package using a dep that other content also requires?

Yes, that is one of the key advantages of this approach.  Dependencies need only be installed when required by a package you want.  The program will handle the installation and removal of dependencies automatically.

 

Metadata are text-based YAML files, so very easy to read and write.  To contribute, you really just need a text editor and some familiarity with git.

[memo 1,440]

On 10/28/2023 at 7:02 PM, Aesonus said:

As for parsing command line output, I don't see any issues with making that happen, provided the output is consistently formatted. Of course if there was an issue with it, I would submit an issue on github.

I am happy to announce that Sc4pac now has a JSON-based API. This should make it much easier for a GUI program to interact with Sc4pac. This would have been difficult with the CLI, especially since the CLI makes use of interactive prompts. Now, with the API, you can start a local server and interact with Sc4pac through regular HTTP.
 

Also, a big shout-out to @Flann who added metadata for a whole bunch of packages, including almost all of mattb325's civic buildings. There are more than 350 packages available, so far.
 

On 11/6/2023 at 2:42 PM, Flann said:

Metadata are text-based YAML files, so very easy to read and write.  To contribute, you really just need a text editor and some familiarity with git.

To add to that, let me link to an actual example: radio-station.yaml. I encourage anyone interested to look around and see how the metadata is set up. With a Github account, you can edit the YAML files directly in your browser, taking Git largely out of the equation.

[smf_16 1,835]

This thing is one of the best things *ever* to happen to SimCity. Seriously.

I've used npm (the package manager for JavaScript) a lot in my professional life and have always thought that it would be awesome to have something like that for SC4 as well. I've often thought about writing something like that myself, but it's an enormous amount of work so I never got to it. So glad to see it's finally been done!

[smf_16 1,835]

EDIT:

I should've just inspected the code better before writing my post. I found out that it's already possible with environment variables. See also https://github.com/memo33/sc4pac-tools/blob/0.4.0/src/main/scala/sc4pac/Constants.scala#L41 if people are looking for it.

Original post:

I was thinking a bit about the download limit on Simtropolis of 20 files if you're not authenticated. As I understead, ST authentication is being worked on, but I think I've found something that should work in the meantime.

It looks like when downloading a file, ST only needs the ips4_IPSSessionFront cookie to be set. For example, the curl command below worked for me when I was above the 20 file download limit

curl "https://community.simtropolis.com/files/file/36180-growify-plopped-buildings-cheat-dll-plugin/?do=download" -H "cookie: ips4_IPSSessionFront=<session-id>;"

If there would be a way to pass the value of the ST session id somehow, then it could be a temporary solution until ST authentication is fully implemented. Something like

sc4pac update --st-session-id=<session-id>

which then just sets a header

cookie: ips4_IPSSessionFront=<session-id>

when it actually performs the download.

Unfortunately I don't know Scala, but I'll see if I can figure something out and create a PR in the GitHub repo.

[smf_16 1,835]

@memo I've noticed you added documentation about how to use the cookies. That will surely be useful!

As for a definitive solution, I was thinking about something like this. Ideally every user on Simtropolis would be able to generate a token somewhere (could be a jsonwebtoken) which can then be injected in sc4pac somehow. We could use environment variables for this, or use a configuration file (e.g. .npmrc files with npm, see https://docs.npmjs.com/using-private-packages-in-a-ci-cd-workflow#create-and-check-in-a-project-specific-npmrc-file)

Adding the possibility to generate such tokens will require some server side coding though, so I'm going to tag @Cyclone Boom and @Dirktator to see what could be possible. I'm definitely willing to help here.

Apart from the possibility to generate tokens, the server side protection for 20 downloads should also be changed so that if the server detects a valid token, it will ignore the download limit (as if the user was logged in).

[memo 1,440]

On 11/6/2023 at 12:20 PM, rsc204 said:

What is needed to get at the Metadata required? As in how can others help to fill in the gaps here?

There is now a detailed documentation page on adding metadata that explains this process.

 

@smf_16 Thanks for the kind words. Regarding authentication to ST, it's not something I'm currently working on. The cookie method works, but is a bit too convoluted for most users I imagine.

With a little help from Simtropolis admins, your proposal would be a nice solution. Something that is a bit easier to implement server-side would be setting the download limit based on user-agent (which has obvious implications, though).

Another option would be using password-based authentication, but there does not seem to be an API for this on Simtropolis. For now, I've deferred these plans till the implementation of a graphical UI.

[TheLetterZed 1]

This is a phenomenal project, just getting started using it today. This is exactly what I have always wanted for SimCity 4 mod organization!

There's two features I'd love to see:

• I'd love to be able to add mods based on their original Simtropolis post or their original Simtropolis download link. Something like this:

  sc4pac add https://community.simtropolis.com/forums/topic/761020-1st-beta-opera-house-fix-for-modpacc-zero/?tab=comments#comment-1784801

  Which would be equivalent to:

  sc4pac add mz:opera-house-fix

  Alternatively, on the Sc4pac Channel that lists all the available packages, it would be helpful to be able to search by original Simtropolis URL from the metadata

• Additionally, are you accepting metadata submissions for gigantic guides? I want to be able to install something like all the mods in Indiana Joe and the Quest for the Ultimate Plugins Folder - SimCity 4 General Discussion - Simtropolis with a single command, but don't know how realistic that would be. I guess in cases like that, it would also be useful to explicitly exclude certain mods if you don't agree with the creator's choices.

[Ulisse Wolf 4,408]

11 minutes ago, TheLetterZed said:

Additionally, are you accepting metadata submissions for gigantic guides? I want to be able to install something like all the mods in Indiana Joe and the Quest for the Ultimate Plugins Folder - SimCity 4 General Discussion - Simtropolis with a single command, but don't know how realistic that would be. I guess in cases like that, it would also be useful to explicitly exclude certain mods if you don't agree with the creator's choices.

Information title:

Currently that guide is partially supported pending its replacement by this final guide which should support the new ecosystem as well as being designed to be compatible with the new sc4pac management system

In addition, before adding new features sc4pac still requires support of Simtropolis infrastructure and at the moment only supports SC4Evermore infrastructure

[smf_16 1,835]

26 minutes ago, TheLetterZed said:

Additionally, are you accepting metadata submissions for gigantic guides? I want to be able to install something like all the mods in Indiana Joe and the Quest for the Ultimate Plugins Folder - SimCity 4 General Discussion - Simtropolis with a single command, but don't know how realistic that would be. I guess in cases like that, it would also be useful to explicitly exclude certain mods if you don't agree with the creator's choices.

This is definitely be possible with sc4pac, at least if all the individual content of that guide is available. In that case, it's just a matter of creating a package that contains all the content as dependencies.

I've been doing this for example with @Diego Del Llano's content (see https://github.com/memo33/sc4pac/blob/main/src/yaml/diego-del-llano/complete-collection.yaml). If you sc4pac add diego-del-llano:complete-collection, it will install all of his content. The same can be done for all plugins mentioned in that thread. It's just a matter of adding metadata for all packages (which is an enormous task), but @Indiana Joe's list is a great place to find content which you can create metadata for.

As for installing mods by their url, I'd actually do it the other way around and add the sc4pac add command to Simtropolis itself (which can be copied to the clipboard or so, next to where the download button is now or something). This could also encourage modders to immediately create a metadata file when they upload new mods.

[memo 1,440]

On 4/4/2024 at 10:53 PM, TheLetterZed said:

Additionally, are you accepting metadata submissions for gigantic guides? I want to be able to install something like all the mods in Indiana Joe and the Quest for the Ultimate Plugins Folder - SimCity 4 General Discussion - Simtropolis with a single command, but don't know how realistic that would be. I guess in cases like that, it would also be useful to explicitly exclude certain mods if you don't agree with the creator's choices.

Yes, metadata submissions are appreciated. The main part that takes time is adding package metadata for each individual plugin. I'd love if we manage to add packages for many of the plugins in Indiana Joe's guide. I'd be hesitant about accepting a single package into the main channel that installs all the plugins from that guide at once though, as that feels a bit too rigid and non-cohesive to warrant the maintenance overhead. As long as there's metadata for each individual plugin, the better approach is to just share a list of those package identifiers. You could squeeze them into a single command and skip any undesired packages or alternatively host such a giga package in a secondary channel.

On 4/4/2024 at 11:29 PM, smf_16 said:

As for installing mods by their url, I'd actually do it the other way around and add the sc4pac add command to Simtropolis itself (which can be copied to the clipboard or so, next to where the download button is now or something). This could also encourage modders to immediately create a metadata file when they upload new mods.

I like this idea. This would make it more apparent which STEX files are installable with sc4pac. Searching by URL may sometimes be useful as well.

[junspud 1,066]

I always thought it'd be amazing if SC4 had something similar to what OpenTTD has for plugins/addons, and this is nearly as close to that as is possible. Definitely going to use this when I finally get around to redoing my absolute disaster (bordering on unplayable) of a plugins folder, very cool to see this.

Even in its current state it's incredibly useful. I also never realized the VIP terrain mod was even available to download anywhere, completely missed the fact it was in a dropbox link.

[junspud 1,066]

Apologies if this has been posted about before, but Having trouble working out how to set the environment variables so as to download more than the daily limit.

 

I tried following this guide but replacing the relevant variables, among other things.

https://stackoverflow.com/a/17142065

 

But I don't think I did it correctly, as the update command keeps failing.

 

Any advice would be appreciated, no experience with setting these, thanks again.

 

Edit: Thanks, moving the variable and removing the quotes worked.

[memo 1,440]

@junspud Thanks for asking. I think it should work if you move the SC4PAC_SIMTROPOLIS_COOKIE environment variable to the User Variables instead of the System Variables and if you remove the surrounding quotes ("…") which shouldn't be part of the Value.

Alternatively, you could edit the "sc4pac.bat" file and add the following line somewhere in the middle of the file (anywhere before the java call at the end), with the <value> placeholders replaced:

SET "SC4PAC_SIMTROPOLIS_COOKIE=ips4_device_key=<value>; ips4_member_id=<value>; ips4_login_key=<value>"

 

[nos.17 5,112]

Hi memo,

I have finally found some time to dig into this and start adding some of my own packages. It's a super impressive tool. Some relatively minor things I noticed:

1. When testing, the channel add and channel remove commands provide no confirmation feedback it was successfully added or removed.
2. Is there a way you can support parsing file paths with back slashes as well as forward slashes? Would save an extra step with Find+Replace in Notepad replacing my back slashes.
3. The timezone will fail to parse if there is an extra space, like " 2023-11-11T08:07:11-08:00" resulting in an error when you query the package.
4. A null pointer exception is thrown when querying the package if you have an extra "---" at the end of the yaml file after the last asset.

I'm in the process of writing a little web-based UI that can help me generate the yaml files, so I can take care of some of that input parsing on my end as well too.

[memo 1,440]

4 hours ago, nos.17 said:

Is there a way you can support parsing file paths with back slashes as well as forward slashes? Would save an extra step with Find+Replace in Notepad replacing my back slashes.

Thanks for the feedback. Do you mean in the include/exclude patterns in the YAML files? The problem is that the YAML format requires backslashes to be escaped and, since those patterns are regular expressions, you need to escape them twice. A single backslash \ would need to be written as a quadruple backlash \\\\, so there wouldn't be a real benefit. In practice, I tend to keep those patterns short anyway, either just the file name or folder name, but not the full path. This makes it less likely you need to touch them if a new version of an asset is uploaded.

I'm going to see what I can do about the other issues, though.

4 hours ago, nos.17 said:

I'm in the process of writing a little web-based UI that can help me generate the yaml files, so I can take care of some of that input parsing on my end as well too.

Awesome. I was hoping there'd eventually be a tool like that, but I haven't gotten around to implementing it. (That's also why there are wizard references on the metadata documentation page.) Note that @smf_16 posted a Node.js script that collects some of the information to include in the metadata.

 

Meanwhile, thanks to a recent contribution by smf_16, sc4pac is now able to extract files from Clickteam installers, so all commonly used formats are now supported.

The number of available packages is steadily growing as well, with recent contributions by smf_16, jflann and nos.17.

Even some major mods like CAM and Colossus Farming are now available, which by traditional means are a bit more complicated to install.

[junspud 1,066]

I noticed the CAM page on SC4pac says it uses a DLL to fix the IR bug, I was wondering which DLL this is?

 

Thanks..

[Null 45 3,026]

43 minutes ago, junspud said:

I noticed the CAM page on SC4pac says it uses a DLL to fix the IR bug, I was wondering which DLL this is?

It is the submenu DLL. The relevant exemplars are patched when the game loads them, which removes the need to modify SimCity_1.dat. Here is the readme from the CAM_2.1.0_Stand_Permanent.memo-exemplar-patch download:

Quote

This patch file

    CAM_2.1.0_Stand_Permanent.memo-exemplar-patch.dat

uses Exemplar Patching to apply the IR-Fix that comes with the Colossus Addon Mod 2.1.0 (CAM).
It supersedes the file

    z_CAM_2.1.0_Stand_Permanent Merge.dat

that is traditionally merged into `SimCity_1.dat` during the CAM installation process to avoid applying its effect twice.

Now, using Exemplar Patching DLL magic, there is no need to merge the patch into `SimCity_1.dat` anymore.
If you have already merged the original IR-Fix into `SimCity_1.dat`, it is recommended to revert to the original file.
Even if you do not revert, it still works correctly with the Exemplar Patches as long as you do not have any unnecessary duplicates of the RCI Exemplar files in your Plugins.

Additionally, the file corrects one of the properties of the original IR-Fix.

-memo 2024-03-24

[Somnolence 81]

[Preface: I run Linux.] Works great when downloading from STEX alone, but it seems not to download from SC4E properly. Whenever there is a SC4E dependency, it returns an error along the lines of

Operation aborted. Failed to extract <home directory>/.cache/sc4pac/coursier/https/www.sc4evermore.com/index.php/downloads%3Ftask%3Ddownload.send%26id%3D3%3Asc4d-LEX-legacy-bsc-common-dependencies-pack

The file is recognized by the system as an archive, but cannot be extracted manually either. The error itself doesn't provide much info on how I could fix it :/

[Edit: I should point out that when I rename the archive, I can open it manually, but then it won't get recognized by the program.]

  Edited June 8, 2024 by Somnolence