[sc4pac] Let's write our own package manager

[memo 1,444]

1 hour ago, Somnolence said:

Whenever there is a SC4E dependency, it returns an error along the lines of

Operation aborted. Failed to extract <home directory>/.cache/sc4pac/coursier/https/www.sc4evermore.com/index.php/downloads%3Ftask%3Ddownload.send%26id%3D3%3Asc4d-LEX-legacy-bsc-common-dependencies-pack

The file is recognized by the system as an archive, but cannot be extracted manually either. The error itself doesn't provide much info on how I could fix it :/

By any chance, did you run out of disk space during the operation? The archive file in ~/.cache is 1.7 GiB in size. Its contents are extracted into the "temp" folder. When the Jar file contained in the archive is extracted, it temporarily takes up another 1.7 GiB inside the "temp" folder.

If this is the problem, either try to make some space on your current partition or change the location of the "temp" folder by relocating it to a larger partition. The location of the "temp" folder is configured in the "sc4pac-plugins.json" file, which can be edited with a text editor.

If this was not the problem, make sure the file downloaded completely by checking its integrity. The sha1sum of the BSC common dependencies 9a file is 0db8b990739346a168607cf94b7bf03cd93cbe1f.

Edit: Another thing you could try is enabling debug mode. This will print a stack trace for the particular error in question.

SC4PAC_DEBUG=1 ./sc4pac update

 

[Somnolence 81]

I deleted the cache and tried the update command again, no issue.

Could have been an incomplete download due to very poor connection that I am having today for some reason. Thanks for replying quickly anyway!

[memo 1,444]

A new version of sc4pac has just been released: 0.4.4.

This adds support for the Apple Silicon M1 platform. The 3rd-party 7zip library that sc4pac uses for extracting NSIS installers did not include native support for Apple M1 – until now. Therefore some packages relying on NSIS installers couldn't be extracted before. Thanks to @bhmantan for reporting this issue and testing this change.

Moreover, I've renamed the "050-early-mods" subfolder to "050-load-first" for clarity, as was suggested on Discord. The old folder isn't removed automatically, but you can safely delete it.

Also many thanks to @nos.17 for adding a large amount of additional dependency packages and for developing this online tool that can help in creating correct new metadata.

 

On 6/8/2024 at 7:59 PM, Somnolence said:

Could have been an incomplete download due to very poor connection that I am having today for some reason. Thanks for replying quickly anyway!

Coming back to this issue, the root cause seems to be with the SC4Evermore server configuration. The admins are aware of the problem, but there's no solution yet. This also affects ordinary downloads using a web browser. For anyone else encountering this problem in the meantime, my recommended workaround is to delete the incomplete file from the cache and start the download from scratch. A faster internet connection can also help avoid the problem.

[Zasco 105]

For those interested, I just launched my own channel:

Zasco's sc4pac channel

_{* Follow the link for more details.}

[memo 1,444]

23 hours ago, Zasco said:

For those interested, I just launched my own channel:

Zasco's sc4pac channel

This is an exciting development. Thanks for assembling all this metadata and sharing it.

 

Additionally, there are a few other exciting developments that have been going on in the background recently. @Cyclone Boom implemented a new STEX API that allows to query some information about STEX entries. This is huge – as it helps a lot with maintaining the sc4pac metadata in the long run. For example, it's possible to check whether any of the collected metadata is out-of-date. Many thanks.

 

Moreover, I've started implementing a graphical UI for sc4pac. It's not finished yet, but the source code is already on Github: sc4pac GUI. I'll show some pictures once I've made more progress.

It's developed with Flutter. If anyone would like to contribute, drop a quick note about your intentions, for example by opening a GH issue, so we can coordinate. Otherwise, I'm planning to follow the steps outlined in the roadmap in the next weeks.

[nos.17 5,120]

On 10/10/2024 at 1:17 PM, Zasco said:

I just launched my own channel:

How does a channel work in practice - is it a mutually exclusive set of packages? From a user perspective, say I am installing packages from the default channel. If I want to install some packages in your channel do I need to manually switch over? I get that the same metadata could be mirrored across the two channels, but I guess what I am wondering about is the scope of a channel. Can channels "see" another's content?

7 hours ago, memo said:

Additionally, there are a few other exciting developments that have been going on in the background recently. @Cyclone Boom implemented a new STEX API that allows to query some information about STEX entries. This is huge – as it helps a lot with maintaining the sc4pac metadata in the long run. For example, it's possible to check whether any of the collected metadata is out-of-date. Many thanks.

Do you have details you're able to share on this and how to access it?

  Edited October 12, 2024 by nos.17  

[Zasco 105]

@memo is likely better suited at answering you, but here's my try:

3 minutes ago, nos.17 said:

How does a channel work in practice - is it a mutually exclusive set of packages?

Not it's not. In fact, it's quite the opposite. Channels can be complementary or concurrent, depending on metadata and load order. Mine is an extension of the default channel; it only provides packages not listed in it.

5 minutes ago, nos.17 said:

From a user perspective, say I am installing packages from the default channel. If I want to install some packages in your channel do I need to manually switch over?

No you don't. You can have both channels registered simultaneously. In fact, you can have an unlimited amount of channels registered.

6 minutes ago, nos.17 said:

I get that the same metadata could be mirrored across the two channels, but I guess what I am wondering about is the scope of a channel. Can channels "see" another's content?

Yes, packages from one channel can depend on others in a different channel. Although, you need to have that "different channel" registered for it's metadata to be fetched.

 

More details about channel registering here.

[nos.17 5,120]

26 minutes ago, Zasco said:

Not it's not. In fact, it's quite the opposite. Channels can be complementary or concurrent, depending on metadata and load order. Mine is an extension of the default channel; it only provides packages not listed in it.

That's great to hear, thanks!

By the way, I made a little UI to help create the yaml files. It's still a work in progress and there are some bugs, but you might find it helpful.

[memo 1,444]

6 hours ago, nos.17 said:

How does a channel work in practice - is it a mutually exclusive set of packages?

Zasco's explanation is correct. Generally, it's possible to overwrite package definitions of another channel by defining a package of the same name – if there's ever a need for that. More commonly, secondary channels would define packages that don't exist in the main channel.

7 hours ago, nos.17 said:

Do you have details you're able to share on this and how to access it?

As it's an authenticated API, my suggestion for anyone interested is to get in touch with @Cyclone Boom or @CorinaMarie.

[CorinaMarie 29,605]

13 hours ago, nos.17 said:

Do you have details you're able to share on this and how to access it?

You've been added to the development PM.

And for right now, we're going to keep the group small and do invite only since CB just finished the API and we still need to really see how it goes under load testing and potential features tweaking.

[nos.17 5,120]

2 hours ago, CorinaMarie said:

You've been added to the development PM.

And for right now, we're going to keep the group small and do invite only since CB just finished the API and we still need to really see how it goes under load testing and potential features tweaking.

Thank you!

[nos.17 5,120]

I have updated my UI tool for creating YAML files and it can be found at a new site with a much better name: https://yamleditorforsc4pac.azurewebsites.net/. Also, thank you to @Zasco for their pull requests and work in helping to improve the site!

There's a lot of small improvements I'd like to do still, but it should be much easier to navigate now with a new clickable tree view to navigate through all the packages and assets in the file. There's also some new drop downs for the subfolder and group thanks to Zasco.

[memo 1,444]

@nos.17 That's great. Thanks for the updates. I've added links to your YAML editor as well as to @Zasco's channel on the main website.

 

Meanwhile, I've made a lot of progress on the GUI front. I'm ready to show some pictures of what's currently implemented and what it looks like. So far, I've been focusing on the core functionality, which is nearing completion. The overall looks might ultimately still change a bit.

 

[nos.17 5,120]

2 hours ago, memo said:

@nos.17 That's great. Thanks for the updates. I've added links to your YAML editor as well as to @Zasco's channel on the main website.

Thanks!

2 hours ago, memo said:

Meanwhile, I've made a lot of progress on the GUI front. I'm ready to show some pictures of what's currently implemented and what it looks like. So far, I've been focusing on the core functionality, which is nearing completion. The overall looks might ultimately still change a bit.

This is so unbelievably cool! It's things like this that cause me to want to go make 15 sock puppet accounts so I can give you more than one useless reputation point.

[Propfam 2,288]

12 hours ago, memo said:

@nos.17 That's great. Thanks for the updates. I've added links to your YAML editor as well as to @Zasco's channel on the main website.

 

Meanwhile, I've made a lot of progress on the GUI front. I'm ready to show some pictures of what's currently implemented and what it looks like. So far, I've been focusing on the core functionality, which is nearing completion. The overall looks might ultimately still change a bit.

 

😯🤯. You did the God's work! Also, it'd be nice if we can port this stuff into SC4 so we can manage our plugins inside the game, possible using embedded Chromium. Hope for the best, @memo!

[Null 45 3,041]

On 10/27/2024 at 11:57 PM, Jidan said:

Also, it'd be nice if we can port this stuff into SC4 so we can manage our plugins inside the game

Why would you need to manage plugin inside the game?

SC4 performs all of its plugin loading on startup, so even if there was an in-game way to manage plugins SC4 would still have to be restarted for the changes to take effect.

[Zasco 105]

1 hour ago, Null 45 said:

Why would you need to manage plugin inside the game?

SC4 performs all of its plugin loading on startup, so even if there was an in-game way to manage plugins SC4 would still have to be restarted for the changes to take effect.

Unless it's something you can fix with your DLL magic.

[Ulisse Wolf 4,412]

4 hours ago, Null 45 said:

Why would you need to manage plugin inside the game?

SC4 performs all of its plugin loading on startup, so even if there was an in-game way to manage plugins SC4 would still have to be restarted for the changes to take effect.

As is the case in Cities Skylines 2. You install a mod and have to restart the game to make that mod available.

 

3 hours ago, Zasco said:

Unless it's something you can fix with your DLL magic.

It cannot be done. We get into the OpenSC4 regime and so you can't do it with mod DLLs.

[Null 45 3,041]

5 minutes ago, Ulisse Wolf said:

It cannot be done. We get into the OpenSC4 regime and so you can't do it with mod DLLs.

Theoretically it could be, you can load and unload DBPF files at runtime. But forcing the game to reload all/most plugins at runtime would be a large amount of effort for minimal benefit.

[matias93 7,535]

2 hours ago, Null 45 said:

Theoretically it could be, you can load and unload DBPF files at runtime. But forcing the game to reload all/most plugins at runtime would be a large amount of effort for minimal benefit.

Maybe only on a very limited way, it could be useful for A/B testing with minimum plugin folders, but that's more the scope of a development tool than a gameplay plugin.

[Zasco 105]

How would you handle the addition of the Day-Night Mod by Gizmo? The "nightmode" variant currently has "standard" and "dark" defined. This mod also is a dark night mod. Adding a new variant would mean refactoring of every already written package to add it. Otherwise it would then throw a linting error. I think it also reveals coupling, meaning a package has to be "aware" of every other package.

[memo 1,444]

@Zasco There are two approaches that I have in mind:

Either add Gizmo's mod as an override to 900-overrides. This first requires to figure out whether the mod can indeed safely be used as an override on top of simfox:day-and-nite-mod. Like for Simfox's mod, you'd add two variants nightmode=standard and nightmode=dark to Gizmo's mod as well, where the former installs nothing, to ensure people cannot accidentally end up with a DN mod and MN buildings simultaneously.

The alternative is to add a new variant to simfox:day-and-nite-mod itself. For example:

variants:
- variant: { nightmode: standard }
- variant: { nightmode: dark, simfox:day-and-nite-mod:provider: simfox }
  assets:
  - assetId: "simfox-day-and-nite-modd"
- variant: { nightmode: dark, simfox:day-and-nite-mod:provider: gizmo }
  assets:
  - assetId: "gizmo-day-night-mod"

(Instead of just defining an asset for Gizmo's mod, a distinct package gizmo:day-night-mod could be added as well. This package would have the same three variants and the two night mod packages would have to mutually depend on each other. The main benefit of this is that the package would get its own package info page with website link etc.)

Coupling can indeed be a problem, but in this case the coupling is only between the two night mods. All the DN buildings only care that a DN mod is installed, but not which DN mod is installed. So there's no need to modify any of the packages containing DN buildings.

[Zasco 105]

13 hours ago, memo said:

There are two approaches that I have in mind:

Respectfully, I don't agree with either. You may however be more qualified than me to adjudicate on this. Here is why:

• It makes Gizmo's "secondary". Why this way and not around? What if a better and more modern "dark night" mod is released? Would then @SimFox's be relayed to secondary?
• With option 2, @SimFox's metadata would have to be refactored everytime a new "dark night" mod is added.

All of this is code smell (on your approaches) in my opinion. I think an approach like the following would easily handle this and ensure decoupling:

variants:
- variant: { nightmode: maxis.default }
  assets:
  - assetId: "group-package-mn"
- variant: { nightmode: simfox.day-and-nite-mod }
  assets:
  - assetId: "group-package-sn"
- variant: { nightmode: gizmo.day-night-mod }
  assets:
  - assetId: "group-package-gn"

I could have shortened the variant values to only the provider, but this way allows for a provider to release a new night mod (no matter how likely it is). Please note dots can be replaced by colons in the above example.

13 hours ago, memo said:

The main benefit of this is that the package would get its own package info page with website link etc.

That's in part the why of my approach.

13 hours ago, memo said:

Coupling can indeed be a problem, but in this case the coupling is only between the two night mods. All the DN buildings only care that a DN mod is installed, but not which DN mod is installed. So there's no need to modify any of the packages containing DN buildings.

Coupling may not be a problem here and now (<1000 packages), but may become overwhelming later on (>10000 packages). The above example doesn't take that into account. However, this one does:

variants:
- variant: { nightmode: standard }
  assets:
  - assetId: "group-package-mn"
- variant: { nightmode: dark, nightmode-mod: simfox:day-and-nite-mod }
  assets:
  - assetId: "group-package-sn" # Or "group-package-dn"
- variant: { nightmode: dark, nightmode-mod: gizmo:day-night-mod }
  assets:
  - assetId: "group-package-gn" # Or "group-package-dn"

This way it wouldn't require packages to define any "nightmode-mod" variant but would still allow it which would also allow for different "dark night" assets.

_{* Note that the linter of the current implementation of sc4pac in 0.4.6-SNAPSHOT will accept this, but not the channel builder.}

 

Why is the "conflict" field of string format? Shouldn't it be an array like "dependencies"? This way, sc4pac could check if any two conflicting packages are installed and throw an error.

[smf_16 1,835]

On 11-10-2024 at 6:56 PM, memo said:

Additionally, there are a few other exciting developments that have been going on in the background recently. @Cyclone Boom implemented a new STEX API that allows to query some information about STEX entries. This is huge – as it helps a lot with maintaining the sc4pac metadata in the long run. For example, it's possible to check whether any of the collected metadata is out-of-date. Many thanks.

@memo @Cyclone Boom can I find some information about this somewhere? I've been playing with the idea to create a tool that accepts a Simtropolis link and then generates metadata from it with minimal manual intervention needed. I could just clean up my script a bit, but it parses the information from the html, which is a bit hacky. If an api is available, this would make the task obviously a lot easier.

[CorinaMarie 29,605]

13 minutes ago, smf_16 said:

 @Cyclone Boom can I find some information about this somewhere?

I can answer this. Right now it's still in the beta testing phase and each person is assigned a key in order to get a response from the API. It's not ready for release to the public (and likely the direct access will always stay with developers). We've got a private message going for this and we'd love to have you join in. I'm not sure when CB will be available to get you setup, but we'll then add you into the convo.

[smf_16 1,835]

Cool, thanks. I'll see when it arrives then. Good to see you back by the way. 

[Zasco 105]

2 hours ago, smf_16 said:

I've been playing with the idea to create a tool that accepts a Simtropolis link and then generates metadata from it with minimal manual intervention needed.

Just to let you know, I've already implemented it, I just didn't commited/pushed it yet. I'll show you in PM.

[memo 1,444]

 

17 hours ago, Zasco said:

• With option 2, @SimFox's metadata would have to be refactored everytime a new "dark night" mod is added.

There's no way around that with the current implementation of sc4pac.

I agree with your second approach though. I've created a pull request for that proposal. Please have a look.

17 hours ago, Zasco said:

Why is the "conflict" field of string format? Shouldn't it be an array like "dependencies"? This way, sc4pac could check if any two conflicting packages are installed and throw an error.

The conflicts field is intended for highlighting potential conflicts – to make those easier to see at a glance than if they were included as part of the description. These conflicts can be very specific, for example

Incompatible with CAM, compatible with IRM.

but, most of the time, they are quite generic, referring to a whole family of packages or even mods outside(!) of sc4pac. Some examples:

The seasonal controller variant is only compatible with terrain mods that include the Seasonal Flora Patch.
Incompatible with all other tree controllers – only one tree controller may be installed at a time.

Must load after other Maxis Lot replacement mods such as CAM.

Only compatible with game version 1.1.641, the Windows digital edition.

Incompatible with the macOS version of the game.

Incompatible with mods blocking Maxis lots

Having a way to define conflicts such that sc4pac can automatically detect them is a feature that doesn't currently exist, but it could be useful. However, if e.g. every tree controller needs to maintain a list of incompatible other tree controllers for this, that's just more coupling, so not ideal.

[Zasco 105]

On 11/3/2024 at 6:22 AM, memo said:

There's no way around that with the current implementation of sc4pac.

I know, but maybe it means the source code has to be changed to allow this. I can open an issue on this, but I don't quite know how to formulate it.

On 11/3/2024 at 6:22 AM, memo said:

I agree with your second approach though. I've created a pull request for that proposal. Please have a look.

I'll review it asap.

On 11/3/2024 at 6:22 AM, memo said:

but, most of the time, they are quite generic, referring to a whole family of packages or even mods outside(!) of sc4pac.

I agree keeping this feature is important! Maybe the solution is to have both?

On 11/3/2024 at 6:22 AM, memo said:

Having a way to define conflicts such that sc4pac can automatically detect them is a feature that doesn't currently exist, but it could be useful.

And it wouldn't be as complicate to integrate to the code and as computing heavy as versions resolving.

On 11/3/2024 at 6:22 AM, memo said:

However, if e.g. every tree controller needs to maintain a list of incompatible other tree controllers for this, that's just more coupling, so not ideal.

I think in the case of conflicts it's pretty much unavoidable as one package has to "be aware" of another to declare a conflict with it... In such case, I don't think it's coupling as changes in one wouldn't affect the other.

 

Souldn't "180-flora" be renamed to "180-mmps"? "flora" is a bit restraining for what might belong in this subfolder...

Allowing multiple sources for an asset would be great for redundency. See for instance the plugin right below. It is hosted on both Simtropolis and SC4E.

What should be done following recent update of this file?:

 

[memo 1,444]

On 11/3/2024 at 9:52 PM, Zasco said:

I know, but maybe it means the source code has to be changed to allow this. I can open an issue on this, but I don't quite know how to formulate it

I've opened https://github.com/memo33/sc4pac-tools/issues/10 with some ideas for discussion.

On 11/3/2024 at 9:52 PM, Zasco said:

I agree keeping this feature is important! Maybe the solution is to have both?

Open an issue please.

On 11/3/2024 at 9:52 PM, Zasco said:

Souldn't "180-flora" be renamed to "180-mmps"? "flora" is a bit restraining for what might belong in this subfolder...

For this one, I'm leaning towards 180-flora, as it's more readable, all MMPs are technically Flora, too, and 180-flora also includes god-mode Flora.

On 11/3/2024 at 9:52 PM, Zasco said:

Allowing multiple sources for an asset would be great for redundency. See for instance the plugin right below. It is hosted on both Simtropolis and SC4E.

True, but it's actually quite a tough problem to ensure that both exchanges are in sync. The exchanges do not keep older versions around and do not provide checksums that prove that assets are identical. Until something like that is implemented on the exchanges, I don't think it makes much sense to spend an effort on this. As currently the majority of files is only available on one server, this would rarely be of use.

On 11/3/2024 at 9:52 PM, Zasco said:

What should be done following recent update of this file?:

I'd just wait for the update of the Common Dependencies, as that's going to require a metadata update anyway.