simcity 4 Sharing Plugins Folders - Discussion
By
Haljackey, in SC4 - Custom Content
681 posts in this topic Last Reply
Highlighted Posts
Recommended
Okay, so here are those thoughts I mentioned. They’re in… some kind of order. Apologies in advance for the War-and-Peace length of this post.
***
OVERVIEW
The core goal of this project would be to develop a viable system by which larger content collections could be shared through the Simtropolis and/or SC4Devotion communities, under terms that address the core concerns of content creators.
This system would be developed and tested through a one-time pilot project, which would be led by a small team of community members with oversight from STEX and LEX admins. Depending on the results of the pilot project, the team may then work with STEX and LEX admins to develop long-term policy changes that would allow other content collections to be developed and shared.
DETAILS
The project would have seven stages, with each stage being conditional on the outcome/success of the previous stage.
1. Recruitment. This stage would involve developing a team of volunteers to work on the project. Ideally, this would be a group of 3 to 5 experienced SC4 players who share a goal of making larger content collections accessible.
2. Initial Planning. Following recruitment, the team would develop a high-level plan for the pilot project collection(s). At this stage, planning would be focused on
The overall scope and goals of the pilot project
Guiding principles for obtaining content permissions for the pilot project
Guiding principles for packaging and distribution of the pilot project collection(s)
Guiding principles for support and documentation of the pilot project collection(s)
This plan would then be brought to the STEX and LEX admins, as well as key stakeholders (namely, content creators) for feedback. The plan would then require authorization from the STEX and LEX admins to proceed to step #3.
3. Detailed Planning. After getting sign-off on a basic framework for the pilot project, the team would develop the plan for the pilot project in more detail. This would include planning which content will be included in the pilot project itself and obtaining permissions to this content. Active creators would need to provide permission for their content to be included in the pilot project. Permissions from non-active creators could be issued through the STEX and LEX admins, depending on the terms agreed upon in Step #2. Any such permissions would be restricted to the pilot project.
4. Pilot Project Development. At this point, the team would begin to actively develop the pilot project collection or collections.
The guiding principle for development would be to wait until the project is authorized and key permissions are secured before performing the bulk of the work.
As with any larger content collection, this stage would entail compiling and organizing materials, creating new materials (if needed), testing content packages, creating documentation, etc.
5. Pilot Project Release. Once completed, the pilot project collection(s) would be released under the terms agreed on by the STEX and LEX admins in the initial planning phase. Following release, the team would track the results of the release, including:
The popularity of the collection(s) among existing, new, and returning SC4 players
Feedback from users of the collection(s)
Feedback from the existing community, with a particular focus on content creators
Problems or complications that arise from the collection(s)
The effect (if any) of the collection(s) on the overall SC4 community
Note that the STEX/LEX admins they would have final say over the release. If they felt that the collection(s) could not be released for whatever reason, the release would be tabled or canceled.
6. Policy Drafting. If the pilot project is released and well-received by the broader community, the team would then work with STEX and LEX admins to draft a proposed update to the community’s policies around sharing larger content collections. Content creators involved on the pilot project could be invited to collaborate on these policies. The team could also engage with the broader community during policy development. Again, STEX and LEX admins would have authority over this process.
7. Final Decision. Once the policy is drafted, the STEX and LEX admins would have final say over how it was ultimately implemented, including:
Conditions that must be met for the policy to be approved
The final terms and language of the policy itself
Whether or not to implement the change in policy at all
Again, if the STEX/LEX admins felt that these policy changes could not be implemented for whatever reason, the policy changes would be tabled or canceled.
MISC. THOUGHTS
This approach is meant to overcome some of the obstacles and objections that have emerged over the course of this thread. Below are some of the main advantages that I think this approach would offer:
Involvement by STEX and LEX admins will be necessary for this project to work. Because of this, I don't think community members are going to view this project as viable and consider volunteering until STEX and LEX admins get involved in some way. Launching this as a joint STEX/LEX project addresses this issue and would hopefully offer enough momentum to get this thing off the ground.
The pilot project would offer a chance for the team to work with active content creators to try and address concerns around content permissions and distribution. The limited scope and conditions of the pilot project would make this much less imposing to creators than simply asking them to sign off on major changes in community policy. Content creators would be able to take a "wait and see" approach, whether they're involved with the pilot project or not.
The project is structured to minimize wasted work. Each stage is contingent on the success of previous stages, and the workload increases incrementally from stage #1 to stage #4, so that the most time-intensive work only takes place if the pilot project is viable. For example, nobody is going to spend thousands of hours compiling a megapack only to find out they can't get half the permissions they need. Those questions should be (largely) covered before the megapack is compiled.
The pilot project approach would allow the team to work through many of the issues raised in this thread from a practical, hands-on perspective. To my mind, that’s going to be a much more helpful and realistic way of figuring out how this might work (or if it’s workable at all).
Even if the project does not lead to broader changes in community policy, the collection(s) produced by the pilot project could be highly beneficial unto themselves—particularly if they are focused on making the game more accessible to new/returning players.
***
Hopefully that all makes sense!
I’d be happy to know if anyone else thinks this approach might be workable. I'd also be interested in hearing if there might be ways to build off of this or improve what I’ve got outlined here.
Thanks again to Cori, CB, Tibi, and Tarkus for the thoughtful responses.
CoriBoom™ Approved proposal for a prospective way forward. This is a potential pathway to success!
Recommended by Cyclone Boom
Recommended
Something that has been mentioned several times in this thread is the idea of using a package manager for distribution of custom content (@matias93 @Handyman @T Wrecks @Tarkus). As a returning player myself, creating a new plugins folder from scratch is off-putting and a package manager for SC4 content would be immensely useful here.
For anyone unfamiliar with the concept, think of a sophisticated installer that allows you to select what you want to install and then downloads and installs exactly what you need, including all dependencies, into the correct locations in your plugins folder. Package managers are the standard way to install software on Linux distributions and are a joy to use, as it only takes one person to figure out how to install something, so that everyone else can benefit from it. You always have the option to install software without a package manager as well and customize it to your liking, but often there is no need for that.
Setting up such a system is much more achievable than you might think, as long as we use a package manager that already exists. A good candidate for that might be Conda, as it is cross-platform, comes with a graphical front-end and has a large user base.
I've set up this GitHub repository which guides you through the installation and illustrates how this can work for SC4 custom content by installing some example packages. This is already functional. I encourage you to go through the process or at least read the steps in the README as this has a lot of potential. The initial setup might look a bit daunting and the graphical front`end has some quirks, but that seems tolerable compared to some of the quirks we have gotten used to over the years (dependencies, puzzle pieces, crowded menus, etc). (Side note: I have not tested this on Windows yet, but in principle it should work or can be fixed easily.)
Such a repository could host metadata for a selection of custom content (such as MODPACCs and their dependencies). The metadata contains the information that is needed to install plugin files in an automated fashion. As Conda is a fully fledged package manager, it is possible to automate even the selection of dark-night/standard-night model variants or CAM/no-CAM variants.
What is missing to make use of this right now is a way to distribute the actual custom content in the form of Conda packages. A Conda package consists of the actual data, like a props pack .dat file, as well as metadata. There are mainly two options going forward:
Host the Conda packages alongside the original zip files directly on the STEX and LEX. In the same vein as Project ZIP (many many thanks to the herculean efforts by @Tyberius06), this merely constitutes a repackaging of existing content which seems to be an accepted practice in our community. Alternatively, seek explicit permission from authors to rehost the files somewhere else.
Instead of including the .dat files directly, just include download links to the files on the archives. Conda places those download links into the plugins folder and another program must be used to automatically bulk download all the files. Such a program would have to be written -- maybe LEXDownloaderX could be repurposed for this. This approach subverts many features of Conda because it installs files Conda does not know about, so such a setup would be a bit brittle, but might be a short term client-side solution which is even site-agnostic.
The first option would be most preferable. It would require a collaboration of many people, especially the admins of both sites, but the potential outcome seems well worth it for the longevity of our community (the NAM Team has ideas for 10 years to come).
TL;DR Have a look at this step-by-step guide https://github.com/memo33/sc4pac-example
Potential new SC4 package manager idea explained.
Recommended by Cyclone Boom
Recommended
Happy to see some activity in here again, and I suppose that means I should share my visions and ideas towards a "Mod Manager" of sort; it's something I've been thinking and planning about for nearly a year mostly independently, but more recently with two other people.
My vision
A while ago I had the pleasure of reading through this discussion in its entirety, and while quite long winded at times, I found some excellent kernels in there that sparked some ideas, and I began wondering how and what I could build off of. While most of the rest of the thread was constructive as far as the "soft" concepts and ideals for the system, I pulled out a handful of posts which I think were trending the right way and were the most constructive as far as physical needs and requirements. Reading through it all, I couldn’t help but to see strong parallels between things brought up earlier in this thread and this Prop + Texture Catalog concept which I have built and slowly expanded a few times.
I’m envisioning some kind of system that
a) keeps track of all content uploaded to STEX/LEX,
b) keeps track of content of items in a player’s Plugin folder, and
c) is able to bridge the two (downloading content, managing dependencies, etc.) Regardless of how the end system gets implemented (some sort of package manager solution sounds great to me), the system ultimately still needs to know what additional things are required for a given upload. For that, there needs to be something major in place first - some sort of database of information where things are managed and linked. For this database, I think we’d need at least 3 groups of tables. One for the uploads themselves and metadata about them (name, URL, date uploaded, version, date modified, maybe even tags/categories, possibly dependency breakdowns, definitely incompatibilities, and/or other metadata as appropriate). A second for all of the TGIs in each upload (kind of like a much-expanded Prop & Texture Catalog but for all other exemplars too including buildings and other required files). The third is a list of the TGIs currently in the player’s Plugin folder. Exactly whatever the data structure ends up looking like is not important for now, and I am focusing for now on just massing an index of everything I can. Some in the Sharing Plugins Discussion have offered the opinion that such a task would be herculean; I don't necessarily agree with that. It certainly will take a lot of time and investigative work, but from what I have tested quickly so far it seems doable. I think that should maintain even as the scope scales up. I'm making some initial assumptions about the degree of detail this theoretical dependency manager would go to (include to the individual building/prop/texture/LUA/etc exemplars for lots and ignore the FSH/BMP files which are technically dependencies for models). My implementation so far I loved the idea of MODPACCs discussed, and figured bundling all of my B62 content would make a great start. First, I started a C# library to read and parse DBPF files. This was mostly a personal challenge to learn and accomplish, and I believe it will prove invaluable to me as I work towards achieving many of my ideas, from this plugin sharing to reversible datpacking to improved PIMX. This code library allows me to look into a DBPF file to actually read its contents - it won't matter if someone has datpacked or renamed or done anything else - we still should be able to know what a specific file is and what it contains. Additionally, I will need to transition to something actually suited to handle this huge amount of data - my JavaScript filtering in the Prop + Texture Catalog which while pretty neat and something I am proud of will definitely not be up to the task. It’s served its purpose. With assumptions I mentioned above in terms of scope and detail, I'm wildly guessing the end result will be somewhere on the order of tens of millions of records. A proper data management implementation should eat that up just fine considering that we'd only be adding a few thousand more records each month at the most, assuming roughly the current rate of uploads. (Timeout here to send a huge thank you to @memo - your jDBPFX library was invaluable in helping me getting started on my own library - it was a tall task learning C# and how to analyze DBPF at the same time and you were a huge help for the latter). Anyways, using csDBPF, I created a barebones program to use that library to parse through the B62 content and create CSV tables as placeholders for data tables in an actual database I described above. The first table examined all of the LotConfiguration Exemplars and listed all of the IIDs for rep 13 of the LotConfigPropertyLotObjectData properties – this is essentially a list of IIDs of all buildings, props, textures, flora used on the lot. This is the list of all LotConfigPropertyLotObjectData used in the B62 Remastered lots. The second table or set of tables is a list of TGIs and some identifying information about the file they come from, and associated tables detailing metadata about each upload. Here all TGIs uploaded ever are represented. After referencing the first table against the second table, I can have the exact list of dependencies for a given item. If this list is complete, then by definition I also know which packs the IIDs are located in and where to download them. (Tangentially, I would also be fascinated to see a heat map type visual of which dependency packs are most utilized and even which props are most referenced too - tons of minimally useful yet fascinating knowledge is hidden here). A third table could contain all of the TGIs you have in your plugins folder. This can be cross referenced between the first two tables to determine what you already have in plugins, and what needs to be downloaded. I've put together a quick prototype using all of the uploads in my B62 Remastered series. It's located here in Power BI, but the gist is that i can know the required dependency packages for any chosen lot. It's also embedded below for you to play with. Despite my csDBPF library being in its infancy, I am pretty much already able to accomplish putting together table one (IIDs used for a given lot or set of lots) and table three (all TGIs in a plugin folder) listed above. Table two (the list of all TGIs uploaded) is where things get interesting... Indexing all known uploads Looking from the LEX side of things for a moment, @CasperVg has built a fantastic system with API access to LEX file data (example [LEX login required]). This makes it fantastically easy to index and cache information on each upload. I'd be trivial to iteratively look at each item and store relevant information in a database for referencing. The STEX database stores a lot of this same information, but because there's no custom API, it's a little bit more difficult to get to everything. Critically it also does not list the most important thing either: dependencies. On a related side note: A long time ago, CB kindly shared a sample of it with me. Here's a link to what it looks like. Basically there's two main tables: one for each file listing containing the listing information, and one for "attachments" to each listing including screenshots and the uploaded file(s) to download. Circling back to figuring out what is required for each upload, this is where I am currently stuck right now. It's possible to iterate over the STEX database and programmatically download, extract and analyze each file to get a rough idea of what IIDs are used (and thus what dependencies are required), there's no way of knowing if that item is the original prop/texture/building or if it is an override and actually refers to some other upload. Note that this also does not cover other dependencies which may be required like queries, sounds, LUA scripts, or other kinds of resources, though I'm sure that can be figured out in time. All of this might look something like @catty-cb has done with her Catalog, but would be more comprehensive and would include the LEX and possibly other non English language sites as well. This section is still kind of up in the air while I work to improve and simplify and expand my csDBPF library and further build out a draft of the database schema expressed in the Power BI report linked above. I've pretty much taken a break from both of those for the summer, but towards the fall I'll be diving head first again into those projects and ramping up collaboration with others on this and related projects. Conclusion The way I see it there's really three aspects to a sort of "mod manager" that can manage downloads and dependencies: the part the user interacts with, the storage of the files, and the data that is the foundation for the system. Regardless of what the first two parts turn out to look like, I've chosen to get started on the data component because it seems reasonable to accomplish and make significant progress without having to come to some sort of consensus as a community. That was quite a lot, but it basically summarizes most of what I have been thinking and slowly working towards for nearly a year. That's what has been floating around in my head, and I'm super excited to hear what yall might think about it, and curious about any other potential issues or shortcomings I might have missed.
a) keeps track of all content uploaded to STEX/LEX,
b) keeps track of content of items in a player’s Plugin folder, and
c) is able to bridge the two (downloading content, managing dependencies, etc.) Regardless of how the end system gets implemented (some sort of package manager solution sounds great to me), the system ultimately still needs to know what additional things are required for a given upload. For that, there needs to be something major in place first - some sort of database of information where things are managed and linked. For this database, I think we’d need at least 3 groups of tables. One for the uploads themselves and metadata about them (name, URL, date uploaded, version, date modified, maybe even tags/categories, possibly dependency breakdowns, definitely incompatibilities, and/or other metadata as appropriate). A second for all of the TGIs in each upload (kind of like a much-expanded Prop & Texture Catalog but for all other exemplars too including buildings and other required files). The third is a list of the TGIs currently in the player’s Plugin folder. Exactly whatever the data structure ends up looking like is not important for now, and I am focusing for now on just massing an index of everything I can. Some in the Sharing Plugins Discussion have offered the opinion that such a task would be herculean; I don't necessarily agree with that. It certainly will take a lot of time and investigative work, but from what I have tested quickly so far it seems doable. I think that should maintain even as the scope scales up. I'm making some initial assumptions about the degree of detail this theoretical dependency manager would go to (include to the individual building/prop/texture/LUA/etc exemplars for lots and ignore the FSH/BMP files which are technically dependencies for models). My implementation so far I loved the idea of MODPACCs discussed, and figured bundling all of my B62 content would make a great start. First, I started a C# library to read and parse DBPF files. This was mostly a personal challenge to learn and accomplish, and I believe it will prove invaluable to me as I work towards achieving many of my ideas, from this plugin sharing to reversible datpacking to improved PIMX. This code library allows me to look into a DBPF file to actually read its contents - it won't matter if someone has datpacked or renamed or done anything else - we still should be able to know what a specific file is and what it contains. Additionally, I will need to transition to something actually suited to handle this huge amount of data - my JavaScript filtering in the Prop + Texture Catalog which while pretty neat and something I am proud of will definitely not be up to the task. It’s served its purpose. With assumptions I mentioned above in terms of scope and detail, I'm wildly guessing the end result will be somewhere on the order of tens of millions of records. A proper data management implementation should eat that up just fine considering that we'd only be adding a few thousand more records each month at the most, assuming roughly the current rate of uploads. (Timeout here to send a huge thank you to @memo - your jDBPFX library was invaluable in helping me getting started on my own library - it was a tall task learning C# and how to analyze DBPF at the same time and you were a huge help for the latter). Anyways, using csDBPF, I created a barebones program to use that library to parse through the B62 content and create CSV tables as placeholders for data tables in an actual database I described above. The first table examined all of the LotConfiguration Exemplars and listed all of the IIDs for rep 13 of the LotConfigPropertyLotObjectData properties – this is essentially a list of IIDs of all buildings, props, textures, flora used on the lot. This is the list of all LotConfigPropertyLotObjectData used in the B62 Remastered lots. The second table or set of tables is a list of TGIs and some identifying information about the file they come from, and associated tables detailing metadata about each upload. Here all TGIs uploaded ever are represented. After referencing the first table against the second table, I can have the exact list of dependencies for a given item. If this list is complete, then by definition I also know which packs the IIDs are located in and where to download them. (Tangentially, I would also be fascinated to see a heat map type visual of which dependency packs are most utilized and even which props are most referenced too - tons of minimally useful yet fascinating knowledge is hidden here). A third table could contain all of the TGIs you have in your plugins folder. This can be cross referenced between the first two tables to determine what you already have in plugins, and what needs to be downloaded. I've put together a quick prototype using all of the uploads in my B62 Remastered series. It's located here in Power BI, but the gist is that i can know the required dependency packages for any chosen lot. It's also embedded below for you to play with. Despite my csDBPF library being in its infancy, I am pretty much already able to accomplish putting together table one (IIDs used for a given lot or set of lots) and table three (all TGIs in a plugin folder) listed above. Table two (the list of all TGIs uploaded) is where things get interesting... Indexing all known uploads Looking from the LEX side of things for a moment, @CasperVg has built a fantastic system with API access to LEX file data (example [LEX login required]). This makes it fantastically easy to index and cache information on each upload. I'd be trivial to iteratively look at each item and store relevant information in a database for referencing. The STEX database stores a lot of this same information, but because there's no custom API, it's a little bit more difficult to get to everything. Critically it also does not list the most important thing either: dependencies. On a related side note: A long time ago, CB kindly shared a sample of it with me. Here's a link to what it looks like. Basically there's two main tables: one for each file listing containing the listing information, and one for "attachments" to each listing including screenshots and the uploaded file(s) to download. Circling back to figuring out what is required for each upload, this is where I am currently stuck right now. It's possible to iterate over the STEX database and programmatically download, extract and analyze each file to get a rough idea of what IIDs are used (and thus what dependencies are required), there's no way of knowing if that item is the original prop/texture/building or if it is an override and actually refers to some other upload. Note that this also does not cover other dependencies which may be required like queries, sounds, LUA scripts, or other kinds of resources, though I'm sure that can be figured out in time. All of this might look something like @catty-cb has done with her Catalog, but would be more comprehensive and would include the LEX and possibly other non English language sites as well. This section is still kind of up in the air while I work to improve and simplify and expand my csDBPF library and further build out a draft of the database schema expressed in the Power BI report linked above. I've pretty much taken a break from both of those for the summer, but towards the fall I'll be diving head first again into those projects and ramping up collaboration with others on this and related projects. Conclusion The way I see it there's really three aspects to a sort of "mod manager" that can manage downloads and dependencies: the part the user interacts with, the storage of the files, and the data that is the foundation for the system. Regardless of what the first two parts turn out to look like, I've chosen to get started on the data component because it seems reasonable to accomplish and make significant progress without having to come to some sort of consensus as a community. That was quite a lot, but it basically summarizes most of what I have been thinking and slowly working towards for nearly a year. That's what has been floating around in my head, and I'm super excited to hear what yall might think about it, and curious about any other potential issues or shortcomings I might have missed.
Concept of mod manager and file parsing implementation details.
Recommended by Cyclone Boom
-
Similar Content
-
Recently Browsing 0 members
No registered users viewing this page.
