Going to WAR to put custom content in a JAR

[Romaq 18]

I mentioned this in the Mac forum, but one of the issues that really causes me annoyance with SC4 and the STEX, the LEX, the PLEX, and pain in the NEX, is the issue of how to package custom content.

I think it's safe to say that Maxis had NO PLAN for SC4 being a killer hit with custom content. They had no clue that ... could people just post how big their /plugins/ folder is? The Menu Management scheme is something that should take a separate thread when we have some direction on MC's take, but just to make the point: Maxis did NOT plan on custom content having so large an impact on the SC4 community.

Well, we're here, and it's easy to say there is a LOT of 'stuff' one may really enjoy using.

I hope Monte Cristo is allowing for a 50 fold return on their basic content. That is, if CU's actual content is half a gig, Monte Cristo should allow for custom content to fill out 25 Gig at least. Actually, I don't think 500 fold return on custom content is 'unreasonable'. But let's take that as a starting point.

25 Gig is a *LOT* of data to manage. Something that grieves me with SC4 is that you have to use a third party program like the SC4 Startup Manager to segregate that content by region or even by city. An example I give is that I want PEG's Mountain Theme Package in mountain terrain. But I want to explicitly exclude that content from the Valley and Bay locations. I'd prefer to not even have it clutter up my menu.

Then there's the issue of... where'n'hell did THAT building come from? Where did I download so'n'such? I would HOPE they mention that within the documentation that I *DIDN'T* put in my /plugins/ folder. Different groups have different filename and organization conventions. Some individual packages have no convention at all, or documentation to go with it either.

And then there's that one word that causes pain and terror on the STEX: DEPENDENCIES. The use of dependencies can save lots of redundant work, time, effort, system memory, and whatnot. Dependency MANAGEMENT within SC4 is... I'll just call it by the technical term 'nonexistant'. You didn't fulfill the dependency, you get a brown tur... er, package in your game. Don't know what's missing? It sucks to be you. So in addition to third party package management, we get third party dependency tracking.

One 'resolution' to some of these issues has been to create .exe installers that make the attempt to figure things out for you. But you are on a Mac? Oh, I'm sorry, did I break your concentration? Well, go to this OTHER website and download the .zip file. That is, if you are on a Mac, or you are someone who is really REALLY paranoid about downloading other people's .exe files. Or you are someone who want's to know what's going into plugins and is making a vain attempt to NOT have so many directories within the /plugins file. AND someone who really really hates having to go through an installer and have that stuff show up in his 'control panel | add/ remove programs' list. And I'm one who really REALLY hates that.

SC4 didn't build the concept of a metadata file or a consistant packaging scheme. I would really, REALLY like for CU to plan on killer custom content of the quantity and quality they see among a dozen SC4 *EX (many different EXchange) sites.

I offer as example Java 'WAR' and 'JAR' files. Not that CU should *USE* those 'as is', but borrow from the concepts those things which are useful and apply. For the purpose of this discussion, I'll just refer to WAR files, but you can get more details and inspiration from those formats from http://java.sun.com/j2se/1.4.2/docs/guide/jar/jar.html.

Some things to note: A 'package.jar' file may be renamed to 'package.zip', and all the contents are easily available and manipulated by any PKZip utilitity including Windows Explorerer, which is a part of any Windows OS that will run Cities Unlimited.

The JAR file contains a 'META-INF' directory that holds configuration data, files that explain what is in the package, where everything is found, signatures to verify WHO created the package and so on. It's not my intention to explain JAR files, but to suggest a way to resolve the current mess that SC4 custom content is in. A really freakin' cool feature I could see with this? What if I uploaded a 'CU Package' to the STEX, and by doing so the STEX would insert a tag that would point people to where it will be downloaded? I don't KNOW that information the first time I upload something. But if the STEX had a means to insert that into my data, then when someone downloaded my package, they would automatically have a means to go back to that link on the STEX. If a particular building was causing problems, or maybe it just looks really good and you felt the urge to comment on it, it may be possible to build into the building menu a means to access the 'where did it come from?' info.

But if you don't have that information WITH the package in the first place, none of the rest of it matters.

Something ELSE I could see the STEX doing... suppose instead of having to put in a description on upload as well as offer two images, suppose the STEX were to insist that information be found IN the package? So I upload my package, and everything the STEX needs to know to present my content... the images, the text and so on... it's right there in the package to be found. The data you find when you look something up in the STEX is data that really REALLY ought to be built into the package. A consistant package format that everyone expects would make it possible for the STEX to simply require the data is there for it to pull out and put on display. For that matter, the STEX software wouldn't have to hold separate images and test. It would be possible to extract that 'on demand'.

Guess what? It would be very trivial for a third party program, if necessary to do the SAME in terms of simply making a 'STEX-like' view of what'n'hell is IN that freakin' 25 Gig folder of custom content on my PC.

You may also have a place for documentation, key words, and so on. I'd really like this thread to be a place for people who MAKE the content to play some 'what-if' games in their minds. 'What if' the STEX as such didn't exist yet? 'What if' we were just given the specs on how to create the content, but all we had to do was come up with some sane way to organize it? Knowing what we know since about March 2003, how can we make the custom content package scheme work for US instead of the BSC folks having to do their dependency checker, the Startup Manager and ... the rest of this which can only be called a total mess from which we are four years too late to do anything about it.

Here's a chance for us to do something about it, and I hope Monte Cristo understands the scope of the problem as well as has an interest to work with us on avoiding it with their product.

And I also really, REALLY hope Monte Cristo plans to see 250G of custom content for CU be the standard for the hard-core CU player in 2012. Too bad Maxis never hedged their bets to be that successful with SC4.

[Zwabberaar 18]

This is way too much text to read. Can't you make a summary?

[fern244 1]

agree, way too much

[lasergoat 0]

This is very important and thank you for bringing this up. 

Please take the management of custom content very seriously Monte Cristo.  If this is a popular game, then the amount of custom content one uses will be incredible.  I used to have a 2 gig plugin folder and out of frustration I'm reduced it to 300 megs after performance and management issues.

It would be neat if there was a common repository everyone could update their base plugin folder making a common base of plugins.  In that repository would be custom content from the community that is high quality and balanced.

[Mr_Maison 2,426]

I also agree that this is a major issue to consider. It would be cool if all available dependencies were put into one regularly updated package. In my opinion, dependencies should have a separate installer that actually updates the plugins folder the same way we update virus programs or MS content. Call it a "Dependencies Updater". And there can be a place in the options menu to activate what downloaded content we want to use. I dont mind dependencies sitting in my plugins that I may not be using at the time, but it's better to have them and not need them than to download a brown package. Besides....there's stuff that came with SC4 that I never used, but at least the option is there.

[MayorTim 16]

If your too lazy to read something just because its long, don't spam the thread complaining. That drives me nuts.

Anyway, I read through it, and there are some very good points and ideas in there. It's written almost like a rant, but I'm sure your not the only one who's frustrated with the way custom content is handled in SC4.

This is going to have to be a very major aspect of Cities Unlimited if they truly want to make a game with replayablilty. They should spend a lot of time on this, and if they could make an intuitive "manager" for custom content, it's going to really help this game.

[BurroDiablo 222]

Although I didn't understand all that War and Jar file stuff, I definitely agree that more stable plugin system would be a bonus in CU.

[KeanoManu 42]

It would be awesome if they could provide us with an menu-editor. So that we can change everything in the menu, add new categorys, change in which category a building is and so on. It would be cool if they could let us do our own zones. Just add a new zone-type in that editor and then say which buildings that can be built when zoned with that zone.

[Androv 6]

Yeah, my 'parks' menu option had 200+ options in it :-(

[north country dude 205]

Have to agree with MayorTim on the childish "it's too much text to read" replies. They were totally unnecessary and instead just informs the rest of us that some people have the attention span of a goldfish.

Moving on,...

Romaq makes some very good points and suggestions for the developers to follow.

However, Custom Content management and Dependencies are not just an SC4/Rush Hour problem. Having played many different simulation types that encourage Custom Content by the users, I have found that the problems of missing dependencies, nonexistant readme files, and incorrectly labeled files to be a common problem when such content is created by 3rd( or 4th, 5th, 6th, etc) parties, in other words, custom content created by the average user will have some errors (see my signature).

As an example, (if you are familiar with the Trainz Railroad Simulator 2006), TRS2006 ships with a program called the Content Manager Plus, or CMP.  This program works with the main download site at Auran (the developer) and supposedly "helps" you get the custom content and looks to see if it has dependencies and gets those for you if you don't have them installed. It also keeps track of every file currently available on the main site, and sometimes other sites as well. This is called the "Asset Database" and is updated every time you run the CMP program.

Currently, there are over 85,000 pieces of custom content available within the database. The CMP lists all content and notes its location (your PC or the Download Station), it's status ('ready to use' or 'open for edit'), and its condition( no faults, faulty, missing dependencies). When it works, it's really quite handy.

BUT, it cannot keep track of content made and posted elsewhere. It cannot keep track of "payware". While it does tell you what is "faulty" with a certain file, it does not tell you how to fix the faults. On the plus side, it does have a built in editor that allows you to repair the file if you know what you are doing.

Another drawback for the whole system is the fact that this content is made all over the world. The content is generally always labeled in the creators native tongue. If you don't happen to speak German, or Italian, or French, or Spanish, or etc, you have no idea what some of the stuff listed in the CMP database is.

There is a classification system that divides the objects into categories, but the categories are assigned to the custom object by the creator when they post it for download. I have found in the category "Scenery:Flora", telephone poles???, or roads???, or traffic lights???. Strange, I didn't know these things were grown.

So you see, while there is an elegant program available for handling the large amount of custom content in Trainz, it still comes down to how smart or savvy the individual user is that uses it.

I'm not saying "don't try for perfection", but I am saying "don't expect perfection".

[Mr_Maison 2,426]

Is it wrong for the sites that host custom content to require modders to follow rules for uploading their content for the end user's sake? Otherwise it won't be uploaded and we won't know what we're missing anyway. I do not ask this in sarcasm. It's just a question.

[dasilva 3]

Originally posted by: KeanoManu It would be awesome if they could provide us with an menu-editor. So that we can change everything in the menu, add new categorys, change in which category a building is and so on. It would be cool if they could let us do our own zones. Just add a new zone-type in that editor and then say which buildings that can be built when zoned with that zone.quote>

I like that idea... I'll check with the development team if it's something in their plans. If not, maybe make a suggestion...

On the main topic about custom content handling, I cannot give you much information on this but we have been working in something that should just make this issue you met in Simcity 4 a no brainer: CU should handle this issue for you.

However, I cannot see a way to solve the situation where you download so much content that your computer doesn't meet performance issues: you are only limited by the power your machine can deliver...

[fern244 1]

ok, finally read it all and totaly agree. I have about 300+ landmarks in a single menu! Another thing i hope they do is let you have different terrain/water mods. For example in simcity for me i can only use one terrain mod at a time, what if iwant my land to be half snow half desert? My game constently crashes because my plugin folder is so big.

[WackoJak 0]

I agree with KeanoManu that a menu-editor would be a very cool and helpful feature, but drawing up script for a custom UI for it would be horrendous I imagine.

An advanced plugin manager in CU would benefit everyone greatly in the long-term and if there were some way to ask for it personally I would second it immediately.

[Romaq 18]

I'm sorry if it was too big a chunk to drop at one time, but I've been stewing over how custom content is packaged since I it the 1G mark.

The reason I brought up Java 'Jar' and 'War' files is because SC4 custom content is in 'some package format', it contains different objects, images, text, whatnot... but I can't do the following:

'rename sc4thing.dat sc4thing.zip' and be able to examine WITHOUT the use of a thirdparty kit everything in the file. A JAR file lets you do this quite nicely.

sc4thing.dat does not let you store config data within the file so you could have just ONE file of MULTIPLE 'Peg Rocks', for instance, and then just edit the 'config' data in the package to choose which rock texture you want displayed. JAR files have that notion built into the package.

sc4thing.dat does not have any notion 'metadata file info' (who made this, what version is it, how can I get help). JAR files do, and the culture of Java expects that data to be htere.

sc4thing.dat does not store documentation, text or HTML with images within the package. SO you either have to use a Windows installer program OR pull that stuff out of the .zip file. JAR files allow and expect documentation within the package.

sc4thing.dat does not have a notion of dependency assistance. Either that dependency you need is there or it sucks to be you. Stare at a brown package, sport! Java has dependency management built into the Jar package specification.

This is NOT to say that CU should be using Java Jar file format for custom content. What I'm saying is that custom content should have within the package the same advantages and features that Jar *can* do, and if we develop this as part of the culture of CU custom content (same as there's pressure to behave one's self on the STEX) then, while things won't be perfect, it will be a better launch point for going forward. It's also handy to have CU specifiy minimum requirements for a custom content package to even be looked at.

The problem of 'hardware doesn't cope with 250G of custom content' is an issue of 'it sucks to use the rig you do', and as one fills up on custom content to the point it drags the game, one must make choices. And, of course, a better way to manage said content will make THINNING OUT said content easier, especially in the realm of dependencies.

Concerning menu issues, it really wasn't my intent to cover that specific problem. I simply point out the menu problem as a symptom of Maxis NOT planning on SC4 being the success it turned out to be, at least with regard to custom content. Where Custom Content meets menus sounds like a good thread for someone ELSE to make. I've caused enough grief here in this thread.

The really REALLY short summary is that I hope Monte Cristo plans on CU being a huge success with an endly supply of custom conten from fans, content that will HAVE to be managed 'somehow'. The better they plan on so huge a success, the easier we'll be on using it when the success of CU arrives.