To Pack or Not To Pack

[Vern 29]

So I'm not referring to the Second Amendment (the right to bear arms) of the United States of America. 

In my search to resolve an issue that I previously had with SC4 CTD I stumbled upon dat-packing using wouanagaine's SC4 Dat Packer Utility found here . Since I still download lots of content part of my process is to "pack" after I've thoroughly tested for compatibility. 

Recently I've decided to use the DAMN menu and ended up using my "unpacked" directories to try and fix some little glitches I was experiencing with those DAMN menus.  My question is this:  How many people are still packing?  Does it make sense these days to continue packing? I mean our CPUs and GPUs have come along way since the SC4 Dat Packer Utility was published.  Is it still being used by the community?

BTW, I use Startup Manager since there are scenarios where I don't want to load certain plug-ins.

[ilikehotdogsalot 311]

I packed my plugins once and it improved the game but it was a little difficult when I wanted to change things in my plugins because I packed pretty much everything.

If you want to, maybe pack the NAM, terrain mods, or things that have a lot of files to them.

[Vahr 341]

maybe pack the NAM

quote>

Do not datpack the NAM

On the main note, its up to you. When I packed my 2GB some plugins, I had no improved performance, and it was a hassle to change things. But that's just me.

[Timmystwin 201]

I pack the DESC, SAV and LOT into 1 using reader. only the LOT is needed but it gets rid of them from mah pc. I used to pack the models but i never bother

[joshriddle 29]

What I am trying to find out is this:

I have an office program that allows you to open any unopenable file in the Text Editor part of it. What I am trying to find out is to just open the reader, and then save an empty dat, and copy all the stuff from the other dat's into that one. I would definately not do it, but just wondering.

Dat Packer might work for me. The thing is I am not going to go out of my way to get those extra .dlls. I had trouble enough installing DX to use Terraformer. Like Blue said, too much of a hassle.

Also I don't see where there would be any performance improvements. You pack the dats. Say you pack 2 of them. Each is 200 KB. Isn't still gonna be 400 KB? Nothing is changed in the amount of plugins you have, it is all just compressed into one.

Not to mention how do you unpack them? If you use it for awhile and it quits working and or if you decide that you don't like it anymore and see another one that is better so to speak, then what are you going to do? To me Dat Packer is, well, hate to say but nothing I would want to use.

[Vern 29]

Originally posted by: Blue Lightning

maybe pack the NAM

quote>

Do not datpack the NAM

On the main note, its up to you. When I packed my 2GB some plugins, I had no improved performance, and it was a hassle to change things. But that's just me.quote>

This is exactly the info I was looking for. TX!! Anyone else feel the same way?  Now that you did mention it I do remember seing somewhere (probably the readme file) not to pack the NAM. I've packed a lot of the dependencies

[Vern 29]

Not to mention how do you unpack them? If you use it for awhile and it quits working and or if you decide that you don't like it anymore and see another one that is better so to speak, then what are you going to do? To me Dat Packer is, well, hate to say but nothing I would want to use.

quote>

You wouldn't unpack them. You would have kept your unpacked folders.  One of the pitfalls of this is that you now are using more diskspace but with the cost of  HD being so low it may not be a factor. Whey you need to add  another plug-in you would add it to your unpacked folders and then repack it.  This is a pain if you are trying a lot of plug-ins.

Also I don't see where there would be any performance improvements. You pack the dats. Say you pack 2 of them. Each is 200 KB. Isn't still gonna be 400 KB? Nothing is changed in the amount of plugins you have, it is all just compressed into one.quote>

I think the improvement is suppose to come from having fewer dat files to load since it basically will create one dat file for each folder.  Blue Lighting has details in the post below.

The readme file for it has some good info on how SC4 loads and tips.

[Vahr 341]

Also I don't see where there would be any performance improvements. You pack the dats. Say you pack 2 of them. Each is 200 KB. Isn't still gonna be 400 KB? Nothing is changed in the amount of plugins you have, it is all just compressed into one.

quote>

There is basis behind this: The speed it takes for the game to request a file from Windows, Windows to give it the file, and the game to unload the file. The game does this upon startup:

1. Load DBPF

2. Read DIR

3. Unload DBPF

4. Repeat until all is finished.

Datpacking essentially, for instance, you have 500 files. That's 500 directories to load and 500 calls to Windows to load, and 500 calls to unload.

Packing them into 1 DAT reduces it to 1 call to windows, 1 large DIR to read (however, DIR reading is extremely efficient), and 1 call to unload. Multiply that over thousands of times and you get a large speed increase.

[Vern 29]

Originally posted by: Blue Lightning

Also I don't see where there would be any performance improvements. You pack the dats. Say you pack 2 of them. Each is 200 KB. Isn't still gonna be 400 KB? Nothing is changed in the amount of plugins you have, it is all just compressed into one.

quote>

There is basis behind this: The speed it takes for the game to request a file from Windows, Windows to give it the file, and the game to unload the file. The game does this upon startup:

1. Load DBPF

2. Read DIR

3. Unload DBPF

4. Repeat until all is finished.

Datpacking essentially, for instance, you have 500 files. That's 500 directories to load and 500 calls to Windows to load, and 500 calls to unload.

Packing them into 1 DAT reduces it to 1 call to windows, 1 large DIR to read (however, DIR reading is extremely efficient), and 1 call to unload. Multiply that over thousands of times and you get a large speed increase.quote>

That being said has the HW improvements we have today diminished the performance gained from dat packing?  Your thoughts Blue?

@Blue Lightning - Would you happen to know if Startup Manager loads all the plugins in memory and just hides the unused ones from SC4 or are they all loaded regardless and access to them is blocked?  I think Startup Manager does it from the systems point of view since if I try to unzip a file into a directory I will always get a error if its running.  Although it does the operation anyway.

[A Nonny Moose 8,261]

Generally I use ilive's reader to pack my BAT lots.  You need to include only the model, desc, and lot files.  It is possible, I suppose if you are adroit with the reader, to put the stuff from the desc file on the lot file and then you only need the model (graphics) and the lot.

[mproca 0]

1. zip em all first on multiple categories like:

Buildings (RC +/- I)

Agriculture

Industry (yes... i added it twice since this is the main pack for industry and it is harder to make 3 separate R C I packs)

Airports

Seaports and Canals

Stations

Other

...

keep em in this format !

2. now select what u need and copy em in a folder and zip it. again

3. in the end take this last zip files and pack em to dat

Or u can just skip pass 1 and put em on 2 categories like

Buildings 1

Builgings 2

Buildings 3

...

Anyway i have a pain too sorting out these plugins.

U pack em now... one year later u will have twice as many plugins and u want to repack em and take away some plugins no longer nice

[SC4BOY 21]

I DatPack absolutely EVERYTHING outside the root directory. There will NEVER be a problem from doing this. Where problems come about is people not understanding or being sloppy about how DatPacker works and how it relates to the files in the plugins. The DatPacker in almost every iota processes the files EXACTLY like SC4 does (the exception is certain infrequent issues such as certain ascii characters in file names such as might occur if you are doing Japanese or Korean bats .. but it is well-behaved in this matter and will TELL YOU if you pay attention to the "process window" in DatPacker). Doing maintainance on DatPacked files is where the risk comes in. If you are at all in doubt, just "fresh DatPack" the plugins EVERY TIME.. don't take any shortcuts trying to be cute... DatPacker is MUCH SLOWER than SC4 for reasons I don't know.. perhaps ask Wounagain if you are actually curious.. I'm not.... I am a patient man.. and if it takes an hour to DatPack my plugins, I could care less.. I do it rather infrequently. That's my two cents worth.. its a great program and works quite well.... The point where it save time is loading the game and city files and in view changes.. it doesn't make a LOT of difference in other areas. If you try it and it doesn't save you "play time" its simple enough to drop it.. but it has made big differences to my play.

[joshriddle 29]

OK so I found out I DID have the required dlls to make it run properly. It only took me like 10 minutes to datpack everything. Then I wound up with only 12 files instead of 678. I wanted to datpack those 12. Which it would not let me do. It might be neat to have a 1 file plugin that holds everything. Might be dangerous too.

So thankfully, it stopped the operation of packing my 12 files to 1, cause heeding Blue's warning not to datpack the NAM, it meant to move the NAM files to a different location when I did pack them. Turns out me being stupid they were sitting there about to be packed... so that was a sigh of relief. Allthough I don't know the unfortold crahes or whatever will happen when you datpack the NAM Files.

I must say on the other side it was convenient. Cut back on my plugins folder size 100 MB even. So that is good. Probably those are an accumulation of some 200 folders in there.

[ilikehotdogsalot 311]

Woops, listen to the one who knows what they're talking about, don't pack the nam

[z1 566]

Originally posted by: SC4BOY

I DatPack absolutely EVERYTHING outside the root directory. There will NEVER be a problem from doing this. Where problems come about is people not understanding or being sloppy about how DatPacker works and how it relates to the files in the plugins. The DatPacker in almost every iota processes the files EXACTLY like SC4 does (the exception is certain infrequent issues such as certain ascii characters in file names such as might occur if you are doing Japanese or Korean bats .. but it is well-behaved in this matter and will TELL YOU if you pay attention to the "process window" in DatPacker). Doing maintainance on DatPacked files is where the risk comes in. If you are at all in doubt, just "fresh DatPack" the plugins EVERY TIME.. don't take any shortcuts trying to be cute... DatPacker is MUCH SLOWER than SC4 for reasons I don't know.. perhaps ask Wounagain if you are actually curious.. I'm not.... I am a patient man.. and if it takes an hour to DatPack my plugins, I could care less.. I do it rather infrequently. That's my two cents worth.. its a great program and works quite well.... The point where it save time is loading the game and city files and in view changes.. it doesn't make a LOT of difference in other areas. If you try it and it doesn't save you "play time" its simple enough to drop it.. but it has made big differences to my play.quote>

I would agree with this completely, right down to the capitalization, with only two exceptions.  I don't DatPack either the NAM or RTMT.  The game will still work fine if you do; it's just that I make frequent changes in both areas, and that's a lot more difficult if they're datpacked.  Also, both the NAM and RTMT are relatively small, so DatPacking them would make no perceptible speed difference.

Also, DatPacking is not an all or nothing deal; it's very easy just to DatPack individual folders (and their subfolders) when you change something.

One huge advantage of DatPacking that most people don't know about is the difference it makes during actual game play.  Specifically, returning to the normal view from the Zones view can take several minutes if you don't DatPack your files.  If you do, it takes no more than a few seconds.  The same goes for the NAM Subway Building View, which is based on the Zones view.

As for fast computers, I've got a speed demon myself, but I still wouldn't think about giving up DatPacking.

[SC4BOY 21]

There are some warnings I failed to mention, but related to the issue of "pack or don't pack". How you tend to play the game will determine how you should handle these with DatPacker.

1. There are programs which modify the parameters for SC4 "on the fly" such as the traffic simulator tool which expect to find parameters in certain "slots" so they can be properly modified. Since DatPacker modifies the files, the info is no longer in the "slots" where it is expected.. those areas cannot be packed or your 'changes" will fail and certainly may cause unpredictable results.

2. Files that have "structure modifiers" such as the NAM.. which really is a set of "parts" which can be put together and taken apart depending on what checkmarks you make when running or re-running the "installer". For the same reasons above, namely that DatPacker modifies and rearranges the info, those "install/modify" packages will not yield the expected results if you try to run them on packed dats.

3. There is a third relatively obscure piece of info.. that is if you make a single "dat file" by processing a batch of folders and the single files gets over about 750Mbytes, the game will not handle them properly (or possibly DatPacker yields some error.. the end result is the same either way). About the only place this will happen is the massive BSC megapacks which are all stuffed into single folders with tons of subfolders. For that reason I break BSC stuff into "bits" so that I don't get those large dat files. I simply break them down alphabetically so that if a dat file is in danger of exceeding the "evil cliff", I divide the folder into two or more. This is a bit of a maintenance headache but is just part of the territory given that BSC has stated they will not modify how they organize their files.

Once again it is a matter of understanding what is going on.. the more you understand, the more you can "wing it" on packing... and in turn the more able you are to judge where you messed something up if it doesn't work correctly. If you have no understanding of these things, I'd suggest only packing the plugins AFTER you've made whatever changes you are going to make. Good practices are also required for having backups, maintenance folders, etc... if you're bad at doing those, again that's a warning flag to suggest you not fool with it.. It certainly isn't "fool proof"

As more and more "toys" are used, the closer it comes to being "flexible to the point of collapse"

[mproca 0]

i packed the nam! and put it in first in the plugins and i had no troubles so far... well... sometime when i go with nam puzzles over the transport enabled lots i win a free crash to desktop... but i was thinking this is about the nam.

but i told u... everything u pack keep em in zip files. why? because futher u may want to update the pack.

[z1 566]

Originally posted by: mproca

i packed the nam! and put it in first in the plugins and i had no troubles so far... well... sometime when i go with nam puzzles over the transport enabled lots i win a free crash to desktop... but i was thinking this is about the nam.quote>

This has nothing to do with DatPacking, and it's not unique to the NAM, though that's where it's most often seen.  It's a game bug, pure and simple.

but i told u... everything u pack keep em in zip files. why? because futher u may want to update the pack.quote>

I just keep my original Plugins folder structure intact in a folder named Plugins Disabled.  That way, I can create my DAMN menus from there (you can do this while your files are DatPacked), I can download new files into there, and when I'm ready to update my actual plugins, I just DatPack the containing folder.

To be a little more specific about one of SC4BOY's points, yes, you can DatPack the NAM, but you can't update it in its DatPacked form, nor can you run the Traffic Simulator Configuration Tool on a DatPacked NAM.  Similarly, RTMT has a customization feature, which will be expanded in RTMT V4, and that will not work on DatPacked files.

[Vern 29]

Originally posted by: SC4BOY

3. There is a third relatively obscure piece of info.. that is if you make a single "dat file" by processing a batch of folders and the single files gets over about 750Mbytes, the game will not handle them properly (or possibly DatPacker yields some error.. the end result is the same either way). About the only place this will happen is the massive BSC megapacks which are all stuffed into single folders with tons of subfolders. For that reason I break BSC stuff into "bits" so that I don't get those large dat files. I simply break them down alphabetically so that if a dat file is in danger of exceeding the "evil cliff", I divide the folder into two or more. This is a bit of a maintenance headache but is just part of the territory given that BSC has stated they will not modify how they organize their files.

quote>

I think the 2008 version (which is the latest) addressed creating large megapacks and now truncate it at 392MB.  See the snippet below from the readme file. 

Revision history

2008 version

based on v5

very fast reading and writing as in Anniversary version

sort the file in correct order to respect SC4 loading order

add an 'optimize space' option

add a 'force repack' option

you can choose which folder you want to pack

splitting of big megapack to max 392Mb, great if you have a 1Gb subfolder

[Vern 29]

Originally posted by: SC4BOY

. DatPacker is MUCH SLOWER than SC4 for reasons I don't know.. perhaps ask Wounagain if you are actually curious.. I'm not.... I am a patient man.. and if it takes an hour to DatPack my plugins, I could care less.. I do it rather infrequently. That's my two cents worth.. its a great program and works quite well.... The point where it save time is loading the game and city files and in view changes.. it doesn't make a LOT of difference in other areas. If you try it and it doesn't save you "play time" its simple enough to drop it.. but it has made big differences to my play.quote>

Wow. My Plugins directory in My Documents is currently about 700MB unpacked. It takes about 22 seconds for Dat Packer to pack it.  You are a patient man to wait for it for an hour.  So I'm thinking that you may have one or all of the following conditions in relation to me.

1- Larger Plugins directory

2-Slower CPU performance

3-Less RAM

Or you are saying that the process of packing as a whole takes you over an hour.

It looks like so far there are still lot of people packing.  I should have made this a poll.  As for myself, I think I will continue to pack.  I take advantage of the region playstyle and have to change views often so I do get the benefits of packing.  So for now they will have to pry Wouanagaine's SC4 Dat Packer Utility from my cold dead hands before I give it up.  Tx to all your input. For everyone else feel free to put your two cents  in.

[chocolate_city 93]

Isn't it possible to take the packed file and any updates of the packed file into a new folder then repack it?

[Vern 29]

^^ It probably depends on what the update is doing. If its adding files then no consequence. If its deleting a file or renaming a file then you have to have access to the unpacked files. If it uses the same name then it will use whichever it loaded last with the same name. In this case the consequence would be redundant files using up diskspace and possibly adding one more task to the process.

Also, Im not sure on this so someone can correct me if I'm wrong, but if you delete a file from your unpacked folders and repack it, unless you check the "force repacking" option it will not delete the file from your packed dat.

[joshriddle 29]

@SC4Boy: It is the same yet somewhat slower even for me. However menu scrolling isn't to big a deal anymore. It is easier. And I recently datpacked all the NAM components except the traffic simulator I built with the TSCT. Before I couldn't use the RHW. Now I can! I couldn't use the NWM. NOW I CAN! So I don't see the worries in datpacking the NAM...

[Vern 29]

Originally posted by: joshriddle

I must say on the other side it was convenient. Cut back on my plugins folder size 100 MB even. So that is good. Probably those are an accumulation of some 200 folders in there.

quote>

So, safe its safe to say you are a convert and that you are now packing.  Welcome to the fold.

[joshriddle 29]

I have like TONS of BSC stuff. Close to 500 MB of it, and I recently packed that one. What I noticed was it packed 398,000 KB of the BSC stuff and then created a new .dat called Bsc001, which contained 60,000 things of data. I am wondering if the packer can only cram so much into one file...

[Vern 29]

@joshriddle - Yes, the last version of DAT Packer packed up to 392MB and the remaining are renamed and index. Its in the readme file. Its probably the threshold for performance optimization and the prevention of CTD.

[joshriddle 29]

Originally posted by: Vern

@joshriddle - Yes, the last version of DAT Packer packed up to 392MB and the remaining are renamed and index. Its in the readme file. Its probably the threshold for performance optimization and the prevention of CTD.quote>

I only read readmes if there is a problem and or I don't know how to use it so I didn't know that.

I was hoping to pack all my plugins into one file but now I know why my computer locked up when I tried to.

I also wonder if I can run it on other files...

Wait, no I wouldn't...

[Phunkadelic77 7]

Isn't it possible to take the packed file and any updates of the packed file into a new folder then repack it?

 

Absolutely this is possible.. that's what the "Force Repack" option is for...  This is how I handled my ever-growing plugins folder. 

 

A) Before ANYTHING else, I create a new folder outside of the My Documents\Sim City 4 folder... for example, I made a New Folder in the Root of My Documents, but separate from the default Sim City 4 folder.  I named this new folder "SC4_Master Directory"

 

B) With my newly created folder, I create sub-folders.  One of which I named Plugin_Source Files - and this is where I store the original .ZIP Archives I obtain from downloading custom content.  Inside that folder, I created several more sub-folders organized categorically (For a good category Listing Scheme, I used the same Format used on SC4 Devotion/Lex - that way it is easy to go back in case of issues and locate the source files.

 

Here is an example of the folders I created for storing all of my Archived_Source Files

 

CAM_Agriculture

CAM_Commercial Offices

CAM_Commercial Services

CAM_Industrial

CAM_Main Files

CAM_Residential

Civics_Education

Civics_Health

Civics_Safety

Commercial

Commercial_Highrise

Commercial_Midrise

Commercial_MEGA

Commercial_Small

Commercial_W2W

Commercial_WFK

Dependency

Dependency_Props

Dependency_Textures

Documentation

Farming

Farming_SF

Industrial

Industrial_MEGA

Industrial_WFK

Landmarks

Landmarks_Functional

Maps_Standard

Maps_Terraformer

Models_3DS

Models_gMax

Mods_General

Mods_Graphics

Mods_Transport

NAM

Parks

Parks_BSC Compliant

Parks_Eye Candy

Parks_Sportsfields

Residential_Highrise

Residential_Midrise

Residential_MEGA

Residential_SF

Residential_Small

Rewards_BTE

Rewards_General

Tools_Data Creation

Tools_Data Management

Tools_Data Modification

Transport_Air

Transport_Rail

Transport_Road

Transport_Water

Utilities_Power

Utilities_Garbage

Utilities_Water

 

I narrowed it down some as I don't use custom content from all of these categories, but it was a very logical template to follow.

 

C) Now that I have a locale for storing all of my Source Files, going back to the "SC4_Master Directory" I created in My Documents, I create a Second Sub-Folder named "Plugins_backups" and the VERY FIRST thing I did before using DAMN or DatPacker was to take my plugins folder (which has been thoroughly tested and confirmed to be glitch free, and I made a COPY of all of my plugins in their original, default format.  When installing certain items - especially stuff that includes an installer, the content will be installed to a very specific folder.  An example would be the folder titled "bsc"

 

A lot of dependencies reside in that folder as well as content from numerous modders/lotters, etc. and a lot of content that is downloaded for use in Sim City 4 comes with an installer anyway, so it is good to have a solid backup of all of my plugins in their ORIGINAL, UNALTERED STATE.  Mostly, this is used for troubleshooting issues should they arise, but also a good thing to have for anyone who uses a lot of plugins, etc.

 

D) OK, so I made a copy of my plugins folder in its original state.  The next thing I did was set about to manually "tidy up" my plugins.  Using the same category list I use to store the actual Source Files seemed like the most logical thing to do, so everything inside my plugins folder that wasn't part of a big package (i.e, bsc, NAM, Simgoober, Paeng, PEGPROD, etc - those folders are pretty well self-organizing because of the installers that put the files to the right location) but all of the "loose" plugin files I have for various Stand-Alone Lots, Mods, etc. I set about to create corresponding folders which as I already stated match the folder structure of my Source File Library....

 

E) After "Consolidating" my "Loose" plugins and stand-alone items, I make a SECOND copy of my plugins folder in its NEW format and put it back in my SC4_Master Directory\Plugins_backup folder... naming it something like Plugins_User_Consolidated (the first copy I made of my unaltered plugins I put in a folder named Plugins_Unaltered

 

F) OK, so now that I have 2 backups of my plugins handy (just in case) I can go about fiddling with the Tools (DAMN, DatPacker, etc.)

By the way, I should also mention that next to my Source_Files_Library where I keep the ZIP Archives for safe keeping, I created another Folder specifically just for storing all of the Tools and Utilities available for Sim City 4 (iLive Reader, SC4 Node, DatPacker, Lot Editor, BAT, Gmax, etc. etc.)

-so I don't have to re-download these if I want to reinstall or whatever (unless an updated version comes out)

 

BUT to get to my point, the way to Repack an item once its already been DatPacked should you add any AddOns or Updates is to take the .dat file out of your Plugins_Compressed Folder and put it in a New Folder of the SAME NAME in your ROOT plugins Folder along with the NEW Files you want to add to it.  Then Launch DatPacker and select the new folder you just created making sure to check the box "Force Repack" and that will essentially combine the new folders/Files with the .dat file previously created.  Make sure you remove the original stuff back out of your plugins folder and then put the new .dat file (with its original name) back into your plugins_compressed folder.  Simple as that.

 

On a side note....

 

I have tried running the game using a DatPacked NAM and so far have had no issues.  I think it's a matter of personal preference whether or not to do it.  One word of caution though - if you have to reinstall NAM or want to change installed items, etc, you will have to do it fresh or just repaste the original from your saved backups back into the plugins folder first..... Personally, I keep my NAM uncompressed but that's just my preference - it should work either way if you do it correctly to begin with.

 

I love the DAMN Menu interface and just recently started using it... I am using a hybrid combination of some Datpacked items (mostly dependencies, props, mmps, etc and then keep all of my zzz_"folder name" files in their original structure - seems to run very smoothly!

 

Anyway, that was a very lengthy post to make such a simple point which I could have done in about a fraction of the sentences (sorry for going off on tangents like that - its the OCD) lol

 

 

 

On another side note, I found a great little app that is useful for taking a listing (like the one above of my folder names) and rather than creating each folder manually one at a time, this app will take a list made in notepad (a .txt file) and create a folder for each one on your list!  Very handy little tool and it's Freeware... here's a link for anyone interested...

 

Text2Folders

  Edited July 9, 2013 by Phunkadelic77  

[infamousjbe 593]

Not sure if you've noticed, but this thread has been dead for almost 3 years. I some how doubt that the OP is still wondering about all this. But I suppose it could pertain to some others that have not seen the thread.

[A Nonny Moose 8,261]

Generally I don't pack, but if I want to just pack up a specific BAT lot, I use a program by Ilive called files2DAT.  It is cumbersome, but it works.  Saves the number of file entries needed on the system, and you can use it to pack up several .dats.