Extracting files

[mrbisonm 1,620]

I don't know if this is the right place to ask this, but to me it is the most logical place. I also searched here and some other places and couldn't really find anything detailed enough on this subject.

I was wondering, is there a way to extract model files from .dat's?

I want to extract a model file from a datfile so that I can make my own lot with a specific model , without having to keep the original datfiles and its lots.

If it is possible please explain in details how to do it. Thanks in advance...most appreciated.

mrb

[T Wrecks 10,209]

First things first: Yes, it is entirely possible. I already used it extensively to boil down many of those prop packs that contain mostly main buildings (something rather annoying IMHO) to pretty handy sizes by deleting models I didn't like at all and extracting those I wanted to use independently.

Unfortunately I'm not at home right now, so I can only give you some general pointers off the top of my head. Whatever you are up to, iLive's reader is the tool you need to get the job done. Are you familiar with it?

What you do is basically open the reader, select the DAT to open and click "Fill the List". Then you'll get a list of all files that this BAT contains in the left part of the UI. In the right part you'll see a preview of the respective file.

Let's pretend you have a DAT where the model file, its lots, and the exemplar files containing the game-relevant parameters of the building have all been put into one file. In that case the easiest way would be to delete everything that doesn't belong to the model itself and save the file under a new name.

"Everything that doesn't belong to the model itself" means all PNG files (lot preview for Lot Editor + menu icon(s), if applicable), all exemplar files (for lot stats and desc stats), and all LTEXT files (for menu description / building name). Highlight and delete all of them, re-index the list, and save the file under a new name. Then you should be able to use the model like any .sc4model file.

I can give you more precise pointers once I'm back - which won't be before tomorrow, though.

[Andreas Roth 605]

I think the key to be able to extract a single model from a prop pack etc. is understanding how the SC4 files are structured internally. As T Wrecks said, you need the Reader for this process, and you should start by opening various SC4Lot, SC4Desc and SC4Model files with the reader and examine the internal structure. If the files were not modified by the creator, you should find the following:

- SC4Model files contain a bunch of S3D and FSH files, those are the actual 3D model and the textures. Each model also contains two BMP and two JFIF files, those are preview pics that are not needed by the game, so you can safely delete them. Do not delete the XML file if you intend to make new lots, though, as the information that is contained in the XML file is used by the plugin manager to determine the size of the model etc. For the game, the XML file is irrelevant, so you can delete it after you created the SC4Desc file in the Plugin Manager.

- SC4Desc files are created by the Plugin Manager and usually contain one Exemplar file (either a building exemplar file that contains the properties of a building or a prop exemplar file that contains the properties of a prop). It also creates an XML file which I think is used by the official exchange for reading the properties, so you can delete that one.

- SC4Lot files contain a lot exemplar file that contains the layout of the lot, a PNG preview image (not needed by the game, but you might want to keep it for reference purposes), an LD file that is created by the Lot Editor, which contains some basic information about the lot (not needed by the game, so you can safely delete it as well), and in case of a ploppable another PNG file (the menu icon) and a copy of the building exemplar file (in this case, you can safely delete the loose SC4Desc file). Modded files may also contain LTEXT files (text files for menu description, query windows etc.), UI files (custom queries), WAV and TRK files (custom query sounds), LUA files (reward scripts) and probably more.

Now, let's assume you have a large prop pack and want to extract a single model. What you need is finding the S3D and FSH files, plus the related prop exemplar file. When opening the DAT with the Reader and filling the list, you should be able to recognize a certain file order, but unfortunately it might differ a bit, depending on how those models were packaged. The most common is a mix of S3D and FSH files, all with the same group ID, and an prop exemplar file that is either somewhere in that list, or further up or below. In order to determine to what files the exemplar file belongs to, click on it and have a look at the "Resource Key Type 1" property. It should tell you the ID of one of the S3D files. Just look at the group ID (the one in the middle of the tree IDs) and compare it with the rest of the files.

Now open a second instance of the Reader and click on "File/New" in order to open a new file. Switch to the first Reader instance, highlight all files that belong to the model you want to extract in the first instance (press [CTRL] and select them all), right-click and select "Copy file(s)", then go to the second Reader instance, right-click again and select "Paste file(s)". Attention! Do not attempt to click on any of the S3D files after inserting them, because this might crash the Reader. Go to the "File" menu" and select "Save as..." in order to save the extracted files into a new DAT. You have successfully extracted the prop model along with the prop exemplar file now. If you were extracting a building model file, make sure to select the XML file as well, so you can use it in the Plugin Manager for making other buildings or props from it.

There are some occasions where the method of looking up the group ID doesn't work - i. e. if the creator used 3DStudioMax instead of gmax. Usually, all models will have the same group ID in that case, so you need to find the exemplar file, and examine the instance ID (the third one from the left) in the "Resource Key Type 1" property more closely. The first four digits if the instance ID are the same for all S3D and FSH files of one model, while the last four are used for the different zooms and rotations. S3D and FSH files of one model are usually located in a single block in the file list, though, so you should be able to tell where the next model file set begins. In some cases, the XML file was deleted by the creator; if you want to create a new building or prop from the model in that case, you can create a new exemplar file by using another model and then change the ID in the "Resource Key Type 1" property manually.

[mrbisonm 1,620]

I don't quite understand all this and can't seem to find certain things......I guess that is over my capacities of filetweaking.

I have a so called datfile that is called *Ressource* and that is the only file in that package that seem to have model files in it. On it are 3 different models plus some other grassy like pictures on there. There don't seem to be any png, xml, ld, bmp nor JFIF.... etc files, all there is in there are some Exemplar files, a lot of SFH and S3d and 1 DIR file. Btw, how do I know that they belong to the same model, is it the Group numbers that have to be the same or the Type number? Sometimes the Type Nrs are the same but then it changes into another group Nr further down....oufff, that looks messed up I say.....lol

I also realized that I cannot find them as props in the LE.....hm......

Jey , thanks guys for helping me so far. Most appreciated simply because my computer knowledge is about the same as a 7 year old. (and even then I think my youngest grandson is better than that.....lol)

mrb

[Andreas Roth 605]

It would help if you post the filename (or even better the download source), so we can point you into the right direction with some screenshots or so.

[Andreas Roth 605]

mrbisonm told me which file he was about to modify, so I made a few screenshots to illustrate the process. The file in question is tne "PEG-MTP_CS$$2_CornerShop_RESOURCE_100.dat", which contains tree nice corner building models. Since PEG deleted the XML files that are necessary for the Plugin Manager, it's not directly possible to use the models on other lots. But with the help of the Reader, we can circumvent that.

Start up the Reader, open the file and click on "fill the list". You should see that the list of files starts with a "DIR file" entry, which is basically an index of all compressed files that are inside the DAT. You can ignore it, as the Reader updates it automatically whenever you save the file. The next three entries are building exemplar files, which will play an important role during the process of making the models available for new custom lots. Since there are three exemplar files, you can guess that there are three building models included as well. Obviously, this is not a strict rule, but when examining the DAT further, you will see that the assumption was correct.

Let's have a quick look at the first exemplar file. As mentioned before, the "Resource Key Type 1" property is essential for determining the files that belong to the individual models. The second "REP" (value) of the property is 0x3647FB0E - this is the group ID of the model. All files that belong to this model have the same group ID. Fortunately, they are neatly arranged below each other, first the FSH (texture) files, and then the S3D (3D model) files.

Scroll down further until you reach the files with a different group ID (marked in green). If you check the Resource Key Type 1 property of the second exemplar file, you will see that this one has the same ID in the second REP. So now you got a basic grasp how the DAT file is structured, and how to find the files that belong to a certain model. For all models created in gmax, the process is very similar, but in some cases, exemplar files, FSH files and S3D files might be scattered around a bit more.

To make things easier, we are now going to copy all files that belong to one model into a new DAT. For this, start another instance of the Reader (so you will have two Reader windows on your screen) and select "File/New" in the second Reader instance. This will give you a blank DAT file. Highlight all files of the first model (FSH and S3D files) by selecting the first, scrolling down and selecting the last one while pressing the [sHIFT] key. Now right-click somewhere in the list and select "Copy file(s)" from the context menu.

Switch to the second Reader instance, right-click and select "Paste file(s)", then select "File/Save as..." and save your new file under a new name. You can use a *.SC4Model file extension, but since it can't be used in the Plugin Manager due to the lacking XML file, maybe it's better to give it a *.dat extension - for the game, it doesn't matter anyway. Also, make sure not to click on any S3D file before you saved your DAT, otherwise the Reader might crash due to the lacking DIR file (this will be created when the file is saved). If you want, close the second Reader instance now, but keep the file in your plugins folder. Leave the first Reader window open, though, since we still need it for reference purposes.

For creating a new *.SC4Desc file, we simply use any "regular" model that still has its XML file as a template for our new building exemplar file. If you use a model that has roughly the same size as that one you just extracted from the DAT, you won't have to adjust as many properties, however, it is a good idea to use an external reference (such as SC4Tool's building datebase) in order to get proper values. So look for an *.SC4Model file, put it into the plugins folder and start the Plugin Manager. Drag the model to your desired type of building (i. e. R$$), adjust the values and exit the Plugin Manager again.

Now open the newly created *.SC4Desc file with yet another Reader instance (if you are an avid modder, you probably assigned the SC4 file extensions to the Reader anyway, so you only need to double-click them in order to open them in the Reader). We will now edit some properties in the new building exemplar file, so it links to the PEG model instead of the one that you used in the Plugin Manager.

Switch to the resource DAT that is still open in the first Reader instance, select the first buidling exemplar file and double-click the "Resource Key Type 1" property. Now, highlight the "Value as text" line and copy the IDs (right-click -> copy). Switch over to the Reader with the *.SC4Desc file, open the "Resource Key Type 1" property there and paste the IDs that you just copied. Repeat the same steps with the "Occupant Size" property. This is important, because it will adjust the footprint of the building (the grey rectangular that you see in the Lot Editor) accordingly.

Also, change the "exemplar name" to something more fitting, if you haven't done so already, and finally save the modified *.SC4Desc file. Repeat all these steps with the other building models as well. You can remove the original resource file now and open the Lot Editor to make lots four your extracted buildings.

Of course you don't have to extract the files into new DATs if you want to use all three models, in that case, it's easier to delete the old exemplar files and the overlay textures that PEG also included in his resource file. If you scroll down to the bottom of the list, you will see another bunch of FSH files, but with a group ID of 0x0986135e. Highlight those and the three exemplar files, right-click and select "Remove", then save the DAT under a new name and remove the old file from your plugins folder.

[mrbisonm 1,620]

Oh man......this works like a charme, just great,  Andreas. 

The way you have explained these steps of extracting the .datfile is just perfectly done (computer-dumbproof), even for someone like me (that has no tweaking knowledge at all) to understand. You really know how to make a tut clear, detailed and fully understandable. Can I hire you now?.........lol. 

That was really great, Glenni helping me with gmax this morning and Andreas tonight with the reader and extracting .datfiles. You know, there are so many .datfiles out there that I would have liked to transform to .descfiles, now I can and I don't have to prive myself anymore............btw and to prove that I was able to make my OWN lot now with an extracted datfile (lol)....voila, I included a picture of the first model.

Thank you very much Andreas, that was really greatly appreciated and now I owe you one!

Fred

[bixel 1,221]

Did you know you can also extract actual 3ds models? I've done it. The automata (cars and trucks) are actual 3d models in the game and using the Reader you can extra those as a mesh and import them into 3ds Max - you can extract the textures they use and slap them on the models. (these cars are the ones used in the U-Drive-It NOT the props. I did that to get a bunch cars in to be rendered in the parking lots of my structures.

Other real mesh models in the game that can be extracted fully is the Highway and the highway textures used. Other possibilities are avenue pieces, bridges, and maybe the ports but I am not sure.

Buildings are not real models just an assembly of pictures so they cannot be extracted like model.