Jump to content

  • Log in with Facebook Log in with Twitter Log In with Steam Log In with Google      Sign In   
  • Create Account
  •  
Photo
* * * * * 11 votes

SimCityPak - Modding Tool

588 replies to this topic

#1 oppie

oppie
Awards
  
  
  
Offline

Posted 04 February 2013 - 04:17 PM

*
POPULAR

What is SimCityPak?
Simply put, it's a work-in-progress tool that will hopefully eventually be the iLive Reader equivalent for SimCity 2013. It will eventually allow you to view and edit the game assets contained in .package files.

What does the tool allow us to modify?
While there are still some file formats that need to be reverse-engineered, property files and text files (javascript, json) can already be modified and saved back into the game.

What will we eventually be able to modify?
Nobody knows - the only way we'll be able to find out is through experimentation. People have expressed concern that because the game partially runs on Maxis' servers, we won't be able to modify anything at all - although experiments have revealed that the city simulation runs only on local machines and that Maxis has no systems in place that verify wether a file is modded or not.

Will I be banned for modding?
This is a hot topic that I've seen come up in almost every discussion about modding so far. Despite being asked repeatedly, Maxis hasn't given any straight answer yet. On the other hand, they've said in the past that they appreciate the modding community and (probably the biggest sign that they're tacitly allowing mods) so far, nobody has been banned for modding yet. So, the best answer we can give so far is; use mods at your own risk, even though that risk is very small.

How can I contribute?
There's still many things to do, so every contribution is more than welcome. If you think you've got what it takes, take a look at the Issue Tracker (http://simcitypak.co...item/list/basic).

Also, experiment! There are already many possibilities for mods - all it takes is dedication and a lot of digging into package files. However - remember to make backups often and to always test in a non-public, disposable region.

----

First of all, this tool was born when I realized that SimCity uses the same DBPF version as Spore, and that there were already a few Spore Modding tools out there that I could base my research on. I was lucky to come across SporeMaster (https://code.google.com/p/sporemaster/), an open-source .NET based tool. While this tool was made specifically for Spore (and there are quite some differences between the .RW4 model files for example) I managed to reuse a lot of the underlying object model to build my own tool from the ground up (I've been in contact with the creator of SporeMaster and have permission to use his work). From there, I've been trial-and-error'ing my way through the files to see what I can open and how.

For those of you who want to help building the tool (which is in C#/.NET/WPF with a dash of Xna thrown in), I've uploaded the source code over at https://simcitypak.codeplex.com/ . I'm not sure if anyone can just download the source code using TFS, but if you need access, give me your codeplex account and I'll make it so.

I've also uploaded an executable version of the tool. Let me know if it doesn't run or if anything goes wrong when opening a package. There will undoubtedly still be hundreds of bugs but most of them will be caused by unexpected data in the package files.
 
LINK TO THE LATEST VERSION: https://simcitypak.c...ses/view/107793

BASIC MODDING TUTORIAL: https://simcitypak.c...e=Documentation

Changelog:
 
0.003
  • Added Hex view to all file types (even the already identified ones)
  • Added Hex view to subsections of a RW4 file to identify the unknown ones
  • Added 3D preview for RW4 Mesh sections (warning, meshes aren't read correctly yet, so it still looks like crap)
  • Fixed most bugs when loading properties. Only 'Transform' type properties still have some issues
  • Added a context menu to Key Properties so that you can open the file it links to in a new window (right click the value and select "open in new window..."
  • Ability to export textures. From the RW4 Texure view, you can select a file and save it to the .dds format.
  • 0.004
  • Added 3D view for RW4 meshes
  • 0.005
  • Added first attempt as a viewer for RASTER files (which are surprisingly not building textures... wondering where those are hidden)
  • Added instance regisrty (Add custom names and comments to specific instances by right-clicking on the instance number in the main grid to make identification easier)
  • Modified properties view to be easier on the eyes
  • NOTE: While the properties files are being read a lot better than in previous versions, it still has a few bugs that I hope to be able to iron out soon.
  • 0.006
  • Export to package - export one single TGI index to its own seperate package file
  • Properties files will now be read correctly - all properties will be read
  • Added relevant information to RASTER and PNG file displays (should be expanded)
  • Started refactoring of back-end object model code
  • 0.1.0.0
  • Added textures in 3D viewer for vehicle models
  • Added Loading of Locale files from Settings screen (will load localized strings for PROP files)
  • Added About window
  • 0.1.0.1
  • Added search for specific values in property files
  • Added first attempt at saving modified data to a property file (warning, still buggy)
  • 0.1.0.2
  • Ability to edit Text files (JSON, Javascript etc...)
  • Ability to edit property files (still preliminary)
  • Advanced Search for property files
  • Ability to export multiple assets at once
  • Ability to save packages to a new file
  • 0.1.0.3
  • Allows opening multiple package files at once
  • Includes editors for most properties
  • Improved user interface
  • Viewers for 8-bit maps
  • Import function for PNG's
  • 0.1.0.4
  • Added Lighting Editor (first version)
  • Added possibility to add/remove properties
  • Identified many more properties/instances
  • New Viewer for 3d files
  • Many general UI tweaks
  • Many general bugfixes
  • 0.1.0.5
  • Lighting editor expanded to simple Lot Editor that allows editing of lights,props and decals (note; work in progress)
  • Fix for searching property files
  • Added Copy Key, and Delete to context in main list
  • Added fuction to delete indices
  • More identified properties/instances
  • Many general bugfixes and UI tweaks
  • 0.1.0.6
  • Export/Import Textures
  • Export meshes to .obj format (no import yet, though)
  • 0.1.0.7
  • Import .OBJ format 3d models. When importing, make sure that the model file contains normals and texture coordinates. Materials are not needed. Tangent and binormal for normal maps are exported.
  • Save button in hex editor
  • 0.1.0.8
  • Import BMP color palettes for vehicles
  • Import RASTER file (uncompressed 8.8.8.8 DDS files)
  • View different channels of RASTER files or preview of all layers combined
  • Find text in javascripts
  • TGA viewer
  • Ground textures added to lot editor
  • Many additional identified instances and properties


#2 oppie

oppie
Awards
  
  
  
Offline

Posted 06 February 2013 - 03:55 AM

It all depends on how much of the game we'll be allowed to modify (will there be checks server side?) but I'm optimistic that sandbox mode will provide the freedom we're looking for. From what I've gathered looking through the game files, this opens up a lot of interesting possibilities. With a bit of luck, we might even be able to mod in terraforming (the tree tool writes to the 'forestheightmap' - all we have to do is get it to write to the normal 'heightmap' instead).

Anyway, if you've been using SimCityPak, you've probably noticed that some areas work better than others. I thought I'd write up a short summary of the current state of the game file deciphering progress for the most important file types:

Package files:
The DBPF structure is completely deciphered and I have yet to get any error there. While I haven't tried writing to a package file yet, the Spore object model I'm using seems stable enough to work for SimCity.

Property files:
There are some major differences between how these worked in Spore and SimCity. While I've got most of the specific property types to work, ArrayProperties are still a matter of concern. The problem is that if one property fails to read, all subsequent properties will also fail (because I can only pick up reading where the previous property left off). This means that to get properties to work a 100% of the time, all of them need to be read 100% correctly. This means analyzing the file byte by byte, and while I've made some progress there, I wouldn't at all call it 'stable'. Nevertheless, while there might be missing properties in a lot of the files, a lot can be done to analyze them. For example, I've managed to figure out how the game uses TextsProperties to link to a certain localization string, something I hope to integrate into the tool at some point.

RW4 Files:
Besides the textures, not much of these is readable. While all the individual sections can be read, the contents are either unknown or outdated for the Spore object model I'm using.

RASTER Files:
These contain textures (for buildings, I'm assuming), but are still unreadable. I've got a python script that is supposed to be able to translate them into something viewable, but I haven't yet had enough time to look at it.

#3 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 09 February 2013 - 07:26 AM

One thing that confuses me is that I cant see much (if any) difference between the latest DBPF 2.x and the new DBPF 3. I wonder why they increased the version number at all?

 

And I tried access the source code on codeplex, but it says its not published, my account's username is jon889.

 

And what is the difference between TGIRegistry and TypeRegistry?



#4 oppie

oppie
Awards
  
  
  
Offline

Posted 09 February 2013 - 08:49 AM

One thing that confuses me is that I cant see much (if any) difference between the latest DBPF 2.x and the new DBPF 3. I wonder why they increased the version number at all?

 

And I tried access the source code on codeplex, but it says its not published, my account's username is jon889.

 

And what is the difference between TGIRegistry and TypeRegistry?

 

Yeah, I can't see much difference there either - I don't know what warranted the version change but I don't mind - makes it easier for us.

 

For now the project hasn't been published yet. I've given your account developer permissions  so you should be able to access it now.

 

TGIRegistry was a previous attempt of mine to save information about the types/groups and instances that didn't work out as well - only TypeRegistry.xml is used right now.

 

Also: I'll be uploading new versions of the tool often, so check this thread and the changelog for updates.



#5 oppie

oppie
Awards
  
  
  
Offline

Posted 10 February 2013 - 01:41 PM

I've uploaded a new version - I managed to find out why the RW4 mesh information wasn't being read correctly, so I was able to add a 3D viewer for the meshes. This should make identifying which property file represent which in-game object a lot easier!

mesh1.jpg

#6 oppie

oppie
Awards
  
  
  
Offline

Posted 12 February 2013 - 04:34 PM

Once again I've uploaded a new version of the tool.

Also, feel free to post any comments you might have; bugs, ideas for the user interface, interesting observations - anything is welcome!

#7 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 13 February 2013 - 10:11 AM

The link to 0.005 doesn't work. 

 

It would be good if you could add the ability to export a subfile, I've added this myself but I cant for the life of me get Team Explorer/CodePlex to work with Visual Studio Express.

 

Also I find it impossible to open large files, it eventually stops completely, and I've got out of memory exceptions, so a size column would be good to have too.

 

Also I can't build the project because WriteableBitmap has no method "SetPixel" (and also there is no XNAControl, though I have the XNA framework installed).

 

The ability to search through the whole package would also be good, even if it isn't quick.



#8 oppie

oppie
Awards
  
  
  
Offline

Posted 13 February 2013 - 04:42 PM

Thanks for the heads-up! I've fixed the link to the latest version.

The reason why you're getting build errors is that some of the referenced dll's pointed to locations specific to my system. I've done a quick update of the code to remove all unnecessary references (like the xnacontrol) and moved the others to a location that can be referenced as a relative path. I've also included the referenced libraries and the folder they're in in the Team Foundation Server. In the checked-in code, I've also added the 'export' context menu option and managed to get all the individual properties to read correctly. (only very few files still have an error). A search function is also something I'd like to have, but for now I'm going to try to focus on saving/loading package files and importing/exporting textures so that I hopefully have something to test with for the beta this weekend.

What kind of troubles are you having with Team Explorer? I'm using the built-in version from Visual Studio 2010 premium - but I read somewhere that for Express you have to use some kind of stand-alone version?

#9 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 16 February 2013 - 08:24 PM

When I make a change in VSExpress, and then go back to the stand alone Team Explorer, it doesn't show any changes for me to act on.


I've just tried replacing a texture in a RW4, when importing I couldn't see any files (including .dds files) so I had to remove the filter from the open dialog. Then when I did import the new texture I get a NotSupportedException in ViewTexture.xaml.cs on Line 234 (textureSection.Write(null, section, strm) ;)

#10 oppie

oppie
Awards
  
  
  
Offline

Posted 17 February 2013 - 07:54 AM

Hmmm, strange - I haven't really worked with the standalone version of Team Explorer a lot, but I'd assume that it'd compare the files in TSF with the ones in your local working folder and update the changes accordingly.

The texture import function doesn't really work yet. I wanted to get it to work for a test in the beta, but I was a bit too optimistic as to how easy it would be to implement.

#11 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 17 February 2013 - 09:17 AM

I think it's because you have used write() rather than read(), on the import texture function. Write() means write the data already in the rw4 file to a stream. And Read() takes a stream and "reads" it into the rw4 file. (I think anyway)

I changed the write to read, and it worked a little more, but then it didn't like that the first parameter was null. But I don't know enough about WPF to know how to get the RW4Model from the parent MeshView to the TextureView. (the whole DataContext and WPF confuses me)

#12 oppie

oppie
Awards
  
  
  
Offline

Posted 17 February 2013 - 10:15 AM

I think it's because you have used write() rather than read(), on the import texture function. Write() means write the data already in the rw4 file to a stream. And Read() takes a stream and "reads" it into the rw4 file. (I think anyway)

I changed the write to read, and it worked a little more, but then it didn't like that the first parameter was null. But I don't know enough about WPF to know how to get the RW4Model from the parent MeshView to the TextureView. (the whole DataContext and WPF confuses me)

 

Yeah, I already changed that in my own local version, but like you said, it still doesn't work - it's because the RW4Model splits some of the header info that defines what kind of texture it is (directx compression, size) from the actual texture data, which it then stores in different sections of the RW4 file.

 

WPF is pretty cool once you get the hang of how it works. I use it for most of my own projects these days. It's especially great when combined with the MVVM pattern, but in this case I went with a much more quick and dirty approach because I wanted to crack those package files open. Speaking of quick and dirty - while I'm trying to refactor a lot of older code, some parts of the application are still a bit of a mess. Eventually I'd like to have a simple and straight-forward object model for package and TGI manipulation that people can build on.



#13 oppie

oppie
Awards
  
  
  
Offline

Posted 20 February 2013 - 04:59 PM

I've uploaded a new version (to codeplex this time, because my own site seems to be down for the moment, let me know if you can't access it). I've adopted a new (more sensible) version numbering, so this version is called 0.1.0.0. The most exciting new features of this version are the addition of a texture viewer to the 3D mesh preview (this only works for vehicle models for now):
ztoFMOZ.jpg

Also, to help identify properties, you can now load your locale files from the new Settings... dialog. This will load the localized strings into memory and will display them appropriately in the PROP files. This should make identifying those a lot easier.

NOTE: I will be on vacation from 23-feb to 11-mar
This not only means that I'm sadly not around for the game's release, but also that as of today, I won't be able to do any more work on the tool. Warrior, I've given you coordinator permissions on the codeplex project so that if needed, you can manage that. I've also given simcool developer permissions, so if he wants to he can join in on the development as well. Feel free to make any changes you think are necessary.

Like I said, I also won't be around on the game release date. I was originally hoping to make the tool open source on that same day - and honestly that could still be possible. From what I can see on CodePlex, warrior should be able to publish the project (which should be done before march 5th because otherwise it will be deleted) and if Dirk unhides this forum (I'm not sure what the plan is for release date), everyone should be able to find it.

#14 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 21 February 2013 - 08:19 PM

Wow thats brilliant, strange that it only works for vehicle models though? Are they in a different format or something? And the localized settings sounds interesting,

 

sorry I haven't replied much, I'll publish the project, but I'm not sure about making the forum public on the release day, at least not without seeing to what extent modifications can be made, not sure how happy Maxis will be if suddenly the game breaks because everyone is fiddling with it :) 

 

If it's that kind of vacation, I hope you have fun :)



#15 oppie

oppie
Awards
  
  
  
Offline

Posted 22 February 2013 - 12:18 PM

It only works for vehicles because the RW4 meshes are different for different types of models - the information stored in a vertex, mainly. While the x, y, and z data is always stored in the same bytes, the uv mapping, normals and bone weights seem to be different from model to model - and even then there's more data in them that I just ignore for now. Besides that, the vehicles are the only objects where the textures are reliably stored in the RW4 file itself.

...I'm not sure about making the forum public on the release day, at least not without seeing to what extent modifications can be made, not sure how happy Maxis will be if suddenly the game breaks because everyone is fiddling with it :)

I've thought about this, but I don't think it can do much "damage" to make this forum public. First of all, the tool currently doesn't really support any fiddling with the files, because it's still much more of a viewer than an editor (speaking of which, making the tool able to save packages and the readable file types should be a priority for the next new feature). I've experienced that finding people who are willing to participate is very hard when you're doing it in secret, and I'm hoping that the more people are experimenting with the tool, the more we'll find out about the modability of the game. Besides, Maxis already knows that we're working on modding, and I'm sure they're well prepared for it. But since I'm gone on the release date, I'm also fine with waiting a week or so.

If it's that kind of vacation, I hope you have fun :)

Thanks! I'm going to South Africa, I'm sure it's going to be amazing!

#16 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 22 February 2013 - 02:16 PM

I've downloaded the latest version and the loading from locale file makes everything a lot clearer :)

Though you would seem to have a different locale file than me, because the tool crashes when it came across a comment in the JSON files in the locale package, I had to add a check for comments in the code, which implies you didn't come into that problem? Windows says my locale package is 450,408 bytes (not "bytes on disk"), from beta 2 files (the crash also happened on beta 1 files).

 

There is one more thing I'm curious about, in the screenshot below, it says the value refers to a JS file:

 

K5fchZ9.png

 

But when I search the blue number as an instance ID no file is found (in any of the SimCity_* packages), so how did the tool know it will be a javascript file?



#17 oppie

oppie
Awards
  
  
  
Offline

Posted 22 February 2013 - 02:38 PM

I have Dutch locale files, so that's probably why it didn't crash for me.

The key properties sometimes contain a full TGI link, from which I can identify the file type the key should be pointing to. When you click the link, it searches the current package for that exact TGI instance. If it's not found, nothing happens..

#18 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 23 February 2013 - 10:05 AM

I'm not sure if you'll see this, and I feel bad for saying on your vacation. But only the project owner can set the license, which has to be done in order to publish it.

 

Otherwise, I managed to get the Team Explorer working by using a trial of VS 2012 (horrible UI though), and I committed a change that makes the property links work in the ViewWindow.



#19 oppie

oppie
Awards
  
  
  
Offline

Posted 24 February 2013 - 02:17 AM

No worries, I have wifi from time to time. Could you try to find out which license would be best for our project?

#20 oppie

oppie
Awards
  
  
  
Offline

Posted 01 March 2013 - 10:34 AM

I've picked a license and published the project so it doesn't get deleted.

I see you've done some checkins, keep up the good work!

#21 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 05 March 2013 - 12:46 PM

It's been quite addictive, but I haven't the skills to do anything more complicated. But I hope I've made it easier to work thing out, at least with the PROP files.



#22 AlikMalix

AlikMalix
Offline

Posted 06 March 2013 - 03:59 AM

You guys are cool.  Your interest in this (modding SC13) is highly appreciated. I was trying to read your progress, but I dont understand much.  If you ever figure out how to bring the joy of SC4 to SC13, devs should pay you because you'll keep this game around forever like modders did for SC4.  Good Luck. 

 

Oh a few ideas for future thinking:

1. Allow/change game to upload/save to local drive

2. Expand city size limit, or build outside the dredged white dotted line.

3. Farms

4. Self dependant city

 

I know that any of that stuff is probably impossible (you dont have to explain why), but seriously to us simple people - modders are the last hope for enjoying something new like the SC13.



#23 jorday

jorday
Offline

Posted 06 March 2013 - 04:50 AM

2. Expand city size limit, or build outside the dredged white dotted line..

 

If they could do this it'd be amazing.



#24 Bipin

Bipin
  • Member
  • Bureaucrat

  • 1,171 posts
  • ← I wouldn't touch that with a ten foot pole.
  • Locationis irrelevant unless you are some sort of stalker. In that case, drop me a line and we can hook up... ;-)
Awards
  
  
Offline

Posted 06 March 2013 - 07:27 AM

While I'm not sure if you could focus on it with this tool, I am sure that two of the biggest things would be the ability to create your own buildings and the ability to save to your own computer, offline. It sounds like you've all got some work done on the former, so kudos to you guys! It's projects like these that provide the best chances for salvaging this game. great work! :)


bipin2.jpg         Need to contact me? Head on over: http://www.simpeg.co...profile;u=53285


#25 warrior

warrior
Awards
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Offline

Posted 06 March 2013 - 08:34 AM

Expanding the city size is unlikely. while it's possible that the city size is stored in the package files, creating a region with these larger tiles seems highly unlikely. Perhaps it will be possible in sandbox mode, and I'm 99% certain we can't expand the city sizes for multiplayer and single player, as Maxis will have put checks in place for modifications to preserve the integrity of the leaderboards.

 

There's more chance we'll be able to change things like the textures of the roads (Euro mod for example), but this may not work because the game may not allow any changes, included pure cosmetic ones in the non sandbox modes. 

 

Saving locally is heading into the piracy side of things, and isn't something that would be done by a tool for package files.

 

I'm in Europe so I don't get the game until Friday, and Oppie is on vacation atm, but when I get the game I'll be seeing what we can modify in what modes (if someone hasn't already).

 

If we are able to mod things, then the amount of what we can change looks like more than SimCity 4 (the UI seems less connected to the EXE than in SC4 for example)



#26 Bitteresthat

Bitteresthat
  • Bitteresthat
Offline

Posted 06 March 2013 - 01:55 PM

God I can't wait for modding tools to be available.. 



#27 Indiana Joe

Indiana Joe
  • LocationNorth Carolina, USA
Awards
  
Online

Posted 06 March 2013 - 08:46 PM

Once modding does become possible, most likely the only things that will be able to be changed are textures and skins for various objects or buildings.  Adding completely new content probably won't be possible due to the EA server connection.  And of course changing any of the physical abilities of the game is unlikely.

 

But amazing work so far!  It took years for SC4 to be modifiable; who would have thought someone would start on this game in a few days?  Bravo!


Scruffy's gonna die the way he lived.

#28 Karybdis

Karybdis
Awards
  
  
  
Offline

Posted 06 March 2013 - 09:20 PM

Well, that was quick.

 

I'll probably get in touch with some of the others and ask their opinion on rejoining dev work, but otherwise, I'm in as usual.



#29 mazman34340

mazman34340
Offline

Posted 07 March 2013 - 10:42 AM

Been a long time since I posted here.

 

My college has Maya, UDK, Z-Brush, and Photoshop on all their computers. While I don't know anything about character rigging (yet) or programming, I would like to start making content for the game for the sake of my portfolio, as a hopefull future game designer. Joining a mod team would be great too. I haven't purchased simcity yet, but I may this summer.

 

Would you mind explaining what exactly can be done with the program, other than previewing buildings and textures?

Also, will you write a short user guide anytime soon?



#30 damaindude

damaindude
Offline

Posted 07 March 2013 - 02:14 PM

Just seeing the names Oppie and Karybdis in this thread took me back 10 years. We are in good hands here.


2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users