DatGen - DBPF Editor - The NEXT version

[DarkMatter 21]

Hello modders! Its been a while since I've posted on SimTropolis, but it wasn't because I've dissapeared. The last couple months, I've been redesigning DatGen to be a much better, more flexible editor. The entire application is being rewritten from the ground up, with a new core library for loading, editing, and saving DBPF format data files, as well as a new interface. The core library is only about 15% done so far, but it is far more dynamic and extensible than the previous one. It will allow for dynamic loading and creation of file type objects, so that new file types can be written and added to the system without requiring a recompile of the core library and DatGen program. The DatGen interface is going through a massive redesign to improve its flexability, allow modification of the layout to suit personal needs, and present a much cleaner interface to modders.

 

My hopes here with this new dynamic library are two-fold. First, the ability to write new file type classes without requiring a recompile should assist in the discovery and development of filetypes for SimCity 4 savegames. This is the last frontier for file type decoding in SimCity 4, although many known filetypes still need to be decoded and have file type classes written for them so they can be loaded into DatGen. In addition to assisting development of file types for SimCity 4, DatGen's new core library is flexible enough to expand into editing other games that use the DBPF format. Currently, the Sims Online and the Sims 2 use the DBPF format, and its quite possible that other Maxis/EA games use it as well. The ability to extend the types of files known by the core DatGen library without a recompile expande DatGen's usefullness far beyond what it used to be.

 

The new DatGen (DatGen.exe) and core DatGen library (DatGen.DBPF.dll) are still in very early stages of development, Alpha 1 age. Most of the base design of DatGen's new interface is complete, though, and I have a few screenshots for everyone. I've greatly improved the interface, allowing parts of it to be collapsed to nothing, allowing the layout to be changed, etc.

 

Datgens standard, opening interface. Take a close look at the splitters between different sections of the screen. Clicking the

center part will collapse that part of the screen, as you'll see shortly. Also take note of the Default Namespace item in the file

browser's tree. The new DatGen will include a feature called Dat Namespaces, which allow you to manage data files as groups,

load them together, save them together, list their files together in other browsers and/or editors. Namespaces will allow you to

load .dat, .sc4lot, .sc4desc, and .sc4model files that are related to a given project, and maintain the layout and state of each screen

when you save.

 

DatGen allows the layout of the main screen, the file browser, to be changed to meet personal preferences.

 

Here is another layout, resembling Ilive's Reader layout more, for those who are more familiar with it.

 

The collapsible panels and parts of each screen can let you quickly clear away clutter and just view or work on exactly what you need.

The sidebar, which takes up a fair amount of window space regardless of what screen your on, can be collapsed to provide more work

area for every screen.

 

 

As you can see, there are no files listed in any of these screens. The core DatGen library is only about 15% done so far, so DBPF files arn't yet loading into the program. This version of DatGen also only contains two built-in viewers/editors for files: The standard ASCII text and standard HEX editors. All other viewers and editors will be in the form of plugins, which remove the need to recompile DatGen itself whenever a change to an editor is made. This will mean smaller downloads in general whenever a viewer or editor is debugged or has features added.

 

Many of the viewers that exist now will be ported to the new DatGen, rather than beeing completely rewritten, to save time and effort. Others will be completely rewritten, either because they won't port very well, are too buggy, or because current viewers will be changed to a new plugin type, an editor. Editors will show up under the Editors sidebar button, and will be more capable at editing things like S3D files, tweaking model skins, editing UI files or Lots, etc. Ports of others programs, if permission is given, like Larsz TrafficCop, can also be included as editor-type plugins.

 

Finally, DatGen will include a fairly impressive array of command line options and tools. For those who wish to script certain operations with .bat files, DatGen will be quite capable. Files will still be able to be loaded from the command line, so double-clicking an associated DBPF file type will load it in DatGen as usual. In addition to the command line options, DatGen will eventually include a finalized scripting engine and API, allowing power modders to accomplish just about anything they need without having to request an additional tool for DatGen.

 

Thoughts and comments are welcomed, and more updates will come as the program evolves. Let me know what you think, what you would like to see fixed or changed from the current version of DatGen, what new features you think are most important, etc. Thanks!

[Equinox 1,211]

Whoo!  It's great to see some pics of progress!  I think having functionality for TS2 and mebbe even other future Maxis games is incredibly useful, especially when one becomes familiar with the interface (not much of an issue anymore, lol, you could make it practically any way you want ).  I hope you can get lots of help from other programmers and modders to make this a really useful program for everybody.  Right now, reskinning features sound pretty nifty ...

[DuskTrooper 314]

Well, I cant wait for this, DM!

[Karybdis 34]

I've said it before and I'll say it again, this new version kicks arse

*happily waiting on whatever updates appear*

[MrTruesage 27]

Will there be help files included this time round to help market the functionality, useability, scaleability etc

Look forward to your new creation!

Cheers

[cow 1]

this looks pretty sweet. i can't wait to see what you come up with.

as for those of us who know very little about modding lots through actual exemplars and whatnot, will there be a possiblity that easytweaker will work with this new version--hopefully with transit enabling working (as far as i know, the current version still doesn't work)?

[DarkMatter 21]

Thanks for the comments, everyone. To answer the questions.

The help files will be here on SimTropolis, in the Modding Open Discussion forum. There are actually two threads already, one for DatGen modding tutorials, and one for DatGen documentation. The help for datgen will go in these threads. Updated tutorials will be made for the existing tutorials to comply with the new DatGen. Documentation is as big a task as writing the program itself, so it will kind of evolve along with the program. This new DatGen will come out in chunks. First the main program will be released, then individual plugins. The program will be refined, as debugged as possible, and as cleaned up as possible before any plugins are written. DatGen itself won't really do much initially, other than list files in .dats. After DatGen is stable, then I'll work on one plugin at a time, starting with the few essentials like an Exemplar viewer, S3D viewer, etc.

That should answer the other question. EasyTweaker will be ported to the new DatGen, and it will be much better than the current version. It will be expanded with features that allow it to replace the plugin manager, making it easier to create buildings for use in SC4 and the Lot Editor with the BAT.

[cow 1]

this is awesome darkmatter. i'm really glad to see that you are still standing behind your easytweaker program, because it is truly wonderful for those of us who don't want to spend too much time getting into the technical aspects of lot editing...

i think the current version is great as it is except for the problem with transit enabling and that it can't read/edit .sc4desc files. will it be able to read .sc4desc files with the newer datgen (i'm assuming so since you said in the first post that it can read .dat, .sc4lot, .sc4desc, and .sc4model files)?

[DarkMatter 21]

The new EasyTweaker will load anything that conforms to the DBPF spec. As the title says, DatGen is a DBPF Editor....not just an SC4 .dat editor now. The .dat, .sc4lot, .sc4model, .sc4desc, and .sc4 file formats are all DBPF files, as are .package files from the Sims 2 (an I assume other extensions, as well as files from the Sims Online). DatGen will read them all, and both DatGen and EasyTweaker will allow you to configure what extensions you can browse for in the open and save dialogs by editing a .xml configuration file. So yes, it'll read .sc4desc files, as well as .sc4model files. Depending on what type of file, exactly, that you open, EasyTweaker will present you with a different interface. If you open a .sc4model file, you'll be presented with steps to create a building exemplar, and you'll be able to SAVE a new .sc4desc. If you open a .sc4desc, you'll be able to tweak the building exemplar inside it much like you would if you opened a .sc4lot file.

 

While DatGen is expanding and becomming a DBPF editor, EasyTweaker is still SC4-centric. It will still be explicitly for editing SC4 files.

[Karybdis 34]

Ah well, as long as I don't have to work out the specs for TSO and TS2 Will leave that to their two respective communities. hehe

[121759 2]

Very nice, can't wait to get my hands on it.

[Buggi 684]

I'm gearing up for TS2... can't WAIT to get my grubby modding hands on it..

MUAHAHAHHAHAHA

[Delphy 0]

Karybdis,

Don't worry - I'm working out the file formats for a lot of the Sims 2 stuff at the moment.

[ABCD12345 1]

great DM[:0]

[Buggi 684]

Hey DM.. would it be possible to get my grubby modding hands on this new version soon?

[Karybdis 34]

Delphy: saw your site. very nice Good luck with the TS2 work. Maybe if I ever get finished with this stuff (and maxis doesnt release another EXP or SC5), I can join up with you and Buggi to work on it

Looking forward to this a lot, especially the capabilities it has to be put together in parts without updating the main program

[DarkMatter 21]

I'm glad people are excited about this. There isn't anything tangible to post for this program yet, and probably really won't be for a little while yet. Its in pre-alpha stage, and once it gets to beta stage I'll start posting versions again.

I've been posting cookie crumbs all over the net at TSO and TS2 sites, and people are starting to come out of the woodwork. I think the debugging efforts of the next version of DatGen should proceed smoother than the last, with more people working on using the program, and on deciphering file formats. Delphy should have a lot of company once DatGen.info is fully up and running, as should all the file format decoders here on ST. DatGen.info will have all the facilities to support file format decoding, and can serve as a permanent repository for all the data we come up with, for any game.

Anyone interested in the internal architecture of DatGen should check out the DatGen section on datgen.info (Click the arrow next to the DatGen menu item for more options). I've started the DatGen Design page, and posted some diagrams. I'll add more tonight when i get home from work.

[Delphy 0]

The current state of DatGen the new version sounds like my own program (HWSE2) for 3d models. In other words, the framework is there, some of the plugins are written for importing/exporting to various formats, but the actual program isn't done yet.

I know how complicated a rewrite can be

Thanks for the links on your site, and the notes here.

Regards
Delphy

[cow 1]

any further progress on datgen and easytweaker? i haven't heard anything or seen updates on the website for quite some time, and i'm really anxious to try out the ported easytweaker...

please give us an update on your progress.

[DarkMatter 21]

www.datgen.info

 

I've put up some alpha and beta versions in the download section there. Still a little bit of core coding to do, then I can port EasyTweaker over to the new architecture. The site also has some forums where you can discuss DatGen, plugins, and modding.

[Ryan B 35]

DM - Best of luck with this!  Once it's completed, it'll be a great modding tool for SC4 people the world over.

 

I just came up with an idea concerning stop signs and street signs (I won't go into detail here).  I ran the idea by Equinox, and he said that once this comes out, maybe I can put my idea into motion!

Once again, good luck - I can't wait to see what this turns out to be!

[GoaSkin 2]

I downloaded version 0.7.3 (and 0.7.2) from www.datgen.info and it was not usable for me.

No plugins were loaded. I could browse DAT files but only the ASCII and HEX editors were available.

I tried to register the plugins manually but it failed with the following error message:


Stack trace: at DatGen.Apps.PluginManagerTool.cboPluginType_SelectedIndexChanged(Object sender, EventArgs e) in D:\Data & Stuff\My Documents\Visual Studio Projects\DBPF Editor\DatGen.Apps\Tools\PluginManager\PluginManagerTool.cs:line 349

at System.Windows.Forms.ComboBox.OnSelectedIndexChanged(EventArgs e)

at System.Windows.Forms.ComboBox.WmReflectCommand(Message& m)

at System.Windows.Forms.ComboBox.WndProc(Message& m)

at System.Windows.Forms.ControlNativeWindow.OnMessage(Message& m)

at System.Windows.Forms.ControlNativeWindow.WndProc(Message& m)

at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[DarkMatter 21]

Get the latest debug from www.datgen.info/files/debug.rar . This is FAR more up to date than 0.7.3 (I'm up to 0.7.8 at the moment, with 0.7.9 in the works). There still arn't many plugins for SC4 yet. DatGen is an immense project now, as it covers both TS2 and SC4. The TS2 community consumes tools and generates custom content like the SC4 community never did, and probably never will. I've been putting a lot of effort into making DatGen a very flexible tool so adding new capabilities is easy.

I'd like to start developing tools for SC4 again, but at the moment I dont; think I will really be able to for several more months. That being the case, I DO have time to teach people in the SC4 community how to write file type handlers and plugins for DatGen, and give you guys the freedom to create any plugin you want. It make seem like a lot of work, but in reality, DatGen does a lot of it for you. You only have to worry about your own logic, designing the interface for your own plugin. DatGen handles loading, retrieving, and saving DBPF (.dat) files, and the stuff inside them, for you. It has an extensive infrastructure to make using .dat files very easy, too, either one at a time or in huge groups.

 

I'd like to get people here on ST interested in writing DatGen plugins. There are so many tools that can be written to open up new areas of modding for SC4, all we need are people to write them. DatGen's API is very simple and strait forward, and if you have programmed a little before, you should be able to pick it up very easily. Join irc.thesimsresource.com, channel #datgen, if you want to help.

[GoaSkin 2]

I have a bit programming experience but focussed on X11 applications using GPL tools. I installed a virtual machine with W2K only for modding SC4 but I haven't used Windows since over eight years before. I currently think about writing a filter for FSH files for the free image manipulation software GIMP.