Mojave Texture Development: Phase 0b

[Ganaram Inukshuk 906]

I dont understand a lot about pixels and all of that.. but what I do know is that I like those HD roads, and having them would be awesome.. so, whatever you do, dont drop this because its awesome.. the question is, will it be NAM compatible?

 

Texture development's kinda separate from NAM development (but knowing the insides of the NAM helps with tracking down what textures to replace), so, at least with the first part, it's completely independent of the NAM. (Odd thing is that there are already 128×256 textures included with the NAM, and no one's ever reported an issue with those.)

 

The only issue is playing with ANY HD textures on standard detail and crashing as a result, so if your machine's able to play SC4 on high detail and doesn't have any issue with HD textures, there shouldn't (in theory) be any issue, NAM or otherwise.

[Shy Dude 149]

Shouldn't the width on the lines be multiplied by 1.33 since it appears everything relating to cars in the game is scaled up by 33%? (which has made making suburban BATs quite an annoyance). I've never really cared for HD textures, I love the charm of well crafted low res textures, but you gotta do what you gotta do I guess.

[Ganaram Inukshuk 906]

Shouldn't the width on the lines be multiplied by 1.33 since it appears everything relating to cars in the game is scaled up by 33%?

 

Maybe, but probably not by 33%.

 

Watch: Average RHW width is 35 pixels, or 4.375 meters, and average non-RHW width on non-Street networks is more like 36 pixels, or 4.5 meters. US Interstates have a 3.7 meter standard, so dividing the two (4.5 ÷ 3.7) gives a number that's more like 1.22, IE, average SC4 lane widths are 22% wider than US interstate standards.

 

Transferring that figure to MUTCD recommendations of 4-6 inches, or 10.16-15.24 centimetres (and considering that the MUTCD implies that lines be this width, no matter what the lane width, be constant), line widths would only increase to 12.36-18.6 centimetres, a change of 2-3 centimetres.

 

Considering an SD pixel is 12.5 centimetres (and SD lines are typically 2 pixels wide), a 2-3 centimetre change is basically undetectable. With HD pixels being 6.25 centimetres, it could register a slight change, but not enough to be that noticeable, either.

[Shy Dude 149]

Oh, I made the wrong assumption that the driveways of the houses in my area are of pretty typical proportions, I never realized they were really narrow compared to driveways from across the nation. Yeah, it should be around 20% wider just going by how many parking lot spaces can fit in 16 meters. It still makes a big difference.

[MandelSoft 2,769]

There is one thing that HD textures may affect negatively other than filesize, and that is performance. I can imagine that these textures require more RAM, and this may make the game a bit more crash-sensitive. Although, this is just an assumption and not tested.

[Ganaram Inukshuk 906]

There is one thing that HD textures may affect negatively other than filesize, and that is performance. I can imagine that these textures require more RAM, and this may make the game a bit more crash-sensitive. Although, this is just an assumption and not tested.

 

I once had a theoretical HD-SD intermediary: convert every single network into an S3D-based network, implement SD wealthification, but the S3D textures are HD. The only thing that's HD are the S3D textures (and that'd be the easiest way of implementing an HD RRW with the least amount of work), and with RKT0, you'd need only one mipmap.

 

Thing is, S3Dification of every single network and converting all of the paths, and the risk of HD-SD Z-fighting.

 

Nothing's gonna be easy, which is why I still have the option to have SD available alongside HD, in the event everything barfs back at me or anyone else.

[NielsC007 499]

Personally I prefer the HD ones, I think the filesize and potential performance decrease is worth.

 

Good luck with further development!

[Tarkus 6,932]

As I've been telling Ganaram in our private dev boards, the RHW and NWM almost jumped on the HD bandwagon back around 2009-2010, when RHW 4.0 and NWM 1.0 were being developed.  However, when we found out that the filesize was essentially squared compared to the standard SD compressed textures, we scrapped all HD plans, especially as null45's then-new FSH libraries produced much cleaner SD results than were previously available (on par with the legendarily obscure EAgraph tool, that EA made for FSH imports into NBA Live). 

 

A full set of HD RHW textures at that juncture would have probably run up close to 400MB (it'd be more now, as RHW 4.0 is a fraction of the size of the NAM 32 RHW)--you'd be into gigabyte territory with a full set of Maxis and NAM textures.  At that time, the LEX and STEX had fairly strict filesize limits, due to bandwidth costs.  The STEX and LEX for a long time had 10MB limits (I don't know what the LEX software now allows, but the STEX limit is about 244MB--it had jumped up to 25MB and back down to 20MB around that time), and the NAM itself was in its old "modular" form, with the core running about 22MB, so jumping to a few hundred MB just for graphics would have probably caused Dirk's and Jeronij's heads to explode then.

 

It's more feasible now to go HD than it was 5 years ago, but there still may be some distribution hurdles to clear.  It might have to go on ModDB (4GB limit) instead of the STEX/LEX route. 

 

-Tarkus

[Ganaram Inukshuk 906]

It's more feasible now to go HD than it was 5 years ago, but there still may be some distribution hurdles to clear.  It might have to go on ModDB (4GB limit) instead of the STEX/LEX route.

I've already considered breaking up everything into pieces ever since day 0, and that's because of how workflow works out to be.

For those curious about how it works, here's how it goes:

• Phases 0 to 4: Dimensional specifications for HD and SD textures, One Way Road arrow textures and HD sidewalk textures, Road, One Way Road, Street, and Avenue textures
• Phase 5 to unknown: NAM content; likely to be divided up into pieces as well; roundabouts, turn lanes, and TRAM are all considered separate phases and therefore separate downloads; intersections with Street, Road, and Avenue that are added by the NAM, as well as diagonal streets, are a priority
• Phases unknown+=1 to unknownier unknown: RHW and NWM; legacy support for RHW is questionable, particularly for puzzle pieces slated for obsolescence

Phases 0 to 4 focus solely on just the textures found in the 0x00######, 0x04######, 0x05######, and 0x09###### ranges (corresponding to Road, Avenue, Street, and One Way Road, respectively), and thing is, not all of the textures found in the SimcityX.dat files are used, particularly those in the 0x00 and 0x09 ranges.

 

Because of also how NAM-related texturedev is already separated from phases 0-4, it means that the whole thing is not monolithic; everything's gonna be downloaded as individual parts, because of how I handle everything as individual parts. It also means (and this is pretty obvious by now) that if you only ever use one part of the NAM, you don't have to use/download extra textures for something you don't use.

 

Also, being space-conscious, I'm looking towards using .7z files to package the downloads with. Using regular old .zip files only compresses things down by about a fourth of the original filesize; .7z files, with the highest compression settings selected, could compress things down to nearly a third of the original filesize.

 

This is a comparison of compressing HD and SD textures; note that this is one of 8 .DAT files that make up the Road texture phase, phase 1:

 

 

SD uncompressed: 650KB

SD compressed: 420KB

Percent compression: ~65% original size

 

HD uncompressed: 1974KB

HD .zip: 1555KB: ~79% original size

Percent compression for .zip: 

HD .7z: 705KB

Percent compression for .7z: ~36% original size

 

It's hard to predict what those figures will be for the whole texture set; this is only 1/8th of phase 1, so those figures could change wildly. Still it looks like that 7zip-ing things will be the most viable option.

 

If there's another compression option anyone would like me to know of, or there's anything else important I need to know, please let me know.

[rsc204 21,808]

Whilst I do see filesizes as a potential issue, if the plan is to support SD textures, it's only an issue for those who want them so from a user perspective it's a non-issue. Having watched through your thoroughly informative video (some bits a little too technical for me, but the general gist is clear enough), I am certainly on the HD side of the fence, just seeing the differences between the medians when re-scaling clinches it alone for me.

[Tarkus 6,932]

One other thing to follow up with here:

(Odd thing is that there are already 128×256 textures included with the NAM, and no one's ever reported an issue with those.)

 

The only issue is playing with ANY HD textures on standard detail and crashing as a result, so if your machine's able to play SC4 on high detail and doesn't have any issue with HD textures, there shouldn't (in theory) be any issue, NAM or otherwise.

 

The game is set up to handle base/overlay (transit) textures with resolutions up to 256x256 without problem.  The Maxis terrain textures are actually 256x256 (and the HD terrain and rock mods out there are actually 512x512, IIRC).  It's once you go beyond 256x256 that issues are encountered--if you're on software rendering, textures larger than that will cause CTDs.  Those SD textures with the overhangs that are running 128x256 will end up being 256x512 when made HD, so those will actually cause the software render CTDs.

 

-Tarkus

[noahclem 262]

Nice to see development pick up again Ganaram   As for whether to focus on, SD v. HD, I'd recommend HD unless you personally don't plan to use it. Alex makes a useful point on the overhanging 128x256 textures likely causing software rendering crashes when doubled in size but given the modular release plan people running software rendering and wanting HD textures could pick SD versions for those network types. I wouldn't imagine a significant performance problem using HD textures since textures 256-1024 are common with terrain mods and my experience using HD park, plaza, and sidewalk mods (and now transit texture mods) hasn't included any noticeable performance difference in areas heavy on those texture types. Disk space seems the primary problem but I'd expect transit textures to remain a relatively small percentage of most users plugins if they're changed to HD.

 

I remember hearing that 4x4 pixel blurring problem with FSH compression in the past but forgot about it--thanks for the reminder and another point for the HD case

 

I've spent a lot of time pondering how best to do the pavement markings as well. Basically, after some public consultation, I've decided to design them to look as good and realistic as possible in HD while not putting much emphasis on meshing well with the Maxis textures. I might come to regret that as I have no plans currently to redo the road or OWR textures, but so far I'm happy with that route. I imagine they'll look fine scaled to SD but haven't even got around to checking as that's relatively low priority for me. There are some differences in color (model-based v. texture-based differences), width, and spacing in my textures which I think adds a bit of character but might bother some.

 

Out of curiosity: does working with Inkscape allow you to change resolution however you like without requiring additional effort? One of the big annoyances for me using GIMP is all the extra effort needed on details that wouldn't show in SD.

[APSMS 1,892]

Here's the thing...I can only notice the compression artifacts in Z6, or when you use that sizeof cheat in Z4. I never spend that much time that close to the textures.

 

It's been shown that HD building produce zoom compression artifacts at lower (far) zooms. How do your HD textures show up when viewing the city from a distance, say Z3? The size factor is not inconsiderable. I use most of the NAM components, and as the NAM installation already runs around 700mb, asking for an additional 1GB of space just for textures is a lot to ask for, even if the results are as you show. Keep in mind that I already have a 6.5GB plugin folder, and to be honest I'm not that enamored with your textures.

 

Maybe it's just me getting used to the potholes around my home, but I find it a rare day when the city streets all conform 100% to MUTCD standards; I appreciate the effort to make the roads uniform (because, we can't have randomly dilapidated roads in SC4), but I don't understand the nit-picking. I know the median lines aren't 8 inches wide where I live, but never have I seen the sides of the streets so clean.

 

Of course, having said all that, if your going to move forward with the project, then I have to recommend you follow through with HD development. It would be 1.) a shame to spend a very large amount of time on a product you won't be completely satisfied with and 2.) the HD does look much better when viewed up close. If you're going to be working with these textures for potentially the next 6 months, then I suggest you work with ones that don't make you want to pull your hair out every time that you look at them.

[Ganaram Inukshuk 906]

Out of curiosity: does working with Inkscape allow you to change resolution however you like without requiring additional effort? One of the big annoyances for me using GIMP is all the extra effort needed on details that wouldn't show in SD.

Inkscape's got the ability to export at different resolutions, and I've optimised the spacings such that there's absolutely no changes required when exported at SD.

Thing is, most of the textures exported through Inkscape look like this:

A number of textures, such as the bistubs and particularly the weird Road intersections are actually stitched together using multiple textures. It's easier not to waste Inkscape resources on something that could easily be edited using Paint.NET. The one exception is that with certain bistubs, the alphas had to be created using Inkscape to prevent any loss in colour.

In fact, all of these textures aren't exported directly in Inkscape, but are instead trimmed down and edited using Paint.NET. Part of this reason is to put a sidewalk background on every texture (unless there's a way to do that in bulk) because without it, zero-slope bridges will either show sidewalks as either absolute white or absolute black. Those alpha masks don't actually delete out colours entirely, they just set their respective alphas to 0; the actual colours still exist, but it takes something like a zero-slope bridge to make them appear again.

The other reason is becasue GoFSH doesn't work with .PNGs, and I keep .PNG versions to be compatible with the Batch PNG to FSH tool, and the .PNG versions are a record of what the textures look like without any background at all, in case someone wants to put a different background on them. Plus the filename doesn't work with the aforementioned tool and I haven't memorised the long TGI that GoFSH uses.

 

If I wanna bulk resize these textures so that they turn out to be SD, I'd actually first export everything as HD first, convert that to .FSH, and then take the individual wealth textures and alphas created from that process (because GoFSH generates wealth textures and alphas), bulk resize them using Irfanview, and then reconvert the SD textures into .FSH using GoFSH again.

[Ganaram Inukshuk 906]

Maybe it's just me getting used to the potholes around my home, but I find it a rare day when the city streets all conform 100% to MUTCD standards; I appreciate the effort to make the roads uniform (because, we can't have randomly dilapidated roads in SC4), but I don't understand the nit-picking. I know the median lines aren't 8 inches wide where I live, but never have I seen the sides of the streets so clean.

Just some clarification: it's only with HD textures that the median lines fall within MUTCD standards with a line width of 6.1 inches (0.1 inches over the max, but who's gonna notice 0.1 inches?) and under SD textures, those lines balloon to 8 inches.

Also, good luck finding consistency within my area; there's unmarked superwide streets (or unmarked superwide roads, I can't tell), shoulder lines aren't always painted, superasymmetry that doesn't even exist in the NAM (TLA-4's), places where the outermost lanes are three times wider than the innermost lanes to accommodate parallel parking, places where curbs don't even exist-- there's a lot, and there isn't even a limited access road in my area; I have to go to the Sonoran Desert to find one, and it's just as hard to find potholes.

[APSMS 1,892]

Just some clarification: it's only with HD textures that the median lines fall within MUTCD standards with a line width of 6.1 inches (0.1 inches over the max, but who's gonna notice 0.1 inches?) and under SD textures, those lines balloon to 8 inches.

Also, good luck finding consistency within my area; there's unmarked superwide streets (or unmarked superwide roads, I can't tell), shoulder lines aren't always painted, superasymmetry that doesn't even exist in the NAM (TLA-4's), places where the outermost lanes are three times wider than the innermost lanes to accommodate parallel parking, places where curbs don't even exist-- there's a lot, and there isn't even a limited access road in my area; I have to go to the Sonoran Desert to find one, and it's just as hard to find potholes.

 

I understood the SD vs HD argument as regards the issue with lane marking widths.

 

My point was that the difference is hard to discern in further zooms, and I am still unclear as to how the textures appear from a distance (do compression artifacts show up like they do for HD buildings?). And yeah, I've been to the cities/area where you live, and of course it's chock full of discrepancies. Not that I think anyone should mind driving on a 80ft-wide two-lane road.

[Ganaram Inukshuk 906]

My point was that the difference is hard to discern in further zooms, and I am still unclear as to how the textures appear from a distance (do compression artifacts show up like they do for HD buildings?).

There's practically no difference on lower zoom levels (around zoom 1-3 or 4), but the difference to be had is whether you zoom in (around zoom 4-7) at all. That's one key factor over whether to use HD or SD.

[rsc204 21,808]

The other reason is becasue GoFSH doesn't work with .PNGs, and I keep .PNG versions to be compatible with the Batch PNG to FSH tool, and the .PNG versions are a record of what the textures look like without any background at all, in case someone wants to put a different background on them. Plus the filename doesn't work with the aforementioned tool and I haven't memorised the long TGI that GoFSH uses.

 

I must say the lack of transparency in the previews is one of the things I dislike about using BMP's, that said the more I work with explicit alphas, the more I start the see the benefits. It might be worth noting that GoFSH can process a special "preview" set of textures if you wanted to see the results easily. It's also possible to use the regular -A0 alphas to automate changing the background if desired, just set an appropriate texture for the background and process with the EDIT operation to change the background layer. The -A0 alphas need converting to RGB images, removal of the Green RGB Channel and then renaming from -A0 to -DO. Through a combination of Photoshop actions and a bulk file-rename utility I have this fully automated.

 

As for files names, you just need to add the Type and Group in front in this format TTTTTTTT-GGGGGGGG-IIIIIIII-C0.bmp, provided they are all transit textures, again a bulk file-rename utility can automate that simply. If you use the PNG to BMP feature, provided you've used the ID setup 0xIIIIIIII.png for PNGs or the above T-G-I-C0 for BMPs, GoFSH will automatically ID any bitmaps or PNGs that are converted. You can use the scripts to build and package the final DATs too, that way you don't need to worry about PNG to FSH tools, it's all handled for you as part of the generation process. In any case, if you've a set of FSH files created in GoFSH it should export a reader_load_list file, you could simply import them directly into the reader too.

[Ganaram Inukshuk 906]

I must say the lack of transparency in the previews is one of the things I dislike about using BMP's, that said the more I work with explicit alphas, the more I start the see the benefits. <>

 

As for files names, you just need to add the Type and Group in front in this format TTTTTTTT-GGGGGGGG-IIIIIIII-C0.bmp, provided they are all transit textures, again a bulk file-rename utility can automate that simply. <>

I should mention that any sort of namechanges I have to do with the filenames (TGI-GoFSH format to 0x######## format), I have a preset for that in Irfanview, so that's entirely automated; that, and I have it such that I can convert 0x########.PNG files to 7AB50E44-1ABE787D-########-X0.BMP; essentially change the filetype AND the filename at once. (Though I checked if GoFSH can do that; it can, so... derp)

Also, I have already been working with GoFSH since day 0 so I already know all the ins and outs with using GoFSH (at least all the ins and outs I think I know); A0 and D0 alphas, .PNG and .BMP conversion, the fact that GoFSH can pack and unpack .DAT files, customisable wealth textures; I already know it all. There are a bunch of shortcomings (IE, no dedicated colour channel for dedicated wealth level 7 textures, no selectively choosing wealth levels 0, 1-3, and 7 independenly of 4-6, no ability to use .PNG files directly instead of .BMPs throughout), but I can't do anything about that, either, so I have to roll with that.

The one thing I do like to know is this: if I directly convert a 0x########.PNG file that was freshly exported from Inkscape and trimmed down using Paint.NET, any true alphas will be converted into either white or black once converted to 7AB50E44-1ABE787D-########-C0.BMP. Is there a way to automatically do that without having me to manually edit every single image one by one? Because so far, if there is a process for that, I don't know how it goes, and if it's not through GoFSH, I can't replicate it because I have no means to.

[noahclem 262]

To bring back earlier discussion, I'd say the HD/SD quality debate is almost completely about Z5 and Z6. The less-well-understood element is if it's possible to mitigate some of the 256px HD disadvantages in zoom 5 without further quadrupling things by going to 512 textures.

 

I'm still surprised by how much others use paint.net, and even irfanview, though it's neat that there's such diversity in how people tackle the same types of projects. The obvious thing though, is that I really need to try out gofsh sooner rather than later. Sucks I'm not able to scavenge much time at the moment.

 

edit: punctuation typo

[Ganaram Inukshuk 906]

I'm still surprised by how much others use paint.net, and even irfanview, though it's neat that there's such diversity in how people tackle the same types of projects.

Thing is, for me at least, if there's a way to cut out a million extra steps, I'd go for it. I've already figured out how to cut out Irfanview entirely out of the process (GoFSH has a PNG-BMP conversion/renaming tool built in). Currently waiting for GoFSH to... uhh... be able to cut out the Paint.NET step I described above...