Announcing GoFSH 3.7.x

There is a new version of GoFSH which has two things you might be interested in (particularly if you make big mods).
 
1. Performance Improvements.
Triggered by an offhand comment I read about the duration of a recent build, I've had a major go at improving performance in the the scripting process. As might be imagined, the overhead of showing things as we go is useful when developing things, but it turns out there is a 200% penalty incurred to display on the run in a script. By offering the option of dropping all preview display while running a script I have cut runtimes to 1/3 of what they were so e.g. 18 min should now be close to 6 min. To be honest I was surprised it was such a large effect - I was only expecting 2x as fast. Of course, the faster it runs, the bigger the jobs we will feed it.. 

As part of the work below I improved the FSH<>DAT section speed by caching the file in one go - large files will come up much quicker - however there is always a delay when viewing Tex Sets as this is due to the microsoft code (constraints) doing its thing in the viewlist. Once that's been populated its fast. It will display about 12000 texsets before falling over so the combined EURO Textures set of 24500 can only be viewed as a list of files, unless its split up. I dont have a fix for that either. Also when saving it pays to be working on SSD, as the filesystem can otherwise find itself waiting for disk when dropping several thousand files into a folder at once.. - LOL.

2. Filleting FSH
As I now need to update my various modds to be up to NAM40 standard I have the onerous task of finding out what has changed that I need to update. I then need to extract and convert the new things for US and EUR; yadayada you know the pain this is. For those of you with up-to-date source archives its probably less of a problem. So what I have done is made a tool to save me a lot of work, that, given a folder full of fsh or a given .dat, determines the C0/D0 pairs, puts them into folders and produces a script needed to reconstruct those FSH. 	This takes a couple of minutes for sizeable DATs, and saves many hours of manual error-prone labour. Ironically it now takes longer to unpack a .dat to disk than it does to fillet the C0/D0 pairs out - this is because 10000 FSH may reduce to less than 400 C0/D0 bitmap pairs. Building on the other hand has this factor as a multiplier. It will use about 2 times the storage of the original folder of fsh/.dat but cleans up after itself. 

Of course there is always a price to pay. Because it relies on the regularity of design of Wealth 0, Wealths 1-3 and 4-6 and the lack of wealth 7 while it can get 98% right, the 2% that fools it needs eyeballing and fixing. It gets confused with numbering schemes that look like wealthing but are in fact serial to some extent - e.g. During testing I came a cropper on the series starting 57Dx, 5E1x - RHW ramps which have full series of *E files that are all separate textures, and I'm sure there are others in this camp that need to be listed as exceptions. I've allowed for this with a NWWealthingExceptions.csv in the Profile\Textures Folder. These exceptions show up as bogus W17 sets, and odd colour schemes in the D0 files when unrelated alphas are merged - fortunately they stick out - humans are good at spotting misfits. The original Zoom4 FSH are still there to be used in checking if necessary. There is also a DEBUG log put to disk that can be reviewed to pin down weirdness, albeit laboriously. Nevertheless in testing I have managed to get most all of EuroRHD textures processed without crashing - albeit in pre-cut chunks as its a very big file and at the limit of what GoFSH can display at once.

Other confusers are:
- items where the color C0 changes with the wealthing - examples I've seen are Pedestrian Crossings on Road intersections for wealths 456 but not for 123 - no way to find these except by eye.
- it cannot resolve the generation D0 (although it produces one) of any textures with no transparency i.e. a White D0. This follows as all clues are gone after generation of these - and often they are EDIT candidates because of baked in Grass or Paving. It will put these into NoGen folders, but need manual intervention to turn them into something useful. Robin is the expert on these as he's seen hundreds of them. However, there are now new Generate Options to be able to make thse from scratch as part of an archive.
- color corrections for Puzzle pieces are visually obvious but need manual splitting out for scripting.
- things that are Network groupID textures but not roadways and a large number of lot textures are oddly enough handled well enough. BAT Textures on the other hand generally go wrong - but then we don't need them.
- Rail related stuff (RRW, GLR, HSR etc) will all come out as uncoloured D0s - you actually need the RRW Reskin sources of these D0s - all the hard work has already been done there - except for the recently unearthed Vol2 of the CD Sidings textures. GLR and the like have yet to be formally tackled beyond the work Robin (MGB) has done.

What this is useful for is:
-	Making/Updating Master Archives from DATs - Color Quality of some old stuff is pretty ordinary, but newer textures come up sharp, especially RHW. The sources exist, but the D0s would otherwise be a pain to make just to have a masterset. Update cycle is make a new archive, copy the previous over it - protect those things you know are now obsolete.
- As a Roundtrip test to make sure a generation run or a merge process actually did what you expected - it will pick up missing Wealths in sets that should have them - these show as all wealths except the missing one in UNSURE - I found some in the zEURO RHD sets around 57022xxx and 57896xxx, there's also a couple of US ones there. 
- Another thing that shows up are color differences in tiles presumably because different authors had different monitor calibrations e.g. Many Euro tiles have a green bias. Some have over-prominent white lines e.g. NWM RD-3.  We can now fix this in a bulk manner with this tool, Photoshop actions and rebuilding with scripting.
- Prepping for a modd - particularly if you are altering the originals or making a modd and need all of the sources but only have access to production FSH, or quickly want D0s. Everything can now be made from D0s including no generation defaults for w0 and w7.
- Quick mods - If you are clever about what you choose to play with, (or lucky), you may be able to Fillet and Rebuild with new grass or paving without any significant interventions.
- Preparing for bulk conversion to GoFSH Scripting - prepare your archives with Fillet, then copy your original color sources over the filleted ones, and fix any D0s that need to be specific and update your scripts. This beats making D0s manually hands down. ;-)


So there is a GoFSH 3.7.3 in my archive here. This is an installer that creates a breaking change in the csv files in \Textures. Make a backup of your \Textures Folder first. After installation if you need to go back to the previous version you'll need to uninstall 3.7 and reinstall that previous version. I have updated the manual for the new features, some of which are only detailed there.

Could I have feedback on - make speed, stability and things its failed on. Also, if you broke it, how you broke it - I'm sure it's not bulletproof yet, but I have been using it for a month or so now with no surprises. Exception handling of the new Filleting stuff is functional but arguably fragile - don't use it on anything valuable unless its a copy of said work.

In particular too, please tell me texture ranges known to be non-standard IID series w.r.t. wealthing. I intend keep that exception file maintained.

Here too is a latest version of the archive - All the updated stuff is near the bottom, and older versions have now been removed. Note no changes in Rail archives at this time - they are stable. So far I have done all of my custome SAM sets, and Im working on the Main Street Mod - a lot has changed in the last three years.

   cheers Ron