Try to add 7-Zip support for file uploads

[Windfarmer 18]

It compresses better and therefore saves bandwidth for both sides client and server.

[A Nonny Moose 8,262]

Well, I don't have a comparison, but my Swiss Army Knife file-roller application will pretty much decompress anything.  Certainly .zip, .rar, .7zip, etc.  The system distribution default for this version of Linux is .tar.gz and it is very tight compression indeed.

Perhaps you could back up your request by compressing a BAT set up with both .zip and .7zip and show the difference in file size to improve your request.  Bald request like that always need to be backed up by data.

[rsc204 21,868]

No this is definitely right, 7-zip is vastly more efficient that WinZip. It's also open-source and multi-platform, so it would make more sense on a number of levels.

For example with my TGN mod, I could take an 19mb zip and it only took around 10mb space. Given that I uploaded 5 variants and many users mistakenly seem to download them all when only one should be required, that's a lot of bandwidth that could have been saved.

The problem is users, it's the same reason why a couple lot files will be bundled into an installer rather than in a simple zip file. If you upload 7zip or RAR files, prepare for a raft of complaints from users who don't understand the first thing about files/folders.

[A Nonny Moose 8,262]

The basic problem is that, while Microshaft can expand *.zip files, they are too cheap to add the necessary code to unpack other type than their private compressions.  This is yet another reason to dump those money mongers and switch to a Linux distribution.  The archive manager (file-roller) is a Swiss Army Knife when it comes to archive formats.

Otherwise you have to download the programs to handle these formats.  Fewer people these days know enough to do this.

[Windfarmer 18]

http://www.tomshardware.com/reviews/winrar-winzip-7-zip-magicrar,3436-7.html

Read this.

It seems that the .7z format compresses better than any other format, but you can't upload .7z files on the STEX.

[A Nonny Moose 8,262]

Subjectively, that is correct.  However, there are some private (unpublished) encryption schemes that may be better.  I would be careful about making flat statements like that.

I have no object to *.7z formats on the STEX provided that a link is given for users to acquire the necessary program.

You might find this article interesting, especially the last paragraphs on the format *.tar.gz.  This format is used by Ubuntu to distribute all of its software including massive version updates.

[Windfarmer 18]

What I'm going to do will not be a BAT setup, but a TXT, DOC, ODT, and PDF document having 55,440 lines of "The quick brown fox jumped over the lazy dog 1234567890". I will use PeaZip and 7-Zip (I would have used PowerArchiver instead of PeaZip).

[A Nonny Moose 8,262]

4 hours ago, Windfarmer said:

What I'm going to do will not be a BAT setup, but a TXT, DOC, ODT, and PDF document having 55,440 lines of "The quick brown fox jumped over the lazy dog 1234567890". I will use PeaZip and 7-Zip (I would have used PowerArchiver instead of PeaZip).

If you are running Windoze, you should be able to produce a perfectly good *.zip file by simply right clicking and requesting compress and changing the extension.  Never heard of PeaZip.

[GMT 389]

Good compression ratios are nothing but a matter of libraries, 42.zip proves this in a - literally - stunning way. And it works just fine with winzip!
Also, there's plenty of other tools to further improve packing, beyond what 7zip does. You just have to dig deep enough into geekery to find amazing stuff.
People manage to pack whole DVDs into formats fitting on a regular CD... multiple times. I know it first hand from my teenage filesharing times; 2DVD-sized games packed on 1 CDRom. EDIT: not trying to encourage, just stating experience.

But you're right as far as to say that, on a one-click average-user basis, 7zip is much much better than winzip, albeit it needs a 3rdparty software to be used.

[Windfarmer 18]

On 12/28/2015 at 6:57 PM, A Nonny Moose said:

If you are running Windoze, you should be able to produce a perfectly good *.zip file by simply right clicking and requesting compress and changing the extension.  Never heard of PeaZip.

PeaZip is free file compression software.

I am actually thinking about converting to a Linux distro.

[A Nonny Moose 8,262]

Take a look in the Linux Corner before you do that.  Whatever you do,  pick a free one.  There is support in the community and from some distros.

Some Linux distros can be loaded in parallel to Windoze, and this gives you a lot of choice.

[Windfarmer 18]

On 1/3/2016 at 3:47 PM, A Nonny Moose said:

Take a look in the Linux Corner before you do that.  Whatever you do,  pick a free one.  There is support in the community and from some distros.

Some Linux distros can be loaded in parallel to Windoze, and this gives you a lot of choice.

I mainly use Win 8.1, looking at Linux. I've ran Ubuntu 14.04 LTS in a VM for like 2 days.

[T Wrecks 10,210]

In the end, the question is if the advantage in terms of file compression warrants the hassle resulting from the foreseeable flood of help requests from people who are not actually familiar with the concept of compressed file archives.

Since we don't live in the times of squeezing as many files as possible onto as few floppy disks as possible or downloading through a 56k dial-up modem any more, you could argue that a file size reduction of even 25% is a negligible gain at the scale we are dealing with on the STEX (anywhere from ~250kB for lots to 2-5 MB for the vast majority of BATs).

Mind you, I'm saying all this as a long-time 7zip user...

On a side note, many thanks to Microsoft for that utterly stupid idea to display ZIPs as folders by default, dumbing down users and educating them into thinking that a ZIP is just a folder with a special icon and some fancy name, when actually the concept of compressed archives is very simple and the users would be perfectly capable of understanding it if only they were educated a bit in that direction. *sigh*

[A Nonny Moose 8,262]

11 hours ago, Windfarmer said:

I mainly use Win 8.1, looking at Linux. I've ran Ubuntu 14.04 LTS in a VM for like 2 days.

The VM is totally unnecessary, as the Ubuntu loader would have happily installed GRUB for you, eliminating the VM overhead.  It really doesn't take very long to reboot into a second system.

1 hour ago, T Wrecks said:

In the end, the question is if the advantage in terms of file compression warrants the hassle resulting from the foreseeable flood of help requests from people who are not actually familiar with the concept of compressed file archives.

Since we don't live in the times of squeezing as many files as possible onto as few floppy disks as possible or downloading through a 56k dial-up modem any more, you could argue that a file size reduction of even 25% is a negligible gain at the scale we are dealing with on the STEX (anywhere from ~250kB for lots to 2-5 MB for the vast majority of BATs).

Mind you, I'm saying all this as a long-time 7zip user...

On a side note, many thanks to Microsoft for that utterly stupid idea to display ZIPs as folders by default, dumbing down users and educating them into thinking that a ZIP is just a folder with a special icon and some fancy name, when actually the concept of compressed archives is very simple and the users would be perfectly capable of understanding it if only they were educated a bit in that direction. *sigh*

The only point of compressed files these days seems to be transmission.  As a long time Linux users, I have become indifferent to compression schemes because the file-roller (archive manager) pretty much handles all known types.  Traditionally Linux distros use *.tar.gz double compression.  This is unlikely to change for software distribution because it is not broken, and therefore does not need fixing.

I agree with Twrecks that Microsoft keeps trying to make itself into a nanny-state.  This alienates me from them more and more.  I abhor over-simplification.  Things should be made as simple as possible, but not one jot simpler. (A. Einstein).

[Windfarmer 18]

Quote

The VM is totally unnecessary, as the Ubuntu loader would have happily installed GRUB for you, eliminating the VM overhead.  It really doesn't take very long to reboot into a second system

I was very new to Linux.

[matias93 7,535]

In any case, I just made the experiment with simcity files: I compressed my Mattb325 plugins folder (that has almost every plugin released by him in SC4 or Simtropolis) using 7zip x64 for windows with this configurations:

(a more informed user could suggest that the parameters aren't comparable in both configurations, but I'm not completely informed about the combined function of all of them)

Fortunately for our experimental interests, I'm a very irresponsible plugins folder manager, so this compressed file contains images and html files aside from dat, sc4lot, sc4desc and sc4model, so it makes a credible proxy for downloadable content in a SC4 exchange.

The results are next:

As you can see, 7z format efectively archieves a better compression ratio and final file size in my non-scientifical experiment, but if the span of improvement makes it significatively better, as to justify a new standard in the site is up to debate. Certainly, more novice users would find that *.7z files suppose a barrier in the use of the site's custom content, and new technologies make download times increasingly faster. Maybe the best motivation to use a better compression method could be to preserve storage space in the exchange and our backup download folders.

Interestingly enough, the compression method used by datPacker creates a 52.279 Kb file for the same folder, discarding unuseful content.

[A Nonny Moose 8,262]

Technology improves over time.  There is no question that things get better in the text/file manipulation and archiving sector as time goes by.  Ideally we can someday hold the Library of Congress on a single chip.  Here is the current Wiki on this.

Technical messing about is not always good.  While some formats are better than others, *.zip is the most commonly used method.  Adding some side issues may be interesting, but since this site is turning out to be rather fragile, I'd rather reduce the number of things it does than increase them. 

In fact, I am beginning to believe that the program has become organic, and needs to be rethought from top to bottom.  You can't please everyone all the time, so maybe it is time to just please the largest number and to start ignoring squeaky wheels.

[Windfarmer 18]

I have only one of the compressed files available for download. compress.tar.gz

https://www.dropbox.com/s/g9tjf5u0dvhphes/compress.tar.gz?dl=0

I will always use Dropbox for sharing files in the forums.

[A Nonny Moose 8,262]

And the point of all these quick brown foxes is?  And BTW are you running Linux?

[rsc204 21,868]

The thing is, 25% is not negligible at scale, I don't know what the bandwidth/hosting costs are, but I think you'd be hard pressed to find a case in business not for saving 25% for such minor drawbacks. Of course this is disingenuous, in that the likihood of modifications to existing content are practically 0. However, I still feel there is a case for it.

I don't blame MS for allowing people to uncompress zip files without a third party application. When they introduced this with XP I was very happy about It in fact. Why shouldn't an OS make life easier for day to day tasks like this, it's not like they blocked your option to install another? People want to use computers, they don't all want to be hobbyists, the best applications/software/hardware serve as enablers for the masses. Much like how cars are more popular now you just buy one and don't need to understand how it works to use it.

Yes it would likely end with user requests, but then that's part of dealing with end users. Many of the same questions (already answered) come up time and time again already. I think with an appropriate post to direct users too, within a while most would be up to speed with it and it'd be a minor inconvenience.

[A Nonny Moose 8,262]

Well, as you know, my system (Linux) doesn't much care which compression scheme you use.  I also have unlimited Internet usage so there is no problem there.  The problems that arise, and I am in favour of fixing any that can be, are those Internet suppliers who cap data transfers.  I think this is an outrage, but the market is captive so there isn't much an individual can do except look for a better supplier.

I don't do much, if any streaming.  As a result, I am unlikely to encounter problems in the future.  The outfits that have found the Internet to be yet another source of revenue are to blame for loading the net down with streaming content.

I have only one question to ask: Could you manage to live without the Internet for any length of time?  Say a month?  If the answer is no, I suggest you need to get a life.

[Jasoncw 12,028]

The SC4 plugins system, dependencies, the STEX, and everything else is a PITA as it is. 

"oh, you'd like to add these nice buildings to your game! ok make an account! Now download a random 3rd party program to open up this file format that you've never heard of before. Yep, that's an .exe inside! Run it of course! If they're just game files why are they in an .exe you say?? They just are, trust me!" And this is *after* the user has already spent hours sifting through the STEX trying to find what they want to have in their game.

It's very cumbersome and sketchy sounding. 

[rsc204 21,868]

A zip file is no less sketchy than an .exe. At the end of the day it's about trust in the site to keep you safe from harm. And in the individuals who upload content, not to do malicious things. If you are downloading a file, regardless of what the file type is, there is the potential for nasties. 7zip is hardly a random 3rd party application, to my knowledge there should be no need to sign up to download it.

Now if you want to reopen the debate about useless installers that just make life so tedious. Especially for new starters, who not only don't need an installer, but have to fix every stupidly organised file it spews out, I'm all ears.

*hypocrisy alert* I've just uploaded 7 mods all that come with installers. To be fair though, they are pretty complex packages that will only get more so. I won't be expanding this for simple mods/lots containing a few files. I've also taken care that readmes and non-SC4 files are kept well away from the plugins, ensured no modifications are made to the registry and released the source scripts.

[A Nonny Moose 8,262]

Actually, considering the level of computer "expertise" these days, a self-extracting installer (*.exe) is probably a better choice than any other   form of archive.  It can be a lot of work the first time you do one of these, but once you have the framework, it should be pretty straight forward. 
The main thing is to avoid putting anything in that travesty of a database called "the Registry".  It was a bad idea when it came out, and is worse now.

When this game came out, most computer users were programmers.  No longer the case, and probably a very good thing.  Most people these days have a brain freeze if you say "script".

[Cyclone Boom 17,032]

An interesting discussion...

What might be a way forward is to create a self-extracting archive (SFX). This is an option is most compression tools, including 7-Zip. It allows files to be compressed, and contains the framework needed to extract them, without needing an external program. Everything required is bundled with the created file. As it includes this ability, there's a small increase in the file size. Although for large archives, this is fairly negligible.

The file created would be an .*exe which gives the ability to self-extract. Once run, it works like a standalone program, and allows the contents to be copied to a specified location.

In fact, this is just what the NAM does:

Compressed size (SFX):  260 MB
Extracted size:  1.09 GB

Compression ratio:  4.19:1
Space saved:  76.15%

In comparison, using Windows to compress the installer into a .*zip gives the following:

Compressed size (zip): 466 MB

Compression ratio: 2.34:1
Space saved: 57.25%

For this, it means 7-Zip saves a total of 206 MB (44.2%) compared to a standard zip created in Windows.

With a large package like the NAM installer, the extra compression has great benefits. Being also one of the most downloaded items on the STEX, as you can imagine, this saves a considerable amount of bandwidth over time.

To put this in perspective, the NAM is downloaded around 120 times per day...

Estimated Bandwidth:

Uncompressed:  130.8 GB
Standard zip:  55.9 GB
7-Zip SFX:  31.2 GB

Of course, this will become even more significant over time.

So for files which warrant the extra compression, I'd say there's good evidence here. The question is whether it's worthwhile for smaller files...

In some cases, the SFX may actually create a larger archive even compared to a *zip, which kind of defeats the purpose. Sure you can just do it as a *.7z, but 7-Zip (or another tool) would always be a dependency. I don't think it's fair to expect users to download a program to open a downloaded file. It's all about accessibility at the end of the day, something which we're striving to improve for the STEX.

[Windfarmer 18]

7 hours ago, A Nonny Moose said:

Actually, considering the level of computer "expertise" these days, a self-extracting installer (*.exe) is probably a better choice than any other   form of archive.  It can be a lot of work the first time you do one of these, but once you have the framework, it should be pretty straight forward. 
The main thing is to avoid putting anything in that travesty of a database called "the Registry".  It was a bad idea when it came out, and is worse now.

When this game came out, most computer users were programmers.  No longer the case, and probably a very good thing.  Most people these days have a brain freeze if you say "script".

1. I don't have unlimited internet. It is limited to 30 GB, far too low for today's standards.
2. Some people's internet  is very slow.

 

[matias93 7,535]

20 hours ago, Cyclone Boom said:

Of course, this will become even more significant over time.

So for files which warrant the extra compression, I'd say there's good evidence here. The question is whether it's worthwhile for smaller files...

In some cases, the SFX may actually create a larger archive even compared to a *zip, which kind of defeats the purpose. Sure you can just do it as a *.7z, but 7-Zip (or another tool) would always be a dependency. I don't think it's fair to expect users to download a program to open a downloaded file. It's all about accessibility at the end of the day, something which we're striving to improve for the STEX.

Also SFX files aren't the best option for users of non-Windows OSs, even if basically every Mac SC4 player uses File Juicer in a daily basis...

[rsc204 21,868]

16 hours ago, Cyclone Boom said:

With a large package like the NAM installer, the extra compression has great benefits. Being also one of the most downloaded items on the STEX, as you can imagine, this saves a considerable amount of bandwidth over time.

I don't know what magic the NAM team use to compress the NSIS installer further using a 7zip SFX archive. I suspect that it handles compression instead of the installer since I only got errors trying to mirror that setup. In the end I settled for letting the installer handle it. Despite being an installer .exe, my installers can be opened in 7Zip just like a regular archive. I hope Mac/Linux users would find this useful, because it simply doesn't make sense to make 3 installers. The multi-platform apps for installers I could find all have compromises I couldn't get along with.

I believe such an SFX archive should also be possible to open in 7Zip. But the NAM is special and simply can not realistically be manually installed.

[A Nonny Moose 8,262]

This is a very good discussion. 

Now let's add that most virus scanners and other malware detectors are generally against *.exe files from random sources.  However, all of these packages, AFAIK, have white lists in which the downloading site can be presented as a valid downloader.  A single line edit should fix this, and the result would be that the files would be accepted.

In any case, systems can be set to always ask about such downloads, leaving the decision to the loose nut holding the keyboard.

[Windfarmer 18]

I think downloading 7Zip would we worth it. Less traffic would be conducted on the servers, meaning Simtropolis and so many other websites could become faster.