Sharing / Moving / Managing big modded builds

[AmiPolizeiFunk 446]

Let's talk about managing and sharing heavily modded builds. I'm glad that CO has given us new buttons in the SAVEGAMES menu, but they appear to work oddly. Recently I wanted to move my Berlin build (~2500 custom assets, ~70 mods) to my second computer to assist me in making cinematics. It was an interesting experience and it took me an entire day before I got it to work. There were many pitfalls to overcome, like finding all of the mod data files and copying them over. 

There was one thing that I found very odd. After you share a savegame and download it on a new machine, you can click "Subscribe All." I was really hoping that this would only subscribe to the assets that are used in the savegame, but sadly, it doesn't. It subscribes to ALL assets that the uploader had active when the savegame was uploaded. This is a huge problem, because as far as I know, there is no easy one-click way to isolate used assets from enabled assets. Yes I know that LSM puts out a great hotlinked record of every savegame that you load, but it is still an unfeasible way to do what I want to do (going through the record asset by asset still takes forever, and the fact that some assets are embedded in packages with other assets makes things even more difficult). Ideally, we need an automated way to isolate mods, assets, and settings that are used by a particular savegame, so that they can be shared as a complete package. It would be really cool if steam collections from this package could be generated as well.

 

Have you guys tried to move or share big builds? If we could streamline the process, it could open up many possibilities.

[Elektrix 42]

All right, I think I've worked out how to do this efficiently-ish. This process allows you to unsubscribe from only the assets that are unused (allowing you to publish the gamesave with only used assets), and then resubscribe later from a collection. The only issue I could see arising is embedded items in packs being removed by mistake. However, these are few and far in-between.

1. Open the LSM assets report.

2. Run this code in the JavaScript console (remember to note how many assets there are):

// written by Elektrix; clipboard API fallback from https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript
var unused = document.getElementsByTagName("h1")[2];
var nextLinks = [];
var elem = unused.nextElementSibling;
while(elem){
	if(elem.firstChild && elem.firstChild.firstChild && elem.firstChild.firstChild.nodeName && elem.firstChild.firstChild.nodeName.toLowerCase() == "a") nextLinks.push(elem.firstChild.firstChild.href.substr(55));
	elem = elem.nextElementSibling;
}
console.log(nextLinks);

function fallbackCopyTextToClipboard(text) {
	var textArea = document.createElement("textarea");
	textArea.value = text;
	document.body.appendChild(textArea);
	textArea.focus();
	textArea.select();
	
	try {
		var successful = document.execCommand('copy');
		var msg = successful ? 'successful' : 'unsuccessful';
		console.log('Fallback: Copying text command was ' + msg);
	} catch (err) {
		console.error('Fallback: Oops, unable to copy', err);
	}
	
	document.body.removeChild(textArea);
}

console.log('There are ' + nextLinks.length + ' unused assets.');
fallbackCopyTextToClipboard("var items = " + JSON.stringify(nextLinks) + ";")

Code Explanation: This code (1) finds unused assets from parsing the LSM report, then (2) copies the result to the clipboard as a code block.

3. Create a collection.

4. [IMPORTANT] Make the title and details of the collection anything you want. Make sure "Items that work together" are selected under "What kind of collection is this?"

5. Go to the next screen and reopen the JavaScript console.

6. Paste and run the contents of the clipboard.
Code Explanation: This code creates an array of all the item ids that correspond to unused assets. It was generated in step 2.

7. Wait for the code to finish. It is finished when you get "Attempted to add item" repeated in the console as many times as there are assets. This took about 5-ish minutes for me, but it depends on your internet speed and the connection speed to the Steam servers.

for(var item of items){
	jQuery.ajax({
		type: "POST",
		url: 'https://steamcommunity.com/sharedfiles/addchild',
		data: {
			id: (new URL(location)).searchParams.get('id'),
			sessionid: window.g_sessionID,
			childid: item
		},
		success: function(response){
			console.log("Attempted to add item");
		}
	})
}

Code Explanation: Takes the array of items pasted in step 6 and tells the steam servers that each one should be added.

8. Reload the page. All the unused items are now in the collection. Then publish the collection.

9. The page should reload; hit Unsubscribe All. This will unsubscribe you from all assets that are unused in the gamesave. Note: If you forgot to do step 4 or did it incorrectly, the Unsubscribe All button won't exist. 

10. Load C:SL; publish the gamesave (you won't be subscribed to any assets that aren't used, so they won't show up in the gamesave)

11. If you so choose, go back to your collection and you can resubscribe there (Subscribe All) or delete it if you don't want the collection lying around.

 

That should work. I tested it on my computer and it worked fine. And hey, at least it's better than unsubscribing all the assets by hand.

[Elektrix 42]

Just as an afterthought, you can use the same technique to generate collections of items that are in the gamesave so that other people can subscribe to them quickly... Just switch the first part of the first script for this one (very similar):

// written by elektrix
var unused = document.getElementsByTagName("h1")[1];
var nextLinks = [];
var elem = unused.nextElementSibling;
while(elem && elem.nodeName.toLowerCase() != "h1"){
	if(elem.firstChild && elem.firstChild.firstChild && elem.firstChild.firstChild.nodeName && elem.firstChild.firstChild.nodeName.toLowerCase() == "a") nextLinks.push(elem.firstChild.firstChild.href.substr(55));
	elem = elem.nextElementSibling;
}
console.log(nextLinks);

But since there's a built-in option for this now, it's less useful to create collections like that.

[AmiPolizeiFunk 446]

On 10/15/2018 at 7:48 PM, Elektrix said:

7. Run this code until you get "Attempted to add item" repeated as many times as there are assets.

I appreciate trying to solve this problem with a web-based approach, but this step in the process is exactly these kind of repetitive and time-consuming activity that I'm trying to avoid. I'm hoping for a system where we can load and unload heavily modded savegames and share them with one another easily and often. Perhaps we'll need CO's help and some updates to the system, but I don't think that's out of the realm of possibility. 

 

Right now I'm having the problem that the ASSETS->"Disable All" button isn't working for me. I have 2500+ assets enabled, and when I click "Disable All," my computer will spin and hang up for a minute or so, but eventually it will come back to life and the "On/Off" buttons will all be greyed out and it looks like the action worked. But then when I EXIT and re-load, or load into a savegame, I find that the action didn't work at all, and all of the assets are enabled again. 

[Elektrix 42]

6 hours ago, AmiPolizeiFunk said:

I appreciate trying to solve this problem with a web-based approach, but this step in the process is exactly these kind of repetitive and time-consuming activity that I'm trying to avoid. I'm hoping for a system where we can load and unload heavily modded savegames and share them with one another easily and often. Perhaps we'll need CO's help and some updates to the system, but I don't think that's out of the realm of possibility. 

[snip]

1

Wait-wait-wait— I didn't mean to run the code 2500+ times. I'm sorry, that's not what I meant to say at all. All you have to do is run the top code once and the bottom one once, and just wait while the bottom one finishes. I could see how repeating the same code over and over and over could be super time-consuming, and as a developer I would never make someone repeat the same code 2500 times, lol. I'm way too lazy for that. I edited my post to clarify what I meant.

This is the part that iterates over all the items (a basic loop)

for(var item of items){
	// ... there is stuff here
}

 

If you mean that waiting for the program to finish takes too long, then I can understand that. Since web development is my forte, I can usually paste code into the console a lot faster than other people, and I didn't really take that into consideration while working. Sorry about that.

I'm currently looking into the source code to see where it could be detoured to fix savegames' functionality— it seems that the asset list for the savegame is generated during saves. It includes all loaded assets (instead of used assets) and is then sent straight to the workshop while publishing (without any formatting or editing). If the asset list were removed of all unused assets in the SaveGameMetaData before the game is saved during play, then they wouldn't be included in the workshop entry.

[AmiPolizeiFunk 446]

Wow, thanks for looking so deeply into this. I'm kinda surprised that more people aren't talking about it or pushing for it... because many of us must want to share our savegames or easily load savegames by other people. Thanks for clarifying how your code works too... I thought I would have to re-run the portion of it that that adds assets to the collection. You've given me more confidence now and I pledge to give your method a shot 

 

My working theory right now about the "enable all" and "disable all" buttons is that it somehow stores the list in a separate register, because when you quit and re-load, everything is back to the way it was before you clicked "disable all." I really have no idea what's going on tho... and it's an important part of being able to unload the build that you are currently working on. 

[Elektrix 42]

Yay! You're welcome; I love helping out with anything I can (especially when it's something that I'm also frustrated with, and definitely when it allows me to procrastinate on my calculus homework ).

As for the enable/disable all button issues, it could be a variety of problems, although I personally suspect that the game is unable to disable the assets at all. From my understanding of it through a cursory glance at the code, the game first turns off all the checkboxes and then disables assets, so if it couldn't disable the assets then they might still appear to be disabled in the UI. Never mind, I took another look and I'm pretty sure it might be broken because of a mod's detour. I'm having a hard time coming up with ideas why that specifically would fail.

Also, exploring the source code may not be as complicated as you might think...
The source code for the Enable/Disable All buttons are in ContentManagerPanel.cs, CategoryContentPanel.cs, and EntryData.cs.

[Stanil 15]

@Elektrix: I tried your code, but unsucessful.

It seems to create the array, but it fails when I copy it into the collection creation tab console and run your code of step 7.

(Don‘t worry about the shorter array in screenshot 2. I shortened it in my second attempt because I suspected that maybe I might have gone over the top with the amount of subscribed assets  But that‘s the reason why I'm in this thread.)

[Elektrix 42]

@Stanil Thanks for pointing that out! I forgot to update my original post when I added some stuff locally for simplicity of use when I was testing; that's my fault, sorry.

I updated the original post with the correct code, it should work perfectly now 

[Stanil 15]

@Elektrix: Thanks a lot, now I'm one step further.

I still do have a problem though. I don‘t know when the code is finished. I waited some time, but there is no feedback. So I reloaded the page and I saw that 239 items were added to the collection, that's about 5 percent of the total number of items in the array. So I guess I reloaded too early? But how do I know when to reload?

Thank you for your help so far! 

[Elektrix 42]

var a = 0, i = 0, s = new Date(), d;
if(!items || items.length == 0) throw "Error: Either you didn't input an item array, or the length is 0. Either way, this script can't do anything with your current input.";
items = [...new Set(items)];
function addItem(items, index){
	var item = items[index];
	jQuery.ajax({
		type: "POST",
		url: 'https://steamcommunity.com/sharedfiles/addchild',
		data: {
			id: (new URL(location)).searchParams.get('id'),
			sessionid: window.g_sessionID,
			childid: item
		},
		success: function(response){
			console.log("Succeeded to add item with id " + item + " (" + (a + 1) + "/" + items.length + ")");
			a++;
		}
	})
}
var int = setInterval(function(){
	if(a + 1 == items.length && !d){
		var t = Math.abs((new Date().getTime()) - s.getTime()) / 1000;
		alert('Done loading; took ' + t + ' seconds.');
		clearInterval(int);
		d = !d;
	}
	if(i < items.length){
		console.log("Queued item " + items[i]);
		addItem(items, i);
		i++;
	}
}, 30);

All right, this should provide some more feedback. When an item is successfully added, it will give you info like this: 

The new script also loads the items separately instead of all at once, which should hopefully decrease wait time.

Edit: I just got home, and tested the newest script on my full asset list; it works wonderfully and took just over 4 minutes for 1000 assets. I also updated it again to improve performance; duplicates (e.g. packs) are merged so that you don't have to wait for as many assets. Woohoo!

[Stanil 15]

@Elektrix: Cool, I will try this tomorrow.

Unrelated to your code update I tried it again with the old code without feedback, and I think I ran into another problem unrelated to your code. There seems to be an item limit for workshop collections of 1,000 items. I have found this out because my collection created by your (old) code is exactly of this size (which is odd), and also a short google search brought up this link:

https://steamcommunity.com/app/255710/discussions/0/152391995408268435/

I know I can work around this issue by creating several collections (in my case: 5). Still cumbersome, but of course that's not your fault 

[Elektrix 42]

Wow, I didn't know that limit even existed! That's good to know. I think I can probably code around it; I'll work on it a bit tonight and hopefully I'll have something soon 

[AmiPolizeiFunk 446]

1000 item limit good grief .... another hurdle 

 

keep up the good work guys, I'm following w keen interest!