Potential 'Dependency Manager' system

[concrete_mielie 0]

Hey everyone,

I just want to share my experiences in general, and run something by you, see if anyone's started a project like this and if not would be willing to help me do so.

As an avid SC4 player, I love my addons. I really do. But at the time when i started downloading addons I had dial-up and now it isn't much better since i'm on 3G so my bandwidth is at a prime.

I have hundreds of addons, from the no-brainer NAM to BSC Parks to buildings galore. I'm a hoarder.

But the problem is, and probably is for other hoarders, there are so many dependencies out there, some in megapacks, some in updated megapacks, some moved from STEX elsewhere, etc etc.

I would like to create a central repository that is useful for mod creators, teams, and general players alike, that can help us all to better organize our stuff. I realise that everyone has their stuff sorted out in their own way, and have their own system, but trust me ... as someone who is reinstalling mods dating from 2005 onwards from scratch to ensure that I have all the dependencies ... keeping track of files is a nightmare (and re-downloading the same thing over again because the filename is different is a waste of bandwidth)

So I am thinking that I could create a PHP-MySQL driven website (I am a webdesigner/developer by trade) that has multiple purposes:

• Mod creators can list their mod on the site (with a direct link to their host of choice)
• The mod creator lists all their dependant files as individual files
• Dependencies get listed on the site too, as individual files. Super/Mega packs can be set up too, but as a 'collection' of individual files already listed, along with an upload date.
• When dependencies, mods or packs get updated, anyone who "has that mod" can get alerted as to the status of the file(s) he/she has vs the ones available for download.
• Store user-experienced mod incompatibilities, for example when viewing info for Mod Y it says "I see you have Mod X. Mod Y is known to not work with Mod X." etc.

At the moment that's all i can think of. I certainly don't wish to take anything away from STEX or LEX or any of the websites that host files, my idea would point people in the right direction (if someone moves their files, they can indicate where it moved to, or if it became part of a pack, etc) and to be able to, at a simple search, ensure that they have all the dependencies that they need.

Any thoughts?

[doodlea21 31]

You may find that Sc4devotion already has something alike to what you have described. The 'brown boxes' link will take you there. All the dependency packs, including the ones that hae been merged into mega packs, are listed in alphabetical order with a relevant link. And of course SC4devotion has a 'my downloaded files' page, where you can keep track of your downloaded files and it will tell you if any have been updated.

[DocRorlach 25]

While doodlea21 is right in principle, in practice this does not quite work because the SC4D link first of all focuses on BSC & related dependencies; and secondly it requires membership. Thus Pegasus' dependency, for example, are not listed, nor are ton's of dependencies from other independent creators.

Such a central, public, and most of all independent repository would be a huge boon if it managed to avoid site and team politics (which, strangely enough still abound). Such excellent tools as Cleanitol, which would make finding dependencies a lot easier if - again - it would be used by every content creator who uses dependencies )and there are few who don''t).

If such a repository ever comes into being, I would gladly focus the attention of the DIC (Download & Installation Catalogue) in its direction. As it is, I cannot..pity that..

[warrior 169]

I thought about this, (but I dont have anything near the skills to do it) and it would a massive task, as there are thousands of custom content on the exchange (and lots of dependencies)

But if this could be done it would be great, and could practically remove the problems with dependencies. And a user friendly interface to add downloads to the database would make it much easier for a group of people who say dont have the skills to make databases input data.

Jonathan

[JakeMD 107]

Interesting idea, I sent you a PM.

[DocRorlach 25]

While I might as well be wrong, I do think I could do this, at least in terms of programming, if I had some help in terms of what I am looking for within a given exemplar. I've recently started the (immensely complicated) study of iLive's genius code.

But - correct me if I am wrong - the far greater task would be to assemble the locations of the dependencies, and to get permission to access the different databases programatically.

Finally, this would be team work (and you;re wrong, Jonathan, you do have the skills, I know that for a fact), so, if there would be joiners..???

[Neddiggis 1]

Originally posted by: DocRorlach

While doodlea21 is right in principle, in practice this does not quite work because the SC4D link first of all focuses on BSC & related dependencies; and secondly it requires membership. Thus Pegasus' dependency, for example, are not listed, nor are ton's of dependencies from other independent creators.

quote>

 

Doc, have you looked at the list on SC4D ever?  

It contains every single dependency available on the STEX, PLEX and BSCLEX up to around July last year I think.   I have also linked to external sites like Gizmo's for the more common dependencies, although it's possible they have fallen off the list when it was converted to Joomla.  There has never been, and never will be, a site bias to the list, although I am not able to update any of Peg's PLEX dependencies myself although his STEX ones would be added.

I started it when I was in the same position as concrete_miele November 2006.  It was originally hosted at Peg's, and the intention was to do something similar to what he is suggesting, however during Peg's hosting problems and conversion to SMF it got dropped and Jeronij kindly offered to host it.  I have recently gained access to the source and hope to update it soon, but it's a mamoth task and takes some time to do and my life is a little manic at present.  

When I update it I will also provide a downloadable PDF version so people can print it off and keep track of what they have downloaded.

[DocRorlach 25]

Diggis, I stand corrected. Gladly so. No longer being a member of SC4D (for my own reasons) I am not certain I would have access to that list. Again, I could be wrong.

But rather than providing a PDF, would a CSV or plain text - aka Cleanitol file - be of more use. Folks could then use Cleanitol to locate missing dependencies. Admittely, I am asking out of ulterior motives: the DIC could use such an accessible list to make it easier for everyone. Especially if that would be a truly global list that grows over time. I fully understand that it is a gargantuan undertaking - if I can help in anyway, let me know. 

While we are at subject - is there any way to identify a dependency by the crazy identifier SC4 throws up whenever it is missing something that does not produce a brown box but is considered as a "missing plugin"? Again, I am asking as a programmer..

[Jezus53 18]

I am really interested in this task. At the moment I'm learning computer code and html code for fun (weird sense of fun ). Just one question, where are you going to host this site and the files? That's gotta be a lot of data to store up. Or are you talking about a data base that sends you to the place that has it instead of keeping it on hand? If so, then in theory, it should be that hard to set up the list, just time consuming. Making the user friendly interface...well, I'm still working on that.

As a little side note, I'm curious if any of you know of any good sites or books that teach how to work with java script or with html code? I have limited recources (books from school) and I have no idea where to start!

[BarbyW 40]

DocRorlach, you are welcome at SC4D any time. Your account is still intact so you can go and look at the Dependency List and anything else that takes your fancy.

[concrete_mielie 0]

Hey all

Thanks for the responses. Let me address some of the questions in general.

1) I am aware of the SC4D list, and while it certainly serves it's purpose in fantastic fashion, it is basically updated by (if i'm not mistaken) one man (isn't that you Neddiggis? ) and sporadically based on the mammoth nature of the task. I'm looking to make a system that is always up to date and essentially moderated by the suppliers of the content. Also, a printable PDF is an option, but it's static. I think a solution that runs itself may be more work but will also solve a great need, and will even make the lives of those that are organized ever so slightly easier.

2) Part of the issue for me has been the whole 'readme' problem, or 'dependency list on the web' thing. There is no consistency as to where dependencies are displayed, and for idiots like me who are download-whores, sometimes you lose track of your downloads, where you got them, and what dependencies it uses. Tracking that down is a nightmare. This system would be developed not to link to LEX and STEX, but for STEX and LEX to be able to link to the system. However the whole system relies on the user entering quality data.

3) This would be a resource system, not a storage facility for the files. So for example, i make a awesome new powerplant BAT. I am a registered mod maker on "The List" (which we'll call it for now) ... so as a part of uploading my creation for others to download i also index my product on The List, and I reference whether or not it contains any dependencies (or explicitly states that it is completely self-contained). It appears as a BAT on there, but more importantly it adds the BAT to the database including information about all its dependencies automagically. I would make sure that it is easy to do this. Theoretically, I could then supply a direct link to The List to show information (which can also track downloaded files and with that in mind can also track if there are any updates to megapacks, as well as which megapacks contain what files.) So instead of having to be bothered to write full dependency lists on whatever websites I upload my BAT to, I can basically store my information on The List and link to it. Or, depending on how people name their stuff and if they are consistant with spelling, dirktator and Jeronij or anybody else could add a generic 'search based' link to The List as a standard button which opens in a new tab/window (i.e. www.thelist.com/search.php?query=Full+Mod+Name+Here) which does a search for the name. If it matches, it redirects. if not, it presents 'plausible options'.

4) It would definately be a team project, my real-life job is providing solutions exactly like this ... customised database / web application development. So it'd be a nice challenge for me too.

5) DocRorlach - Honestly i'm not very clued up on the technicalities of mods and exemplars and identifiers, but yes, it could be a set of fields that people who create prop packs and so on (general dependency files) can actually enter. This could then be searchable. But once again, this relies on the quality of data entered, but we could also have a dedicated and passionate team of editors that enter that data for them. Hahah. So if the data is there, we can make it a parameter.

6) Jezus53 - Google is handy. hehe. Try www.pixel2life.com ... lots and lots of tutorials on just about everything programming/design/IT

7) I'm excited for this, honestly. This could work very nicely

[DocRorlach 25]

CM, you obviously have really thought this through, and I for one wholehartedly agree with your approach. Your assesment is correct in pretty much all aspects. Inf fact a lot of the data already exists, in the rightly famed Cleanitol files that a lot of the better known teams already use. NAM is a typical example.

As you said, it will depend on the quality of the input, but if this where an online database, independently hosted it might actually possible to convince the various site admins/owners to permit uploads only once the product has passed through the database (call it content certificate perhaps?). Since it is a database, with a simple form to face it, it would not become particularily large and since it would be hosted away from the forums and exchanges, bandwith and speed would not be a problem at all. Most of the free ASP/PHP hosts around could be home to it, and it would need no more than a couple of admins who regularily check the new entries. Since the most likely candidates would be PHP/MySQL I cannot help, I only know ASP/ASP.NET & SQL/Access. But I'd be willing to lend a hand in any other way.

There will be hold-outs, for sure, since there are always "personalities", but that can be overcome in time. 

BTW: My "examplar" approach would not come into this - I am attempting to scan incoming downloads or installations for MIA's and errors - and I've not been verys successful thus far.

---

As an aside: BarbyW, I've pm-ed you, since I don't think that topic befits this forum.

[sixers33 3]

concrete_mielie, that has been discussed between me, doc and a few others several times before. If you need help, i can provide my php & mysql knowledge

[concrete_mielie 0]

Doc: Ironically, it kinda just flooded to me in a very short amount of time, I've spent a few hours contemplating HOW to do it, but not what to do. Heheh. I like the concept of "content certificates" ... i just dont want people to feel like they have to "jump through hoops" to share their content. Having said that, the success of the system may just depend on those uploaders being willing to go that extra step, which most of them won't have a problem since I will ensure that the process of adding to their 'profile' is extremely simple and quick.

I could write a simple script where STEX and any others can actually add a field for checking to see if the certificate 'number' provided matches that which is on The List site, and require a match in order to upload the file. But would people respond to that inconvenience?

sixers: thanks for your offer of help, it will definately come in handy I'm a PHP/MySQL developer myself, so a partner in crime will be great

to all interested in helping: I think the important thing to do, something which i've already started doing, is working out exactly how the flow will work and what is actually needed in order to make it viable, as well as feature requests. So if you have any ideas, fling them out, outrageous as they are, lets see if we can do it.

I'm going to, at some stage, get a blog going on this. Unless someone wants to start one in the meantime ... hehe

[warrior 169]

Coincidentally we've been learning about Databases in IS at school(for the first time I've ever learn't anything in the lesson : ))

By database would that mean like MS Access

And how would the database be set out?

Like in one column the lot/mod and in the next it's dependancies?

But like I said before I haven't used a database before.

Jonathan

[DocRorlach 25]

Probably not MS Access, not for an online DB. MY guess is, since most of the sites use PHP, the database would be MySQL, with two tables that possibly need no more than a few fields each: first the dependencies table (dependency name, url where to get it, reference number to the upload table), and the upload table (upload file name, author/team, type: manual/exeutable installation, number of depencies, verfication code). The process ought to be simple, so it's not much work. Perchance the script  would issue the uploader with a code that would then be used by he various to see that it is a verified upload.

[warrior 169]

So how would the upload table link to the dependencies table (I feel like I'm hijacking the thread and taking it off topic a bit, sorry )

Jonathan

[DocRorlach 25]

No, your no hijacking: it's easy enough, and it is related:

With the dependencies in the main table, and the - for example - lot files in the related table, it'll look like this (perhaps - up to the designer):

Dependency	URL	ID
dependency_A.dat	somehwhere.url	20202
dependency_B.dat	somehwhereelse.url	20203
megapack_Cdat	somehwhere.url	20204
dependency_D.dat	somehwhere.url	20205

upload table

Upload	Type	ID	Dependencies
upload_A.zip	ZIP	20202, 20203, 20304	3
upload_C.zip	EXE	20205	1
upload_D.zip	ZIP	n/a	0

The upload table is thus related to the dependency table via the ID. Hence "Relational Databases".

[warrior 169]

What if another upload has a dependency of Dependency_a.dat ?

Then you'll get duplicate entries of the dependencies, which doesn't seem efficient to me?

Jonathan

[Jezus53 18]

Warrior, I think they want to set it up so that it tells you if you already have the dependency. Kinda like on SC4D shows you when you already downloaded the file. So that way you don't accidentally install it twice.

CM: Thanks for the site. I'll definitely look into it. Very very interesting stuff guys! I would love to take part once I learn a little bit more about this. You said you were going to make a blog, could you tell us where that is?

[warrior 169]

I meant on the inside/ back end. But Doc explained it to me, thanks for that

Jonathan

[concrete_mielie 0]

Warrior: Doc has it pretty much on the money, essentially it's a series of several interlinked tables. One BAT can use one, many or no dependencies, and one Dependency can be a part of one, many or (technically) no BATS. But it gets slightly more complex in that dependencies can be found in mega packs. So the site will tell you that you should download MegaPack X ... and by downloading MegaPack X you will automatically 'have' all dependencies that are part of that pack already, and when you view the page for "My First BAT" it then has ticked off Dependency A,B,C,D  etc etc.

Oh the magic of Left Join queries, hahahah.

The only 'downfall' to this is that for obvious reasons I wont be hotlinking to the files directly, so for users it's a tool that they have to USE - i.e. mark the file as downloaded manually. In an ideal world, I would be able to provide a direct link to the file. But the world isn't ideal, is it ... -sigh-

MS Access is pure, unadulterated evil. Step towards the light, use the principles taught to you and learn MySQL. Heheh

Jezus: You're welcome, i learnt a lot there! I have created the blog: http://sc4list.blogspot.com ... it's pretty empty for now but I'll fill it up when I have some time with a full-on wishlist etc.

[DocRorlach 25]

The only 'downfall' to this is that for obvious reasons I wont be hotlinking to the files directly, so for users it's a tool that they have to USE - i.e. mark the file as downloaded manually. In an ideal world, I would be able to provide a direct link to the file. But the world isn't ideal, is it ... -sigh-quote>

Not necessarily a downfall: once the system exists and works, the Download &Installation  Catalogue (DIC) could be modified to allow users to run an automatic check of their existing plug-ins against the online database and thus mark their own installation history.

If you take the "joys" of inner joins a step further you could add a date (last updated) to the dependencies, as well as a boolean field for active/superceded because especially megapacks do receive periodic updates. The BSC Downloader for the LEX already uses such a system. Again, the DIC use tthis info to warn usesr that they have to download the latest version.

All I have to do is learn about MySQL (aargh - I feel a migraine coming my way..)

[concrete_mielie 0]

Hahahah. No worries, i'll help you with whatever MySQL you need But yes, you're spot on. And having DIC checking for updates etc etc and marking the database ... great idea! What is DIC programmed with, and i'm assuming you can connect to an ODBC source?

[DocRorlach 25]

Programmed with .NET (VS 2008) and yes ODBC should not be a problem

[warrior 169]

So the database will not provide you with links to download the Dependencies?

Who thought of megapacks that just complicates the thing even more!!!

Shouldn't the website just have a database and a frontend for uploaders to add their uploads to. Then the front end that users who are downloading is a program such as the DIC (this would allow more flexibility wouldn't it?)

Maybe the website could have a basic frontend, but the database should have a program you can download to do more complicated tasks such as downloading it al for you and checking for duplicates and other stuff?

And I thought Access would probably be considered Evil, as is most Microsoft products. But it seems the easiest to use, I tried myPHPadmin with mySQL and it's not exactly simple.

EDIT: What would be mega useful would be a database of the plugins IDs(that window you sometimes get you open a city and it says about missing plugins. But I doubt it will be possible for such a mamoth task.

Jonathan

[DocRorlach 25]

Originally posted by: warrior What would be mega useful would be a database of the plugins IDs(that window you sometimes get you open a city and it says about missing plugins. But I doubt it will be possible for such a mamoth task.

Jonathanquote>

 

Absolutely agree with that last one, although I have no idea how that could be achieved. I vaguely recall a threat on SC4D where RebeccaTSLynn did some explaining of that due to one of my questions.

As for the database. MS Access suffers from the fact that - online - you need either ASP or ASP.NET to control it (am not sure, though) and as product it is quite "heavy". 

But the dependency database ought to provide links to where a particular pack or dependency can be found. If I understand CM correctly, he was planning on doing just that. However, that will totally depend on the cooperation of all sites - the biggest dependency of them all..!

[sixers33 3]

This doesn't seem all that complex that you guys think it is

[JakeMD 107]

If this is to be a low-cost solution, php and mysql would be best. Most hosts that offer .net as a web based scripting language are pretty spendy.

[concrete_mielie 0]

Hey all.

ASP (.NET) is not an option. Not only do i not really know it, but the bit that I have worked with it, it's been a nightmare filled with bulky excessiveness.

Sixer: It's not really that complex at all, you're right, but it's better to hash it all out to start with ... even the obvious stuff ... so that we can cover all the bases. I dont know much about the technical side of BATS and dependencies and so on, so anyone mentioning ideas about dependency ID's makes me add to my thought process that (for example) there's a need to search by ID. Easy peasy to integrate, but i never thought of it before

Speaking of which ... Warrior: of course there will be links, provided the 'original host' (i.e. STEX, SC4D etc) are willing to participate. If not, it's pretty useless...although at least having the name there 'would be a start'. Good idea on the ID. If i'm not mistaken, when someone makes a file that is a dependency, it gets given an ID ... and that ID is known by the maker, what's stopping them from entering it into the system (other than sheer laziness ... but then they can't upload, hahahah)

Doc: You 'can' connect to .accdb files but it's a nightmare via PHP. ODBC is an option as well. But truthfully, it's slow as hell. Access is a desktop-optimized database system, i've always found that it works well for local use in a small database, but with sharing and permissions issues, user access and large tables ... we should go with a lighter solution. Plus, Linux hosting with PHP/MySQL is simply cheaper. As for ODBC and connecting DIC to the database ... that might not even be necessary, you can get your program to execute a web-based script and the script can return XML (or any other data format, prefer XML though) which DIC could interpret.