This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

How can we keep our NES development resources around?

How can we keep our NES development resources around?
by on (#242817)
Recently in another topic some discussion started about preserving some of the learning resources people used to get started with NES development. This conversation stemmed from the links to some of the nerdy nights sound tutorials going offline.

This led me to question: what if the nerdy nights tutorials went offline? What if nintendoage had a catastrophic database failure, or someone missed paying a bill? Or what if the downloads for the tutorials were lost?

I ended up writing a quick-and-dirty mirror site for the nerdy nights tutorials (including sound) to help mitigate this risk. It's plain html/js, and I offer a zip download of the entire site (with tutorial zips, etc) for offline use and/or to bring the site back up if my host goes down.

http://nerdy-nights.nes.science

Nerdy Nights is probably the most commonly-cited resource, but I think most of us learned from a lot of other places too. (Such as nesdoug's tutorials, topics here, various texts on the wiki, and some other sites I'm a little afraid may have already gone offline)

So, I'm wondering: should we do something to try to preserve all of this? Assuming the answer is yes... what resources do we want to preserve, and how do we want to preserve them? The backup website approach could work, but the answer might be as simple as making sure everything is on archive.org too. Let's discuss!
Re: How can we keep our NES development resources around?
by on (#242818)
The thing that terrifies me more is if NesDev as a whole were to go down, because there's over a decade's worth of information on this forum alone, and that's before you consider how indispensable the wiki is. :P

Really, redundancy is the way to go, whether it's in the form of backups or mirrors. If Nerdy Nights is such a handy tool for newcomers, why not host it here too?

There's also code repositories, such as Github or BitBucket, which can serve as a nice anchored place to host a project, Nerdy Nights could potentially live there too, which gains the benefit of discussion and multiple contributors.
Re: How can we keep our NES development resources around?
by on (#242820)
I mirror the wiki but not (yet) the forums.
Re: How can we keep our NES development resources around?
by on (#242822)
For 3rd party links I've cited on the wiki, I've started adding (commented out) links to backups of these 3rd party links using archive.is, and replacing the original link with the backup when I become aware that the original went away.

The Internet Archive has fairly good coverage of the wiki ("200000 URLs have been captured") and forums ("100000 URLs have been captured").
Re: How can we keep our NES development resources around?
by on (#242823)
The archive's weakness being their retroactive robots.txt policy. A squatter buys the domain, says no crawling, and all the previous saved content is deleted by them.
Re: How can we keep our NES development resources around?
by on (#242833)
Internet Archive changed how it handles /robots.txt in 2017. See "Robots.txt meant for search engines don’t work well for web archives" by Mark Graham and my answer to "How can an attacker use robots.txt?" on Information Security Stack Exchange.
Re: How can we keep our NES development resources around?
by on (#242850)
I should probably note that the site I made is on github, but I have it private while I do a little bit of cleanup. (The scripts I used to grab the content are also there and they're a bit ugly/unclear right now)

I fully plan to release it though!
Re: How can we keep our NES development resources around?
by on (#242851)
I do make backups of the forum database, but maybe I should do it on a more regular schedule. If the worst were to ever happen, we would lose months, rather than decades of info. I probably don't have any backup at all of the old forum though. Originally this website, and my NES projects, were backed up to an LS-120 SuperDisk.

I actually did lose my earliest NES stuff. What's funny is that I had posted one of the ROMs I made on Usenet. Sometime later I went back there and asked "anybody have testr.nes?". And yep, someone replied with it attached.
Re: How can we keep our NES development resources around?
by on (#242876)
Drag wrote:
The thing that terrifies me more is if NesDev as a whole were to go down, because there's over a decade's worth of information on this forum alone, and that's before you consider how indispensable the wiki is. :P

I was going to say the same, except it's closer to two decades than one, if you include the WWWthreads boards before 2004. Also the NESdev main page is basically an archive, while everything covered by it is covered in some other form by the wiki, it'll never be the same.

Considering the major issues with the server and the inability to contact WhoMan, this is really worrysome.
Re: How can we keep our NES development resources around?
by on (#242884)
I've been trying to collect my own old one-off test ROMs in a GitHub repository called little things.
Re: How can we keep our NES development resources around?
by on (#242968)
lidnariq wrote:
For 3rd party links I've cited on the wiki, I've started adding (commented out) links to backups of these 3rd party links using archive.is


Slightly relevant, if a user happens to be using Cloudflare's 1.1.1.1 DNS resolver, these archived links will be unreachable (something to do with Cloudflare not supporting EDNS and archive.is treating such requests as invalid). I actually switched away from using 1.1.1.1 once I learned that.
Re: How can we keep our NES development resources around?
by on (#243170)
I know some MySQL. A system administrator of forums.nesdev.com could do a full extraction of the topics, posts, forums, categories, public user profile data, avatars, uploads, etc., and export all of that into some sort of an XML/JSON file for each "page" (category view, (sub)forum view, topic view, profile view, etc.) and there would be a HTML+CSS+JS frontend for navigation.

I don't know about programming the frontend part XD I suck at CSS. But I could make an Windows EXE as a reader of those XML/JSON files. This could make it super easy for stuff like searching. Searching by multiple criteria such as must-have-exact-phrase or avoid-this-exact-phrase and such; or regular expressions or whatever. I think that that would be seriously cool as there wouldn't be that need of wasting a server's CPU resources just to find what you need.

Due to this being very retro, I say we stick to some Windows XP-times HTML/JS/CSS standards + browser security considered. Better yet, a Windows XP-compatible EXE application (MIT LICENSED!) for browsing that stuff (I'll have to code it so that it stays MIT licensed :3). Windows 10 isn't very sustainable in terms of privacy and such. Windows XP is the latest Microsoft OS that I deem good regarding retro. We could make it even more retro by making a CD/DVD with an Autorun running that EXE Viewer.

So if you have some old computer that's off-grid and you wanna snuggle up inside the comfort zone of offline life, you will be able to do so. This is how my school had these computer magazines with new software and hardware and games and fashion and there was always that CD/DVD which had freeware, demo, trial, shareware and other software as well as some other good resources.

And in these days of great data compression, compressing the plain text could be relatively easy! The zlib library could compress the data for the CD/DVD and the EXE could decompress it. Alternatively, if one doesn't have a Windows machine to run the EXE but Linux or BSD, the solution is to use wxwidgets to make the Viewer because wxwidgets can cross-compile to multiple platforms.

What do you think?

BUT! Before that, I wish to revisit my old posts to delete some things that I wish I hadn't posted XD