This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

FDS Redumping Project

FDS Redumping Project
by on (#24933)
scanty and I have been discussing this for the past few days, and have decided to post it here.

FDS files are by definition flawed. Mitsumi's quick disk format, as used on the fds, literally stores a stream of data (MFM or RLL encoded) on the disk, in a spiral track. All that current fds files contain is the data from all of the disk blocks, concatenated together in a big stream.

This is wrong for a few reasons:
1. The crc16 values at the end of each block are missing. at least one game checks for this as part of its copy protection scheme.
2. The pre-gaps before each block are missing. again, its possible to check this.
3. Data which is not encoded in the standard 'fds format' of nintendo fds disks, which is sometimes present at the end of the disks, is missing. at least one game REQUIRES this data in order to function!

We're planning to modify an FDS unit by combining it with a copynes, to be able to inject our own code and read the ENTIRE DISK, sequetially, to the parallel port. This should produce dumps which are somewhat larger than the existing dumps, but should be 100% accurate reflections of disk contents.

What we still need is this:
People with expertise on FDS programming. marty from nestopia has some information on this, I'm hoping to get his input
Someone with access to LOTS of FDS disks. X-or from the nesdev forums seems to know someone who does.
Input from people with experience. You can help too! Your Ideas are welcome!

Lord Nightmare
Re: FDS Redumping Project
by on (#25392)
Lord Nightmare wrote:
Someone with access to LOTS of FDS disks. X-or from the nesdev forums seems to know someone who does.
Input from people with experience. You can help too! Your Ideas are welcome!

Lord Nightmare
Due to the hi price of FDS disks, Yuki and I bought, dumped and sold immediately in order to buy more. A lot of sealed disk were "sacrified" to get cleaner dumps, and we lost a lot of money when reselling those. We are perfectly aware about the flaw in the current FDS dumps but there is nothing we could do about it. We did what we could with what we had, this is sad but that's it.

If you look for someone with a huge collection, I'm afraid you will need to search elsewhere.

Today the No-Intro FDS database is the cleanest database on the web, and we will be glad if you can contribute cleaner dumps.

About the missing data, once a few "proper" dumps are created, we will probably be able to rebuild the missing data in our clean dumps. Calculating a checksum and adding it is no problem.
Re: FDS Redumping Project
by on (#25395)
X-or wrote:
About the missing data, once a few "proper" dumps are created, we will probably be able to rebuild the missing data in our clean dumps. Calculating a checksum and adding it is no problem.

That's not how it works, rebuilt images should be immediate grounds for redumping since you CAN'T rely on the structure of the disks. The only way is to read a disk byte for byte until the end of the track. It's also useless to redistribute new dumps until a file format is adopted, I use a file per side with EOF signifying the end of the track, that seems the most logical to me.
Re: FDS Redumping Project
by on (#25432)
kyuusaku wrote:
X-or wrote:
About the missing data, once a few "proper" dumps are created, we will probably be able to rebuild the missing data in our clean dumps. Calculating a checksum and adding it is no problem.

That's not how it works, rebuilt images should be immediate grounds for redumping since you CAN'T rely on the structure of the disks. The only way is to read a disk byte for byte until the end of the track. It's also useless to redistribute new dumps until a file format is adopted, I use a file per side with EOF signifying the end of the track, that seems the most logical to me.
assumption made from thin air
until we get a complete dump there is no way to tell

by on (#25442)
What are you talking about? The logic is sound. Converting your dumps won't do you any good because you'll eventually have to redump them. Checksums may be bad, if you fix them you'll be making a bad dump which is hard to verify since it's "correct"!

If you want to go ahead and create bad dumps before people even start redumping, go ahead, all the info you need is in this document: http://nesdev.com/fds-e.txt

by on (#25455)
I know those infos. Just saying that before we get the chance to analyze some full dumps, there is no way to tell if the missing bytes can't be rebuild. :wink:

Time will tell.

by on (#25459)
I have full dumps that I dumped myself, they are more or less the same as what's out there floating around, they also have a lot of garbage data left on them from when I wrote other games to the disks. That makes them an unreliable reference since I have no idea what was originally on the disks, except for a few games which don't use saves. I don't know how I can say this any clearer, rebuilding is no better than .fds files.

by on (#25481)
kyuusaku wrote:
I don't know how I can say this any clearer, rebuilding is no better than .fds files.

Except as test cases for emulators until the redumps get out there, right?

by on (#25505)
Right if any emulators support true disk images.

by on (#26113)
kyuusaku, if you can you make "full" dumps (1:1 copies) of FDS, please buy and dump a sealed disk of Zelda no Densetsu.

there is no garbage data in sealed disks

I will analyze your 1:1 copy against a FDS copy and see if anything can be rebuilt

=========

edit: according to a friend, Pasofami can burn a FDS file into a real disk
doesn't that mean real disk missing data (crc and gap data) can be rebuilt?

by on (#26114)
No, I will not buy a sealed Zelda.

Yes, Pasofami + hardware can write FDS disks.

Yes, Pasofami recreates the disk structure.

No, it's not true to the original disk!

No, you can't assume there isn't "garbage" in sealed disks.

by on (#26119)
I know the question is *really* unimportant, but has anyone thought of a good extension for the format yet? If so it'll be alot more easier to label this project.

by on (#26120)
kyuusaku wrote:
No, you can't assume there isn't "garbage" in sealed disks.
None of the 18 sealed disks we dumped so far had any garbage data so it's safe to assume. At least I can confirml Zelda no Densetsu has no garbage.

kyuusaku wrote:
No, it's not true to the original disk!
Did you redump a 1:1 copy of a reburned disk to know that? What has changed from the original 1:1 copy then?

kyuusaku wrote:
No, I will not buy a sealed Zelda.
Thanks for helping! :lol:

by on (#26123)
This is the last time I'll post in this thread because it's going nowhere, you're going around in circles and I'm just repeating myself.

A perfectly dumped game can have bad formatting or bad CRCs. If you want a perfect dump, you cannot take a .FDS file and convert it because you do not know if the original disk has perfect formatting or not. Everyone is 100% sure some games do not follow the conventional format. It should not be assumed that all games have perfect CRCs for the files either! As a ROM scene guy you should know that a good dump can have a bad CRC. If you fix that CRC, you have just created a new bad dump.

by on (#26125)
If you say so, I suppose you have at least one 1:1 ROM image with bad checksum.

And as a scene guy I know that 100% of licensed commercial Super Famicom cartridges and 100% of licensed commercial Nintendo64 cartridges use correct checksum(s). For both of those systems, there is not a single instance of a licensed commercial cart with bad checksum. At all.

But that were times where Nintendo had the monopole of media manufacturing. I suppose those disks with bad format were not manufactured by Nintendo since at that time every company could produce their own media.

by on (#26126)
Simply put, why waste sealed because extra data called "Garbage"? If the games work fine? Now you see why I lost interest in posting here.

by on (#26127)
NotTheCommonDose wrote:
Simply put, why waste sealed because extra data called "Garbage"? If the games work fine? Now you see why I lost interest in posting here.


I agree. It's not like the game wil behave differently because of this "garbage" I mean the games were designed to work with garbage around them.

by on (#26140)
NotTheCommonDose wrote:
Simply put, why waste sealed because extra data called "Garbage"? If the games work fine? Now you see why I lost interest in posting here.
what are you talking about?
Garbage data has nothing to do with the topic. We're talking about missing data, GAP and CRC. Get your facts straight.

by on (#26182)
So basically the intent is to produce 1:1 images of sealed FDS disks (including any incorrect CRCs or "garbage"/unused data that may be present), the results of which could be entered into a goodnes style database as [!] "100% correct dumps"?

by on (#26272)
jonwil wrote:
So basically the intent is to produce 1:1 images of sealed FDS disks (including any incorrect CRCs or "garbage"/unused data that may be present), the results of which could be entered into a goodnes style database as [!] "100% correct dumps"?
correct, except I don't see the use of a [!] which means nothing. I pefer to release proper documentation of the dumps in pdf format, this is what I'm doing currently.

I'd be interested to know how tepples got his 1:1 copies.
If there is a hardware that does such copies, can you link me to it?
Tototek copier (currently a prototype) aims at doing 1:1 copies. I hope they succeed someday.

by on (#26281)
X-or wrote:
I'd be interested to know how tepples got his 1:1 copies.

What 1:1 copies? The only perfect copies of FDS games I know about are the ones that used to be inside Nintendo's "writing system" kiosks. This would be the Holy Grail of FDS emulation, but to the best of my knowledge, no writing system ROM ever leaked to the public.

by on (#26283)
oh sorry tepples, not you. I meant kyuusaku


kyuusaku wrote:
I have full dumps that I dumped myself, they are more or less the same as what's out there floating around, they also have a lot of garbage data left on them from when I wrote other games to the disks. That makes them an unreliable reference since I have no idea what was originally on the disks, except for a few games which don't use saves. I don't know how I can say this any clearer, rebuilding is no better than .fds files.


supposing he meant 1:1 copies by "full dumps"

by on (#27191)
Ok, X-or, you're being silly.
JUST because EVERY SINGLE DUMPED non-kiosked FDS disk has CORRECT crc16s and has NO data at the end, does NOT, by ANY MEANS, mean that *EVERY* non-kiosked FDS disk does. Its like trying to prove global warming.
The disk HAS the crc16 data on it, and it CAN be read off of the disk. So the point is IF WE CAN READ OFF THE DATA, WHY DON'T WE?

Also, we should read off the pre-gap timing data too. Yes, it's just lots of zeroes, but its a SPECFIC NUMBER of zeroes.

Not backing up data 'just because everything else has the data correct and so we assume the data is always correct here' is an incredibly stupid idea. Its on the order of "hey, just because all non-prototype snes roms have the checksum correct, we'll just remove the checksum from the rom image because we know its correct and can regenerate it, and we save four whole bytes!"

LN

by on (#27193)
If you were to take two sealed copies of a particular FDS game, dump them, and then compare the results, would the two dumps contain the exact same data at the exact same position on the disk, bit by bit? (No, I'm not asking anyone to actually do this, unless he/she really, really wants to...)