This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

No-Intro NES/Famicom databasing community.

No-Intro NES/Famicom databasing community.
by on (#99858)
I'd just like to bring to light something that has been bothering a lot of us over the years about No-Intro's commendable effort to database the known world of NES and Famicom (legit or unlicensed) cartridges.

We know that they are databased without any form of header often. And they believe that moving forward the NES/Famicom console format should be entirely headerless. It appears that they have put all their eggs into certain communities or emulator authors' unwilling baskets to support this overall effort; if you do something en masse it's up to the world as a whole to adapt to the change.

Since flash carts are becoming the enthusiasts option of choice, we know that a headerless format for the NES/Famicom would be entirely impossible unless processors/fpga-ish devices become faster/smaller/cooler and memory becomes cheaper. With current technology a NES/Famicom flash cart needs headers. Not only to this fact, but a header format takes the burden off emulator coders to database games themselves, keep up-to-date with good/bad dumps, allow dumps to be released without updating their code, allows not having to rely on a databasing community for updates to their database, and frankly allows a life outside of NES development. The NES/Famicom console ROM dump format and emulation could be much more intensive without the simple yet necessary innovation of the iNES header.

No-Intro could in fact be the agent to bring alive the NES 2.0 format or even beta-test the DotFami format. Instead it seems that they want to steer NES/Famicom into the dark.

As quoted by a No-Intro administrator in a thread by someone else attempting to make the effort to fix the NES/Famicom headers:

BigFred wrote:
I basically agree but the main goal was a completely headerless format. Just datting PRG/CHR and providing an external XML with header info accessible by emulators. I think it can be done with Nestopia or MESS.

I mean no disrespect to anyone, but my goal is to hopefully steer them away from this asynchronous step that is just doing more harm than good.
Re: No-Intro NES/Famicom databasing community.
by on (#99859)
What's wrong with this sort of son-of-ZapFC format? As I understand it, a ROM might look like this:
  • thwaite.prg: 16384 bytes
  • thwaite.chr: 8192 bytes
  • thwaite.board is an XML representation of the data in the iNES or NES 2.0 header
Is it that an 8-bit CPU has more trouble processing the XML representation than, say, NES 2.0? Or what else am I missing?

Anyway, you might want to mention that the maintainers of the NES 2.0 spec itself (kevtris and myself) are more easily contactable than Quietust. You might mention that the iNES format is in essence an archive format, just as zip and tar are archive formats. The header specifies whether certain predefined objects (PRG RAM trainer, PRG ROM, CHR ROM) are present and how big they are, and it's followed by the objects in that order.
Re: No-Intro NES/Famicom databasing community.
by on (#99867)
Well, opening this discussion has brought that to mind. I'm glad that you and kevtris are available for comment and aid. Quietust was a valuable source of information; and still makes a good emulator on the DL.

We have to consider that a flash cart will always be preferred to an emulator. Will a flash cart on older hardware be able to decompress 2 to 4 files and then look at an XML file, as in son-of-a-ZapFC to tell how the mapping logic and expansion chips function with given current technology and then be produced for a commercial market? Even so, that XML file is more or less a header but not existing at one end of the archived file. My conclusion is that something other than the PRG, CHR, and possibly sample ICs needs to exist to tell the emulator or hardware what to do with the file. And if it's a small header or XML file that exists in the archive, why not just keep headers and use the most useful databasing community as a vehicle to increase the capability and compatibility of that set of dumped images to aid the emulation and new commercial, NES/Famicom hardware.
Re: No-Intro NES/Famicom databasing community.
by on (#99871)
Ultimately, you'll end up needing a PRG, CHR, and .verilog source for the mapper, and binary file to send to the FPGA inside the flash cartridge (built on a PC).
Re: No-Intro NES/Famicom databasing community.
by on (#99877)
I am not sure you have anything to worry about. People spend a lot of time talking about formats, but its the availability of content that matters. Probably most people are going to stick with iNES 1.0 a long time yet.

What happened with ZapFC anyway?
Re: No-Intro NES/Famicom databasing community.
by on (#99902)
Dwedit wrote:
Ultimately, you'll end up needing a PRG, CHR, and .verilog source for the mapper, and binary file to send to the FPGA inside the flash cartridge (built on a PC).

Exactly my point, but then that format wouldn't be very useful for PC/tablet/phone/console emulators. There needs to be one standardized format that just works. And as Kev and Quietust put it, NES 2.0 happens to also be "future-proof."

rainwarrior wrote:
I am not sure you have anything to worry about. People spend a lot of time talking about formats, but its the availability of content that matters. Probably most people are going to stick with iNES 1.0 a long time yet.


In the vein of "future-proof", I very much do have something to worry about. I'm also a NES/Famicom dumper with my CopyNES. And since I'm not an emulator author and I'm not entirely certain which mappers are available (if any) I am very reluctant to dump my stack of Brazil, Taiwan, Russia, etc. originals that would need another mapper number. Being that iNES 1.0 has grown too big for its britches, much of the new and interesting software cannot be dumped, coordinated with emulator authors, for mapper numbers.

I have to give props to NES 2.0 and DotFami. Both of them look pretty damned solid. Just considering the fact that ZapFC uses XML makes me wonder if a hardware flash cart would take kindly to it.
Re: No-Intro NES/Famicom databasing community.
by on (#99958)
tepples wrote:
As I understand it, a ROM might look like this:
  • thwaite.prg: 16384 bytes
  • thwaite.chr: 8192 bytes
  • thwaite.board is an XML representation of the data in the iNES or NES 2.0 header
.....You might mention that the iNES format is in essence an archive format, just as zip and tar are archive formats....
In that case, could it be made 7-Zip to extract these three files (plus the trainer file if it has one) from a .NES file? (7-Zip can already extract many archive formats, including .msi, .iso, .rpm, and even Macintosh disk images, and more.)

Dwedit wrote:
Ultimately, you'll end up needing a PRG, CHR, and .verilog source for the mapper, and binary file to send to the FPGA inside the flash cartridge (built on a PC).
It may be possible to compile DotFami .cart binaries into Verilog, too. The ines.map file can then be used to convert iNES files into DotFami (or use the DotFami directly if it is already in that format), and then into Verilog and compiled to FPGA binaries.

B00daW wrote:
In the vein of "future-proof", I very much do have something to worry about. I'm also a NES/Famicom dumper with my CopyNES. And since I'm not an emulator author and I'm not entirely certain which mappers are available (if any) I am very reluctant to dump my stack of Brazil, Taiwan, Russia, etc. originals that would need another mapper number. Being that iNES 1.0 has grown too big for its britches, much of the new and interesting software cannot be dumped, coordinated with emulator authors, for mapper numbers.
If you have things that require another mapper number, there are two ways to proceed, either NES 2.0 first or DotFami first.

If you do NES 2.0 first:
  • Figure out the mapper.
  • Dump the PRG ROM and CHR ROM, as well as any others which might be necessary.
  • Write a document and/or DotFami .cart describing the mapper.
  • Post a message to ask for a mapper number.
  • Fix the NES 2.0 header with the specified mapper numbers.
  • If you want to, write a module to use with ines.map.

If you do DotFami first:
  • Figure out the mapper.
  • Dump the PRG ROM and CHR ROM, as well as any others which might be necessary.
  • Write a DotFami .cart describing the mapper (possibly using the Haskell library I plan to write for this purpose).
  • Combine them into a .fami file.
  • Optionally write a document describing the mapper.
  • If you want to, post a message to ask for a mapper number, and then write a module to use with ines.map. (Otherwise, someone will assign a mapper number later on if they want to.)

B00daW wrote:
I have to give props to NES 2.0 and DotFami. Both of them look pretty damned solid.
DotFami is incomplete, but I am working on it. Both are good formats.

B00daW wrote:
Just considering the fact that ZapFC uses XML makes me wonder if a hardware flash cart would take kindly to it.
Perhaps it is best to first use another computer to convert the format to whatever a specific flash cart needs.
Re: No-Intro NES/Famicom databasing community.
by on (#99982)
Well, with a little persuasion I have helped shed some light on the situation to No-Intro.

http://forums.no-intro.org/viewtopic.ph ... =10#p11812

The administrators are now aware of it understand that we need to reconsider the headerless idea and iNES 1.0.

They are in support of re-databasing the known collection of NES/Famicom games with the NES 2.0 format. This entails a lot off legwork but ensures to give emulator authors incentive to support NES 2.0. Since NES 2.0 is already proven to work and is more or less complete, I believe it's the format we should turn toward so we can continue dumping and emulating previously undiscovered NES/Famicom software without running out of mapper numbers. This way it could also make a lot of room for community custom mapper numbers to be assigned and developed under.
Re: No-Intro NES/Famicom databasing community.
by on (#99997)
I like NES 2.0, I was just saying I don't really see iNES 1.0 as obsolete yet. It works pretty well for most games, and emulators have workarounds for most of the ones that don't. We'll need new mapper IDs eventually, but there's still a few left for the time being.

Homebrew comes out slowly, but most (all?) homebrew is for existing mappers. I've seen a lot of custom mapper ideas being thrown around, but I haven't yet seen somebody actually make a game with one (if you know any, list them). We've got a slow stream of increasingly obscure games popping up with new mappers (esp. unlicensed chinese carts), but I don't imagine homebrew will be pushing the spec much.

DotFami is an interesting idea, but ultimately I kinda dislike the custom/generic mapper concept. I mean, I like the idea that it might account for new mapper configurations that we may need in the future, but I think it's just as likely that something new will come up requiring a spec revision and reimplementation. I think it would also be subject to homebrew abuse which may expose and complicate the internal implementation (compare: NSF multi-expansion). I much prefer the simplicity of an order-by-number mapper like you get with iNES.
Re: No-Intro NES/Famicom databasing community.
by on (#100000)
rainwarrior wrote:
Homebrew comes out slowly, but most (all?) homebrew is for existing mappers.

Just because a board uses an existing mapper doesn't mean it can be represented in original iNES. Size of PRG RAM and CHR RAM is the first big stumbling block. Original iNES supports SNROM and SUROM fine, but there's no widely supported way to specify the extra PRG RAM of SOROM and SXROM, and at least one homebrew release by Neil Baldwin needs the extra RAM of the SXROM board. An MMC3 board can be rewired to support large CHR RAM, but not in original iNES.
Re: No-Intro NES/Famicom databasing community.
by on (#100002)
Which Neil Baldwin release was that?

I'm not trying to poo-pooh efforts to move to a new format. I'm just saying that for most people iNES 1.0 isn't creating a lot of problems, and adoption of a new format is going to take a while yet. There are a some notable popular games that aren't properly handled (e.g. Startropics), but for the average user these problems are taken care of "under the hood" via CRC checks and whatnot. Not an ideal solution by any means, but I do think it slows the need for a format revision.

Do we have a list of games/releases somewhere that iNES is inadequate for? I guess I could go through FCEUX's CRC list, though I know many of those entries are just to correct improper iNES headers, rather than deal with iNES limitations...

By the way, why was MMC6 never given its own mapper number? I'm kinda curious about this. Is there some historical reason it doesn't get reassigned from iNES mapper 4?
Re: No-Intro NES/Famicom databasing community.
by on (#100003)
iNES and iNES 2.0 where the original fails is the way to go. There is no need for silly XMLs or other useless things. Considering the licensed/non-pirate collection of games is described quite well with iNES 1.0, iNES 2.0 helps with those that fell through the cracks and gets the job done. The whole idea of another format or a "headerless" format are never going to work. If you literally just had two files, one PRG and one CHR (or just PRG) you would have to hash it against an included database which is already a problem with iNES 1.0 for detecting games like Startropics or you have this XML file which is just a disguised header file.
Re: No-Intro NES/Famicom databasing community.
by on (#100007)
rainwarrior wrote:
Which Neil Baldwin release was that?

PR8
Re: No-Intro NES/Famicom databasing community.
by on (#100011)
rainwarrior wrote:
I'm not trying to poo-pooh efforts to move to a new format. I'm just saying that for most people iNES 1.0 isn't creating a lot of problems, and adoption of a new format is going to take a while yet.
Unofficial MagicKit already supports NES 2.0 (but not by default; if it did, it would specify no RAM even if an older .asm file requires RAM on the cartridge), and I will soon add support for UNIF and DotFami as well.

Anyone making other assemblers (such as CA65 and so on) should consider support new formats too.

rainwarrior wrote:
There are a some notable popular games that aren't properly handled (e.g. Startropics), but for the average user these problems are taken care of "under the hood" via CRC checks and whatnot. Not an ideal solution by any means, but I do think it slows the need for a format revision.
I know, that is why we can have NES 2.0 submapper numbers, and is also the reason for the ines.map file (part of the DotFami specification).

MottZilla wrote:
There is no need for silly XMLs or other useless things. Considering the licensed/non-pirate collection of games is described quite well with iNES 1.0, iNES 2.0 helps with those that fell through the cracks and gets the job done. The whole idea of another format or a "headerless" format are never going to work. If you literally just had two files, one PRG and one CHR (or just PRG) you would have to hash it against an included database which is already a problem with iNES 1.0 for detecting games like Startropics or you have this XML file which is just a disguised header file.
OK. I think rips should be the PRG and CHR files and another file describing the other stuff, and then combine them together into the formats needed such as NES 2.0, UNIF, DotFami, and so on; you could uncombine them if necessary (although it might not be necessary). It just seems more logical to me to rip the ROMs separately and then combine them (including the header) afterward (which can be before they are published; when you need to send the files to anyone, only the combined files are necessary).
Re: No-Intro NES/Famicom databasing community.
by on (#100014)
zzo38 wrote:
Anyone making other assemblers (such as CA65 and so on) should consider support new formats too.


CC65 doesn't support any file formats directly, it just outputs binary files as you specify. You build an iNES ROM by creating a binary header in code, same for any other binary format.
Re: No-Intro NES/Famicom databasing community.
by on (#100017)
rainwarrior wrote:
zzo38 wrote:
Anyone making other assemblers (such as CA65 and so on) should consider support new formats too.


CC65 doesn't support any file formats directly, it just outputs binary files as you specify. You build an iNES ROM by creating a binary header in code, same for any other binary format.
In that case, make the macros for NES 2.0 format. But, I did not mean only CC65 I mean other assemblers too, doing in whatever way work for those others assemblers.
Re: No-Intro NES/Famicom databasing community.
by on (#100018)
zzo38 wrote:
MottZilla wrote:
There is no need for silly XMLs or other useless things. Considering the licensed/non-pirate collection of games is described quite well with iNES 1.0, iNES 2.0 helps with those that fell through the cracks and gets the job done. The whole idea of another format or a "headerless" format are never going to work. If you literally just had two files, one PRG and one CHR (or just PRG) you would have to hash it against an included database which is already a problem with iNES 1.0 for detecting games like Startropics or you have this XML file which is just a disguised header file.
OK. I think rips should be the PRG and CHR files and another file describing the other stuff, and then combine them together into the formats needed such as NES 2.0, UNIF, DotFami, and so on; you could uncombine them if necessary (although it might not be necessary). It just seems more logical to me to rip the ROMs separately and then combine them (including the header) afterward (which can be before they are published; when you need to send the files to anyone, only the combined files are necessary).


Rips, what rips? Are you redumping games with a device like CopyNES? The iNES format already is 2 files (PRG and CHR) with a third file (header) describing them. You can already separate them. I don't see any benefit to what you are proposing. Part of why iNES is popular is because support for it can be added to a new emulator or device very easily. You'd need to have quite some benefits to get people to use a different format.
Re: No-Intro NES/Famicom databasing community.
by on (#100019)
MottZilla wrote:
zzo38 wrote:
MottZilla wrote:
There is no need for silly XMLs or other useless things. Considering the licensed/non-pirate collection of games is described quite well with iNES 1.0, iNES 2.0 helps with those that fell through the cracks and gets the job done. The whole idea of another format or a "headerless" format are never going to work. If you literally just had two files, one PRG and one CHR (or just PRG) you would have to hash it against an included database which is already a problem with iNES 1.0 for detecting games like Startropics or you have this XML file which is just a disguised header file.
OK. I think rips should be the PRG and CHR files and another file describing the other stuff, and then combine them together into the formats needed such as NES 2.0, UNIF, DotFami, and so on; you could uncombine them if necessary (although it might not be necessary). It just seems more logical to me to rip the ROMs separately and then combine them (including the header) afterward (which can be before they are published; when you need to send the files to anyone, only the combined files are necessary).


Rips, what rips? Are you redumping games with a device like CopyNES? The iNES format already is 2 files (PRG and CHR) with a third file (header) describing them. You can already separate them. I don't see any benefit to what you are proposing. Part of why iNES is popular is because support for it can be added to a new emulator or device very easily. You'd need to have quite some benefits to get people to use a different format.
I am not proposing anything.
Re: No-Intro NES/Famicom databasing community.
by on (#100020)
I am bugged that all this garbage is always proposed to use MD5 and CRC32 and all that crap. Why in the world does there need to be so much processing needed just to locate what mapper and info to use for it? What a waste. Especially on the NES. I hate the idea of any independent ROM library, it's pretty much stupidity to me.
Re: No-Intro NES/Famicom databasing community.
by on (#100021)
3gengames wrote:
I am bugged that all this garbage is always proposed to use MD5 and CRC32 and all that crap. Why in the world does there need to be so much processing needed just to locate what mapper and info to use for it? What a waste. Especially on the NES. I hate the idea of any independent ROM library, it's pretty much stupidity to me.
I agree it is stupid. We shouldn't use MD5 and CRC32 and all that crap. However, with the old iNES format we would need to. Other formats do not require that, so a program could be made for conversion.
Re: No-Intro NES/Famicom databasing community.
by on (#100022)
The old iNES is the key word. The fixed spec. is already out and handles everything. Blame emulator makers. And the lack of games the iNES 1.0 didn't work for? :wink:
Re: No-Intro NES/Famicom databasing community.
by on (#100024)
It's not all emulator creators' fault though. The same old ROM files still circulate. There are still bad ROMs with !DISKDUDE in the header going around. It's not the emulator programmer's fault if you try to load a game like Startropics and it doesn't work right because the iNES 1.0 header says it is a MMC-3 game. Some will attempt to I.D. the game with a hash or checksum on some or all of the data. Others maybe only rely on iNES 2.0 to specific MMC-6 I suppose?
Re: No-Intro NES/Famicom databasing community.
by on (#100028)
The biggest problem I've seen regarding mappers is the Mapper 206 vs Mapper 4 fiasco. Then you combine it with the GoodNES people deciding to remove any mirroring information from mappers where mirroring is supposed to be hardware controlled, and you get crap like Mapper 4 RBI Baseball with Horizontal mirroring.
Re: No-Intro NES/Famicom databasing community.
by on (#100050)
Mirroring should have been part of the mapper (SRAM too). We'd maybe have 10 more mappers, but a cleaner format.