This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

utility: Fixed Bank Creator, for use with 32kB PRG banks

utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228862)
Since I was writing some GTROM documentation, and wanting to include templates for a few different assemblers, I decided to include a simple utility for duplicating data among 32kB banks, rather than complicate the source code for the templates. I'll be using it myself with CA65. I don't know if anyone else wants it, but here it anyways. C source is included.

edit - bugfix v1.1 added 11-16-2018

Code:
Fixed Bank Creator
    by Memblers
   
V1.0 - 11-14-2018 - initial release
v1.1 - 11-16-2018 - bugfix: blank check fail warning would corrupt source
             bank selector

-- Introduction --

Fixed Bank Creator is a utility designed for NES developers who are using a
mapper that bankswitches PRG-ROM in 32kB pages.  It will allow you to take
code/data from one bank, and duplicate it to the same address in every bank. 
You could do this with your assembler/linker, but the methods for doing so
vary by assembler and can become convoluted, with issues such as having
conflicting label names within the same scope.  Because this program operates
on a binary file, it doesn't matter which assembler you are using.

Whether you've already developed with 32kB banks and are tired of dicking
around with an overly-complicated build process, or you are a UNROM veteran
looking to shrink your fixed bank and free up more memory on boards such as
GTROM, it is hoped that this program could be useful to you.

-- Usage --

Fixed Bank Creator is a command-line program, intended to be run as part of
your normal ROM-building process.  You specify the input ROM file in .NES
format, address range for the desired fixed bank, and (optionally) an output
filename.  You do not need to specify a source bank, as the program will
detect which bank already has data within that range.  If data exists within
that range in multiple banks, a warning message will be output.

    fixbank.exe [.NES file] [begin address] [end address] [optional filename]
   
Usage examples:

    fixbank hotseat2.nes $C000 $FFFF
        Creates a UNROM-style 16kB fixed bank, saved as "output.nes"
   
    fixbank hatetank.nes FFFA FFFF romtest.nes
        Duplicates the vectors only, saved as "romtest.nes"
   
    fixbank romtest.nes 0xE000 0xE7FF
        Duplicates 2kB data, DPCM samples for example, saved as "output.nes"
      
      
When creating your ROM to use the program, what you are supposed to do is
choose the region of memory (anywhere in $8000-$FFFF) that you would like to
make into the fixed bank.  Organizing your program into 32kB banks, you then
leave that region empty (padded with $00 or $FF bytes) in every bank, except
for one of them.  The contents of that one bank will then be duplicated.
       
-- Notes and Tips --

Warning, Error, and Info messages are output to the screen.  The warning
messages should not be ignored, as they are likely indicators of a
non-working result.

'$F000', 'F000', and '0xF000' are all valid input syntax for this program.

After the source bank has been detected, the rest of the banks are scanned to
check if they are blank.  If they are not, a warning is output and data will
be overwritten.

Blank check requires memory to be padded with either $00 or $FF.  If you use
$FF, FlashROM and EPROM chips may be slightly faster to program, as $FF is
the typical blank state of non-volatile memory.

Searching for source data bank begins in the lowest bank.  It's recommended
that you put your fixed bank source data in the lowest (first) bank.

It's best if you set up your assembler to reserve memory in your fixed bank
area, as the error messages from your assembler will likely be more helpful
than the warning messages from this program.

If you need to create multiple "mirrored" parts of memory, just run the
program again on the output file.

This program is free and open source under the MIT license.  Additionally,
any discordian pope may automatically grant themselves a license under the
WTFPL, as desired.

-- Known Issues --

The source bank detection will fail if you were to create a bank that
consists entirely of $00 and/or $FF bytes.  If, for some reason, you wanted
to have nothing but patterns of $00 and $FF bytes in your fixed bank, this is
bad news.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228863)
I never had to manage 32kB banks but there is one thing I would like clarification since maybe some of my assumption were wrong for ca65.

If I write code in assembler with ca65, create a scope then include a file in that scope, wouldn't that remove the label clash and allow the code to be exactly at the same place? I never tried but this is what I would do first if I had this issue.

I may try someday to see how the assembler react, out of curiosity and wanting to know more about it.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228866)
Banshaku wrote:
If I write code in assembler with ca65, create a scope then include a file in that scope, wouldn't that remove the label clash and allow the code to be exactly at the same place? I never tried but this is what I would do first if I had this issue.

Wrapping things in scopes does indeed solve the label clash problems, but there are a few catches. I tried creating a macro to automate this once, but it didn't work. Macros are kinda mystical in ca65, I guess. But it works if you do it in other ways. What I ended up doing was creating my own implementation of PROC, which checks whether the entry point label is already defined or not, to decide between creating a named scope or an unnamed one. With this I didn't have to explicitly wrap stuff in scopes, I simply included the subroutines multiple times and the custom PROC macro took care of creating global labels only the first time.

Anyway, Memblers' idea is pretty cool, but unfortunately it doesn't solve the problems I have with virtual fixed banks. First, having to know the address means I'd still have to change settings manually whenever I added/removed things to/from the fixed bank. I guess I could set a larger size and only shrink it at a later time, when the fixed stuff was probably already set in stone and the lower portions of the banks were getting full. But the other problem for me is that not all fixed banks are equal: while some things are present in all banks, other things are repeated only in specific banks. Banks with CHR data have code to copy/decode that to VRAM, banks with level maps have collision detection routines, stuff like that. It's kinda like another fixed bank below the general fixed bank. Multiple passes of Memblers' tool could maybe handle that, but again I'd have to know all the addresses beforehand and constantly change the settings during development.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228867)
tokumaru wrote:
But the other problem for me is that not all fixed banks are equal: while some things are present in all banks, other things are repeated only in specific banks. Banks with CHR data have code to copy/decode that to VRAM, banks with level maps have collision detection routines, stuff like that. It's kinda like another fixed bank below the general fixed bank. Multiple passes of Memblers' tool could maybe handle that, but again I'd have to know all the addresses beforehand and constantly change the settings during development.

I don't think these other things fit the category of "fixed bank". This is just duplicate code, not something that needs to be at a fixed location. The problem relaxes a bit if you don't include the additional constraint of putting it at the same location.

Stuff at a fixed location can be dealt with in a various ways (e.g. tepples' suggested this "workaround" recently, other possibilities have been discussed in the past, and now this tool is being offered, etc.). Relocatable code that's duplicated in some banks is an easier problem.

For instance, probably the simplest case: in ca65, if the duplicated code is in different modules (i.e. different assembles), as long as you don't export the symbols there is no duplicate name conflict when you link. You can reuse the code via .include without even having to scope it. The name conflict only comes up when you have to use the duplicated code across different assemblies. Just .include the duplicate code where needed, sort of like a C++ inline implementation.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228869)
Just to mention, this is an excellent option for those using cc65/ca65
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228884)
rainwarrior wrote:
I don't think these other things fit the category of "fixed bank". This is just duplicate code, not something that needs to be at a fixed location.

I guess you could have the same code replicated in different addresses if a trampoline in each bank called the correct address for that bank, but calling conventions that don't use a dedicated trampoline (e.g. target address in RAM, or in ROM after the JSR to a generic trampoline) would require the repeated routines to be in the same place, otherwise you'd need a more complex mechanism to select an address based on the bank. For that and other consistency reasons, such as page alignment so that there are no variations in timing across different banks, it makes much more sense to have all instances of duplicated code run from the same address.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228886)
Thanks for making this!

Something that would make this cooler (but significantly more complicated) would be if, instead of specifying an address range, you could specify a ca65 segment. Then the program would read through the generated map file and automatically figure out what it needed to do.

Something like:

Code:
fixbank.exe --by-segment FIXEDBANK [mapfile] [.NES file]


Of course, I guess I could just pretty easily write a wrapper scrap (in something like python that makes string parsing a little easier than C) that looks up the bank and then calls your utility, so I can't complain.

I could see this also being useful for the eternal question of how to handle multiple banks with C, where it needs to call C runtime functions. A post-processor like this would really simplify the question of how to trick the linker into putting the C runtime everywhere it needs to be.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228978)
Had a little copy+paste error in my code, added v1.1 to the first post. bug causes the "data will be overwritten!" warning to ensure you'll have a bad result.

Yeah when I was writing it I had started to consider cases where you'd have code in some banks and not others (like music replay engine in the music data banks), but specifying all that pretty much requires making some kind of per-project config file, and figured it's best to keep it simple for now, and simply copy into every bank.

I like the idea of having it read a CA65 map file to detect things. In another thread, I believe tokumaru showed a way to make a segment auto-aligned/justified to the end of memory. Parsing the map file would be most useful in that situation.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228980)
gauauu wrote:
Of course, I guess I could just pretty easily write a wrapper scrap (in something like python that makes string parsing a little easier than C) that looks up the bank and then calls your utility, so I can't complain.

I could see this also being useful for the eternal question of how to handle multiple banks with C, where it needs to call C runtime functions. A post-processor like this would really simplify the question of how to trick the linker into putting the C runtime everywhere it needs to be.

I've said this before in other threads and I'll say it again: if you use Python, you've now ensured nobody except Linux and OS X users are going to use the tool. You can essentially throw away 95% of your user base. Suddenly there's a "wrapper script" that's included, dangling in front of the face of the non-*IX user (alongside a C program and a Windows .exe), while the tool that generates the said map file is a binary they're already using. Yeah, this is kinda insulting to the developer.

I'm dealing with this exact situation right now, where because NES tool authors seem to be all over the board with their PL use and what OSes they run, I'm forced to use VMs else bare metal systems (some FreeBSD, some Linux), and have to copy files between systems manually. It's a complete waste of a person's time.

In short: either provide statically-linked Win32 .exe files of the Python program (good luck! If it can be done, then awesome, problem solved!) alongside the code, use another PL that can do this (there are many), or just use C.

And no, string parsing in C for this particular purpose (parsing a ca65 map file) is not difficult. fscanf()/sscanf() and strtoul() are not advanced topics. This present-day fear of C is actually more scary than the language itself. If complex data structures were involved here, oh yes, I would definitely be scared, but they aren't.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228982)
The only time on window that it's less an issue is if you use win10 with the linux subsystem but not everyone do that so I can understand your pain.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228992)
koitsu wrote:
In short: either provide statically-linked Win32 .exe files of the Python program (good luck! If it can be done, then awesome, problem solved!) alongside the code, use another PL that can do this (there are many), or just use C.

Would it be acceptable to distribute the tools in source code form without charge but charge for the Windows and macOS executables as a way of recovering $119.99 for a Windows 10 license and $799 for a Mac mini computer (which is the least expensive way of obtaining a new macOS 10.14 license) with which to build and test the executables? True, one can cross-compile a Windows or macOS application using GNU/Linux, and even partially cross-test a Windows application with Wine, but recent experience with focus bugs in BGB (a Game Boy emulator) in Wine 3 that don't happen in Windows or in Wine 1 has reduced my faith in Wine as representative of the behavior of Windows, and there's no way to cross-test a macOS binary because GNUstep was never designed to be binary compatible.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#228996)
koitsu wrote:
I've said this before in other threads and I'll say it again: if you use Python, you've now ensured nobody except Linux and OS X users are going to use the tool. You can essentially throw away 95% of your user base. Suddenly there's a "wrapper script" that's included, dangling in front of the face of the non-*IX user (alongside a C program and a Windows .exe), while the tool that generates the said map file is a binary they're already using. Yeah, this is kinda insulting to the developer.


If I'm writing a weird wrapper script instead of modifying his source, I'm doing it for my own use-case, so I don't care about supporting windows. Linux is 100% of my target user base (me).

Quote:
In short: either provide statically-linked Win32 .exe files of the Python program (good luck! If it can be done, then awesome, problem solved!) alongside the code, use another PL that can do this (there are many), or just use C.


I agree that if you're really targeting Windows users, providing an .exe makes the most sense, but Windows users CAN install python, just like Linux users CAN install Wine. I've never heard a developer complain about python being hard to install. In fact, most developers I know find it easier to install python than to compile an arbitrary C program into a windows command-line executable.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229003)
gauauu wrote:
I agree that if you're really targeting Windows users, providing an .exe makes the most sense, but Windows users CAN install python, just like Linux users CAN install Wine. I've never heard a developer complain about python being hard to install. In fact, most developers I know find it easier to install python than to compile an arbitrary C program into a windows command-line executable.


Yeah, I find it weird too. I don't develop in python, but I always have it installed on my machine for other people's scripts. Never had a problem. It might be an issue with people new to NES _and_ to programming in general. Because console is not what windows users usually use. But really, if you are digging into NES, might as well install Python and learn the console interface.

I personally write my scripts in C#, just because I know this language well, and use LINQPad as script runner, but those are for my personal use anyway. Although could be compiled to mono-compatible exe files.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229007)
gauauu wrote:
I agree that if you're really targeting Windows users, providing an .exe makes the most sense, but Windows users CAN install python, just like Linux users CAN install Wine. I've never heard a developer complain about python being hard to install. In fact, most developers I know find it easier to install python than to compile an arbitrary C program into a windows command-line executable.

You've never seen what happens if you install Python system-wide on Windows (specifically the involvement of Python-centric environment variables), and then use Python-using binaries of commercial apps at the same time (ex. Dropbox, EVE, etc.), have you? :-) "Just install the PL" is not really feasible; *IX is not Windows, and Windows is not *IX.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229008)
In Windows PCs that I've used, I've had no problem using both Python.org Python and Dropbox, as Dropbox successfully sandboxes its embedded copy of Python so as not to interfere with a system-wide installation of Python.org Python. I haven't played EVE Online however.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229012)
I cannot find evidence of it now (because Dropbox revamped their forums at some point), but: for a long time there was a problem where on Windows, if you installed Python system-wide, the DLLs or environment variables (not sure which) related to that would cause Dropbox (which used some version of Python internally) to break in very weird and non-obvious ways (IIRC, dialog boxes wouldn't appear, and some others would appear but have no contents?); smelling of an ABI incompatibility problem. The only solution was to not install Python on Windows, or to create annoying workarounds (like batch files that would set the appropriate environment variables first, followed by running the script under the interpreter, rather than using per-user or system-wide env vars).

Dropbox in particular seems to have a history of this type of problem (on both *IX (including its own shared libraries that may conflict with system-wide ones) and Windows (a.k.a. DLL hell)), but in the thread some developers pointed out that it was a problem with the general model of Python-on-Windows in general and thus applied to more than just Dropbox. EVE just happens to be something that uses Python (quite heavily) as well.

Maybe this is something that was addressed with newer (3.x) Python somehow, I don't know, but when there's no official "we've fixed this problem" ever disclosed, folks are forced to continue assuming the problem exists indefinitely.

If the point was "I can write a Python program to do this for myself and me alone", then yeah, no problem. The situation changes when releasing software publicly, whether the author(s) like it or not.

Sorry for the OT.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229017)
I've never run into a problem like that with Python on Windows, but sure I can believe that Dropbox did something screwy with it at some point (though I've used both for a long while now). In general I've found it fairly easy to install, the 10 times or so I've done it... and the installer itself has gotten a lot better over the years.

...but here's some problems that I have had with it:

1. The Python 2 vs 3 schism. That manifest as a whole host of problems but this actually seems to be disappearing finally. The Python 2 holdouts seem to be very few by now. (For a long while the Windows version had a much harder time switching between the two, if you had both installed, but that was eventually mostly "solved", too.)

2. Basic Python was generally fine, but installing extra libraries (in particular PIL / Pillow for images) has never been "easy". It eventually became a bit better with the command line PIP utility being ported to Windows, but it remains something that I would never ask a "typical" user to do. I'm happy to tell someone "just install Python" to run a script, but if there's an extra library you have to install, I just wouldn't recommend it at all unless the person was a habitual Python user already.

To elaborate a little on that: at best the user has to figure out where and how to run a command line utility to install a package. With a common library like Pillow, it'll probably go smoothly as long as you can get that far. With less common libraries it's still a crapshoot. A lot of stuff is only really maintained for Linux and the installer will fail cryptically with internal jargon or sometimes fail and also make a mess on Windows. Depends on the package, who maintains it, etc. but there's a lot of Python library stuff out there that just doesn't work, or doesn't work reliably on Windows.

(There's also plenty of libraries accessible to PIP that will fail to install properly on Linux too, though. This has always been a mess, in my experience. PIP is a jungle.)

4. As far as solutions for compiling Python to a Windows EXE, in my experience these have always been terrible, and they still were the last time I looked at it. I've successfully gotten programs to run this way, but I've never produced something I would dare distribute to others with these tools. YMMV.

5. The Python documentation is horribly organized. I always have a hard time looking things up in it. Additionally, the online official docs are more or less unsearchable because it mixes in all sorts of cryptic results from documents for internals experts for almost any common keyword.


I find it an extremely useful tool though, and as long as libraries are out of the picture I think it's a pretty reasonable install for a lot of users.

(Of course for the case at hand... I wouldn't mix Python and an EXE, I'd just update the EXE, or rewrite the whole thing in Python. :P)
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229039)
koitsu wrote:
*IX is not Windows, and Windows is not *IX.

You mean, *I*X (*); which is absurd since UNIX is completely obsolete and nobody uses this anymore so you might just as well say Linux.

(*) Even this is wrong if you are case sensitive, in which case it should be * but nobody would understand what you're taking about.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229046)
Just because you believe everything uses Linux and nobody uses UNIX any more doesn't make it true. I said *IX and I meant *IX.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229054)
Solaris is UNIX certified, as is macOS. Another pattern I've seen is "*n?x".
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229060)
And everyone forgot BSD. Might as well call it *** to include all of them :)
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229061)
BSD is as opposed to "system V", which are both *n*xes :p
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229064)
lidnariq wrote:
BSD is as opposed to "system V", which are both *n*xes :p


It is not Unix certified, so it is not Unix, it is *BSD :P
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229115)
koitsu wrote:
Just because you believe everything uses Linux and nobody uses UNIX any more doesn't make it true. I said *IX and I meant *IX.

Since you are apparently for whathever reason excluding Linux, you could as well call "UNIX" by it's whole name instead of using periphrases.
Re: utility: Fixed Bank Creator, for use with 32kB PRG banks
by on (#229118)
Or I could say *IX which encapsulates UNIX, UNIX-like, and Linux all in one string. Here's some history of the term, where the upvoted answer is correct/accurate. But real/true UNIX systems are still in use today. So once again: I said what I meant, and I meant what I said. Heck, in my initial reply I even cited one, right alongside Linux in the same sentence. This conversation no longer serves any value.