This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

Batch files as a build system for ASM6 projects

Batch files as a build system for ASM6 projects
by on (#236898)
I'm porting an audio driver from ca65 to ASM6, in part so that it can be integrated with NESmaker. In the process, I've had to work around numerous limits of its macro system and what appear to be bugs in its forward reference support. I've asked a bunch of questions about ASM6, and I'm pretty sure I'll be asking more. But this question is about build systems.

On Windows, one can install pieces of the GNU operating system to gain the use of things like GCC, Bash, GNU Make, and Coreutils. Ports of GNU to Windows include Cygwin, MSYS2 (from the MinGW team), and Windows 10 WSL. In the past, I have recommended installing some form of MSYS to build my ca65-based NES projects, and nowadays I recommend obtaining Bash and Coreutils through MSYS2 as distributed by Git for Windows.

But I've been told that compared to ca65 users, ASM6 users are less likely to be willing to install a GNU environment. This goes double if they use NESmaker. Instead, they use a batch file as a makeshift build system. During the porting of this library, I have been emulating this practice by using a POSIX shell script.

One big difference between makefiles and batch files is that Make conditions each rule in a makefile on comparing the date of the source file and the date of the target. This lets Make skip a build step if the target exists and is no older than the sources. Skipping a build step is important for some of my projects, which use a somewhat CPU-intensive process to convert and compress assets other than code before the assets can be included into the code.

Is there a good way in a batch file to compare two files for existence and skip if the target file exists and is no older? Answers to a related question on Stack Overflow conclude that batch isn't fit to do this, and switching from batch to VBScript is recommended if you know the user doesn't have Make.

On the other hand, it could be the case that I am overcomplicating things, that developers on Windows who use ASM6 are willing to endure the reconversion of 100+ assets in order to make the smallest code change. Is this the case? What sort of build system do developers of nontrivial ASM6 projects actually prefer?
Re: Batch files as a build system
by on (#236900)
Sounds like a job for cscript.exe, which is the command-line version of wscript.exe, and thus can be safely used in a cmd/batch file. This has come with stock Windows since Windows 98, which means no need to install any software.

The underlying PL that works "out of the box" is either VBScript (.vbs) or JScript (.js) (don't confuse this with JavaScript -- different beast). I'd suggest VBScript only because you're likely to find a lot more examples and tutorials, plus lets you do things like use exit codes via WSH.ExitCode and WScript.echo to echo output (yes, I said WScript.Echo despite cscript being used). Additionally, a lot of Windows SAs use it, else PowerShell (which is its own hellish nightmare and requires installation, and is not particularly user-friendly IMO), so you're likely to get better help from StackExchange or similar sites.

I should add: in VBScript tutorials, you're going to find lots of HTML junk injected into the mix, because it was commonly used as an Active Scripting Language alongside IIS. Ignore all the HTML trash -- just focus on the language itself. You don't need the HTML junk; cscript.exe my_thing.vbs and a .vbs file of raw code is all you need.

Another approach is to simply not worry about any of this at all and rebuild everything unconditionally. Really. You'd have a hard time convincing me a 6502/NES audio driver would have exceedingly long build times. No Windows user is going to really care if all the bits get rebuilt from scratch every time in this particular case; we're talking what, an additional 1-2 seconds tops? If you think I'm kidding, go look at how infiniteneslives builds his own projects on Windows: he's using some form of Make on Windows, but note his process every single time: make clean + make. A lot of us build stuff this way on Windows simply because we just don't care. For huge projects with tons of files (we're talking 10K+) and massive dependencies during build-time, you should be using Ninja anyway (details if you think I'm kidding).
Re: Batch files as a build system
by on (#236903)
koitsu wrote:
No Windows user is going to really care if all the bits get rebuilt from scratch every time in this particular case; we're talking what, an additional 1-2 seconds tops?

In the build process of 240p Test Suite, it takes 10 seconds to recompress 15K of help text alone, as I mentioned earlier. An RPG script could easily grow past that, not to mention converting all background and sprite tilesets from PNG to CHR+NAM-like format and then from CHR+NAM-like format to compressed CHR+compressed NAM-like format.
Re: Batch files as a build system for ASM6 projects
by on (#236906)
But this is not recompressing 15K of RPG script. How many seconds will rebuilding this take?
Re: Batch files as a build system for ASM6 projects
by on (#236911)
My comment stands. If you've got some 6502/NES audio engine that takes several full seconds (like 5+) to build, you're most likely doing something atrocious and horrible. I sure wouldn't expect compressing 15KB of text to be the same thing as a 6502/NES audio engine.
Re: Batch files as a build system for ASM6 projects
by on (#236914)
<sarcasm>Oh no! That will take almost as long to run as a large Unity project that you "don't have to compile". </sarcasm> I figure I spend 30+ mins a day waiting for Unity. It never takes less than 10 seconds to compile (on a 4 Ghz i7), and about the same to dynamically link every time you press the play button. People think this is the bees knees. :-\

Seriously though, I think you underestimate the patience of people that don't know different. If you don't know that a build system could give you vastly better build times and don't want to install one, then you probably won't have a lot of context for what a good build time is.

I'm not going to claim it's the rule (admittedly it's cynical though), but a lot of Windows devs I know would prefer 10 clicks in a GUI every time to make a build than to script something and use a command line tool where it would take 2 key presses to do the same. This extends even to full time software developers who look at code all day, but are idealogically opposed to typing code into a command line or putting into some sort of script. I do the best I can to support a broad set of people with my open source stuff, but you cannot please everybody. I maintain a 2D rigid physics library called Chipmunk2D that was popular a few years back, and had several different Visual Studio projects since MS seems to break the project format for every new version. Occasionally I'd get somebody complaining that I didn't support VS 2009 (or whatever one version older than I did support was). One guy went so far as to insult me since clearly his version was the best version and I didn't know it. I had to roll my eyes at that one and just close the issue in the tracker. (All of the VS versions I dosupport were projects helpfully submitted by other Windows devs, so I'm not trying to make a negative generalization here)
Re: Batch files as a build system for ASM6 projects
by on (#236915)
FWIW NESmaker is already bogging down buildtime by about 10 seconds compared to a free standing asm6 project. Much of it spent before calling asm6. My laptop isn’t the best workhorse though.
Re: Batch files as a build system for ASM6 projects
by on (#236917)
Building a project consisting of just an audio driver and a simple wrapper that displays a static image with a few sprites takes about one second on one core of a quad-core Pentium N3710. (This isn't the best workhorse either, as Airmont architecture is descended from Atom.) I spend more time waiting for Mono to load Mesen. But this demo project is unrepresentative of an actual game. If a single game contained both this audio driver and the text subsystem, a build system based on batch files would have to rerun the text compression even if the only thing that changed was audio.

I have another project that takes a full minute to make clean && make -j4 because it has to convert about a hundred scrolling backgrounds from PNG to the game's streaming compressed format. If the build system were based on batch files, it would 1. take four times as long due to lack of parallelism and 2. have to reconvert the backgrounds even if the only thing that changed was audio.
Re: Batch files as a build system for ASM6 projects
by on (#236920)
Yet another reason why it makes little sense for NESmaker to import .chr files to convert them to BMP:s only to re-convert them back to chr each build. It had been more time efficient to simply use chr:s directly and build bmp:s only if you wish to export and chr:s only if you wish to import. On the other hand, a lot of time has been spent on making the bundled in pixel editor etc revolve around bmp editing.
Re: Batch files as a build system for ASM6 projects
by on (#236921)
But, assuming the goal is to get this audio driver for ASM6 into NES Maker, NES Maker handles the CHR, and the text, and the whatever.

I understand you can come up with cases that add to build time, but are they cases that affect what you are currently doing? Your streaming background code is a rather specific need, even way more specific than text compression, and perhaps anyone that had need of that specific thing would be advanced programmers willing to deal with installing all that it requires.

This audio driver in NES Maker? I'd eat the two seconds.

Edit: Even this audio driver outside of NES Maker, I'd eat the two seconds. I probably waited longer for Famitone2 to do all my music and sound effects, and didn't even think twice about it.

Edit2: I'd probably even eat the 10 seconds for the text compression. I'd eat 15 for both together.

I wouldn't eat the minute for the PNG->CHR streaming, but I also probably wouldn't use it in the first place. If I did, I'd probably be willing to read about/install what it actually requires to keep it from taking a minute.
Re: Batch files as a build system for ASM6 projects
by on (#236927)
If I had something in my build process taking over a minute, I'd probably make that a separate build process and just include the result of that in the main build. Separate batch files could be used to combine the most common build sequences, so you could still build the project in one go no matter what you're editing. NES games are usually fairly small and don't typically take long to assemble, so I don't think there are many things you'd need to handle separately.
Re: Batch files as a build system for ASM6 projects
by on (#236929)
tokumaru wrote:
If I had something in my build process taking over a minute, I'd probably make that a separate build process and just include the result of that in the main build.

Despite what Peter Miller has claimed, a "separate build process" is OK, so long as there's some way for the main batch file to know whether or not to call the "separate build process".
Re: Batch files as a build system for ASM6 projects
by on (#236930)
I think he meant one batch file is build all (assets and code) and the other is just build (code) and you run the appropriate script based on what you are editing.

If not, it's something I've done before to minimize build time on my laptop.
Re: Batch files as a build system for ASM6 projects
by on (#236952)
tepples wrote:
a "separate build process" is OK, so long as there's some way for the main batch file to know whether or not to call the "separate build process".

What I meant is that the main batch file wouldn't need to know anything, you'd actually have different "main batch files" you could call depending on what you did or didn't modify. Not that you'd have hundreds of batch files for all possible combinations, just a handful to set apart the slow stuff.
Re: Batch files as a build system for ASM6 projects
by on (#237145)
You can compare modified times in batch files. However making a .js or .vbs script would be less pain.

msbuild is free and open source, supports linux and macOS X

However I really do wonder what your compression tool is doing to eat 10 seconds for 240p test suite.. did you write it in Ruby? I mean 10 seconds is neither here nor there time wise. If I had a project that took a long time to do a thing, I would make make it so I manually ran the thing that was talking a lot of time, and modify the normal build to not build it.
Re: Batch files as a build system for ASM6 projects
by on (#237162)
Oziphantom wrote:
However I really do wonder what your compression tool is doing to eat 10 seconds for 240p test suite.. did you write it in Ruby?

Close. It's in Python. In a previous topic, the merits of rewriting it in C were explored.

So it appears we have a consensus: ASM6 users are less likely to understand Make than cc65 users. Instead, they prefer a two-step build process driven by Windows Command Prompt batch files, where the first step converts and compresses assets and the second step assembles the code.