This page is a mirror of Tepples' nesdev forum mirror (URL TBD).
Last updated on Oct-18-2019 Download

Wondering: how does the NES handle logic in a single frame?

Wondering: how does the NES handle logic in a single frame?
by on (#236666)
So I was recently watching a YouTube video explaining how the SNES worked, and in one part it was explaining some things about how the system processed its information while the screen was drawing an image.

I had never really thought about it in such depth before, but the way I had always thought of how these consoles processed the game data and rendered the image was wrong. For some reason I was thinking they worked too similar to how modern computers render an image, which basically would "store" the complete image in memory. Well duh the NES doesn't do that, because it would require several KB of video RAM. It processes the render of an image in literally the same moment that it sends the signal to the screen, and thus only processes a tiny part of it. This makes so much sense, and I'm surprised I never really thought about it before. All of the tricks where a system changes properties between scanlines makes more sense this way.

But there's a couple things about this that I'm a little unsure of, and so I thought I'd ask for some clarity.

The biggest thing that is confusing me is how the NES (and other consoles) updated the game logic. In the video it stated that this was done at the same time everything was rendering on the screen. To my understanding, this is everything like updating where the player and enemies are, processing commands for someone attacking, changing the name tables to change the backgrounds, and whatever else needs to be done in a given frame.
But if this is being done at the same time that the image is being drawn onto the screen, um, wouldn't that cause problems? Like if an object moves on a given frame, what if say part of that object was drawn on one scanline, then the code updated that object, and while drawing the next scanline that same object would be in a different position, resulting in its appearance being split on that frame.
I would assume that all of this logic would have to be done during the V-Blank so that the image never gets screwed up. But this also doesn't quite add up. If this were so, it would provide a relatively small amount of time wherein the system can process all the game logic. And furthermore, if the system didn't finish all this before the next frame started drawing, then the system would be drawing an incomplete frame where some objects and things have changed but not all, instead of just repeating an exact copy of the last frame.
But yet, the system DOES just repeat the last frame. So, how does it manage to do this if it isn't storing a complete and whole image of the screen in a memory buffer? And how do these systems manage to keep applying the same effects where they switch data between scanlines if the console is still "busy" processing the game logic?

Plus, as I said, this reserves a rather short amount of time for processing game logic. And all the rest of the time is spent processing rendering for the screen. I suppose it is possible, but it just doesn't sound right to me.

I think I have the wrong idea about what is happening.

Also I have a few other things I am curious about. For one, just how much "final render data" is the NES actually "storing" in its system? And what about later consoles like the SNES, Mega Drive, and PC Engine? Do they "store" a complete scanline before it is sent to the TV, or does it really render each pixel as it gets pushed to the signal out of the console? It seems somewhat reasonable to me that it should have some understanding of at least a few pixels ahead before it sends them through the cable.
But again, maybe I have a misconception about how these systems work.

Another thing I am curious about is: how do emulators manage to handle all this? I mean, sure, even back when I was running NESticle my computer could out-perform the NES a hundred times over, but to function properly the emulator still needs to keep all the timing correct, all while drawing an image that it sends to the screen but in a pre-computed fashion. I'm rather curious how a PC handles this change in how the game needs to be rendered.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236667)
A thorough answer to some of your questions would literally fill a book. Please start at "NES reference guide" on NESdev Wiki.

Marscaleb wrote:
But if this is being done at the same time that the image is being drawn onto the screen, um, wouldn't that cause problems? Like if an object moves on a given frame, what if say part of that object was drawn on one scanline, then the code updated that object, and while drawing the next scanline that same object would be in a different position, resulting in its appearance being split on that frame.

That's called tearing. There are cases where it happens on Game Boy, TurboGrafx-16, and Genesis because these platforms allow video memory access during at least horizontal blanking. But it doesn't happen on NES or Super NES because neither NES nor Super NES allows video memory access outside vertical blanking. You can change scroll position but not video memory.

Marscaleb wrote:
I would assume that all of this logic would have to be done during the V-Blank so that the image never gets screwed up. But this also doesn't quite add up. If this were so, it would provide a relatively small amount of time wherein the system can process all the game logic.

Games fill a buffer in RAM during scanlines 0-240 and copy the contents of this buffer from RAM to video memory during scanlines 241-260. Generally an entire page (256 bytes) of memory is reserved for a sprite display list, also called shadow OAM, and the CPU offers a DMA mechanism to copy it to the PPU during vertical blanking. This takes about 4 1/2 scanlines. There's no DMA to the pattern tables (tile shapes) or the nametables (tilemaps); each game has to implement its own buffer mechanism. Lately, I've used a video memory transfer buffer mechanism that I call Popslide.

Marscaleb wrote:
But yet, the system DOES just repeat the last frame. So, how does it manage to do this if it isn't storing a complete and whole image of the screen in a memory buffer?

The video memory stores a complete and whole image of all 8x8-pixel tiles used in a scene, called the pattern table or CHR data. It also contains a map of what tiles are placed where, called the nametable. Think of the pattern table as the "font" and the nametable the "text". The PPU also contains a list of the position of up to 64 sprites, called object attribute memory (OAM).

Marscaleb wrote:
And how do these systems manage to keep applying the same effects where they switch data between scanlines if the console is still "busy" processing the game logic?

Some games devote 100% of CPU cycles during some parts of the frame to pushing scroll changes between scanlines. Other games contain a timer, which fires once the PPU has reached a particular scanline, causing an interrupt.

Marscaleb wrote:
Also I have a few other things I am curious about. For one, just how much "final render data" is the NES actually "storing" in its system?

Pattern tables are typically 8192 bytes (8 KiB), though a cartridge may store more CHR data than that and make subsets available to the PPU at any given time. Nametables are 2048 bytes (2 KiB) in almost all cases. OAM is 256 bytes. The palette is 28 bytes, though only 25 of them are actually used during rendering. Internal buffers (secondary OAM, sprite shifters, background shifters) are a few more bytes, but I doubt that a beginner would quickly understand a thorough enumeration of the sizes of all temporary buffers within the PPU down to the bit.

Marscaleb wrote:
And what about later consoles like the SNES, Mega Drive, and PC Engine? Do they "store" a complete scanline before it is sent to the TV, or does it really render each pixel as it gets pushed to the signal out of the console?

These platforms follow mostly the same principle, just with different memory sizes and different structures of temporary buffers inside the PPU. The differences among the exact memory sizes of all of these platforms are considerable, and a reply describing all of them in much detail would quickly become tl;dr (too long; didn't read). But suffice it to say that for the TurboGrafx-16, Genesis, and Super NES, the main video memory is 65536 bytes (64 KiB).

Marscaleb wrote:
It seems somewhat reasonable to me that it should have some understanding of at least a few pixels ahead before it sends them through the cable.

This is correct. When constructing the video signal from the contents of video memory, the PPU reads data from video memory into internal temporary buffers. These buffers contain data a few pixels ahead of the electron beam. The precise internal structure of these buffers differs greatly among the NES, Master System, TurboGrafx-16, Game Boy, Genesis, and Super NES.

Your use of both "PC Engine" (a Japan-exclusive name) and "SNES" (an other-than-Japan-exclusive name) confuses me as to which region's name I should use when describing the differences among the video architectures of these platforms. NES was called Family Computer (Famicom) in Japan, Master System was called Mark III in Japan, TurboGrafx-16 was called PC Engine in Japan, Mega Drive was called Genesis in North America, and Super NES was called Super Famicom in Japan. Among the major third and fourth generation consoles, only the Game Boy kept the same name among Japan, Europe, and North America. Why Sega used a Biblical name for one console only in Japan (Mark III) and for its next console only outside Japan (Genesis) beats me.

Marscaleb wrote:
Another thing I am curious about is: how do emulators manage to handle all this? I mean, sure, even back when I was running NESticle my computer could out-perform the NES a hundred times over, but to function properly the emulator still needs to keep all the timing correct, all while drawing an image that it sends to the screen but in a pre-computed fashion. I'm rather curious how a PC handles this change in how the game needs to be rendered.

I will leave it to others to explain emulation in great detail.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236668)
I would simply say that the NES CPU is running and executing code at all times, it cannot be stopped. It is up to the programmer to know when to execute each type of code.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236669)
Marscaleb wrote:
It processes the render of an image in literally the same moment that it sends the signal to the screen, and thus only processes a tiny part of it.

This description fits the Atari 2600 much better than it dos the NES or the SNES: The Atari 2600 has so little video memory that it's only able to store about a single scanline worth of data, so the program has to "race the beam" and change the video memory as the screen is rendered in order to display anything more detailed than colored vertical lines!

As for the NES and SNES, it's true that they don't use a simple frame buffer, instead they optimize space and time by constructing the final image from smaller reusable units (name tables, pattern tables, palettes, etc.), but these consoles have enough video memory to hold at least one entire screen worth of data, so the game code doesn't need to update the video as it's being sent to the TV. The PPU is the one that's constantly busy doing video work, gathering all the different pieces of data to form an image every frame. During rendering, the CPU is normally busy running the game logic.

Quote:
The biggest thing that is confusing me is how the NES (and other consoles) updated the game logic. In the video it stated that this was done at the same time everything was rendering on the screen. To my understanding, this is everything like updating where the player and enemies are, processing commands for someone attacking, changing the name tables to change the backgrounds, and whatever else needs to be done in a given frame.

During rendering the game code is updating the game's STATE, not the graphics. For example, when they say that the positions of the player and the enemies are updated, that means that the internal variables that represent their positions are updated, not their SPRITES.

Quote:
But if this is being done at the same time that the image is being drawn onto the screen, um, wouldn't that cause problems? Like if an object moves on a given frame, what if say part of that object was drawn on one scanline, then the code updated that object, and while drawing the next scanline that same object would be in a different position, resulting in its appearance being split on that frame.

As a game updates its internal state, it also buffers any graphical changes that are needed as a consequence of these state changes. For example, when the player moves, the variables that describe its position are changed, and a new set of sprite entries for it is generated and buffered in RAM. Not video RAM, just regular RAM, where it doesn't affect what's being displayed on the screen at all. Then, when vblank starts, all the buffered information is transferred to video RAM, so that all the changes are visible on the NEXT frame.

Quote:
I would assume that all of this logic would have to be done during the V-Blank so that the image never gets screwed up. But this also doesn't quite add up. If this were so, it would provide a relatively small amount of time wherein the system can process all the game logic.

Exactly. That's why games use buffers... vblank time is too short to do any real processing, so all the heavy logic is done during rendering, and any changes that need to be made to the image are buffered, so the vblank handler can be mostly "dumb" code that just copies the contents of the various buffers to VRAM as fast as possible.

Quote:
And furthermore, if the system didn't finish all this before the next frame started drawing, then the system would be drawing an incomplete frame where some objects and things have changed but not all, instead of just repeating an exact copy of the last frame.
But yet, the system DOES just repeat the last frame.

Games are usually programmed to NOT transfer the buffered data to VRAM if vblank starts before the game logic is finished running. IN this case, the vblank handler will probably just do some light housekeeping before returning and letting the game logic finish, and since VRAM is left untouched, the same frame is drawn again.

Quote:
And how do these systems manage to keep applying the same effects where they switch data between scanlines if the console is still "busy" processing the game logic?

"Real-time effects" such as background distortions or parallax layers do not rely on changes to the contents of VRAM, they are usually achieved by changing the scroll position mid-frame. This indeed has to be done by the CPU (although the SNES has extra hardware to help with that), so you need to use IRQs or timed code in order to sync with the PPU and make the changes at the precise moments. Depending on the synchronization method you use, you can indeed lose some processing time.

Quote:
Also I have a few other things I am curious about. For one, just how much "final render data" is the NES actually "storing" in its system? And what about later consoles like the SNES, Mega Drive, and PC Engine? Do they "store" a complete scanline before it is sent to the TV, or does it really render each pixel as it gets pushed to the signal out of the console?

The NES has very little memory to hold graphics that are being prepared for output... If I'm not mistaken, it can hold sprite patterns for one scanline (64 pixels), and 16 pixels worth of background patterns, and these are combined with the palette data pixel by pixel to calculate the final color that gets sent to the TV. I'm not sure about other consoles.

Quote:
Another thing I am curious about is: how do emulators manage to handle all this? I mean, sure, even back when I was running NESticle my computer could out-perform the NES a hundred times over, but to function properly the emulator still needs to keep all the timing correct, all while drawing an image that it sends to the screen but in a pre-computed fashion. I'm rather curious how a PC handles this change in how the game needs to be rendered.

There are different levels of accuracy when it comes to PPU rendering and CPU-PPU synchronization. A very inaccurate (and fast!) emulator could simply grab a snapshot of VRAM at a given moment in the frame and combine all the information from the different areas (name table, pattern tables, palette, OAM, etc.) and make an "instantaneous" rendering of that state, completely disregarding that the CPU and the PPU run in parallel. Simpler games might work just fine that way, but any game that talks to the PPU mid-frame (changing the scroll, emphasis bits, etc.) will fail. The other extreme are emulators that replicate the exact internal logic of the PPU... all the memory fetches, the state changes, the sprite evaluation steps, etc., all in perfect synchronization with the CPU. Current computers can handle that level of accuracy just fine these days, but if you look at all the NES emulators ever made, you'll see everything in between those 2 extremes.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236674)
Quote:
There are cases where it happens on Game Boy, TurboGrafx-16, and Genesis because these platforms allow video memory access during at least horizontal blanking. But it doesn't happen on NES or Super NES because neither NES nor Super NES allows video memory access outside vertical blanking. You can change scroll position but not video memory.

I'm fairly certain the Super NES allows at least palette changes during HBlank - certain games does this via HDMA.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236682)
Bregalad wrote:
I'm fairly certain the Super NES allows at least palette changes during HBlank - certain games does this via HDMA.

Yes, but on the SNES, the memory used for tiles and nametables (VRAM) and the memory used for palettes (CGRAM) are two separate things, accessed by separate MMIO registers and using separate address spaces.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236685)
Marscaleb wrote:
Just how much "final render data" is the NES actually "storing" in its system?

In terms of actual "final render data", there is the 16 pixel wide shift register for background tiles, and that's it. This keeps getting updated while the scanline is drawn.

When it's time to draw sprites on top of that scanline, it had already gathered 8 sprites, and it knows the sliver of pixels it wants to draw, its X coordinate, and palette selection/priority. This is called Secondary OAM.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236713)
Okay, first of all I want to apologize because I wrote that really late at night (for me) and I didn't quite phrase a few things right, since I was half asleep.

I want to explain what I meant about storing the screen's image.

With a PC from the time, (and technically still so now,) they needed enough video memory store a complete pixel-by-pixel image of the screen. The screen can show a whole image with no repeating tiles across the entire screen, and this whole image is stored in memory. (Which can take a lot of time to change except in small sections, which is part of why so many games were adventure games and other titles where the screen never scrolls.)
But that kind of video memory is expensive, so for most consoles they have their system of using tiles and tables, which can still draw incredible images but doesn't require as much memory. And as a bonus you can make the table a little larger than the screen to enable smooth scrolling. Unless you're an MSX.

That much I already understood, but I didn't quite get some things I saw, like the Genesis changing the palettes mid-frame to simulate a water transparency in Sonic the Hedgehog. Even though I understood that the system split the image into separate pieces, I was still thinking that the consoles had to render and send a whole image like a PC did. So it confused me that the NES could change tiles in the middle of a frame to draw the status bar in Mario 3, because doing so defied the idea of splitting the image into smaller chucks, because now there were more tiles on the screen than can fit into the memory. I didn't get how this was possible because I was still imagining that a single whole image of the screen was being "stored" in memory.

But watching that video cleared up that confusion. The consoles can make those changes in the middle of rendering the frame because they are just changing the live signal being sent to the screen.

tepples wrote:
That's called tearing. There are cases where it happens on Game Boy, TurboGrafx-16, and Genesis because these platforms allow video memory access during at least horizontal blanking. But it doesn't happen on NES or Super NES because neither NES nor Super NES allows video memory access outside vertical blanking. You can change scroll position but not video memory.

That makes sense. It kind of makes me wonder why only those two consoles would think to prevent that. It seems especially surprising for the Game Boy, since it was the same company. I guess maybe it was related to battery consumption.

tokumaru wrote:
During rendering the game code is updating the game's STATE, not the graphics. For example, when they say that the positions of the player and the enemies are updated, that means that the internal variables that represent their positions are updated, not their SPRITES.

Ahhh, there's the other half of the equation!
And this one really shatters my false misconceptions about how these systems work. Yeah, I was really interpreting an actor's location and the location of said actor's sprite as being the same thing. I never really thought that you could change one but not the other.

That actually clears up a LOT of the confusion I had. Separating those two (and likewise a number of other variables) makes it a lot easier to understand how the game data can be calculated and updated while the image is being drawn without screwing that render.

Hmm, a lot of the base-level code for these games just became more complicated than I thought.

tokumaru wrote:
The PPU is the one that's constantly busy doing video work, gathering all the different pieces of data to form an image every frame. During rendering, the CPU is normally busy running the game logic.


And there's the last thing that I didn't get, and I feel kinda stupid for not seeing it before. Of course, there is a whole separate chip handling all the tasks to draw those scanlines, not the CPU.

tepples wrote:
Your use of both "PC Engine" (a Japan-exclusive name) and "SNES" (an other-than-Japan-exclusive name) confuses me as to which region's name I should use when describing the differences among the video architectures of these platforms. NES was called Family Computer (Famicom) in Japan, Master System was called Mark III in Japan, TurboGrafx-16 was called PC Engine in Japan, Mega Drive was called Genesis in North America, and Super NES was called Super Famicom in Japan. Among the major third and fourth generation consoles, only the Game Boy kept the same name among Japan, Europe, and North America. Why Sega used a Biblical name for one console only in Japan (Mark III) and for its next console only outside Japan (Genesis) beats me.


Yeah, that's something I really am chalking up to the fact that it was really late and I was really tired.
Honestly, in my mind the console is always known as "Genesis" first. But a number of years back I realized that since it was only called that in the US the console's "true name" would be the "Mega Drive," so from time to time I try to refer to it as that in an effort to be "more accurate." And being half-asleep I slipped into that because... half-brain logic.
And then I immediately followed that up with "PC Engine" partway because I didn't want to bother trying to remember how to properly spell "Turbografx 16" and partway for the same logic of "that's what the people who made it called it so that must be its real name." Because when you are tired enough all kinds of things make perfect sense! Hey, who wants to talk about politics at four in the morning?

Also, I'm pretty sure that "Mark III" is referring to "Mark" as in "version," not Mark chapter 3.

tokumaru wrote:
There are different levels of accuracy when it comes to PPU rendering and CPU-PPU synchronization. A very inaccurate (and fast!) emulator could simply grab a snapshot of VRAM at a given moment in the frame and combine all the information from the different areas (name table, pattern tables, palette, OAM, etc.) and make an "instantaneous" rendering of that state, completely disregarding that the CPU and the PPU run in parallel. Simpler games might work just fine that way, but any game that talks to the PPU mid-frame (changing the scroll, emphasis bits, etc.) will fail. The other extreme are emulators that replicate the exact internal logic of the PPU... all the memory fetches, the state changes, the sprite evaluation steps, etc., all in perfect synchronization with the CPU. Current computers can handle that level of accuracy just fine these days, but if you look at all the NES emulators ever made, you'll see everything in between those 2 extremes.


That sounds pretty reasonable, actually.
And the more I think about it, I'd probably need a better understanding of how PC's work and render images to satisfy my curiosity of how emulators work.
I remember reading some article that talked about Nesticle using some particular approach that revolutionized emulation and made it really feasible. Makes me wonder where it fell between thsoe two extremes.
I was surprised when I discovered later that another emulator (FCE) had taken over as the head-honcho of NES emulation. I was a little disappointed at first, but honestly I'm really glad. I'm not 14 anymore and I don't want to double-click some nutsack every time I want to play some Mega Man.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236714)
Marscaleb wrote:
That much I already understood, but I didn't quite get some things I saw, like the Genesis changing the palettes mid-frame to simulate a water transparency in Sonic the Hedgehog.

On most of these old consoles, color palettes are stored in a separate area of memory. Some allow changing it between one line and the next, and this can be used for "copper bar" shading.

Marscaleb wrote:
It kind of makes me wonder why only those two consoles would think to prevent that.

For everyone else, being able to write video memory at any time was considered a desirable feature. The Genesis have in fact has slightly reduced parallax capability compared to the Super NES, and one of the reasons is to leave some VDP time open to let the CPU update video memory.

Marscaleb wrote:
It seems especially surprising for the Game Boy, since it was the same company. I guess maybe it was related to battery consumption.

Game Boy was designed by a different team, and in association with Sharp, not Ricoh.

Marscaleb wrote:
Also, I'm pretty sure that "Mark III" is referring to "Mark" as in "version," not Mark chapter 3.

Where you see tepples, expect to see puns ;-)

Marscaleb wrote:
I remember reading some article that talked about Nesticle using some particular approach that revolutionized emulation and made it really feasible.

I don't think NESticle did anything special. The revolutionary part was UltraHLE, a Nintendo 64 emulator that intercepted calls to early N64 games' standard library and ran equivalent native code.

Marscaleb wrote:
I was surprised when I discovered later that another emulator (FCE) had taken over as the head-honcho of NES emulation.

FCE became FCE Ultra, which was forked a bunch of times to add mappers, debugging, and TAS recording, and then the forks were merged into FCEUX. And FCEUX is still a "decent"-class debugging emulator for low-end machines that can't quite run Mesen.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236717)
Biblical jokes aside, and it's offtopic but as it's brought up already so anyway...

The reason the SMS was called Mark III in Japan was that its full name was originally SG-1000 Mark III as it's considered a revision/successor of the SG-1000 console (the revision model SG-1000 Mark II DOES exist), just that people rarely called it by its full name.
It later received an upgrade/revision model in Japan called Master System(confusing?) which used the same case as the international SMS, but its code name during development was Mark IV.
Then the Mega Drive came, and magazines called it Mark V during development (don't know whether the magazines made it up or it's the real development code name).
Actually in Chinese speaking cities, the Mega Drive was just called Sega Mark V because of this.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236728)
So is the Dreamcast the Sega Mark VII or is it the Sega Mark X? :)
Re: Wondering: how does the NES handle logic in a single fra
by on (#236753)
As the Mark I-V were built upon the same tech (the Mega Drive was still compatible with Mark III) they're thus named like this.
Starting from the Saturn, as it's completely different from its predecessors it no longer used the "Mark" code.
Re: Wondering: how does the NES handle logic in a single fra
by on (#236802)
Project Mercury is the Game Gear.
And the Nomad (portable Genesis) is Venus.

Neither of these went by Mark VI or VII. The Mark naming stopped at V.
/offtopic