Temporal halt

I'm currently very busy, that's the main reason that the public betas (and CVS development) only saw one iteration. I've quite a few new code coming, mostly oriented to compatibility / visual fidelity (not speed), as it's what I'm mostly interested now. I'll probably take my a month or so until I can update this again, so have fun meanwhile :P

New beta "policy" on official DeSmuME

Just as a quick note, due to recent developments that I won't go into detail, I decided to start doing CVS builds, from the work in progress between releases, myself and uploading in a small web page that I setup.

I've been thinking for quite some time on doing some serious work on the CVS regarding the 3D core and the general speed. I know I can get easily a 10% speed up and make the 3D games look correct. Of course that's what I can get up shortly (few days/week), but more will come in the future.

Any way, the good points are having trusted builds that will stop a bit before official releases to keep some surprises on the official releases (that's a REALLY IMPORTANT point for me), and having more control over what's in the CVS builds (for example, removing the "report bug" on the menus). Of course, another benefit is getting better testing from a more general user base to avoid official releases being broken or suffering from regression.

Well, some of you might have already read about this already, the new part is that it will start soon, with some long demanded features. Exactly in a few hours from this post: it's going to be an interesting new way to receive feedback, let's see how it does turn.

EDIT: Already uploaded the first build, enjoy.

How do I usually fix stuff in desmume

I've not worked much on desmume lately. I really like low level coding, but I prefer graphics coding, and that's what I've mostly been doing the past 14 years. I still work on desmume from time to time, but it's not my only open project as of now.

But that's not what I want to talk about today, I want to give an example of how I fixed a certain issue on desmume, and the whole process that lead to fixing the bug: usually involves hell a lot of work to fix small bugs. It's strange to fix major problems with little work, or at least that's what I'm used to. Well, let's get to the good stuff.

Recently, NHervé (from ngemu forums) asked if distributing desmume "mods" was ok. The answer was simple: desmume is GPL, an thus, distributing modified versions is ok, as long as full source code is provided for the modified version. After that, he did the first mod. He is quite focused on fixing New Super Mario Bros. To get it ingame, emulating the GFX FIFO irq is needed, which official version doesn't support. Anyway, there was (at least), another bug: when starting mini-games (on main menu), desmume froze. That's what I'll talk about today, how I fixed that freeze. Let me clear up, that blog entry is a readable version of a what I explained in this thread.

The first thing I usually do is double-check where it freezes, obvious, but needed. I've seen plenty of people fixing "bugs" without even checking the exact reason, and it's losing time. First thing I saw, is that it jumped to 0x00000000, and there wasn't anything resembling code: basically it was all zeros. That jump could be on purpose, or the reason of the freeze. I assumed that it was meant to jump there, as I had to pick an option, and both were possible. So assuming than that memory region wasn't being initialized properly, I thought that it was some DMA failing, but after a quick peek at the DS available documentation, I saw that on that region resides the ITCM, which (explained in a simple way) is basically a small area of memory (32kb), where code is executed faster. On the DS, you can't DMA to the ITCM, so that couldn't be the failure. Next step was a bit more hard, and involved reading a bit more on how the internals of the DS work.

As I didn't knew how the ITCM worked, I spent some time reading about it. The most interesting bit, is that by setting it's virtual size (up to 32MB) bigger to it's physical size (32kb), it'll mirror it's contents from 0x00000000 to the virtual size. I guessed New Super Mario Bros was abusing that, by writing somewhere in the virtual memory area of the ITCM, and then reading from 0x00000000, which would be perfectly working code on the DS. I checked if it set the virtual size of the ITCM bigger than the physical size, and seeing that on boot it's set to 32MB by the game, I quickly hacked mirroring. Even with that, the first routine residing on 0x00000000 would freeze too: when called, the return address was set to 0x00000000 too, so when it tried to return to the calling routine, it jumped to itself. Forever.

I started to think that the fault was not on ITCM handling (even if desmume's current implementation is technically wrong as of now), but on the calling routine. After debugging it for a few minutes, one thing was clear: it was reading from uninitialized memory, and that's why it jumped to 0x00000000. Hell, even all the register were set to 0x00000000 due to reading from memory not initialized, which was rather fishy. To make a long story short, I compared that memory region with two other DS emulators, and it was clear that desmume was not copying data there for some reason. First, I assumed that the DTCM (again, a 16kb memory where data is read faster than from main memory) mirroring was the culprit. I was wrong again. I thought that I was really missing something obvious.

I went back to the available documentation, and searched for the address of memory that the game was reading from before jumping to 0x00000000, to be exact, 0x027FFE00 (plus some offsets on the various reads), et voila, it was simpler than I expected: on boot, the DS copies the game card header there. Quickly checked if desmume was copying it too, and saw that it just copied the header partially. I implemented simple copy loop in a minute and I got the mini-games to work. In the end, it was WAY simpler than I expected :/

Still the mini-games have some problems with the sprites looking bad, but that is another task I'll might talk about another day. Oh, and I committed a few fixes (including this one) to the official CVS, just in case anyone wants to check it out. A few screenshots, as usual:

Have fun :)

Texture fix and icon viewer

Yesterday I was rather bored for a few hours (even if I've a lot of real life stuff to do...), so after checking some information from the DS cartridge header (for some unrelated work still to be done), I thought how hard it'd be to add an icon viewer to the "Game Info" dialog. Extracting it from the header wasn't that hard, as the format is quite simple: a 32x32 pixels icon, stored on a 8x8 tile basis, 4bits per pixel, which are used to lookup on a fixed 16 colour palette (per icon).

In fact, converting the icon was easy, but I had major problems due to some recent changes, as I forgot I needed to use GDI+ to draw stuff on dialogs, rather than using plain GDI (that's what all desmume uses on the windows port). When I moved the drawing code of the icon to GDI+, it worked perfectly. I deleted all the MAP/OAM/etc viewers that relied on GDI code, as I plan to rewrite the GUI in the near future anyway, so there's no point in fixing that code.

After that, I wanted to finally fix the last bits of Mario&Luigi to look perfect. A few weeks ago, I cared to make it run where it "freezed" in the past: in fact, it didn't freeze, just took ages to render a new frame, due to a rather important bug in the 3D core. The main problem after that, is that ALL textures looked wrong: for example, where Koopa should be, only textures of arrows were shown.

When approaching the problem, I had only a stupid idea of what could be: due to the way the display lists are handled, there was a slight chance that the texture bind before drawing the polygons was done in wrong order. Well, to make a long story short, I was wrong: I tried changing a few bits, even hacking some stuff, and it didn't fix anything. In fact, the fix was easier and more logical.

I've been coding on PC for quite a few years, and I'm quite used to some stuff working in a certain way. While emulating the 3D core of the DS, I've already fallen on that problem quite a few times: on the DS, setting attributes per polygon (without starting a "primitive block") isn't that rare, for example, transformations (scaling, for example :P). In this case, it was the texture changing while the primitive block was open. I guess that, if you want to draw all the stuff onscreen, and you only use one type of primitive (a quad, for example), then changing the texture within the primitive block isn't that rare.

Anyway, adding the ability to change textures within a primitive block fixed "Mario&Luigi: Partners in time" and "Final Fantasy XII: Revenant Wings", so now both seem to be 100% playable and glitch free (and run rather fast). The usual screenshots:

Have fun :)

Shadows and next step...

Today, after reading Martin Korth's recently released (he hadn't updated it since ages ago) explanation on how DS hardware shadow volumes work, I attempted to implement them, as it was easier than I thought: it's not like the stencil shadows implementations that I've seen on PC, were the stencil is used to block lighting, and usually a shadow volume has to be computed per light. On the DS, it's a two step process: first, a mask step, which basically creates a mask to know where the next step should be applied, and a second step, that it's simply drawing a certain (usually black or dark) volume were the mask lets us. Light computation/s is NOT used at all :P

For example, imagine a cylinder intersecting the floor, the first step would create a mask that would resemble a circle on the floor (in fact it doesn't work exactly like that, but for the explanation sake...), and the second would paint a dark colour on that circle (and nowhere else). Why use a cylinder, and not a circle directly? Because we can't be sure of the topology/shape of the floor, neither if it'll be a floor, or animated: using this approach, we can shadow whatever is inside our cylinder, without the need of knowing the rest of the object/s in the scene.

So here's the result, and that fixes one of the last "big" bugs on Mario 64 DS. In the process I also fixed and important blending/transparency bug, which narrows even more the list of graphic bugs: only small ones left, the "waves" background on the save selection screen don't look exactly as on the DS, probably due to the way lights or normals are treated right now and the writing on Daisy's letter don't fade correctly (2D core related). Anyway, here's two shots of Mario 64 DS, and one of Tak - The Great JuJu Challenge:

About Tak - The Great JuJu Challenge, it's my next target to debug, as it show geometry, but it's completely wrong/uninitialized. Were it should show an animated mesh, it just show a static blob of what seems random data. I've a few ideas about what could cause that, but first I'd like to spend some trying them.

Have fun :)

One year ago...

I was bored a few hours ago, coding and debugging stuff on the DS (not emulation related), and I remembered that I wanted to do some type of "that desmume one year ago" post. That's what you get when I'm too lazy to spend time doing some real work on desmume :P

So... not much to talk about, I just searched for the first public screenshots that were released of a commercial game (I remembered they looked BAD, because I added an ugly hack for transparency, and I didn't know that version of the 3D core would be used for screenshots...), and made new screenshots today with the lastest version. Nothing really new shown on that screenshots, but might be fun for those that saw the original screenshots one year ago.

The screenshots that I'm talking about are on this post on emutalk (my nickname there is synch, that'll see referenced on that post), and you can compare them with this ones:

Speed has increased about 2x-5x (depending on game) in a year, while having way better (or almost perfect) 3D, so I guess it hasn't been a total waste of time... Have fun :)

Speeding up

After abandoning desmume development for a few months, I've been working on it quite a lot lately. Of course, not as much as when I started working on the project, as motivation to develop desmume is rare this days (just check the official desmume CVS, and you'll see what I'm talking about). But anyway, I've done some improvements on a very specific side of the project that I've avoided working on for the last 6 months: speed.

I've to be clear: desmume current way of handling most of the interrupts sucks. For example, timers, dmas and other, are checked PER opcode emulated. In an ideal world, you'd run a batch of instructions (the exact ammount would be determined by some heuristics or whatever), and then you'd do the processing needed by the pending interrupt / timer handling, then again back to running a batch of instructions. Of course, that'd be a less exact than current implementation, but it'd be WAY faster. I did something similar on my Gameboy emulator, and it gave good results, we'll see how it works on desmume, whenever I find time to implement it.

Anyway, I'll start with two screenshots, one before the speed ups I implemented the past 3 days (left), and another one with them (right):

As you can see, it's a practical 33% speed up on this particular case. I must say that it affects every game that I've tested on a considerable ammount, being the minimum a 14% (I'll explain later why). Just as a real world example, tested "Castlevania: Dawn of Sorrow" and it's 30% faster now... So let's talk about the speed ups.

First, desmume used/s GDI to draw the DS screens onto the main window. GDI setup is as easy as it can get, but seems that it's not as fast as I would wish (more keeping in mind that it's only task is to take an image and draw it on the screen). I considered a few options, namely DDraw, openGL or GDI+. I excluded openGL at the beggining, because it would be quite a lot of work to make the current 3D core work correctly with it. So after some discussing it with some friends (more exactly, hearing some DDraw bashing and why I should stop using DirectX7 features), I started working on a GDI+ blitter. In the end it was easier than I expected. Reading the documentation, implementing the blitter and a bit of adjusting was done in an hour. In fact, searching through the documentation was what took longer. The good part, is that it gave a free 14% perfomance gain.

The rest was a matter of fixing some tidbits of how memory is accesed, avoiding a lot of conversions from floating point to integer (which is way more expensive that you'd guess), changing the way the texture cache works, and more stuff I can't remember. I won't be any more concise about this stuff, as it would take quite a few pages, which I'm not willing to write now. Maybe I'll go back to them in a future post.

Just as an ending, it seems that my obsession about fixing all regression bugs that I see is giving good results, as most games that I test have perfect 3D. Here's some proof, Hotel Dusk running perfectly, even with the motion blur effect (I rotated the image with an image editor, because I don't have screen rotation implemented):

Have fun