New official Desmume release (0.9)

First of all, as this is important for me, I want to say that the important thing to remember from this post is that the official desmume team is doing an ammusing work.

So, the news is that the official team (which I'm member of, but haven't contributed much to lately) that is working on Desmume has released a new version. I've committed a minor amount of my work there: a little CPU fix that makes Dead'n'furious playable (I talked about it in the past) and other minor fixes and improvements. I'm too busy to be able to work more on Desmume (or any other project, for that matter), and there's talented people working there that seem to be more focused and able to do impressive work on the project that I am,:any way it's nice to see that the official version is progressing.

Still, it's very different from my version (nor better of worse, just different), as I focused on different stuff, but it's what you get from different branches of the same project. Keep an eye on what that team (and maybe me, if I get some free time and motivation) is going to get done, because it's going to amuse most of you.

Permanent halt

Hi,

Due to reasons that I don't want to explain, I'm stopping any public (or private, for that matter) work on Desmume. I know I'll work on it (or another emulation related project) in the future, but I don't know if it'll take 3 weeks, 3 months or 3 years.

Expect nothing.

Temporal halt

I'm currently very busy, that's the main reason that the public betas (and CVS development) only saw one iteration. I've quite a few new code coming, mostly oriented to compatibility / visual fidelity (not speed), as it's what I'm mostly interested now. I'll probably take my a month or so until I can update this again, so have fun meanwhile :P

New beta "policy" on official DeSmuME

Just as a quick note, due to recent developments that I won't go into detail, I decided to start doing CVS builds, from the work in progress between releases, myself and uploading in a small web page that I setup.

I've been thinking for quite some time on doing some serious work on the CVS regarding the 3D core and the general speed. I know I can get easily a 10% speed up and make the 3D games look correct. Of course that's what I can get up shortly (few days/week), but more will come in the future.

Any way, the good points are having trusted builds that will stop a bit before official releases to keep some surprises on the official releases (that's a REALLY IMPORTANT point for me), and having more control over what's in the CVS builds (for example, removing the "report bug" on the menus). Of course, another benefit is getting better testing from a more general user base to avoid official releases being broken or suffering from regression.

Well, some of you might have already read about this already, the new part is that it will start soon, with some long demanded features. Exactly in a few hours from this post: it's going to be an interesting new way to receive feedback, let's see how it does turn.

EDIT: Already uploaded the first build, enjoy.

How do I usually fix stuff in desmume

I've not worked much on desmume lately. I really like low level coding, but I prefer graphics coding, and that's what I've mostly been doing the past 14 years. I still work on desmume from time to time, but it's not my only open project as of now.

But that's not what I want to talk about today, I want to give an example of how I fixed a certain issue on desmume, and the whole process that lead to fixing the bug: usually involves hell a lot of work to fix small bugs. It's strange to fix major problems with little work, or at least that's what I'm used to. Well, let's get to the good stuff.

Recently, NHervé (from ngemu forums) asked if distributing desmume "mods" was ok. The answer was simple: desmume is GPL, an thus, distributing modified versions is ok, as long as full source code is provided for the modified version. After that, he did the first mod. He is quite focused on fixing New Super Mario Bros. To get it ingame, emulating the GFX FIFO irq is needed, which official version doesn't support. Anyway, there was (at least), another bug: when starting mini-games (on main menu), desmume froze. That's what I'll talk about today, how I fixed that freeze. Let me clear up, that blog entry is a readable version of a what I explained in this thread.

The first thing I usually do is double-check where it freezes, obvious, but needed. I've seen plenty of people fixing "bugs" without even checking the exact reason, and it's losing time. First thing I saw, is that it jumped to 0x00000000, and there wasn't anything resembling code: basically it was all zeros. That jump could be on purpose, or the reason of the freeze. I assumed that it was meant to jump there, as I had to pick an option, and both were possible. So assuming than that memory region wasn't being initialized properly, I thought that it was some DMA failing, but after a quick peek at the DS available documentation, I saw that on that region resides the ITCM, which (explained in a simple way) is basically a small area of memory (32kb), where code is executed faster. On the DS, you can't DMA to the ITCM, so that couldn't be the failure. Next step was a bit more hard, and involved reading a bit more on how the internals of the DS work.

As I didn't knew how the ITCM worked, I spent some time reading about it. The most interesting bit, is that by setting it's virtual size (up to 32MB) bigger to it's physical size (32kb), it'll mirror it's contents from 0x00000000 to the virtual size. I guessed New Super Mario Bros was abusing that, by writing somewhere in the virtual memory area of the ITCM, and then reading from 0x00000000, which would be perfectly working code on the DS. I checked if it set the virtual size of the ITCM bigger than the physical size, and seeing that on boot it's set to 32MB by the game, I quickly hacked mirroring. Even with that, the first routine residing on 0x00000000 would freeze too: when called, the return address was set to 0x00000000 too, so when it tried to return to the calling routine, it jumped to itself. Forever.

I started to think that the fault was not on ITCM handling (even if desmume's current implementation is technically wrong as of now), but on the calling routine. After debugging it for a few minutes, one thing was clear: it was reading from uninitialized memory, and that's why it jumped to 0x00000000. Hell, even all the register were set to 0x00000000 due to reading from memory not initialized, which was rather fishy. To make a long story short, I compared that memory region with two other DS emulators, and it was clear that desmume was not copying data there for some reason. First, I assumed that the DTCM (again, a 16kb memory where data is read faster than from main memory) mirroring was the culprit. I was wrong again. I thought that I was really missing something obvious.

I went back to the available documentation, and searched for the address of memory that the game was reading from before jumping to 0x00000000, to be exact, 0x027FFE00 (plus some offsets on the various reads), et voila, it was simpler than I expected: on boot, the DS copies the game card header there. Quickly checked if desmume was copying it too, and saw that it just copied the header partially. I implemented simple copy loop in a minute and I got the mini-games to work. In the end, it was WAY simpler than I expected :/

Still the mini-games have some problems with the sprites looking bad, but that is another task I'll might talk about another day. Oh, and I committed a few fixes (including this one) to the official CVS, just in case anyone wants to check it out. A few screenshots, as usual:

Have fun :)

Texture fix and icon viewer

Yesterday I was rather bored for a few hours (even if I've a lot of real life stuff to do...), so after checking some information from the DS cartridge header (for some unrelated work still to be done), I thought how hard it'd be to add an icon viewer to the "Game Info" dialog. Extracting it from the header wasn't that hard, as the format is quite simple: a 32x32 pixels icon, stored on a 8x8 tile basis, 4bits per pixel, which are used to lookup on a fixed 16 colour palette (per icon).

In fact, converting the icon was easy, but I had major problems due to some recent changes, as I forgot I needed to use GDI+ to draw stuff on dialogs, rather than using plain GDI (that's what all desmume uses on the windows port). When I moved the drawing code of the icon to GDI+, it worked perfectly. I deleted all the MAP/OAM/etc viewers that relied on GDI code, as I plan to rewrite the GUI in the near future anyway, so there's no point in fixing that code.

After that, I wanted to finally fix the last bits of Mario&Luigi to look perfect. A few weeks ago, I cared to make it run where it "freezed" in the past: in fact, it didn't freeze, just took ages to render a new frame, due to a rather important bug in the 3D core. The main problem after that, is that ALL textures looked wrong: for example, where Koopa should be, only textures of arrows were shown.

When approaching the problem, I had only a stupid idea of what could be: due to the way the display lists are handled, there was a slight chance that the texture bind before drawing the polygons was done in wrong order. Well, to make a long story short, I was wrong: I tried changing a few bits, even hacking some stuff, and it didn't fix anything. In fact, the fix was easier and more logical.

I've been coding on PC for quite a few years, and I'm quite used to some stuff working in a certain way. While emulating the 3D core of the DS, I've already fallen on that problem quite a few times: on the DS, setting attributes per polygon (without starting a "primitive block") isn't that rare, for example, transformations (scaling, for example :P). In this case, it was the texture changing while the primitive block was open. I guess that, if you want to draw all the stuff onscreen, and you only use one type of primitive (a quad, for example), then changing the texture within the primitive block isn't that rare.

Anyway, adding the ability to change textures within a primitive block fixed "Mario&Luigi: Partners in time" and "Final Fantasy XII: Revenant Wings", so now both seem to be 100% playable and glitch free (and run rather fast). The usual screenshots:

Have fun :)

Shadows and next step...

Today, after reading Martin Korth's recently released (he hadn't updated it since ages ago) explanation on how DS hardware shadow volumes work, I attempted to implement them, as it was easier than I thought: it's not like the stencil shadows implementations that I've seen on PC, were the stencil is used to block lighting, and usually a shadow volume has to be computed per light. On the DS, it's a two step process: first, a mask step, which basically creates a mask to know where the next step should be applied, and a second step, that it's simply drawing a certain (usually black or dark) volume were the mask lets us. Light computation/s is NOT used at all :P

For example, imagine a cylinder intersecting the floor, the first step would create a mask that would resemble a circle on the floor (in fact it doesn't work exactly like that, but for the explanation sake...), and the second would paint a dark colour on that circle (and nowhere else). Why use a cylinder, and not a circle directly? Because we can't be sure of the topology/shape of the floor, neither if it'll be a floor, or animated: using this approach, we can shadow whatever is inside our cylinder, without the need of knowing the rest of the object/s in the scene.

So here's the result, and that fixes one of the last "big" bugs on Mario 64 DS. In the process I also fixed and important blending/transparency bug, which narrows even more the list of graphic bugs: only small ones left, the "waves" background on the save selection screen don't look exactly as on the DS, probably due to the way lights or normals are treated right now and the writing on Daisy's letter don't fade correctly (2D core related). Anyway, here's two shots of Mario 64 DS, and one of Tak - The Great JuJu Challenge:

About Tak - The Great JuJu Challenge, it's my next target to debug, as it show geometry, but it's completely wrong/uninitialized. Were it should show an animated mesh, it just show a static blob of what seems random data. I've a few ideas about what could cause that, but first I'd like to spend some trying them.

Have fun :)