Pixel Shader support (more like re-inventing the wheel)...

Today, I sat back and thought to myself, "Hmmm, maybe I should start fixing some internal issues with Cxbx rather than updating the HLE database for a while." I thought back to myself how no matter how hard I tried, I never could get that stupid BumpEarth demo in the XDK to work right! The base texture always rendered fine, but the bump map and the environment map never did work. I decided to investigate. Initially, I assumed that it was a missing/wrong deferred texture state. After browsing the source code for a while, I realized all that stuff was fine. I even compared the code to the PC version. I noticed that the Xbox version of BumpEarth used a pixel shader, but the PC version didn't. Since I never really bothered with Xbox's pixel shader stuff (or even checked Cxbx's) I didn't initially assume that was a problem. I thought that this would be a good opportunity to check out how Xbox's pixel shaders worked. Turns out that Xbox pixel shaders are converted to a structure. Sounds weird to me, but I'm sure Microsoft had a good reason. After finding out that this was so, I checked Cxbx's implementation of IDirect3DDevice8_CreatePixelShader. Turns out that there was no [dynamic] pixel shader support at all! Whoever wrote it just used a fall back pixel shader and that was the end of it. That's better than nothing, but it's time change that (because if I don't do it, it might not get done)! Kingofc did it before, but we have no idea where he's been off to...

"Oh, I'm sure it can't be that bad... can it?" Well, so far, the Xbox's documentation on how it's pixel shaders work are a bit vague. The XDK also supports "Hard Coded" pixel shaders via the D3DPIXELSHADERDEF structure. Here's the definition of it (the X_ prefix was added for Cxbx to prevent collisions):

typedef struct _X_D3DPIXELSHADERDEF
{
DWORD PSAlphaInputs[8]; // Alpha inputs for each stage
DWORD PSFinalCombinerInputsABCD; // Final combiner inputs
DWORD PSFinalCombinerInputsEFG; // Final combiner inputs (continued)
DWORD PSConstant0[8]; // C0 for each stage
DWORD PSConstant1[8]; // C1 for each stage
DWORD PSAlphaOutputs[8]; // Alpha output for each stage
DWORD PSRGBInputs[8]; // RGB inputs for each stage
DWORD PSCompareMode; // Compare modes for clipplane texture mode
DWORD PSFinalCombinerConstant0; // C0 in final combiner
DWORD PSFinalCombinerConstant1; // C1 in final combiner
DWORD PSRGBOutputs[8]; // Stage 0 RGB outputs
DWORD PSCombinerCount; // Active combiner count (Stages 0-7)
DWORD PSTextureModes; // Texture addressing modes
DWORD PSDotMapping; // Input mapping for dot product modes
DWORD PSInputTexture; // Texture source for some texture modes
DWORD PSC0Mapping; // Mapping of c0 regs to D3D constants
DWORD PSC1Mapping; // Mapping of c1 regs to D3D constants
DWORD PSFinalCombinerConstants; // Final combiner constant mapping
}X_D3DPIXELSHADERDEF;

At first glance, the use of each field looks a bit obvious, but transforming this into a pixel shader isn't a trivial task (at least not for me). So far, it's just now starting to make sense (I had to look in the header file instead of the XDK docs). In the end, all pixel shaders on Xbox take the form of the structure you see above.

For those who have been following Xbox emulation history as long as I have, you might recall the day that _SF_ (author of Xeon) released the source code to his Xbox -> PC Direct3D pixel shader conversion code in an effort to prove that his emulator was not fake. I took a look to see how _SF_ did it in Xeon. Turns out that instead of using assembly, he used HLSL instead. It was a brilliant idea, but it's not going to work for Cxbx since it still uses DirectX 8.1 (another reason why I really want to start using OpenGL soon). Assembly shaders will still be possible, but just a bit more tedious.

What have I done so far? I've already written some prelimary code for pixel shader generation. For now, the best thing to do is to test it against pixel shaders written by myself and XDK demos to ensure accuracy. So far, I've written 3 pixel shader examples. Nothing fancy. I've also written some code to output the contents of the structure. They vary from shader to shader. Here's the results.

shader 1:
xps.1.0 // Xbox PixelShader

mov r0, v0 // Move the diffuse vertex colour to the
// output colour register (r0).
output:
PSAphaInputs[8] = 0xD4301010 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSFinalCombinerInputsABCD = 0x00000000
PSFinalCombinerInputsEFG = 0x00000000
PSConstant0[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSConstant1[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSAlphaOutputs[8] = 0x000000C0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSRGBInputs[8] = 0xC4200000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCompareMode = 0x00000000
PSFinalCombinerConstant0 = 0x00000000
PSFinalCombinerConstant1 = 0x00000000
PSRGBOutputs[8] = 0x000000C0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCombinerCount = 0x00011101
PSTextureModes = 0x00000000
PSDotMapping = 0x00000000
PSInputTexture = 0x00000000
PSC0Mapping = 0xFFFFFFFF
PSC1Mapping = 0xFFFFFFFF
PSFinalCombinerConstants = 0x000001FF

shader 2:
xps.1.0 // Xbox PixelShader

tex t0 // Declare texture register t0 (Texture Stage 0)
mul r0, v0, t0 // Multiply v0 and t0 the output register (r0)

output:
PSAphaInputs[8] = 0xD4D81010 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSFinalCombinerInputsABCD = 0x00000000
PSFinalCombinerInputsEFG = 0x00000000
PSConstant0[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSConstant1[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSAlphaOutputs[8] = 0x000000C0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSRGBInputs[8] = 0xC4C80000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCompareMode = 0x00000000
PSFinalCombinerConstant0 = 0x00000000
PSFinalCombinerConstant1 = 0x00000000
PSRGBOutputs[8] = 0x000000C0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCombinerCount = 0x00011101
PSTextureModes = 0x00000001
PSDotMapping = 0x00000000
PSInputTexture = 0x00000000
PSC0Mapping = 0xFFFFFFFF
PSC1Mapping = 0xFFFFFFFF
PSFinalCombinerConstants = 0x000001FF

shader 3:
xps.1.0 // Xbox PixelShader

tex t0 // Declare texture register t0 (Texture Stage 0)
tex t1 // Declare texture register t0 (Texture Stage 1)
mov r1, t1 // Move texture register t1 into output register r1
lrp r0, v0, t0, r1 // Lerp between t0 and r1 by proportion
// specified in v0
output:
PSAphaInputs[8] = 0xD9301010 0x14D8DD34 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSFinalCombinerInputsABCD = 0x00000000
PSFinalCombinerInputsEFG = 0x00000000
PSConstant0[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSConstant1[8] = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSAlphaOutputs[8] = 0x000000D0 0x00000C00 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSRGBInputs[8] = 0xC9200000 0x04C8CD24 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCompareMode = 0x00000000
PSFinalCombinerConstant0 = 0x00000000
PSFinalCombinerConstant1 = 0x00000000
PSRGBOutputs[8] = 0x000000D0 0x00000C00 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
PSCombinerCount = 0x00011102
PSTextureModes = 0x00000021
PSDotMapping = 0x00000000
PSInputTexture = 0x00000000
PSC0Mapping = 0xFFFFFFFF
PSC1Mapping = 0xFFFFFFFF
PSFinalCombinerConstants = 0x000001FF

So obviously enough, the contents of the structure will vary on the contents of your shader. Quite frankly, it reminds me of .fx files. They are very convienent to use, but once again, not available in D3D8.

Now that I think about it, there is one more option for this... Cg! IMO Cg is awesome shader language (I don't understand why so many hate it) and it supports Direct3D 8.1 too! It's not very different from HLSL either. Although it should work fine on ATI cards, it will make Cxbx much more NVIDIA oriented than it already is.

So, that's what I've been working on lately. I really hope that I can get this pixel shader stuff working sometime. Turok is looking a bit "lifeless" without it's lighting shaders emulated properly. Lack of pixel shaders also limit Cxbx's bumpmapping support too. I'll be sure to keep you all posted on what's going on with this latest endeavour.

Shogun.

EDIT: Sorry, blogger is not very code friendly.

I'm back, but I need a favour (for those who can contribute).

Yup, I'm back to work on Cxbx these days. I'm still rather busy, but I can still make updates from time to time. Since the internet is down at my place, I might not be able to update this blog as much as I want to (I'm at the library). Rest assured, I still have lots of things planned.

The day after Christmas I started working on Cxbx again for a bit. Nothing major, but I did get a few noises out of Azurik's Xbox Demo xbe. It's probably not working due to lack of launch parameters (which means Azurik needs to attempt to execute it first). What's the status of this game? It's still not doing anything yet (one of my old betas got it in the Nothing status). So far, Azurik appears to be the H-A-R-D-E-S-T Xbox game to emulate, and if it ever works, I'll die a happy emu author. What it did do is further remind me how lacking XDK 3911 support is. Signatures for XDK 3911 have been rarely added from anyone lately. I've managed to add a few, but the changes from 3911 to the closest XDK I have (4361) are just to great. IMO, XDK 3911 wasn't very organized when it released it, but eventually got a better flow of organized code structure. Adding DirectSound stuff for this XDK has always been a pain to do by hand (especially if you don't have that XDK).

So anyway, I have a few favours I want to ask you all.

1. Does anyone have XDK 3911? It was abundant at one point, but now it's impossible to find. Even the lib files will do okay. I never could get it back then because I didn't have access to broadband (56k only). If you ever want to have hopes of playing DOA3, please ask/look around :)

2. Can anyone generate FLIRT signatures for IDA? I had a copy of the FLIRT SDK, but I don't know what happened to it. This will make working on Cxbx much easier so I can identify missing functions right away.

I really hate to ask favours of people, but at these times, I really need them! Thanks.

Shogun.

Good news and bad news...

Yes, it's that time again. The dreaded "good news and bad news" post... here's what's going on:

Bad: After starting my new job (judging by the hours I'll be working on a frequent basis), I won't have any time to work on Cxbx probably except on Sundays (and maybe not even then) until after Christmas time. So just because you won't see as many frequent updates this month doesn't mean I've stopped working on Cxbx. If I stop, I'll tell you first.

Good: Thanks to an anonymous source, I've got some new XDKs! 5659 and 5788 to be exact. I haven't found many that uses either one, but they'll come in handy later, I'm sure. Also, I'm sure I'll be able to talk Caustik in making a release before Christmas too. You've all been waiting 5 years for a new release, and it's about that time. Just remember, it will be a *BETA*, meaning it's probably going to have random bugs in it. Also, not every game on the compat list is guaranteed to work for everyone due to region differences, okay? For instance, one version of DOAV might give you the dirty disc error, and another version will not so now you know.

Unfortunately, it's about time for me to go back to work. *Sigh*, I just got off of work!

Shogun.

More on Unreal Championship

If you've read my last update on Unreal Championship, you'll have a good understanding by now what the situation is. If not, you might want to read that first :)

So, now what's going on? So far, this is the most interesting situation I've come across when working on this emulator. The solution to it might be even more interesting (or annoying). I'm not really sure which would be the best way to go from here, but I do have two ideas for this scenario (to handle the problem with the xbe trying to re-launch itself). When XLaunchNewImage is called (and when the .xbe being called is itself) and the LaunchData parameter is saved, we can either:

1. Trigger some sort of "restart" process inside the CxbxKrnl so everything except the necessary things are in the initial state at which the game was started. Then jump right back to the game's main entry point and continue as normal. Optionally "Persist" the display until D3DDevice::Present is called.

2. Save the contents of pLaunchData to a file in the same directory of the game (it's size will always be 3KB). Then load the file and restore certain states the next time Cxbx is running the same game. This will require you to restart Cxbx each time you want to go in-game.

Remember, I said I'm looking for the "best" way to handle this, not the most "ideal". I know everyone would rather have me do number 1, but so far I've only had time to try number 2. This is what happened...

Rats! To a certain extent, #2 does work, but it does have one small problem. The controller needs to be re-initialized so that the game will recognize that it's still there. This will require a bit of hacking. So far, it appears that everything else worked fine. All the files required to load the map, etc. were loaded without problems. But so far, it looks like #1 is harder to implement and #2 requires more extensive hacking. Maybe a simple jump could fix everything (for #1)! Since I'll be busy tonight and all day tomorrow, there's no telling when I can actually try it. Hopefully Caustik can jump back in sometime and give a few suggestions, but unfortunately, it appears that I'm the only one on the Cxbx team that has this game.

I'd really like to get more in-depth on this, but atm I have much work to do IRL so just stay tuned and I'll see what I can do. There's a lot that can be done with this game (once it works), but you'll just have to see if/when that happens :)

Shogun.

Unreal Championship progress...

Yeah, it's been a while since I've worked extensively on this game, but yesterday I had lots of time on my hands so I went at it. Before I go any further, I'd like to thank Chrono Archangel find a certain XNet function (XNetGetEthernetLinkStatus). Without his l337 google skills, I might not be posting this right now!

Unreal Championship goes comfortably to the menus (and is more interactive) with little or no problems. So far this is only for the NTSC version. It required a small handful of hacks, but so far it looking good. Since the game uses time based updates, you don't have to worry about it going to fast when VSync isn't enabled :) I also learned a few new tricks when emulating this game! Even though UT2003 for PC is practically the identical in many ways, IMO this game is too different so I say it's safe to consider this game an Xbox exclusive (that's one reason why I own both).

If you look at the screens, you'll notice that it looks as if termites have been eating some of the textures. "What is it?" Not 100% sure, but my guess is either it's another swizzling issue, or it's a palettised 8-bit texture and the conversion from D3DFMT_P8 -> D3DFMT_A8R8G8B8 isn't perfected yet. One thing I noticed was that so far, XACT wasn't really used much after loading one wave bank (which was surprising), so I might not have to implement the whole XACT API after all (which would be a major sigh of relief).

Now for the list of hacks needed. The first one I should have mentioned before. Sorry to those who tried running this game on my branch before with no success.

1. At one point, I was trying to add a missing DirectSound API and I couldn't generate a digital signature for it. So what I did was I made a copy of my xbe and filled that function call with NOPs using a hex editor.
2. Disabled overlay updates so that the second video can play. The first video works without the hack, but when the second one plays without the hack, it will cause a crash inside of the DirectDraw DLL. I couldn't think of a better solution at the time so I did this.
3. For some reason, Unreal likes to lock it's textures at each level until it fails. So I had to tell EmuIDirect3DTexture8_LockRect to fail when the texture level is higher than 5.
4. This last hack probably won't work for everyone, but I added a hack in EmuIDirect3DDevice8_SetIndicies to stop it from crashing because the IndexBuffer pointer appears to be bogus. If you get a CreateIndexBuffer failed error when trying to go passed the first menu screen, try it again.

These were some dirty and totally ugly hacks used (and I had a feeling that this was going to happen anyway). Oh well, as long as it works, right?

"So, what's stopping it from getting us in game?" Okay, here's where it get's complicated. It's bad enough that Unreal Championship uses features that are rarely used in Xbox games (or should I say features that we're not used to emulating for Cxbx?) which makes things a bit more complicated, but it's biggest feature is getting to us. Unreal Championship was the first console game in history that supported [executable] patches for the game. This means that in order to run that patch, it has to execute a new .xbe file. Cxbx is currently not capable of doing that. But what if it's not there? What it appears to be doing is loading the default.xbe over again if it isn't. It may have been designed like this as a cheap way of clearing most data from memory. So, that's the problem... but what about a solution? Here's an idea. I added the function XLaunchNewImage today to see what it was trying to do. Just as I stated above, it's just re-executing itself. Wouldn't that just start everything over? No, not really. Fortunately, the XDKs documentation on XLaunchNewImage is quite adequate, well written and very informative. Since an Xbox .xbe's entry point (void __cdecl main()) does not take any parameters (i.e. no command line), they have to be set using XLaunchNewImage. The function XGetLaunchInfo is used to retrieve the the data set by XLaunchNewImage. Since Unreal appears to be going back to the entry point after attempting to reload default.xbe, there's a chance that we can fool it by saving the params from the last call from XLaunchNewImage so that they can be retrieved by XGetLaunchInfo. This might work, and it might not. It's worth a try, and so far, I don't see any other options as of now. I'll give it a try, but just know this. I can't guarantee that I can get this one in-game. I can only guarantee my best efforts!

One more thing, I'm looking for anything on the .xbx file format. What is an .xbx file? It's a customized texture format for the Xbox. Since Unreal Championship calls XGWriteSurfaceOrTextureToXPR to create it's icons, it would be ideal to be able to create this functionality just in case the icon is actually used. So if you can find any tools, have any source code that deals with this file format, let me know!

Before I conclude this update, I just want to thank all the Cxbx supporters and fans that have been keeping up with us for all this time. Chrono Archangel, Mr. Fabulous, saintseiya, nokiaman, and more, I really appreciate you all!

Shogun.