The macOS Experiment (Part 2)

Around the beginning of the new year, I mentioned what I titled "The macOS Experiment" and some of what I had planned for it.  Over the weekend, I wanted to put a bit of effort into it and see what would happen.  Did I get it working?  Well, almost.  And I'll get to that in a moment.  First, I'll share the git repo for those that want to see what I'm talking about:

https://github.com/blueshogun96/MacBox

There isn't much there but the bare minimum, and it was crappily written (no need to point out the obvious) because I just wanted to quickly see if I could get it up and running.

Basically what it does is this: It serves as a way to launch 32-bit code from the desired base address (for the sake of Xbox, 0x10000) by calling mmap to reserve that desired memory location and range as well as give us read, write and execute permissions on it, then injects the code to be run at the given base address (in this case, a .xbe file).  From there, we point to the .xbe file's main function (not the entry point, which is what Xeon did) with a function pointer and call it.  Actually before we call that function, we need to insert our wrapper functions to HLE the necessary functions in the .xbe.  For now, I just hard coded those to see if I could get a quick proof of concept going.

So to start off, I took the simplest .xbe to deal with, the first D3D tutorial in the XDK, CreateDevice.  All it does is create the D3D device, clear the screen to a random colour, then present it to us.  This is actually simpler than it sounds, considering that you don't have to literally create a D3D device.  All the D3D functions are C-based functions using the __stdcall calling convention.  The only parameter to even worry about there is D3DPRESENT_PARAMETERS anyway to initialize an OpenGL context that we want to match those parameters.  Clearing and swapping the screen are also very trivial things, but as you can see, I didn't literally add any of that code.  Why is that?  Because I needed to make sure that my hooked functions get called correctly, and this is where things got a bit tricky.

Let's take a look at this code where the "magic" begins (see macbox_xbe.cpp on the repo):

void macbox_install_wrapper( void* function_addr, void* wrapper_addr )
{
    uint8_t* func = (uint8_t*) function_addr;
    
    *(uint8_t*) &func[0] = 0xE9;    // JMP rel32 (quick and easy)
    *(uint32_t*)&func[1] = (uint32_t) wrapper_addr - (uint32_t) function_addr;
}

Just like Cxbx does, my code goes to the top of the function that we need to hook, and places a jmp rel32 instruction right there to immediately redirect it to our wrapper function.  Quite easy to do when you know how to encode x86 instructions.

So let's move on.  Next, let's hook some functions:

/* TODO: Move this elsewhere and don't hard code it either */
macbox_install_wrapper( reinterpret_cast(0x195B0), reinterpret_cast(Direct3D_CreateDevice) );
macbox_install_wrapper( reinterpret_cast(0x1A270), reinterpret_cast(D3DDevice_Clear) );
macbox_install_wrapper( reinterpret_cast(0x1ABC0), reinterpret_cast(D3DDevice_Swap) );

How did I get these offsets?  I loaded the .xbe up in IDA Pro using the flirt signatures to spot the D3D functions right away.  It really saved my arse and my sanity when working on Cxbx.

After this, I was ready to go, or at least I thought I was.  When Direct3D_CreateDevice was first called, the parameters contained all gibberish.  On top of that, it crashed soon after the function returned.  I made sure that all the parameters were correct and the same as they were for Windows (at least, from a byte perspective).  But then I remembered that these functions have to use __stdcall in order to work on Windows for Cxbx.  So I had to find the equivalent for Mac.  This is what WINE used to define __stdcall (or also known as CALLBACK) for Mac:

#define CALLBACK __attribute__((__stdcall__)) __attribute__((__force_align_arg_pointer__))

Kinda long, isn't it?  So I used this, and it caused the immediate crash to go away.  The parameters are still gibberish for that particular function (Direct3D_CreateDevice) but with D3DDevice_Clear, the parameters were perfect!  Then after that function returned, something went wonky and the EIP ended up in a weird state resulting in another crash.  This brings me to the next issue.

From what I have read, the stack alignment in macOS is always 16-byte aligned, whereas with Xbox code it was 4-byte aligned.  This seems like it could be the issue.  I took a moment to get some assembly code screen shotted.

From IDA, this is the code that calls Direct3D_CreateDevice:



And this is the code that XCode generated...


Once more, IDA's output from whence D3DDevice_Clear was called:


And this is what XCode generated...


So the green line is actually where the breakpoint is placed at the beginning of the function.  So is the stack getting screwed up somewhere or what?  I'm quite sure there's a way around it considering that WINE was able to pull it off somehow.  Maybe I should ask them on their forum?  Well, I want to figure something out.

Since I didn't have all day to dedicate to this, I thought I'd at least share with you all what's going on.  The sooner I can fix this, the sooner I can move forward.  Frankly, I wouldn't be surprised if it was stupidly simple what I was missing.  If worse comes to worse, I guess I could do something super hacky to fix it, but I want to avoid that if I can.

So that's it for now.  Thanks for reading, and let me know what you think.

Shogun.