Showing posts with label XQEMU. Show all posts
Showing posts with label XQEMU. Show all posts

Monday, January 2, 2017

The macOS experiment (Part 1)

Lately, I haven't been doing much emulation wise.  Well, to be frank, I haven't done any serious emulation work in a long time (and just incase you're reading this JayFox, I'm not talking about xqemu, nor have I ~ever~ said I was doing any real work on it aside from a few work arounds here and there; hell I've even stated that I'm not an xqemu dev multiple times, so pleeeeeeeeeease spare me of your usual "3 lines of code" lecture because I'm really not in the mood, thanks and no disrespect), but since I'm in between jobs/paid projects yet again, I decided to pick up one of the previous experiments I started before out of curiosity.

What is this experiment?  Well, the goal of this little coding experiment was to claim the typical base address of 0x10000 that every .xbe uses.  For those who don't quite see what I'm getting at, let's take a look at how both Cxbx and Xeon, the first two Xbox emulators, worked from an internal perspective.  They both use HLE, yeah that's right sherlock, in order to run code natively on the host CPU, they both use a .exe to reserve the memory range beginning at 0x10000 and at least 64mb beyond that.  For Cxbx, the generated .exe from the .xbe has the base address set when the header is written.  For Xeon, the main .exe simply reserves that memory address range by forcing it to load at 0x10000 w/ a big global static array and it's overwritten with the .xbe contents.  The actual emulator code is run from a .dll that's loaded into memory well out of that range.  That's the only way they knew how to do it at the time (for windows at least).  Naturally, I started to wonder how would one do that on a Mac?

Since similar experiments for this have already been done on Linux, so I figured Mac could do it also with similar methods.  So I started a thread on ngemu a long while back, and asked for suggestions, hints and ideas.  At the time, I was quite new to Mac development, but at the time of writing this, I have grown to be quite experienced with it (although still not quite as knowledgeable of the lower level and kernel level aspects).  Instinctively, my first try was to use mmap() directly, or vm_allocate().  That didn't work.

Among one of the respondents to my thread back then was Ben Vanik, the author of Xenia.  He was running linux, and recommended this code:

#include 
#include 

namespace {
    void * const MEMORY_ADDRESS = reinterpret_cast(0x10000);
    size_t const MEMORY_SIZE = 1 << 20;
}

int main() {
    void * p = mmap(MEMORY_ADDRESS, MEMORY_SIZE, PROT_EXEC | PROT_READ | PROT_WRITE, MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
    if (p != reinterpret_cast(-1)) {
        std::cout << "Memory allocated" << std::endl;
        munmap(p, MEMORY_SIZE);
    } else {
        std::cout << "Memory could not be allocated" << std::endl;
    }
}

It worked for him, but still didn't work for me on macOS (OSX), so I was still back where I started.  Keep in mind that all these flags were necessary.  I needed to be able to read, write and execute from this memory range.  It also needed to be at that exact address (hence the MAP_FIXED flag), and so forth.  Many were against that last bit, which I'll explain in a moment.

After that, I eventually forgot about it and moved on to other projects.  Mostly working on this indie game that I started less than 2 months of starting that thread.  I eventually did get it working, and the reason it wasn't working was actually quite a testament to my ignorance and stupidity... I forgot to switch the compiler setting from x86_64 to i386!  Duh, no wonder it wasn't working.  I wanted to slap myself afterwards.  IIRC, there's a flag called MAP_32BIT which allows 64-bit programs to access the 32-bit memory address space, but it appears to only be available in Linux, not macOS.  Oh well, no big loss for now.

Now, what I mean by "work" is that mmap() stop failing.  I would finally get that base address, but it would simply crash on a call to std::cout.  Even commenting this out, unmapping the pointer and letting it return from main also resulted in a crash.  I'm sure you could theorize why, but finding a solution was what I was interested in.

Since there weren't many Mac devs on your every day emulation/gaming forum (most of them are extremely anti-Mac; many for ignorant and uninformed reasons, and few for legitimate and well thought out reasons) so I ended up going to a forum that was dedicated to Mac related programming topics; macrumors.com.  While most of them did give me some sound advice, they didn't quite understand why I needed this exact functionality.  Many would suggest removing MAP_FIXED but that would give me a memory address out of the range I'm looking for and would essentially defeat the purpose.  I had to start two threads altogether, and only one person actually understood what I was trying to do, as he was also in the need to write a basic VM while lacking the proper resources to do it.  He tried this code in XCode, and it worked for him, but only after adding the following compiler flag in the "other linker flags" section:

-pagezero_size 0x10000000 -segprot __PAGEZERO rwx rwx

Frankly, I do not fully understand what this did, even though it generally looks pretty obvious.  A google search didn't yield much results either, but it worked.

Now, I understand that what I'm doing is considered "unsafe" and risky.  Even on Linux, it is considered the same to do so.  It's all part of some experiment I put a bit of free time into.  Was there any particular goal to it?  Well, for starters, I wanted to see at one point whether it were at all feasible or possible to bring Cxbx to Mac since using WINE wasn't enough to do it due to the memory map requirements (my assumption; it always crashed for me).  Another idea was to provide proof of concept that HLE and direct code execution was indeed possible on an Intel Mac without the need for a VM library or driver.  There's a VM framework for macOS now, but it requires a 2010 Mac with 10.10+ installed.  Since neither of my comps meet the former requirement, that wasn't an option for me.  Third, I've had the itch to implement just enough APIs in an attempt to run Azurik: Rize of Perathia on my Mac.  I am greatly intrigued by the possibility of playing this game in some hacked form of 1080p or 4k using certain Apple exclusive OpenGL extensions, but hesitant to re-live the pain of trying to get this game working on Cxbx.

So far, it appears to work without issues, but I'm sure there could be some hidden caveats somewhere.  What I'd like to do later on is experiment with a simple .xbe that simply creates a D3D device, and clears the screen and HLE that as proof of concept.  I don't really plan on going too much farther than that, but I am curious to find out if there's anything else that can evolve out of this, even if it doesn't mean emulating Xbox.  Who knows?  Maybe an HLE emulator of something else?  Like one of Sega's recent arcade hardware machines like Europa?  Hey, I just like to learn, and emulating Xbox even on an HLE level has helped me tremendously personally and in my professional career (which is part of the reason I have this job interview coming up; I'll blog about what I mean about this statement later).

I know it's late but happy new year, and happy coding.  It's late so I'm off to bed.

Shogun.

Sunday, August 30, 2015

Jet Set Radio Future


Do you see this?  This is a video of XQEMU running JSRF (courtesy of JayFoxRox).  Looks pretty darn nice, doesn't it?  I'll say.

I know you all want to play it, and so do I.  Now please, pllllllllllllleeeeeeasssssssssseeeeee...... STOP ASKING ME ABOUT IT!!!  lol

Shogun.

Sunday, July 12, 2015

More XQEMU Progress!

Once again, I dare not take credit for most of these (nor would I take full credit for what I did fix), but I spent my Saturday afternoon fooling around with XQEMU, and this is what I managed to do/discover.  I noticed a missing/broken feature or two in the code today, so I went in there, and fixed 'em to the best of my ability.  What happened?  I ended up getting Namco Museum ingame, and Panzer Dragoon 1 is now playable.

I took a video of these two games in action.



JayFoxRox informed me that he fixed the flipping issue that's been going on in some games like Namco Museum.  Not 100% sure what the issue with Panzer Dragoon is, but for now, just enjoy the feeling of playing this game on acid!

Also, I've got to admit, this emulator is progressing much faster than Cxbx has, and the compat list is growing rather fast.  I tested and tinkered with a handful of other games, and I'll update my compat list later because it's 4am, and I'm tired.... and yes, I tried JSRF, it doesn't work yet.

Well, that was a weekend well spent.  Good night everyone.

Shogun.

Monday, July 6, 2015

A quick guide to building XQEMU

Now, this emulator is not easy to build, especially if you don't know what you're doing!  When I first started getting my hands dirty with this one, I had a rather tough time getting it to build due to dependency hell, missing tools, and missing source files.  Now that I have another Mac with better specs in general, I wanted to go about setting it up on this machine.  This time, it's going much smoother, but I think having a more detailed guide on how to set it up would be helpful.

A word of warning: so far, XQEMU appears to work best on MacOSX, which is the OS I will be using for this guide.  JayFoxRox successfully got this built on Linux before (I haven't tried this with espes' branch); but his branch has some files exclusive to him for legal purposes.  Windows?  I hear it's tough, but doable.  If you have any expectations of playing JSRF or Conker right out of the box, chances are you'll be disappointed (but don't let that stop you from trying).  Also, if you don't know how to use a Terminal, git, or even a command-line compiler, chances are you shouldn't be attempting this or even using this wip emulator to begin with.

Now, let's get started.  First things first, always take a look at the wiki: https://github.com/espes/xqemu/wiki/Getting-Started

Well, that Wiki hasn't been updated in almost 2 years though... O_o

If you haven't already, create a folder, navigate to that folder using the terminal, then do a 'git clone' to get the source code from the git.  Once you do that, you need to check two folders within the source folder, and those are:

- dtc
- pixman

If those folders are empty, then you need to clone those sources independently.  You can do so with the following commands while in the source directory:

git clone git://git.qemu-project.org/dtc
git clone git://anongit.freedesktop.org/pixman

So, if you were missing those before, you should have them now!  And now that you have all the source files needed to build XQEMU, and if you want, go ahead and enter this terminal command:

./configure --cc=clang --enable-opengl --disable-vnc --disable-user --disable-kvm --disable-xen --with-coroutine=sigaltstack --target-list=xbox-softmmu --enable-debug

Now, if you got the same problem I had, you will likely run into an error message like this one:

ERROR: pkg-config binary 'pkg-config' not found

There's multiple ways to fix this.  You can either use macports or homebrew.  I chose macports personally.  So I downloaded MacPorts for my OS of choice, and went from there.  To install pkgconfig using MacPorts, use the following command:

sudo port install pkgconfig

There's a possibility you got this error message here:

sudo: port: command not found

If you did, then you need to setup your path correctly.  Try using this command, it worked for me:

export PATH=$PATH:/opt/local/bin

Now, try the previous command again, and it should work.  Okay, now we have that annoyance out the way.  Optionally, you can also install the XCode command line tools if you want or haven't done so already.  To do so, type this:

`xcode-select --install'

It will take a minute or two, and will have a nice GUI pop up instead of terminal text.  At this point, go ahead and try configuring XQEMU again.  If you get this following error pertaining to glib:

ERROR: glib-2.12 required to compile QEMU

That means we need to install that too.  Once again, let's use MacPorts to get this out of the way, like so:

sudo port install glib2

Okay, that'll do it.  Try configuring XQEMU one more time, and it should do so successfully.  If not, then there's some other dependency you are missing.  On Yosemite, I haven't gotten any more errors pertaining to such missing dependencies like Cairo, pixman, etc.  If you do get those in particular (I did), make sure you have the pixman source in the appropriate folder as I mentioned towards the beginning of this article.  If it's something else, feel free to ask me, but I can't promise you a direct answer/solution, especially if you aren't trying this on MacOSX Yosemite.

The next thing you want to do is type make, and xqemu should start compiling.  While compiling, I got an error regarding autoreconf that looked like this:

/bin/sh: autoreconf: command not found

If you get that, let's fire up MacPorts again, like so:

sudo port install automake autoconf libtool

Bam, fixed.  Try compiling it again.  After this, I got a linker error pertaining to pixman, telling me that _prng_state is missing, and other stuff.  Click this link (http://lists.freedesktop.org/archives/pixman/2014-November/003461.html), and it will tell you how to edit the appropriate makefile in the pixman source folder in order to get passed this annoyance.

Okay, if you followed the instructions, then you should have a working build now.  If you are getting a linker error regarding SSE4.2, like the one below:

error in backend: Cannot select: intrinsic %llvm.x86.sse42.crc32.64.64

Then you'll have to comment out the code in nv2a.c that uses it. Search for the following function:

static uint64_t fast_hash(const uint8_t *data, size_t len, unsigned int samples)

Then uncomment the macros, and change the top macro from '#ifdef __SSE42__' to '#if 0'.  Type make again, and that should fix it.  Place the appropriate BIOS files in the right places and you're good to go (of course I'm not going to tell you where to download them here!), now fire up the emu from the terminal and see what you can come up with!

These are all the issues I encountered when building XQEMU the 2nd time around.  If you have different issues, feel free to ask questions below, but remember that I'm NOT going to hold your hand and cater to you personally.  These are things that I figured out for myself simply by googling until I found a solution, all in a few hours.  No offense, but if you can't figure some things out for yourself, then you shouldn't bother.

References:
- https://github.com/espes/xqemu/wiki/Getting-Started
- http://www.fantageek.com/318/install-pkg-config-for-mac-osx/
- http://mikelev.in/2012/10/qemu-mac-os-x-success/
- http://stackoverflow.com/questions/9575989/install-autoreconf-on-osx-lion
- http://lists.freedesktop.org/archives/pixman/2014-November/003461.html

Have at it,

Shogun.

Tuesday, June 30, 2015

Someone is Stealing my Identity!

Okay, this is pretty outrageous.  Who the hell would go this far as to impersonate me on reddit?  Wtf?  Yes, you heard right, someone is pretending to be me, don't be fooled...

Thanks to yun-daddy, I caught wind of this yesterday (maybe he did it, lol).  I didn't get around to reading it before because I was busy (yes, I was actually working on Cxbx for a change), so I had initially assumed someone was making a post in my honour or wanted to inquire about my past work.  Turns out that was not the case.  I should have investigated before things snowballed into what it is now.

Link: http://www.reddit.com/r/emulation/comments/3bhisz/blueshogun96_here_author_of_the_original_xbox/

For those of you who want proof that this is fake, consider these.
  1. I haven't been going to school at all.  In fact, I haven't set foot in a university since 2007.
  2. I stated that I don't have a job earlier on my blog.  Btw, for those of you who were wondering how my interview with 343 went, I didn't get the job because they wanted a more experienced C# dev.
  3. When I make announcements or just want to speak my thoughts on Cxbx, I do it here first, then link it other places.  In fact, I've never had a reddit account in my life until today (and my nickname there is "real_blueshogun96").
  4. When did I EVER claim to be the author of Cxbx or XQEMU?  Never.... Not.  Even.  Once.  These were originally written by Caustik and espes, respectively.
  5. I never even stated that I would have a definite means of working on Cxbx.  My retro gaming PC which I mentioned yesterday isn't truly ideal to work on it.  XQEMU is another story though, but I'd like to work on getting the setup working on my Mac Pro aside my Macbook.
  6. His impersonation of me is terrible.  I sign my statements as "Shogun", I never refer to my favourite console as "OG Xbox" because there's nothing "Original Gangsta" about Xbox.  Lol, I had to make some fun out of this.
 Looking on the bright side, I must really be special if someone is going out of their way to impersonate me of all emu devs.  Besides, they say it pays to advertise!  But in the mean time, I'm going to start another reddit post regarding this imposter as well as link back to this blog post as proof.

So, now that you are aware of all this, don't be fooled.

Shogun.

Tuesday, June 10, 2014

XQEMU Compatibility List (Updated July 21, 2015)

Okay, I've gotten a few requests/inquiries for XQEMU's compatibility, so I'm going to write and maintain a compatibility list.  Personally, I'd like to keep it focused on commercial games right now.

The following is a representation of progress made with all 3 major branches between espes, JayFoxRox, and myself.

IF IT'S NOT ON THE LIST, THEN IT'S LIKELY UNTESTED!  I CAN'T STRESS THIS ENOUGH!

Total Games: 58
Playable: 2
Ingame: 11
Menus: 14
Intros: 16
Nothing: 15


1. 4x4 Evo 2
Status: (Ingame) (*)

2. AMF Bowling
Status: (Intros) (**)

3. Conker: Live and Reloaded (Demo)
Status: (Intros)

4. Dave Mirra Freestyle BMX (Demo)
Status: (Intros)

5. Forza (Demo)
Status: (Intros) (**)

6. Rollercoaster Tycoon
Status: (Menus) (**)

7. Smashing Drive
Status: (Playable)

8. Sonic Heroes (E3 Demo)
Status: (Ingame) (*)

9. Tony Hawk's Pro Skater 2X (Demo)
Status: (Intros)?

10. Wreckless: The Yakuza Missions (Demo)
Status: (Ingame) (*)

11. Jet Set Radio Future
Status: (Intros)

12. Panzer Dragoon
Status: (Playable) (*)

13. Panzer Dragoon ORTA
Status: (Menus)

14. Antz
Status: (Ingame) (*)

15. Aquaman
Status: (Ingame) (*)

16. Apex
Status: (Nothing)

17. Azurik: Rize of Perathia!
Status: (Menus)

18. Blinx
Status: (Nothing)

19. Broken Sword
Status: (Nothing)

20. Brute Force
Status: (Intros)

21. Castlevania: Curse of Darkness
Status: (Nothing)

22. Cel Damage
Status: (Menus)

23. Conflict Vietnam
Status: (Menus)

24. Crazy Taxi III
Status: (Ingame)

25. DaiSenryaku
Status: (Nothing)

26. Dead to Rights
Status: (Intros)

27. DeusEx
Status: (Intros)

28. Dead or Alive 3
Status: (Nothing) (***)

29. Dead or Alive Xtreme Beach Volleyball
Status: (Nothing) (***)

30. Dragon's Lair 3D
Status: (Nothing)

31. Dynasty Warriors 3
Status: (Intros?) (***)

32. EggMania
Status: (Intros?) (***)

33. Finding Nemo
Status: (Nothing)

34. Fusion Frenzy (Demo)
Status: (Menus)

35. Gun Valkyrie
Status: (Menus) (*)

36. Gun Griffon Allied Strike
Status: (Menus)

37. Halo (Demo)
Status: (Menus)

38. House of the Dead 3
Status: (Nothing) (**)

39. Innocent Tears
Status: (Ingame) (*)

40. Kameo (Beta)
Status: (Intros?) (***)

41. Lamborghini (Demo)
Status: (Nothing)

42. Mademan (E3 Demo)
Status: (Nothing)

43. Oddworld: Munch's Oddyssee (Demo)
Status: (Menus)

44. Myst III
Status: (Intros)

45. Nakashima Tetsunari no Othello Seminar
Status: (Intros)

46. Namco Museum
Status: (Ingame)

47. Otogi
Status: (Menus)

48. Outrun 2
Status: (Intros)

49. Petit Copter
Status: (Menus)

50. Pinball Hall of Fame
Status: (Intros)

51. Quantum Redshift (Demo)
Status: (Menus)

52. Rayman Arena
Status: (Ingame)

53. Shenmue II (Demo)
Status: (Nothing) (***)

54. Toe Jam and Earl III (Demo)
Status: (Nothing)

55. Unreal Championship
Status: (Ingame) (*)

56. Unreal Championship 2
Status: (Intros)

57. Whacked
Status: (Nothing)

58. Zapper
Status: (Ingame)

(*) Distorted gfx
(**) Hangs or crashes
(***) No idea what it's doing (flashy or distorted colours, etc).

Now, I won't stop anyone from making requests, but on a serious note, I can't cater to everyone's personal preferences at once.  This list will be updated periodically, so check back every so often.

Shogun.