Iyfjky

Question

Based on my understanding, an emulator is necessary because the machine with the emulator on it (say Windows), doesn't execute the same machine code as the target platform (6502, for example). So what an emulator does is it interprets the code line by line and executes it itself, similar to how an interpreter works.

With this in mind, would it be possible to create the equivalent of a compiler, that would convert (for example) a .PCE file into a .exe file, so that it could run on Windows without an emulator?

You can write a program that could convert a ROM to an EXE, but it would just be bundling the emulator with the ROM in the same EXE file. — 1 hour ago
You would still need to include the emulator code somehow as you need to emulate the hardware of the target platform. So yes, I'm sure it would be possible to have a single .exe, but it would just be a combination of the emulator and the ROM data — 1 hour ago
If you were working at only the CPU level, you could write a 'compiler' to convert 6502 instructions into the equivalent x86 instructions. This could work but gets flakier as complexity increases. You might be able to write a loop in 6502 that counts to 100 and exits, and 'compile' those instructions into x86 code that also counts to 100 and exits. But the moment you tried to 'print the results' you would become platform dependent, and now you need an I/O system, and now your 'compiler' doesn't work anymore. — 1 hour ago

score 3 · Accepted Answer · 2018-11-01 18:30:13Z

^{Note: This answer mainly focuses on the NES, since that's what I'm most familiar with.}

Yes; this is called static recompilation or static binary translation, and it can be done -- jamulator by Andrew Kelly does it.

However, there are some serious challenges involved. Determining the behavior of a program by statically analyzing its code is in some cases provably impossible. Some of the problems faced when statically recompiling a retro video game ROM include:

How do we determine what parts of the ROM are executable code and what parts are data?

The "obvious" way to recompile a NES ROM is to take a single pass, reading a stream of 6502 instructions and outputting a stream of x86 instructions. However, the ROM also includes data bytes, which will produce garbage if interpreted as instructions. These garbage instructions should never be executed, since the program would never jump to that address, but the presence of non-instruction data poses two problems:

1) The program must be able to read its own ROM. This isn't really a hard problem at all; we just need to include an uncompiled copy of the ROM in the compiled program binary to read data bytes from.

2) We can't write a simple single-pass recompiler. Since instructions can be multiple bytes wide, if we try to recompile a block of non-executable data, we might end up misaligned when recompiling the next block of instructions. (As a simple (somewhat contrived) example, in 6502 machine code, the sequence of bytes 69 09 0A is ADC #09; ASL, while the sequence of bytes A5 69 09 0A is LDA $69; ORA #0A. The point at which we start executing drastically affects our results).

So this means we have to perform much more complex code analysis in order to determine the basic blocks of the program and compile them individually. jamulator does this by starting at the interrupt vectors and following all possible branches from there. However, this approach still has problems with:

Dynamic jumps

This is where the address to execute is computed at runtime. In some cases, the runtime-computed address can be looked up and the corresponding x86 code can be found. However, if the only way to access a basic block is via an indirect jump, the recompiler would likely have assumed that block was not executable code and thus not compiled it.

jamulator mitigates this problem using a hack: hard-coded support in the recompiler to recognize jump table implementations in specific games. This works, but is clearly not a general-purpose solution.

Bank switching

Although jamulator only supports NROM games, many NES games with more complicated mappers could switch regions of code and data in and out of the accessible address space. This means that each jump could go to dozens of different locations in the ROM, depending on which bank is mapped to that address at runtime.

Dynamically generated code/self-modifying code

Although this was uncommon for NES games due to the small amount of RAM, if I remember correctly C64 software would occasionally generate and execute code at runtime, or modify code mid-execution. This would be nearly impossible to statically predict.

Other hardware such as graphics and audio

The NES Picture Processing Unit and Audio Processing Unit still must be emulated. jamulator includes an emulator for the non-CPU hardware in a runtime library. The generated code detects writes to the memory addresses mapped to I/O operations and calls out to the appropriate runtime libraries.

However, since many NES games relied on precise timing between the CPU and PPU, the generated CPU code must count the number of NES clock cycles taken by the executed code. jamulator implements this relatively simply: after each NES instruction, it calls out to a runtime library to run the rest of the hardware for a specific number of cycles. This approach is simple to implement, but has a few disadvantages:

CPU writes don't quite happen on the exactly correct cycle relative to the rest of the hardware. The timing is only accurate to the nearest instruction, not the nearest cycle.

Emulating the PPU and APU between every set of CPU instructions greatly impairs performance, since the recompiler loses opportunities to remove, combine, or reorder instructions for better performance, and switching between tasks frequently harms the x86 CPU's cache performance and branch prediction accurately.

A more efficient (but much more complex) approach could predict the times at which synchronization is needed between the CPU and the rest of the hardware and use the predictions to switch between fast and cycle-accurate emulation modes.

Although static recompilation is often possible, it can be extremely difficult. jamulator sometimes falls back to interpretation when the game does something not properly handled by the static recompiler.

The accuracy of static recompilation could be improved by running the game inside of something like FCEUX's code-data logger, which emulates the game while recording the code paths it takes. Data from an actual run of the game can be used to greatly improve the accuracy of static recompilation. However, the "test run" recorded by the code-data logger must comprehensively exercise the possible code paths taken by the game in order to be useful.

Emulators of newer systems, such as the Dolphin Emulator (which emulates the GameCube and Wii) frequently use just-in-time compilation, where the emulator recompiles sections of the game's code at runtime. This generally provides the best of both worlds: we get the performance improvements of recompiled code, and the improved insight of being able to analyze the game's code at runtime.

Well, to avoide a delicate first impression, it may be a good idea to preface this with a clear statement, that the machine emulation is still needed, which is usually the largest part of an emulator - and using recompiled code will raise the challenge, thus ending in an even more complicated process of emulation. Otherwise, nice source of information. — 6 mins ago

Michael Kohne 46616 · Answer 2 · 2018-11-01 17:35:50Z

The problem is that the emulator is emulating a LOT more than just the CPU. So in addition to transpiling the 6502 code to Intel code (and don't think that's simple - making the timing come out right would be a fascinating problem), you also need to provide code (analogous to the standard libraries that any program uses) that provide an emulated I/O environment that the 6502 code can manipulate.

It'd be much easier to just bundle the emulator and program into one binary and ship that. This is what Ian Bogost did with A Slow Year - the 6502 binary is just wrapped up into a binary with the emulator.

The question is also based upon a fundamental misunderstanding that the code in question exists in "lines" that can be compiled. — 1 hour ago

DrMeta 312 · Answer 3 · 2018-11-01 18:32:20Z

Whether or not it's possible isn't the only defining factor that goes into development. Keep in mind that the quality of an emulator is very much connected to it's ability to create TAS, Savestate, use RAMWatch, and so on, all of which would not be possible if the ROM was converted to a .EXE file. With this in mind, developers haven't been interested in creating such a thing, which is the very reason why it isn't "possible" with the knowledge that the community is likely to have.

Jules 8,29222244 · Answer 4 · 2018-11-01 18:36:22Z

Compiling the instructions of an arbitrary processor (e.g. a 6502) into code that will run on a modern PC should be pretty easy. After all, it's not far different from what's done by any just-in-time compiler for a language that's distributed as bytecode (e.g. Java, .NET, and so on) and simpler than a JIT compiler for a language that needs non-trivial parsing (e.g. Javascript). The easy way would be to use an existing code generator (e.g. LLVM or gcc) and just add a very simple front end that read the machine code and generated the appropriate intermediate instructions...

There are three things that make this much more complex than it seems:

First, you'd need to emulate the hardware other than the CPU, so the graphics system, the audio system, etc. Along with this you'd need to provide a pre-translated version of any firmware that's expected to be present on your target system so that it can be linked in to the code you're compiling.

Second, you need to identify what parts of the ROM are actually instructions to be executed, and what parts are data (e.g. sprites, audio tracks, game level maps, etc). The data parts need to be handled differently -- ideally, translated into a more easily accessed format (especially multi-byte numbers, either integers or floating point, which could easily be in a format that the host processor doesn't natively understand).

Third, and possibly more interestingly, all of this may in fact be somewhat less legal than just using interpreted emulation. Many software and hardware vendors have granted permission to allow their code (either firmware or games) to be used in emulators, as long as they remain unchanged (e.g. I know that this is the situation with the ROMs for the Sinclair ZX Spectrum, which have been released under terms like these by their current copyright holder, Amstrad PLC). But recompiling it for a new system does change it. Even when copyright permission hasn't been granted, there are a variety of exceptions to copyright that allow minimal copying to be performed (e.g. copying the contents of a cartridge ROM onto disk for easier access) if that's necessary to make it work -- but recompiling isn't necessary, so even where there's no explicit licence grant available it may be less legal to recompile for a new architecture than just to emulate the old one. (I'm not a lawyer, so take this with a very large pinch of salt, but I have spent a reasonable amount of time learning about copyright law, and I'm pretty confident this position)

lvd 2,474417 · Answer 5 · 2018-11-01 18:44:43Z

Not quite a complete answer; however, many emulators of the consoles like playstation and alike do in some sense the same you are asking about.

Instead of precisely executing the code and precisely emulating the hardware, some typical code pieces (mostly connected with 3D transformations and rendering) are recognized as a whole and the end result is just inserted in the machine state or rendered using host 3D capabilities. The reason in doing so is probably lack of detailed hardware information and the speed of host CPU being not enough to emulate everything precilesy.

score 3 · Accepted Answer · 2018-11-01 18:30:13Z

^{Note: This answer mainly focuses on the NES, since that's what I'm most familiar with.}

Yes; this is called static recompilation or static binary translation, and it can be done -- jamulator by Andrew Kelly does it.

However, there are some serious challenges involved. Determining the behavior of a program by statically analyzing its code is in some cases provably impossible. Some of the problems faced when statically recompiling a retro video game ROM include:

How do we determine what parts of the ROM are executable code and what parts are data?

The "obvious" way to recompile a NES ROM is to take a single pass, reading a stream of 6502 instructions and outputting a stream of x86 instructions. However, the ROM also includes data bytes, which will produce garbage if interpreted as instructions. These garbage instructions should never be executed, since the program would never jump to that address, but the presence of non-instruction data poses two problems:

1) The program must be able to read its own ROM. This isn't really a hard problem at all; we just need to include an uncompiled copy of the ROM in the compiled program binary to read data bytes from.

2) We can't write a simple single-pass recompiler. Since instructions can be multiple bytes wide, if we try to recompile a block of non-executable data, we might end up misaligned when recompiling the next block of instructions. (As a simple (somewhat contrived) example, in 6502 machine code, the sequence of bytes 69 09 0A is ADC #09; ASL, while the sequence of bytes A5 69 09 0A is LDA $69; ORA #0A. The point at which we start executing drastically affects our results).

So this means we have to perform much more complex code analysis in order to determine the basic blocks of the program and compile them individually. jamulator does this by starting at the interrupt vectors and following all possible branches from there. However, this approach still has problems with:

Dynamic jumps

This is where the address to execute is computed at runtime. In some cases, the runtime-computed address can be looked up and the corresponding x86 code can be found. However, if the only way to access a basic block is via an indirect jump, the recompiler would likely have assumed that block was not executable code and thus not compiled it.

jamulator mitigates this problem using a hack: hard-coded support in the recompiler to recognize jump table implementations in specific games. This works, but is clearly not a general-purpose solution.

Bank switching

Although jamulator only supports NROM games, many NES games with more complicated mappers could switch regions of code and data in and out of the accessible address space. This means that each jump could go to dozens of different locations in the ROM, depending on which bank is mapped to that address at runtime.

Dynamically generated code/self-modifying code

Although this was uncommon for NES games due to the small amount of RAM, if I remember correctly C64 software would occasionally generate and execute code at runtime, or modify code mid-execution. This would be nearly impossible to statically predict.

Other hardware such as graphics and audio

The NES Picture Processing Unit and Audio Processing Unit still must be emulated. jamulator includes an emulator for the non-CPU hardware in a runtime library. The generated code detects writes to the memory addresses mapped to I/O operations and calls out to the appropriate runtime libraries.

However, since many NES games relied on precise timing between the CPU and PPU, the generated CPU code must count the number of NES clock cycles taken by the executed code. jamulator implements this relatively simply: after each NES instruction, it calls out to a runtime library to run the rest of the hardware for a specific number of cycles. This approach is simple to implement, but has a few disadvantages:

CPU writes don't quite happen on the exactly correct cycle relative to the rest of the hardware. The timing is only accurate to the nearest instruction, not the nearest cycle.

Emulating the PPU and APU between every set of CPU instructions greatly impairs performance, since the recompiler loses opportunities to remove, combine, or reorder instructions for better performance, and switching between tasks frequently harms the x86 CPU's cache performance and branch prediction accurately.

A more efficient (but much more complex) approach could predict the times at which synchronization is needed between the CPU and the rest of the hardware and use the predictions to switch between fast and cycle-accurate emulation modes.

Although static recompilation is often possible, it can be extremely difficult. jamulator sometimes falls back to interpretation when the game does something not properly handled by the static recompiler.

The accuracy of static recompilation could be improved by running the game inside of something like FCEUX's code-data logger, which emulates the game while recording the code paths it takes. Data from an actual run of the game can be used to greatly improve the accuracy of static recompilation. However, the "test run" recorded by the code-data logger must comprehensively exercise the possible code paths taken by the game in order to be useful.

Emulators of newer systems, such as the Dolphin Emulator (which emulates the GameCube and Wii) frequently use just-in-time compilation, where the emulator recompiles sections of the game's code at runtime. This generally provides the best of both worlds: we get the performance improvements of recompiled code, and the improved insight of being able to analyze the game's code at runtime.

Well, to avoide a delicate first impression, it may be a good idea to preface this with a clear statement, that the machine emulation is still needed, which is usually the largest part of an emulator - and using recompiled code will raise the challenge, thus ending in an even more complicated process of emulation. Otherwise, nice source of information. — 6 mins ago

Michael Kohne 46616 · Answer 7 · 2018-11-01 17:35:50Z

The problem is that the emulator is emulating a LOT more than just the CPU. So in addition to transpiling the 6502 code to Intel code (and don't think that's simple - making the timing come out right would be a fascinating problem), you also need to provide code (analogous to the standard libraries that any program uses) that provide an emulated I/O environment that the 6502 code can manipulate.

It'd be much easier to just bundle the emulator and program into one binary and ship that. This is what Ian Bogost did with A Slow Year - the 6502 binary is just wrapped up into a binary with the emulator.

The question is also based upon a fundamental misunderstanding that the code in question exists in "lines" that can be compiled. — 1 hour ago

DrMeta 312 · Answer 8 · 2018-11-01 18:32:20Z

Whether or not it's possible isn't the only defining factor that goes into development. Keep in mind that the quality of an emulator is very much connected to it's ability to create TAS, Savestate, use RAMWatch, and so on, all of which would not be possible if the ROM was converted to a .EXE file. With this in mind, developers haven't been interested in creating such a thing, which is the very reason why it isn't "possible" with the knowledge that the community is likely to have.

Jules 8,29222244 · Answer 9 · 2018-11-01 18:36:22Z

Compiling the instructions of an arbitrary processor (e.g. a 6502) into code that will run on a modern PC should be pretty easy. After all, it's not far different from what's done by any just-in-time compiler for a language that's distributed as bytecode (e.g. Java, .NET, and so on) and simpler than a JIT compiler for a language that needs non-trivial parsing (e.g. Javascript). The easy way would be to use an existing code generator (e.g. LLVM or gcc) and just add a very simple front end that read the machine code and generated the appropriate intermediate instructions...

There are three things that make this much more complex than it seems:

First, you'd need to emulate the hardware other than the CPU, so the graphics system, the audio system, etc. Along with this you'd need to provide a pre-translated version of any firmware that's expected to be present on your target system so that it can be linked in to the code you're compiling.

Second, you need to identify what parts of the ROM are actually instructions to be executed, and what parts are data (e.g. sprites, audio tracks, game level maps, etc). The data parts need to be handled differently -- ideally, translated into a more easily accessed format (especially multi-byte numbers, either integers or floating point, which could easily be in a format that the host processor doesn't natively understand).

Third, and possibly more interestingly, all of this may in fact be somewhat less legal than just using interpreted emulation. Many software and hardware vendors have granted permission to allow their code (either firmware or games) to be used in emulators, as long as they remain unchanged (e.g. I know that this is the situation with the ROMs for the Sinclair ZX Spectrum, which have been released under terms like these by their current copyright holder, Amstrad PLC). But recompiling it for a new system does change it. Even when copyright permission hasn't been granted, there are a variety of exceptions to copyright that allow minimal copying to be performed (e.g. copying the contents of a cartridge ROM onto disk for easier access) if that's necessary to make it work -- but recompiling isn't necessary, so even where there's no explicit licence grant available it may be less legal to recompile for a new architecture than just to emulate the old one. (I'm not a lawyer, so take this with a very large pinch of salt, but I have spent a reasonable amount of time learning about copyright law, and I'm pretty confident this position)

lvd 2,474417 · Answer 10 · 2018-11-01 18:44:43Z

Not quite a complete answer; however, many emulators of the consoles like playstation and alike do in some sense the same you are asking about.

Instead of precisely executing the code and precisely emulating the hardware, some typical code pieces (mostly connected with 3D transformations and rendering) are recognized as a whole and the end result is just inserted in the machine state or rendered using host 3D capabilities. The reason in doing so is probably lack of detailed hardware information and the speed of host CPU being not enough to emulate everything precilesy.

Search This Blog

Iyfjky

Is it possible to make a ROM converter?

5 Answers
5

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

Your Answer

Post as a guest

5 Answers
5

5 Answers
5

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

Post as a guest

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

Installing NextGIS Connect into QGIS 3?

One-line joke

Category

Random preview

Is it possible to make a ROM converter?

5 Answers 5

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

Your Answer

Sign up or log in

Post as a guest

Post as a guest

5 Answers 5

5 Answers 5

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

How do we determine what parts of the ROM are executable code and what parts are data?

Dynamic jumps

Bank switching

Dynamically generated code/self-modifying code

Other hardware such as graphics and audio

Sign up or log in

Post as a guest

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

Installing NextGIS Connect into QGIS 3?

One-line joke

5 Answers
5

5 Answers
5

5 Answers
5