Restructuring

So the way the plugin loader/linker code currently works (on PS2, psp, Dreamcast) is this:

• We build a main ELF with an absolute position in memory.
• We use a custom ld linker script to link together relocatable plugins. Since the main ELF has absolute addresses, some of the more complicated jumps (like those from plugin code back to the main ELF) can be resolved during this pre-linking without the need for us to write explicit loader code for them.
• Our loader code shifts the location of a plugin in memory and adjusts relocations within the plugin.

Doing things this way (with a non-relocatable main executable and a custom ld linker script for plugins) saves us much of the more extensive and complicated work (such as maintaining a Global Offset Table).

Over the past couple of days, I've come to understand (though I'm not 100% sure) that the Wiz is unlike most "smaller targets" in that its launcher requires Position-Independent-Code. This poses quite a big problem seeing as the way we're doing things depends on the main executable having absolute addresses that the ld script can use. To implement the custom ELF loader/linker on the Wiz, then, will likely be quite an extensive task.

Since (in my proposal) I intended the Wiz to be a testbed for a more crucial target (the DS), I've talked with my mentor about moving on to other work towards supporting the DS and ditching the Wiz (at least, for now).

A good first step in this respect would be starting work on changing the plugin system to support only loading one plugin at a time. Some discussion has gone on as to how to achieve this and I've posted my current (under construction :-)) plans for it on my wiki at http://wiki.scummvm.org/index.php/User:Toneman. I'll be discussing things further with my mentor and posting schedule changes there as well (I'm not absolutely sure about whether Wiz work will continue), but for today I'm going to start looking into the DS port (possibly writing a custom linker for it and seeing what ARM relocation types I need to worry about for it).

objdump-ing

So I was trying to figure out which relocation types I really need to worry about when it comes to the GP2x-Wiz today. After making a number of tweaks to the incomplete loader to get it to compile, I compiled and linked (with my modified ARM linker script) the game engines into plugins. I tried using "objdump" on these plugin files to figure out which relocation types were worth my time, but it kept returning all types as "unknown". After a bit of research, I discovered another program "readelf" that successfully dumped relocation types. I noticed, however, that when I do use "objdump -R" which dumps a dynamic relocation table, I get output for both the main "scummvm.wiz" file and the plugin files. This leads me to believe that the binaries may be PIC, but since the version of objdump I'm using doesn't seem to support ARM (as mentioned before, it couldn't detect the relocation types), the reason it's outputting a dynamic relocation table could be because it's misreading the files, not because the files really are PIC. My goal for this weekend, then, is to become more certain about the nature of the generated files, and then to code for the relocations I may need to worry about, which are listed below (the names of some of these relocations also has me worried about PIC code):

• R_ARM_GLOB_DAT
• R_ARM_JUMP_SLOT
• R_ARM_COPY
• R_ARM_PLT32
• R_ARM_PC24 
• R_ARM_BASE_PREL
• R_ARM_GOT_BREL
• R_ARM_PC24
• R_ARM_ABS32

PS2 GP-relative section success!

As of yesterday, the PS2 port would fail to compile with Dynamic Modules turned on unless I put nothing in the GP-relative section by adding a "-G0" flag to the Defines in the Makefile. Last night (after a discussion with Yotam) I made modifications to the linker scripts used when compiling with Dynamic Modules that got rid of these errors. In the linker script for the main engine, I had an extra zero appended to a hex value specifying an offset from the GP register. Fixing this typo (along with rearranging things in plugin.ld so that the .bss section was no longer assigned to the shorts segment, but rather to the plugin segment) seemed to eliminate the problem I was having with "relocation truncated to fit errors".

Quick Update

Got ScummVM booting remotely on the Wiz via a terminal using the stock USB cable and have printf output (haven't tried GDB yet). Continuing work on the loader, I'll post again once it's in a shape where I can start testing. Once it's working, abstracting a more generic ELF-loader should be quite simple :)

Wiz-tle while you work (bad pun)

After a small amount of testing/altering of the PS2 code, I began work on the GP2x-Wiz this week! On the debug front, I've gotten some help from the GP2x forums to get the Wiz to "pretend" to be a USB serial converter and I've successfully logged onto my Wiz via kermit. Booting ScummVM from remote via kermit isn't quite working well yet, but it's close. On the coding front, I've written a plugin linker script (I don't think I'll have to use a custom linker for the main engine like I did with the PS2 code). I've also decided to prototype the custom ELF loader for the Wiz using a Makefile instead of configure since I might be making significant changes to building. I have a Makefile now that successfully builds ScummVM for the Wiz with static plugins, and am working on the loader.

PS2 plugins working

Work went much better after getting debugging facilities running on the PS2. I discovered the problem seemed to be in how the plugins were linked. After making changes to plugin.ld over the weekend, dynamic modules seem to work correctly, at least for the SCUMM and SKY engines, on the PS2. I'll be doing a bit more testing, but things look good!

Tomorrow I plan to better comment out the altered linker scripts with documentation of what the changes are intended to do. Also, after discussing things with Yotam (my main mentor), I've decided to alter my schedule and begin work this week on the GP2x Wiz ELF loader/linker (instead of the work of abstracting a generic ELF loader). After work on the GP2x Wiz is done, I'll be more able to know what things can be abstracted and what things can't.

The altered schedule (which was approved by my mentor) is up on my wiki page: http://wiki.scummvm.org/index.php/User:Toneman

Debug!

I've been hitting problem after problem getting good debugging support going on the PS2. Lesson learned, I've sent off e-mails to the porters for GP2X-Wiz and DS so I'll be more prepared when it comes to how to print debug output, etc. on those systems :-)

The Work So Far

A summary of my work so far on ScummVM:

• After a bit more hassle than expected (which involved things like changing some toolchain installation scripts to use older svn revisions), I got the necessary toolchains/libraries to successfully cross-compile ScummVM for the PS2, DS, and GP2x-Wiz.
• I copied the psp-provider, elf32.h, and psploader code into the ps2 backend and began tweaking them for the PS2.
• The simplest of these "tweaks" was just changing references to "PSP" to "PS2" :-)
• Tweaks so far also included removing psp-specific things from the ps2 loader (like code that I believe was there to ensure the psp didn't suspend during a load) and in some cases replacing calls to code in the psp toolchain with calls to code in the ps2 toolchain (also being sure to replace "includes" that reference psp toolchain files with "includes" that reference ps2 toolchain files).
• I got the default ps2 cross-compiler ld linker script using ee-ld --verbose and tweaked it in a similar fashion to the way the default psp linker was tweaked for the psp plugins (Yotam provided the default psp linker) to generate plugin.ld (for linking together game engine plugins) and main_prog.ld (for linking together the main engine).
• I tested main_prog.ld with static plugins and found that ScummVM didn't launch on the PS2 using the modified default ps2 linker. I saw in the PS2 Makefile that the default linker was switched out using -T with a "linkfile" in the PS2 toolchain. I remade main_prog.ld and plugin.ld by modifying this instead and it successfully launched again.
• I modified the main (in systemps2.cpp) to check if Dynamic Modules are enabled and, if so, to call PluginManager::instance().addPluginProvider(new PS2PluginProvider()).
• I have spent much of my time making tweaks to Makefile.ps2 (i.e. learning how different compiler flags/Defines affect the building, adding plugin variables, directing that the linker use my linker scripts, ensuring that the ps2loader is compiled and linked, etc.)
• I was getting linker errors concerning the GP-relative data section being over-filled when I enabled Dynamic Modules. I don't see a good reason why this is the case but for now I have used -G0 flags in the Makefile to direct the cross-compiler not to put anything in the GP-relative section.

At this point, compiling with Dynamic Modules turned off works as well as always, so at least my modifications haven't broken anything! :-). With Dynamic Modules turned on, yesterday morning ScummVM booted and let you specify a "Plugins Path" in the options, but failed to find engines for any of the games even though the plugins for those engines were linked. After making some modifications to the makefile and fixing some typos in the loader yesterday, ScummVM won't even boot anymore with Dynamic Modules turned on (might be a simple typo or something more sinister). In light of this, I'm planning to modify my schedule a bit after discussing it with my mentor, but progress is being made!

Note: I've made a number of modifications since my last commit but am holding off on committing code until ScummVM is at least launching again on the PS2.

Some Success!

Got ScummVM on the PS2 building with dynamic modules enabled (plugins linking, too!) Now to start actually testing on the PS2.

A modicum of progress

Finally got the main engine linking without errors when dynamic modules are turned on for the PS2! Now to get everything else working... I think I'll be working this weekend :-)

link or swim

I've been having trouble getting the modified "plugin.ld" script to work when compiling the PS2 port with dynamic modules enabled. I'm going to spend a couple of hours now reading a hundred pages of GNU linker documentation. Having a more thorough understanding of the linker should help since I don't completely get what the default linker is doing :)