English is a Minor thing

I have become a medium through which words travel,
not a locus of their temporal deconstruction/assemblage-

a reader:

a plane upon which words flow and diverge and bead and flow again
from (and back to) the infinite and infinitely elusive edges

Belated Blog

Hey all!

It's been a while since I last blogged (whoops!).

But in the time since my last post, quite a bit has happened.

First off, I got the DS port working with loadable modules after reintegrating the thumb-interworking code into the DS port. It was quite simple, really, as the thumb relocations used were PC-relative (and thus needed little to no manual work). I tested a good number of the engines after getting this work done, including the SCUMM, SKY, LURE, QUEEN, and AGI engines. They all seemed to work fine (though I only played through about 10 minutes of each game and I've heard some, like Sam and Max, use more memory later on). The one exception in terms of successful testing is the CRUISE engine. That plugin had a single relocation type (R_ARM_TARGET1) that none of the other engines have. This type is supposed to be treated either the same as R_ARM_ABS32 or R_ARM_REL32 (how it should be treated can differ, even in the same file). Since I saw no easy way to determine how it should be treated during runtime, I added "--target1-abs" linker flags to the ds makefile (the flags specify that R_ARM_TARGET1 should be treated as R_ARM_ABS32 in all cases) and coded for the R_ARM_TARGET1 relocation accordingly, but to no avail ("Cruise for a Corpse" still crashes on bootup, showing only the Cruise cursor). As of now I can't see the console (on the top screen of the ds) at the point of the crash, so debugging is difficult, but I'll continue to work on this.

Another problem with the DS build is that (as mentioned before) I had to remove the "--gc-sections" linker flags since things were being garbage-collected in the plugins and main executable that referred to each other. This unfortunately resulted in a lot of bloat in the main executable. For some engines, this bloat is around 300kb, which is too much to be ignored. There's been some discussion among the mentors concerning different ways to decrease this bloat, including building static builds with garbage collection and dumping the symbols then telling the linker to included only those symbols in the dynamic builds. In any case, this is something that will need to be worked on before the DS dynamic plugins work is reintegrated into the trunk.

Apart from the DS work, I did some work abstracting a more generic ELF-loader last week. It's a little rough at the moment, but works for both the PS2 and DS ports, so at least I didn't break anything :-). I put the methods dealing with relocations into their own files based on processor type (like arm-relocs.cpp and mips-relocs.cpp) and also split the shorts-segment-manager (made to effectively use the gp-relative section of MIPS processors) into its own file. There's still a bit of work to be done for abstraction, including making subclasses of DLObject (like PS2DLObject, which could be a subclass of MIPSDLObject, etc.) and having the different plugin-providers use these different subclasses (right now, I just used "ifdefs" with different ports).

I've also worked on a bit of the plugin design change work. It took me a while of looking through the base code to understand how to best begin implementing the plugin design changes, but I added some new functions to the PluginManager class (a "loadFirstPlugin" and "loadNextPlugin") as well as changing main.cpp and some functions in plugins.cpp to use these functions in a loop when "NEW_PLUGIN_DESIGN_FIRST_REFINEMENT" is enabled. I tested on a Windows build with dynamic plugins enabled and was able to successfully launch games with only one plugin ever loaded at a time!

So the plan for this week is to continue plugin design change, abstraction, Cruise for a Corpse, and DS memory problem work,

Tony

Good News for the DS port

Last week, I restructured the DS loader to use a SeekableReadStream for file reading. Once that was in place and working, I restructured the PS2 loader to use a SeekableReadStream as well and tested it (this needed to be done anyway and it helped me ensure there was nothing inherently wrong with the way I was using SeekableReadStream in the DS loader).

I then had problems reading the symbol table in successfully. After some wasted time :P, I realized it was just a silly mistake. I had neglected to initialize _symtab_sect to anything in dsloader.h and the value of uninitialized ints is undefined in C++. Thus, the loader was detecting, from the value that _symtab_sect had, that the symbol table was already read in when it wasn't.

At this point the loader was getting to the point where I had to deal with the relocations. I used consolePrintfs to make sure that the relocation types the loader was detecting were the same as the ones detected by arm-eabi-objdump (they were). I then began coding for these relocations. After a day of work on the relocations with no success as far as getting games to run with dynamic plugins enabled on the DS, I decided to disable the thumb-interworking on the ds temporarily so I could work on getting the DS with plugins in a runnable state without having to deal with the complication of thumb instructions/relocations.

I worked further on the relocations yesterday; I started dumping the instructions to be relocated and immediately noticed a MAJOR problem. Every instruction was 0x00000000! The addresses of the instructions seemed reasonable, though, so I suspected it had something to do with how the plugins were being built and not how I was retrieving target addresses. I dumped Mapfiles with "-Map" during plugin linking and discovered that whole input sections were being discarded, among other problems. After switching out the linker script for a modified version of the default "arm-eabi-ld" linker script, these problems seemed to be fixed, perhaps because the modified linker script I was using (based on the script used by the main executable) used the MEMORY command to map out different blocks of memory but I was trying to force the start address of the ".text" section to be 0. I am not sure whether switching out this linker script will cause problems.

I then did a fair bit of reading on the ARM instruction set and continued tweaking/testing the relocation code. Things still weren't working like they should have. Yotam suggested comparing completely unrelocated code and semi-relocated code to see what ld was already doing for me, and through a few dumps, I realized I was trying to do A LOT that ld already did redundantly and thus messing up things that were already fixed! After removing this redundant code, I got Beneath a Steel Sky to run successfully with Dynamic Plugins enabled on the DS!

My next order of business, then, is to test the rest of the engines (some of which have relocation types I still haven't dealt with). Then, I'll reintegrate the thumb-interworking and deal with any major complications that brings (if any :D).

~Tony

Also, I plan to update my wiki schedule shortly: http://wiki.scummvm.org/index.php/User:Toneman

a size-able problem with plugins

While working on changing how the loader opens and reads plugin files on the DS this weekend, I noticed that the ".plg" files produced in my builds were significantly smaller than expected (as in under a hundred kilobytes).

So I looked through verbose build output to investigate what the problem could be. The plugin linking output included all the necessary object files, so I knew the problem wasn't there and decided to look closer at the plugin flags.

Last week when I added the PLUGIN_LDFLAGS into the ds/arm9/makefile, I copied over the regular LD_FLAGS to make sure all the special DS optimizations that occur when linking the main executable would be used for the plugin files as well (stuff like "-mthumb-interwork"). One of these flags was "--gc-sections" which I believe is meant to garbage collect sections full of unused functions and/or data. Since both the main executable and the plugin files were using this flag (but plugins aren't truly linked in to the main executable until run-time), functions and data in the main executable and the plugins that point to each other and nowhere else were mistakenly garbage-collected. After removing the "--gc-sections" flag from LDFLAGS and PLUGIN_LDFLAGS, plugins are a much more reasonable size. Unfortunately, this means the main executable is a bit more bloated than before, which runs counter to the goal of using as little memory as possible on the RAM-starved DS...

But for now, I'll be moving on to reworking the loader to use "Common::SeekableReadStream" to read plugin files.

~Tony

plugins linking on DS

I worked through some build errors, switched my plugin.ld to use a modified version of the correct linker script and added a number of PLUGIN_LD_FLAGS (including "-mno-crt0 $(DEVKITPRO)/devkitARM/arm-eabi/lib/ds_arm9_crt0.o"), and plugins started linking. I had some trouble with plugin.ld syntax errors, but figured it out (I was doing things in SECTIONS like "{ contents } >region =fill :phdr" when it needed to be "{ contents } >region :phdr =fill"). Using objdump on the plugins, it seems most only have one relocation type to worry about (R_ARM_ABS32), but the cine and cruise engines also use R_ARM_THM_CALL and R_ARM_TARGET1. I'll be working on completing dsloader.cpp (including the ARM-specific relocations) now, my first milestone being getting one of the engines that uses just R_ARM_ABS32 working.

DS Update

Work on the DS dynamic module loader is progressing. In the tail end of last week, I put the basic loader code in for the DS and modified the default arm-eabi linker script to make a plugin linker for the DS. This week I found (in a .spec file included in one of the makefiles for the DS) that the DS port doesn't use the default arm-eabi linker so I'll have to modify the one it does use (ds_arm9.ld) instead. At the opening of this week, I took some time to try to understand the Makefiles for the DS (it's a bit more complicated than the other ports I've worked with) and then started modifying them to use dynamic modules. I had a bit of trouble getting make to attempt to link the plugins, but after correcting some dependencies, linking of plugins is attempted (though not completed successfully, which hopefully will change once I switch to a modified version of ds_arm9.ld for plugin linking). I'll post again once the plugins are successfully linking.

:-)

Tony

This week has not been the most productive thus far (though it's only just begun). I've been trying to get engine plugins building for the ds (using the custom "plugin.ld" linker script) so I can begin work on the ARM-specific relocations needed for the DS, but the way I've modified the DS makefiles, engine plugins don't build. (I currently get things like "warning: overriding commands for target 'plugins/scumm.plg'"). I'll be contacting Niel to better understand the DS make system shortly.