This topic really sucks to search for, way too many different results without any actual clarity, so here’s how I was able to get it working:

Firstly, power down the image and edit the configuration options for your image, click on the “Video options”. Check ‘Enable custom screen resolutions’ and add the resolution (in this case, 1280×800). Make sure the resolution is enabled (checked). See the screenshot below for an example of what it should look like:

Next, boot into the VM image and let’s take a look at the xorg.conf file, here are the lines I changed that actually matter:

#HorizSync 31.5 - 48.5
HorizSync 30.0 - 82.0
#VertRefresh 50.0 - 90.0
VertRefresh 50.0 - 90.0
#Option "DPMS"
ModeLine "1280x800" 80.58 1280 1344 1480 1680 800 801 804 827 -HSync -VSync

These should be in the section right after ‘Section “Monitor”‘. After changing the hsync and vsync as well as adding the mode, I changed the display section from:

SubSection "Display"
Depth 24
Modes "1024x768" "800x600" "640x480"
to:
SubSection "Display"
Depth 24
Modes "1280x800" "1024x768" "800x600" "640x480"

It’s a good idea to change the modes for each of the depths (at least 8, 15, 16 and 24) also.

After rebooting (or killing X with Ctrl+Alt+Backspace), your screen should come up in 1280×800 resolution. Hurray!

You can see an example of my desktop setup for Hex 1.0.3-RC2 here:

You can get a copy of my entire xorg.conf file here. (Note that in this xorg.conf, CapsLock is remapped to additional control because I hate capslock with a passion).

Hope this helps someone out there