Saturday, January 14, 2012

Floating Points in Assembly


Turns out modifying meshes are a bit more complicated. It seems like I will need to modify some DX3D structs that contain floating point numbers for the RGB values. Turns out, I know very little about how floating points are treated in assembly. This has forced me to spend an hour or two building c++ apps with various floating point values and arithmetic and seeing what it looks like in the debugger. It's funny, this is one of those things you just don't foresee being something you'll need to learn when approaching a new subject. When I was writing memory corruption exploits I pretty much skipped over any floating point junk I saw in the disassembly. I guess I have to learn it now!

So I approached this like I have approached other problems. I created some simple C++ code, compiled it, ran it through a debugger to see what it looked like. I started with a simple bit of code:

int main() {
float x = 1.0f;
float y = 0.5f;
float z = 0.0f;
cout << x << " " << y << " " << z << endl;
return 0;
}

I figure that should be easy to identify in the assembly. Here's what it looks like in x86-asm:
.text:004114CE                 fld1                  
.text:004114D0                 fstp    [ebp+x]        
.text:004114D3                 fld     ds:__real@3f000000
.text:004114D9                 fstp    [ebp+y]        
.text:004114DC                 fldz                  
.text:004114DE                 fstp    [ebp+z]        
.text:004114E1                 mov     esi, esp
.text:004114E3                 mov     eax, ds:__imp_?endl@std@@YAAAV?$basic_ostream@DU?$char_traits@D@std@@@1@AAV21@@Z ;
.text:004114E8                 push    eax             ; _Val
.text:004114E9                 mov     edi, esp
.text:004114EB                 push    ecx             ; _Val
.text:004114EC                 fld     [ebp+z]
.text:004114EF                 fstp    [esp+0FCh+var_FC]
.text:004114F2                 push    offset asc_417800 ; " "
.text:004114F7                 mov     ebx, esp
.text:004114F9                 push    ecx
.text:004114FA                 fld     [ebp+y]
.text:004114FD                 fstp    [esp+104h+var_104]
.text:00411500                 push    offset asc_417800 ; " "
.text:00411505                 mov     eax, esp
.text:00411507                 push    ecx
.text:00411508                 fld     [ebp+x]
.text:0041150B                 fstp    [esp+10Ch+var_10C]
...<calls the various stream methods for printing>...

I have no idea what these 'f' instructions do as I've never taken the time to learn it. So I head over to the wikipedia page and see that fld1 loads 1.0 into the stack (which at the time I didn't realize meant the FPU stack), and fstp does a 'store and pop' into where [ebp+x] is. It turned out I didn't really know how the FPU worked, so I had to do some general research on it. I can tell already I'm going to become best friends with floats in my game hacking endeavors, so might as well learn it now. During all of this I came across a nice resource on some general FPU information.

I knew the FPU had it's own registers, but I didn't know they were formed into a 'stack'. I certainly didn't know that you couldn't reference the registers directly. Apparently, you have to load values into the st0~st7 registers, do your calculations, then pop it off from the FPU stack and into some memory location. I also learned that any FPU instruction that ends with 'P' basically does the pop automatically.

So let's look at our assembly again and with some comments of what is going on this time.

.text:004114CE                 fld1                    ; load 1.0 onto the FPU stack
.text:004114D0                 fstp    [ebp+x]         ; pop off st0 and store it in [ebp+x]. (where ever that is)
.text:004114D3                 fld     ds:__real@3f000000 ; load from the data segment into st0
.text:004114D9                 fstp    [ebp+y]         ; pop off st0 and store it in [ebp+y]. (where ever that is)
.text:004114DC                 fldz                    ; load 0.0 onto the FPU stack.
.text:004114DE                 fstp    [ebp+z]         ; pop off st0 and store it in [ebp+z]. (where ever that is)
.text:004114E1                 mov     esi, esp
.text:004114E3                 mov     eax, ds:__imp_?endl@std@@YAAAV?$basic_ostream@DU?$char_traits@D@std@@@1@AAV21@@Z ;
.text:004114E8                 push    eax             ; _Val
.text:004114E9                 mov     edi, esp
.text:004114EB                 push    ecx             ; _Val
.text:004114EC                 fld     [ebp+z]         ; load the floating point value from [ebp+z] into st0
.text:004114EF                 fstp    [esp+0FCh+var_FC] ; pop it off st0 and store it in [esp+0FCh+var_FC]
.text:004114F2                 push    offset asc_417800 ; " "
.text:004114F7                 mov     ebx, esp
.text:004114F9                 push    ecx
.text:004114FA                 fld     [ebp+y]         ; load the floating point value from [ebp+y] into st0
.text:004114FD                 fstp    [esp+104h+var_104] ; pop off st0 and store it in [esp+104h+var_104]
.text:00411500                 push    offset asc_417800 ; " "
.text:00411505                 mov     eax, esp
.text:00411507                 push    ecx
.text:00411508                 fld     [ebp+x]         ; load the floating point value from [ebp+x] into st0
.text:0041150B                 fstp    [esp+10Ch+var_10C] ; pop off st0 and store it in [esp+10Ch+var_10C]

Cool, that makes a lot more sense. Seems sort of contrived due to the fact that you can't reference them directly. You see it does a lot of pushing onto FPU stack, then storing in some memory location, then pushing back into st0 and then storing it somewhere else.

So know that I know how it works, I wanted to look at it in a debugger.

1.0f pushed into st0

So there's the fld1 command loading 1.0 into the st0 register. How about fstp?

1.0f pushed onto the stack and popped out of st0

So there you have it, that's how simple float calculations are handled. You'll also notice that 1.0f is 0x3f800000, good to know!

Right, now back to figuring out DirectX's structs and how the hell I can dynamically set an entire mesh to a solid color.

No comments:

Post a Comment