Common function prototype

It is another step to common internal function fuzzing. To call fuzzed function we need right function prototype. “Right” function prototype means such prototype that can pass parameters to function and function can process this parameters. The called function must remains in stable state.

__stdcall, __cdecl, __fastcall, __pascal etc. There are few commonly used function conventions, but every possible convention can be used in studied program.  For __fastcall (quite commonly used) there is no used registers specification. I’ve seen __usercall in some disassambled piece of code.  It is function call that is not described by any convention.

So, it is impossible to recover function prototype to the source level. There is no such a data in binary file. But we no need such prototype for fuzzing. The next futures should be collected:

• Which function frees stack ( callee or caller )
• How match parameters passed through the stack
• Which registers used to pass parameters

In deed, function prototype in any convention contains all and only information from list above.  Function’s options can be presented add double word:

12222222333333333333333333333333

,  where

1 – Which function frees stack ( 1 – callee, 0 – caller )

2 – How match parameters passed through the stack ( 7-bit number)

3 – Registers using bitfield. A register is used if bit set to 1. Bits relation to registers are the same as in BeaEngine.

Next question – how prototype for fuzzing can be generated. The next method is proposed. Describe it in example. We have next function information:

1. Function takes 5 parameters
2. Callee frees stack
3. First parameter passed through ecs. Second through edx
4. Parameters 3-5 passed through stack

The same in binary representation:

0x83000006

For code generation ( for manual linking ) somebody can use auto generated function interface:

int __cdecl foo__ ( int b, int c, int d )
{
    __asm {
    jmp fooaddr;
    }
}

// The function takes 5 parameters. 1st in ecx, 2nd in edx.
// Parameters  3-5 through stack
int __cdecl foo ( int a, int b, int c, int d, int e )
{
   // some parameters put in registers
   __asm {
          mov ecx, a
          mov edx, b
   }

   // some in stack
   foo__( c, d, e);
}

fooaddr function’s address can’t be resolved while compilation. We can get it while linking. Originally function has __fastcall prototype. It is reason why ecx and edx registers are used. However proposed prototype is equivalent. Parameters are passed through registers ecx, edx and stack. Callee cleans stack.

The method proposed above need to be approved.  The idea is described here to get some critics. So, you are welcome.

Some links about function conventions ( thanks to Ivan Sudakon ): [1], [2], [3], [4].