Now we are on to page 0x03. This one has a little bit more going on, but the previous post prepares us for it.

Here is our code:

sub      rdx,rax
sbb      rcx,rcx
and      rcx,rdx
add      rax,rcx

Our first instruction is sub, which is subtract. It subtracts the second operand from the first operand, and stores it in the first operand. It affects quite a number of flags, too, including CF. We know from the previous post that the following instruction, sbb, pays interest to the CF flag.

x86 uses CF as a borrow flag. Meaning, if you do a-b and a is less than b, then CF is set.

Bringing this back around to our original snippet, we can assume that the second instruction will behavior differently depending on if rdx is less than rax.

Again borrowing knowledge from the last page, we know what sbb will do when both operands are the same and if the carry flag is set or not. Let’s start with the first two instructions.

We have two test cases, so we’ll run through it twice.

Here is a case where rdx is greater than rax:

Initial:
rax = 0x0000000000000005
rcx = 0x8080808080808080
rdx = 0x000000000000000d
rflags = 0x0000000000000202 (CF = 0)

Step (sub):
rax = 0x0000000000000005
rcx = 0x8080808080808080
rdx = 0x0000000000000008
rflags = 0x0000000000000202 (CF = 0)

Step: (sbb):
rax = 0x0000000000000005
rcx = 0x0000000000000000
rdx = 0x0000000000000008
rflags = 0x0000000000000246 (CF = 0)

So, to recap this, when rdx was 13, we subtracted 5 from it, putting 8 into the destination, rdx. Since this didn’t borrow, CF is 0. Next is sbb on rcx. Since CF is zero, nothing gets added to rcx. Then we subtract rcx from rcx, and store it in rcx, effectively zeroing the register.

Let’s try it with rax as 13 and rdx as 5:

Initial:
rax = 0x000000000000000d
rcx = 0x8080808080808080
rdx = 0x0000000000000005
rflags = 0x0000000000000202 (CF = 0)

Step (sub):
rax = 0x000000000000000d
rcx = 0x8080808080808080
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000293 (CF = 1)

Step: (sbb):
rax = 0x000000000000000d
rcx = 0xffffffffffffffff
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000297 (CF = 1)

We start by subtracting 13 from 5, which puts -8 into rdx. This also sets CF to

  1. Moving on to sbb, it takes our value, 0x8080808080808080, and adds 1, resulting in 0x8080808080808081. It then subtracts 0x8080808080808081 from 0x8080808080808080, which is -1, so -1 is put in the rcx register.

Note that in this problem, it doesn’t matter what rcx is set to. After the sbb, rcx will either contain -1 (all bits set) or 0 (no bits set) depending on the previous instruction. This brings us into the next instruction, and.

The instruction does bitwise and between rdx and rcx, and stores the result in rcx. As we just figured out, rcx will either have no bits set or all of its bits set. So rcx will either be zero, or it will be what rdx is.

Finally, we add rax and rcx, and store the result in rax. Let’s see the whole thing in action.

In the first case, rcx is zero, so adding rcx and rax together does nothing to rax.

In the second case, rcx is -8, do we add -8 to 13, which is 5, which was the value of rdx in the first place.

This appears to be a conditional assignment that doesn’t use an actual conditional instruction. In pseudo code, it would like thing like this:

if (rdx < rax) then
    rax = rdx

Let’s see the whole thing in action.

When rdx is 13 and rax is 5:

Initial:
rax = 0x0000000000000005
rcx = 0x8080808080808080
rdx = 0x000000000000000d
rflags = 0x0000000000000202 (CF = 0)

Step (sub):
rax = 0x0000000000000005
rcx = 0x8080808080808080
rdx = 0x0000000000000008
rflags = 0x0000000000000202 (CF = 0)

Step (sbb):
rax = 0x0000000000000005
rcx = 0x0000000000000000
rdx = 0x0000000000000008
rflags = 0x0000000000000246 (CF = 0)

Step (and):
rax = 0x0000000000000005
rcx = 0x0000000000000000
rdx = 0x0000000000000008
rflags = 0x0000000000000246 (CF = 0)


Step (add):
rax = 0x0000000000000005
rcx = 0x0000000000000000
rdx = 0x0000000000000008
rflags = 0x0000000000000206 (CF = 0)

When rax is 13 and rdx is 5:

Initial:
rax = 0x000000000000000d
rcx = 0x8080808080808080
rdx = 0x0000000000000005
rflags = 0x0000000000000202 (CF = 0)

Step (sub):
rax = 0x000000000000000d
rcx = 0x8080808080808080
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000293 (CF = 1)

Step (sbb):
rax = 0x000000000000000d
rcx = 0xffffffffffffffff
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000297 (CF = 1)

Step (and):
rax = 0x000000000000000d
rcx = 0xfffffffffffffff8
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000282 (CF = 0)


Step (add):
rax = 0x0000000000000005
rcx = 0xfffffffffffffff8
rdx = 0xfffffffffffffff8
rflags = 0x0000000000000217 (CF = 1)