Hawkeye Research

Research · ROP

Part 1 of 4

Return-Oriented Programming (ROP)

Exploring technical boundaries related to game security

1. What ROP is — principles and first look

1. Return-oriented programming builds on the detail that ret = pop + jmp. If we place an address at the right stack slot and execute ret, we pop that value into RIP and can steer the CPU in a controlled way.

2. With several addresses on the stack and repeated ret execution, we can change control flow multiple times. That is ROP.

A minimal model:

                    Thread stack                    High address
                    ┌────────────────────────────────┐
                    │  Addr F                        │
                    │  Addr E                        │
                    │  Addr D                        │
                    │  Addr C                        │
                    │  Addr B                        │
                    │  Addr A          <── RSP      │
                    └────────────────────────────────┘
                                                   Low address

Suppose addresses A through F each point to a ret instruction in some module.

  • Address A = any ret in ntdll.dll
  • Address B = any ret in user32.dll
  • Address C = any ret in kernel32.dll
  • …
  • Address F = any ret in dxgi.dll

When ret runs, ret = pop address A + jump to A + RSP points at B = RIP lands in ntdll.dll. Because A itself is a ret, we continue: ret = pop B + jump to B + RSP at C = RIP in user32.dll. The same pattern through C–F keeps firing ret until RIP reaches dxgi.dll.

A pointless hop across modules — but it captures the core idea of ROP: with a carefully built stack layout, you steer the CPU instruction stream.

Two tools first
  • Hawkeye ROP — Windows gadget search (ROP “parts” finder)
  • Hawkeye Lab — kernel-assisted cheat detection / evaluation for games

Next step: suppose we want HDC hDC = GetDC(NULL);. Under the x64 calling convention, rcx, rdx, r8, r9 hold the first four arguments. For GetDC(NULL) we need the first argument (rcx) to be 0 before entering GetDC. The obvious gadget is mov rcx, 0; ret — assign and return in one shot. In real modules, “write register + immediate ret” sequences are uncommon and hard to find. A practical equivalent: place constant 0 on the stack and use pop rcx; ret to load rcx (see the thread stack sketch below).

Overview using notepad.exe as the target process.

1. Hawkeye ROP searches for gadgets (short code snippets). We see many pop rcx; ret hits, as in Fig. 2. We usually prefer gadgets in ntdll.dll. Reason: almost every process loads ntdll.

2. Hawkeye Lab: after discovery, gadgets must be validated. Relocation fixups in executable regions can accidentally match gadget byte patterns — avoid those hits when you can.

3. Gadgets do not require instruction alignment in the sense used here. As Fig. 1 shows, pop rcx; ret encodes as 0x59 0xC3. Address 0x00007ffc8e5877fe can serve as a gadget entry; Hawkeye ROP supports truncated search.

Fig. 1 pop rcx; ret and misaligned entry
Fig. 1
Fig. 2 Hawkeye ROP search for pop rcx; ret
Fig. 2

Thread stack sketch

                    Thread stack                    High address
                    ┌────────────────────────────────┐
                    │  Addr E                        │
                    │  Addr D                        │  (4)
                    │  Addr C                        │  (3)
                    │  Addr B  next gadget addr      │  (2)
                    │  0ull            <── RSP      │  (1)
                    └────────────────────────────────┘
                                                   Low address

Suppose RIP → 7ffc8e58429e and RSP → (1). What happens?

  1. pop rcx pops 0ull into rcx; RSP moves to (2)
  2. ret pops the next gadget address into RIP; RSP moves to (3)

RIP now runs the second gadget. We have completed rcx = 0.

What is the heart of ROP? A deliberate stack layout, advanced one step at a time.

How is ROP different from shellcode?

Compare implementation, detection, and engineering cost:

  1. Shellcode is executable bytes you bring in; you typically allocate or map RWX memory and point RIP there.
  2. Shellcode byte patterns are relatively stable and easy to signature; with DEP / NX, “write then execute” is often blocked.
  3. Shellcode is flexible: you can use push / pop freely without planning every stack slot like ROP.
  4. ROP chains existing in-process gadgets; intent lives in the return-address chain, not a fixed instruction template.
  5. ROP jumps within legitimate module code and is less constrained by DEP / NX; control-flow integrity features such as CET are a separate topic (later in this series).
  6. ROP is harder to build: even with search tools or AI help, missing gadgets force module swaps and compromises.
  7. ROP cannot abuse push / pop gadgets casually — they disturb the stack. Winning moves are careful stack design and stepwise execution.
Summary

We started from ret and the stack: ROP is a chain of return addresses consumed by repeated ret instructions, steering RIP through existing gadgets.

  • GetDC(NULL) on x64 needs rcx = 0; ideal mov rcx, 0; ret is rare — stack 0 plus pop rcx; ret is the practical path.
  • Hawkeye ROP finds gadgets; Hawkeye Lab validates them in a live process. Watch relocation false positives and how “truncated” entries relate to alignment in this article.
  • Vs shellcode: ROP reuses in-process gadgets instead of injecting a shellcode blob, which is friendlier to DEP / NX but demands tighter stack discipline and more engineering; CET will spread with newer hardware and Windows 11 generations.

Next: Part 2 — MessageBox lab & clean exit: we target notepad.exe, build a full ROP chain to show a MessageBox, and exit cleanly.

We also cover shadow space (the x64 home stack) and what it means for ROP layout.

← ROP series · Part 2 →