[Reverse Engineering]

A write up for CyLab(picoCTF) reversing challenge “Gatekeeper”

Sagar Joshi

5 min read

·

May 10, 2026

--

Listen

Share

What’s behind the numeric gate? You only get access if you enter the right kind of number.

File Type

x64 ELF binary

Press enter or click to view image in full size

Execution behavior

Asks for user input for a numeric code > 999:

Wrong code gives “Access Denied”

There is a max and min range set for input

Tracing library calls through ltrace

Based on the calls available, we can interpret that the program checks for length of string passed as input(strlen) > converts ascii to integer using atoi() > prints message(puts):

Press enter or click to view image in full size

Reviewing Disassembly

At <+78> we see the program calls scanf to gather user input.

It evaluates the length of input using strlen() at <+97> which stores the returned length as a 32-bit value at [rbp-0x34] (<+102>).

Call to <is_valid_decimal> performs a check on the user input, based on the result we take a conditional jump <+119> to <is_valid_hex> function call or continue with the execution to convert decimal input to integer using atoi() <+128>.

If we take the path where input is decimal, we jump to 0x555555555647 <main+200> once the ascii to integer conversion returns.

   0x00005555555555c8 <+73>: mov    eax,0x0
   0x00005555555555cd <+78>: call   0x555555555260 <__isoc99_scanf@plt>
   0x00005555555555d2 <+83>: mov    DWORD PTR [rbp-0x38],0xffffffff
   0x00005555555555d9 <+90>: lea    rax,[rbp-0x30]
   0x00005555555555dd <+94>: mov    rdi,rax
   0x00005555555555e0 <+97>: call   0x5555555551b0 <strlen@plt>
   0x00005555555555e5 <+102>: mov    DWORD PTR [rbp-0x34],eax
   0x00005555555555e8 <+105>: lea    rax,[rbp-0x30]
   0x00005555555555ec <+109>: mov    rdi,rax
   0x00005555555555ef <+112>: call   0x5555555553d9 <is_valid_decimal>
   0x00005555555555f4 <+117>: test   eax,eax
   0x00005555555555f6 <+119>: je     0x555555555609 <main+138>
   0x00005555555555f8 <+121>: lea    rax,[rbp-0x30]
   0x00005555555555fc <+125>: mov    rdi,rax
   0x00005555555555ff <+128>: call   0x555555555250 <atoi@plt>
   0x0000555555555604 <+133>: mov    DWORD PTR [rbp-0x38],eax
   0x0000555555555607 <+136>: jmp    0x555555555647 <main+200>
   0x0000555555555609 <+138>: lea    rax,[rbp-0x30]
   0x000055555555560d <+142>: mov    rdi,rax
   0x0000555555555610 <+145>: call   0x555555555369 <is_valid_hex>

Decimal input path

We noted earlier that if we provide a decimal input we jump to <main+200>, where we see the input values is being validated to be > 999 and < 10000.

If both conditions hold true, we make another conditional jump to <main+246> where we perform a comparison between 32bit value at location [rbp — 0x34] with 0x3.

Else, we exit the program and print a message to screen.

[rbp — 0x34] if you refer to the beginning of our analysis holds the length of user input.

Which means given the condition that the input needs to be > 999, a decimal input always will be of length ≥ 4 therefore the CMP result will not be equal and will never execute reveal_flag function instead we exit the program.

   0x0000555555555647 <+200>: cmp    DWORD PTR [rbp-0x38],0x3e7 - 999
   0x000055555555564e <+207>: jg     0x55555555565e <main+223>
   0x0000555555555650 <+209>: lea    rdi,[rip+0xa2d]        # 0x555555556084
   0x0000555555555657 <+216>: call   0x555555555180 <puts@plt>
   0x000055555555565c <+221>: jmp    0x555555555693 <main+276>
   0x000055555555565e <+223>: cmp    DWORD PTR [rbp-0x38],0x270f - 9999
   0x0000555555555665 <+230>: jle    0x555555555675 <main+246>

   0x0000555555555675 <+246>: cmp    DWORD PTR [rbp-0x34],0x3
   0x0000555555555679 <+250>: jne    0x555555555687 <main+264>
   0x000055555555567b <+252>: mov    eax,0x0
   0x0000555555555680 <+257>: call   0x555555555449 <reveal_flag>

Hex input path

When the <_is_decimal> function does not return success, we jump execution to <is_valid_hex>

If the function return true(EAX=1, ZF=0)and we have a valid hex value the execution jumps to function stortol()<+171> where we convert the string to integer, using decimal base.

Else, we exit the program by jumping to <main+181> by printing a message to screen.

   0x00005555555555ef <+112>: call   0x5555555553d9 <is_valid_decimal>
   0x00005555555555f4 <+117>: test   eax,eax
   0x00005555555555f6 <+119>: je     0x555555555609 <main+138>

   0x0000555555555609 <+138>: lea    rax,[rbp-0x30]
   0x000055555555560d <+142>: mov    rdi,rax
   0x0000555555555610 <+145>: call   0x555555555369 <is_valid_hex>
   0x0000555555555615 <+150>: test   eax,eax
   0x0000555555555617 <+152>: je     0x555555555634 <main+181> - EXIT
   0x0000555555555619 <+154>: lea    rax,[rbp-0x30] - user input loaded to rax
   0x000055555555561d <+158>: mov    edx,0x10 - Decimal base
   0x0000555555555622 <+163>: mov    esi,0x0
   0x0000555555555627 <+168>: mov    rdi,rax
   0x000055555555562a <+171>: call   0x555555555200 <strtol@plt>
   0x000055555555562f <+176>: mov    DWORD PTR [rbp-0x38],eax
   0x0000555555555632 <+179>: jmp    0x555555555647 <main+200>

Now, since we have a valid hex input and it is converted to integer(rbp — 0x38) we jump to <main+_200> where similar to decimal input path we validate the input against minimum(999) and maximum(10000) values.

If both validations return true execution jumps to <main+246> where we compare the input string length (DWORD PTR [rbp — 0x34]) with 0x3.

In case the comparison holds true, execution goes on to execute the reveal_flag function <+257>.

   0x0000555555555647 <+200>: cmp    DWORD PTR [rbp-0x38],0x3e7 - 999
   0x000055555555564e <+207>: jg     0x55555555565e <main+223>
   0x0000555555555650 <+209>: lea    rdi,[rip+0xa2d]        # 0x555555556084
   0x0000555555555657 <+216>: call   0x555555555180 <puts@plt>
   0x000055555555565c <+221>: jmp    0x555555555693 <main+276>
   0x000055555555565e <+223>: cmp    DWORD PTR [rbp-0x38],0x270f - 9999
   0x0000555555555665 <+230>: jle    0x555555555675 <main+246>

   0x0000555555555675 <+246>: cmp    DWORD PTR [rbp-0x34],0x3
   0x0000555555555679 <+250>: jne    0x555555555687 <main+264>
   0x000055555555567b <+252>: mov    eax,0x0
   0x0000555555555680 <+257>: call   0x555555555449 <reveal_flag>

Getting the Flag…

Based on the analysis we’ve done so far the conditions for getting the flag are as follows:

1. User input has to be Hex which should be greater than 999.
2. And the user input length must be 3.

Using these conditions, we use the hex input 3E8 — >1000.

Press enter or click to view image in full size

As you would note that the string returned does not look much like a flag; seems inverted and appears to display a pattern of repeated characters.

We use python to reverse the string first, which makes the pattern of characters much easier to identify.

str = "}1a9ftc_oc_ipb50aftc_oc_ip8_99ftc_oc_ip9_TGftc_oc_ip_xehftc_oc_ip_tigftc_oc_ipid_3ftc_oc_ip{FTCftc_oc_ipocipftc_oc_ip"

for i in range(len(str), 0, -1):
     print(str[i-1], end='')

└─$ python rev.py                  
pi_co_ctfpicopi_co_ctfCTF{pi_co_ctf3_dipi_co_ctfgit_pi_co_ctfhex_pi_co_ctfGT_9pi_co_ctf99_8pi_co_ctfa05bpi_co_ctf9a1}                   

We can clean up the string by removing the pattern of characters “pi_co_ctf”.

str = "pi_co_ctfpicopi_co_ctfCTF{pi_co_ctf3_dipi_co_ctfgit_pi_co_ctfhex_pi_co_ctfGT_9pi_co_ctf99_8pi_co_ctfa05bpi_co_ctf9a1}"

newstr = str.replace("pi_co_ctf", "")

print(newstr)

This gives us the flag!

picoCTF{3_digit_hex_GT_999_8a05b9a1}

This was a fun challenge to write up and I hope this was a helpful read as well.

Thank you for reading.