We got a 64-bit ELF for this challenge. Running strings shows the use of Obfuscator-LLVM

Obfuscator-LLVM clang version 3.6.1 (tags/RELEASE_361/final) (based on Obfuscator-LLVM 3.6.1)

The binary expects a valid key!

$ ./wyvern_c85f1be480808a9da350faaa6104a19b 
+-----------------------+
|    Welcome Hero       |
+-----------------------+

[!] Quest: there is a dragon prowling the domain.
 brute strength and magic is our only hope. Test your skill.

Enter the dragon's secret:

For further analysis, I used PIN based tracer. First objective was to find the length. Supplying an input of 100 bytes, the CMP instruction was found using PIN

0x4046b6 : cmp rax, rcx
0x4046b6 : [0x64] [0x1c]

The length of key is 28 bytes. Now lets see if we could find the algorithm by tracking user input using PIN tool. Supplying input as BAAAAAAAAAAAAAAAAAAAAAAAAAAA, below instructions were found:

0x4017e8 : add ecx, dword ptr [rax]
0x4017e8 : [0x42] [0] := [0x42]      -> input B
--
0x402a7f : cmp eax, ecx
0x402a7f : [0x64] [0x42]      -> compared with 0x64

Now supplying input as dBAAAAAAAAAAAAAAAAAAAAAAAAAA

0x4017e8 : add ecx, dword ptr [rax]
0x4017e8 : [0x42] [0x64] := [0xa5]   -> input A and 0x64 from previous operation
--
0x402a7f : cmp eax, ecx       
0x402a7f : [0xd6] [0xa5]

So the algorithm reads each byte of user input, compares with a hard coded array, which is a sum input and previous result

0x00 + input[0] == 0x64
0x64 + input[1] == 0xd6
.....

Lets fetch the array using GDB by setting breakpoint at 0x402a7f, and compute the flag

import gdb

sum_array = [0]
def exit_handler(event):
    key = ''
    for i in range(len(sum_array)-1):
        key += chr(sum_array[i+1] - sum_array[i]) 
    print key
    
def callback_fetch_array():
    EAX = int(gdb.parse_and_eval("$eax"))
    sum_array.append(EAX)
    gdb.execute("set $ecx = $eax")

class HitBreakpoint(gdb.Breakpoint):
    def __init__(self, loc, callback):
        super(HitBreakpoint, self).__init__(
            loc, gdb.BP_BREAKPOINT, internal=False)
        self.callback = callback
        
    def stop(self):
        self.callback()
        return False

HitBreakpoint("*0x402a7f", callback_fetch_array) 
gdb.events.exited.connect(exit_handler)

$ gdb -q ./wyvern_c85f1be480808a9da350faaa6104a19b

gdb-peda$ source re500.py
Breakpoint 1 at 0x402a7f

gdb-peda$ run
Enter the dragon's secret: AAAAAAAAAAAAAAAAAAAAAAAAAAAA

[+] A great success! Here is a flag{AAAAAAAAAAAAAAAAAAAAAAAAAAAA}
[Inferior 1 (process 33655) exited normally]
dr4g0n_or_p4tric1an_it5_LLVM

So the key is dr4g0n_or_p4tric1an_it5_LLVM