Reverse engineering an unknown home-made ciphering algorithm

I was reminded of how fun wargames were in my time, so I decided to treat myself and give a shot at one. I went with Hack This Site. Some levels are more fun than others.

I started with the basic missions which doesn't last too long if you are used to these kinds of challenges. And, well, they are basic, more common sense than hacking. But I was pleasantly surprised when I found a "cryptic" challenge.

Let's crack it

You are given a form with a ciphered password, a hash, and your task is to find the unencrypted version. The hash was 0339fj=;.
You also have two forms; the first one is used to cipher whatever string you'd like and the second one is to send your answer.


Info gathering

I did a couple tries from Chrome, and as it often happens, I thought it would be harder and therefore take me more time, so I wanted to do it comfortably. And nothing better than BurpSuite for that.

I use a Chrome Extension called SwitchyOmega to send traffic to BurpSuite who is listening at localhost at my command.

In BurpSuite I'd leave the Proxy listening and from Chrome, send the string testing to the form so it can get registered, and I can send it to Burp's Intruder tab:

The website replied: Your encrypted string is: 'tfuwmsm'.

Now, within the intruder tab, more than brute forcing the app I want to send a few strings and try to figure out the pattern in the encryptions. So I'll tell Burp where to put my list of strings...


... and also where can it extract the answers from:


It is incredibly easy to use this grep functionality in Burp. Literally selecting the text I want from the test response and Burp works it out on the fly.

Here's the list of strings (payload) I want to send to start with. I put in also a blank string just to see what happens:


Now that everything's all set, we just Start the attack:


Isn't it comfortable? We didn't even have to use our browser. If we had needed 1000 tries, it would've been just as nice :)

Figuring it out

Now that we have a few encryptions we can tell at least a couple things:

  1. The first character is always left untouched
  2. The characters or numbers get bigger

But how do letters get bigger like numbers? - You may think - Well, through the power of American Standard Code for Information Interchange or ASCII. All characters in a computer are but representations of codes.

So, all characters in our strings are interpreted as they normally are: according to the ASCII codes. This is a standard ASCII table:


As you can see they are ordered, just like our regular alphabet, but with plenty more characters.

To say how the cipher works in one simple sentence: every character is its value plus its position number starting from 0.

Now all we have to do is reverse the process with the string they gave us, 0339fj=;.
There are eight characters, from position 0 to 7:

  • Position 0: we have to subtract 0, so it stays the same.
  • Position 1: 3 - 1 = 2
  • Position 2: 3 - 2 = 1
  • Position 3: 9 - 3 = 6
  • Position 4: f - 4 = b (just check the ASCII table and go back four characters)
  • Position 5: j - 5 = e
  • Position 6: = - 6 = 7
  • Position 7: ; - 7 = 4

So that's it: 0216be74. We check the solution, and we won a bit of the internets!