5 min readNov 28, 2021

For this challenge we are provided with a source code and an encrypted file cipher.enc. Also we have a hint that the encrypted file is a JPEG image.

Let’s look at the source code:

We see that the code operates on two files: flag.bin and beaches.txt. flag.bin is obviously the encrypted file, beaches.txt is used to initialize bunch of variables that are lately combined into the key array. After that, the flag data from flag.bin is XORed with the key array and written to the cipher.enc. Also first 20 bytes of beaches.txt is written to the cipher.enc, but I don’t see how that would help us in this challenge, so we will ignore that 20 bytes and assume that cipher.enc is just an encrypted flag.

So basically this is yet another XOR challenge. How do we solve this? Well, the key contains 14 unique bytes, there is too much to brute force. Let’s use the knowledge of the file format. Let’s draft the solution source code:

We know that the encrypted file is a JPEG image. This means that it has to start with ff d8 and end with ff d9 according to the JPEG specification. This means that we can find first two bytes of the flag just XORing first two bytes of encrypted flag with ff d8. So we XOR the first encrypted byte with ff to get the value of c and the second byte with d8 to get the value of o

Next, as we know that flag ends with ff d9 we can XOR last two bytes of encrypted file with these values to determine two more bytes of the key. What bytes exactly? To find that out we calculate

So last two bytes of the flag are encrypted with 21st and 22nd bytes of the key. These bytes correspond to variables a and _ respectively:

To move further we need to use JPEG format specification. I used this document:

According to the document, after ff d8 (SOI) should go ff e0(APP0), so we can use that to find variables t and h:

Now, the APP0 itself is not only ff e0 but also a few other fields

There are two bytes of length (skip those for now), and after those there is JFIF string, which takes 5 bytes (4a 46 49 46 00). Those bytes are 6th, 7th, 8th, 9th and 10th bytes of the image respectively. This reveals corresponding key bytes (values _ (we know that already), s, a (also know that), and y)

After JFIF comes major and minor versions, but we don’t care, because those values corresponds to variables _ and y and we already know them, so move on. We also skip units field, because we already know the value of o.

Next goes field x-density which is 2 bytes long and the document tells us that this is normally 00 01. Could be another value, but let’s assume that this should be 00 01. This field is encrypted with bytes u and _. We already know _ so we just calculate u

After x-density goes y-density, which is also normally 00 01. This field is encrypted with a and r so again we only calculate value r.

Next goes width and height of the thumbnail. We could say that we don’t know these values, but look at our current decrypted file (width and height fields are highlighted with red rectangle):

We see that the thumbnail height is 00 and we know for sure it is because this byte is decrypted correctly (it is encrypted with _). And if the height is 00 we may assume that the width is also 00. We use that to calculate value e

The APP0 segment is finished, but we should take a step back and look at the field length again (bytes 4 and 5). We know 4th byte, but the 5th byte is encrypted with n and we don’t know that value yet. What we know is the length of APP0 segment now, which is 16 bytes (10 in hex)

We use that to calculate value n

Next image segment is ff db (DQT)

We already know the marker identifier field, but length field is encrypted with yet unknown to us values b and i. To calculate these values we search for the next marker. This marker is ff db again and it is on the offset 59.

This yields the length of the current ff db segment: 59 — 16 = 43. We use that to calculate values b and i

Running that code gives us the flag