Discussion:
Detecting the present botnet attacks
David Anderson
2013-07-10 12:14:10 UTC
Permalink
Hi,

As most of us know, there are ongoing brute-force password attacks going
on worldwide, that are distributed and hence can't be prevented with
single-site "deny after X failed logins" heuristics. The individual IPs
only connect to each site once or twice. We are protecting some of our
websites with BruteProtect, but unfortunately not enough people are
using that solution yet to make it effective - in our analysis of our
logs, it catches about 10% of bad hosts. That's better than none,
especially if a site gets broken into, so we'll persevere with it - and
mention it so that others may perhaps take a look at it and help boost
that figure for all the users!

But regardless of that, the multiple attacks on various websites on the
same box are causing significant resource usage - perhaps around 25% of
all resource usage on the server in question. We'd like to kill the
attacks at an earlier stage - before the full WordPress load completes.
We're thinking of adding a single line of code using PHP's
auto_prepend_file feature (in php.ini) that will immediately die if an
attack pattern is detected. This solution has the beauty of being
something we can centrally manage fairly easily (our webserver's
configuration file is written on-the-fly by a shell script, including
per-site PHP-options). (Adding plugins to every site would be much more
cumbersome).

This line would detect that the URL being visited is wp-login.php, and
then apply some other tests. Looking at the logs we've collected from
the attackers, we see these two patterns:
1) The POST payload does not include [wp-submit] => 'Log In', which you
would have if logging in from the normal wp-login.php page.
2) The usernames are all one of:
* admin
* administrator
* the domain name of teh website being attacked
* {domain} (literally - presumably a bug in an earlier of the attacker's
code)

So, we're considering banning all those usernames from the hosted sites.
But, it looks like we could get a quicker result by blocking based on 1)
instead. Question: Does anyone know if that's reliable? i.e. are there
scenarios in which a likely user POST to /wp-login.php does not include
that field? (Note that the popular Theme My Login plugin doesn't POST to
/wp-login.php, so differences on its login form aren't important). (Of
course too, if someone's built some legitimate automated WP tools that
don't include that field, then this'll break them too - but we can deal
with that if it ever happens. I'm just interested in normal Joe User WP
usage).

Of course we could also combine 1) and 2) to reduce the risk. But I'm
still interested in the answer to the question.

Many thanks,
David
--
WordShell - WordPress fast from the CLI - www.wordshell.net
Nikola Nikolov
2013-07-10 13:37:21 UTC
Permalink
Hi David,

When the attacks began I started blocking users on the second incorrect
login attempt for the longest time possible via the Wordfence plugin. That
was sort of ok - but as you stated - it's a waste of server resources.

The following solution is at the WordPress end - I know that this is not
what you're really looking for, but you can take a look at it(and it might
help someone else as well).

At the end I got really annoyed with the attacks and I did a simple thing(
I'm warning you that it's probably not suitable for most of the people out
there, although it might be plausible with a filter that would automate one
part of the solution ) - I manually changed the wp-login.php file.

I know - modifying core files is not a good thing. But hey - I don't want
someone to poke at my site/server.

So my solution was to check at the top of the file for the presence of a
$_GET variable and check it's value. Basically if you go straight to
/wp-login.php you would see "Unauthorized access!" and that's it. You have
nowhere else to go and you can't do anything. If you go to
/wp-login.php?myspecialkeyword=series_of_alphanumeric_and_special_characters
you would actually see the form. From then(that's the harder part) I had to
manually change the "action" attribute of the login form to include the
above $_GET parameter. After that the login form would work.

I'm not sure how well that's working - but I believe that this solution
works better than not using anything...

Okay - I played a bit and made this really quick and dirty mu-plugin. You
can find it here - http://pastebin.com/pTyCMd1z . Put it in your
wp-content's "mu-plugins" directory to make it run automatically. It should
have no impact on your site's performance since it's code only works on the
wp-login.php page. Once you upload the file, change the "define( ..." lines
to something that is actually more secret than the defaults.

This plugin will wp_die() (you can probably switch to just die() ) with
status 500 displaying just the "Unauthorized login" message unless you're
visiting the proper URL - with the defaults it would be "
http://mysite.com/wp-login.php?mysecretkey=myv3ry53cr37n0nc3". Some
JavaScript will then run and fix all of the login-related links and the
login form's action attribute, so that you don't have to.

Once you have that in place, just save the URL as a bookmark and that
should make things a bit easier.

I'm not sure whether the return code 500 has any effect on the attacks
intensity(like discouraging them) - I've had 93 POST and GET requests to
wp-login.php for the last 10 days - I remember that the attacks were very
intensive at some point, so it might have some minor effect.

Hope that helps,
Nikola
Post by David Anderson
Hi,
As most of us know, there are ongoing brute-force password attacks going
on worldwide, that are distributed and hence can't be prevented with
single-site "deny after X failed logins" heuristics. The individual IPs
only connect to each site once or twice. We are protecting some of our
websites with BruteProtect, but unfortunately not enough people are using
that solution yet to make it effective - in our analysis of our logs, it
catches about 10% of bad hosts. That's better than none, especially if a
site gets broken into, so we'll persevere with it - and mention it so that
others may perhaps take a look at it and help boost that figure for all the
users!
But regardless of that, the multiple attacks on various websites on the
same box are causing significant resource usage - perhaps around 25% of all
resource usage on the server in question. We'd like to kill the attacks at
an earlier stage - before the full WordPress load completes. We're thinking
of adding a single line of code using PHP's auto_prepend_file feature (in
php.ini) that will immediately die if an attack pattern is detected. This
solution has the beauty of being something we can centrally manage fairly
easily (our webserver's configuration file is written on-the-fly by a shell
script, including per-site PHP-options). (Adding plugins to every site
would be much more cumbersome).
This line would detect that the URL being visited is wp-login.php, and
then apply some other tests. Looking at the logs we've collected from the
1) The POST payload does not include [wp-submit] => 'Log In', which you
would have if logging in from the normal wp-login.php page.
* admin
* administrator
* the domain name of teh website being attacked
* {domain} (literally - presumably a bug in an earlier of the attacker's
code)
So, we're considering banning all those usernames from the hosted sites.
But, it looks like we could get a quicker result by blocking based on 1)
instead. Question: Does anyone know if that's reliable? i.e. are there
scenarios in which a likely user POST to /wp-login.php does not include
that field? (Note that the popular Theme My Login plugin doesn't POST to
/wp-login.php, so differences on its login form aren't important). (Of
course too, if someone's built some legitimate automated WP tools that
don't include that field, then this'll break them too - but we can deal
with that if it ever happens. I'm just interested in normal Joe User WP
usage).
Of course we could also combine 1) and 2) to reduce the risk. But I'm
still interested in the answer to the question.
Many thanks,
David
--
WordShell - WordPress fast from the CLI - www.wordshell.net
______________________________**_________________
wp-hackers mailing list
http://lists.automattic.com/**mailman/listinfo/wp-hackers<http://lists.automattic.com/mailman/listinfo/wp-hackers>
Jeff Morris
2013-07-11 06:14:28 UTC
Permalink
Post by Nikola Nikolov
This plugin will wp_die() (you can probably switch to just die() ) with
status 500 displaying just the "Unauthorized login" message unless you're
visiting the proper URL
I have a couple of problems with that response.

First, a real HTTP 500 code indicates a fatal internal server error that
could result from anything, such as a typo in your .htaccess. In your
case no such error has occurred, so the 500 code is at best untrue.

Second, the 'Unauthorized login' message upon wp_die() or die() (which I
presume will be read and understood by someone attempting a hands-on
unauthorized login) has an adversarial twang to it that might just
antagonize the scriddies out there.

If you're choosing to deny access to a service for whatever reason, it
might make sense to terminate with an HTTP 503 Service Unavailable, and
the message 'Sorry, this service is unavailable.'

It's honest and it's innocuous, but like any 'solution' it still won't
stop them knocking at the door.
John Blackbourn
2013-07-11 08:26:18 UTC
Permalink
Post by Jeff Morris
First, a real HTTP 500 code indicates a fatal internal server error that
could result from anything, such as a typo in your .htaccess. In your case
no such error has occurred, so the 500 code is at best untrue.
FYI you can pass a different HTTP status code to wp_die() in the $args
parameter. Example:

wp_die( 'Sorry, this service is unavailable', get_bloginfo('name'),
array( 'response' => 503 ) );
Nikola Nikolov
2013-07-11 11:31:29 UTC
Permalink
@Jeff - you're right about the error code, but that's what WordPress
defaults to. As @John said - you can change the response code to anything
you want. I just wrote that quickly and I generally am the only one
logging-in, so I don't expect the users to log-in.

If you have users that are supposed to log-in to the site and you don't
have any alternative set-up(you can always have a front-end login form
which has nothing to do with wp-login.php), you can leave them a visible
message, a sticky post or whatever saying how they can log in.

The attack is massive and is targeted at the very small percentage that is
not protected well(has easy to predict username and password), so I would
guarantee you that at least at the current stage no one will actually stop
and read your site to see if there's a work-around for logging-in to the
site.


On Thu, Jul 11, 2013 at 11:26 AM, John Blackbourn
Post by John Blackbourn
Post by Jeff Morris
First, a real HTTP 500 code indicates a fatal internal server error that
could result from anything, such as a typo in your .htaccess. In your
case
Post by Jeff Morris
no such error has occurred, so the 500 code is at best untrue.
FYI you can pass a different HTTP status code to wp_die() in the $args
wp_die( 'Sorry, this service is unavailable', get_bloginfo('name'),
array( 'response' => 503 ) );
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
Jeff Morris
2013-07-11 12:03:32 UTC
Permalink
Post by John Blackbourn
FYI you can pass a different HTTP status code to wp_die() in the $args
parameter.
Right, John. It is that simple.

My point about HTTP 500 is that it ought to signal very serious "trouble
at t'mill". Ideally, you want to never see HTTP 500 responses in server
access logs.

The wp_die() documentation alludes to this in a round-about way:

"It is not recommended to call this function very often and try to
handle as many errors as possible silently."

And if you're unhappy about the bloated output from wp_die(), you can
replace the entire _default_wp_die_handler() function with an
add_filter() hook.
Jeff Morris
2013-07-11 13:32:00 UTC
Permalink
Post by David Anderson
But, it looks like we could get a quicker result by blocking based on
1) instead. Question: Does anyone know if that's reliable? i.e. are
there scenarios in which a likely user POST to /wp-login.php does not
include that field?
I've yet to see a regular bona-fide login or registration that doesn't
carry the aforementioned field in the $_REQUEST. But now that it's been
pointed out here, maybe we should expect to start seeing it in the
payload ;)

I capture a lot of these admin brute-forces, and boy are they dull. One
day last month I watched one feeding on 'Service Unavailable' for over
five hours before I pulled its plug. Such was its sophistication it just
kept on coming and chewing 403s for a futher 90 minutes.

In the case of a botnet, look for cookie-cutter traits, such as
commonality in the HTTP protocol version, referrer and user agent
fields, inter alia. A conclusion based on a combination of ticked boxes
is bound to be more reliable.
w***@on-e.com
2013-07-11 14:25:25 UTC
Permalink
Post by David Anderson
But regardless of that, the multiple attacks on various websites on the
same box are causing significant resource usage - perhaps around 25% of
all resource usage on the server in question. We'd like to kill the
attacks at an earlier stage - before the full WordPress load completes.
We're thinking of adding a single line of code using PHP's
auto_prepend_file feature (in php.ini) that will immediately die if an
attack pattern is detected. This solution has the beauty of being
something we can centrally manage fairly easily (our webserver's
configuration file is written on-the-fly by a shell script, including
per-site PHP-options). (Adding plugins to every site would be much more
cumbersome).
This line would detect that the URL being visited is wp-login.php, and
then apply some other tests. Looking at the logs we've collected from
1) The POST payload does not include [wp-submit] => 'Log In', which you
would have if logging in from the normal wp-login.php page.
* admin
* administrator
* the domain name of teh website being attacked
* {domain} (literally - presumably a bug in an earlier of the attacker's
code)
auto_prepend_file would be better than letting Wordpress actually boot
up, but I would suggest thinking something even earlier in the process
(if you can) - mod_security, which runs very early in the http request
process.

You could probably adapt your rules into mod_security rules, plus you'd
get all the other benefits as well.
Brian Layman
2013-07-11 19:02:49 UTC
Permalink
No black list is going to be perfect, but you might try:
http://www.projecthoneypot.org/httpbl_implementations.php

I've had some success with it recently shutting down attacks Joomla
sites. Plus it is an Apache module, so no plugin and it works
automatically for all sites on the server. Plus projecthoneypot has
been around for a long time and has a fairly large database.

I've seen no speed or resource issues on the server I've installed it
on. That particular server hosts about 100 business sites each with a
typical traffic level for a non-interactive informational site.

Brian Layman
Post by David Anderson
Hi,
As most of us know, there are ongoing brute-force password attacks
going on worldwide, that are distributed and hence can't be prevented
with single-site "deny after X failed logins" heuristics. The
individual IPs only connect to each site once or twice. We are
protecting some of our websites with BruteProtect, but unfortunately
not enough people are using that solution yet to make it effective -
in our analysis of our logs, it catches about 10% of bad hosts. That's
better than none, especially if a site gets broken into, so we'll
persevere with it - and mention it so that others may perhaps take a
look at it and help boost that figure for all the users!
But regardless of that, the multiple attacks on various websites on
the same box are causing significant resource usage - perhaps around
25% of all resource usage on the server in question. We'd like to kill
the attacks at an earlier stage - before the full WordPress load
completes. We're thinking of adding a single line of code using PHP's
auto_prepend_file feature (in php.ini) that will immediately die if an
attack pattern is detected. This solution has the beauty of being
something we can centrally manage fairly easily (our webserver's
configuration file is written on-the-fly by a shell script, including
per-site PHP-options). (Adding plugins to every site would be much
more cumbersome).
This line would detect that the URL being visited is wp-login.php, and
then apply some other tests. Looking at the logs we've collected from
1) The POST payload does not include [wp-submit] => 'Log In', which
you would have if logging in from the normal wp-login.php page.
* admin
* administrator
* the domain name of teh website being attacked
* {domain} (literally - presumably a bug in an earlier of the
attacker's code)
So, we're considering banning all those usernames from the hosted
sites. But, it looks like we could get a quicker result by blocking
based on 1) instead. Question: Does anyone know if that's reliable?
i.e. are there scenarios in which a likely user POST to /wp-login.php
does not include that field? (Note that the popular Theme My Login
plugin doesn't POST to /wp-login.php, so differences on its login form
aren't important). (Of course too, if someone's built some legitimate
automated WP tools that don't include that field, then this'll break
them too - but we can deal with that if it ever happens. I'm just
interested in normal Joe User WP usage).
Of course we could also combine 1) and 2) to reduce the risk. But I'm
still interested in the answer to the question.
Many thanks,
David
Nicolás Badano
2013-07-11 21:12:04 UTC
Permalink
We too have been having quite a headache with the bot attacks recently.
In our case, what we did was installing the wp-fail2ban plugin (no more
than two lines of code that log unsuccessful login attempts in the
auth.log file) and configured fail2ban to monitor that logfile with the
regex included in the plugin. Three failed logins, and we shut down the
server for that IP (Deny from XX.XXX.XXX.XXX in the main .htaccess). An
iptables ban would probably accomplish the same thing, or the denyhosts
action. As we don't have an admin or administrator account, we are
looking into banning tries using those accounts right away from the
first try, but I don't have code for that just yet.

It's less sophisticated than stopping the botnet on its tracks by
identifying a pattern (that would be GREAT) but it did help containing
the bot invasion. We are not getting that many failed logins these days.
I like how the Project Honey Pot looks like though: I'll probably give
it a try, specially if it doesn't hurt performance too much.

My two cents!
Tangren, Gerald Vernon
2013-07-11 21:22:46 UTC
Permalink
How do some of the methods suggested here the last week or so compare with
an inclusive plug-in such as Better WP Security?

http://wordpress.org/plugins/better-wp-security/
--
Jerry <***@wsu.edu>
WA State University-Tree Fruit Research & Extension Center
Wenatchee, WA
509-663-8181 x 231
USDA Cold Hardiness Zone 7a (during the current phase of the Pacific
Decadal
Oscillation)
http://www.tfrec.wsu.edu/pages/webdev/Favorites


³It¹s folks knowing so much that ain¹t so.² - Henry Wheeler Shaw
Post by Nicolás Badano
We too have been having quite a headache with the bot attacks recently.
In our case, what we did was installing the wp-fail2ban plugin (no more
than two lines of code that log unsuccessful login attempts in the
auth.log file) and configured fail2ban to monitor that logfile with the
regex included in the plugin. Three failed logins, and we shut down the
server for that IP (Deny from XX.XXX.XXX.XXX in the main .htaccess). An
iptables ban would probably accomplish the same thing, or the denyhosts
action. As we don't have an admin or administrator account, we are
looking into banning tries using those accounts right away from the
first try, but I don't have code for that just yet.
It's less sophisticated than stopping the botnet on its tracks by
identifying a pattern (that would be GREAT) but it did help containing
the bot invasion. We are not getting that many failed logins these days.
I like how the Project Honey Pot looks like though: I'll probably give
it a try, specially if it doesn't hurt performance too much.
My two cents!
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
Les Bessant
2013-07-11 21:23:41 UTC
Permalink
I've been using fail2ban, but I'm still seeing numerous single attempts to log on to my site - and they're not trying for "admin", they're actually targeting the user name that I post with. Getting one attempt at a time from numerous addresses.

Looks like it's time to go back to using bad behaviour.

--
Les Bessant ***@lcb.me.uk
Losing it - http://losingit.me.uk/
Les Bessant Photography - http://lesbessant-photography.co.uk
We too have been having quite a headache with the bot attacks recently. In our case, what we did was installing the wp-fail2ban plugin (no more than two lines of code that log unsuccessful login attempts in the auth.log file) and configured fail2ban to monitor that logfile with the regex included in the plugin. Three failed logins, and we shut down the server for that IP (Deny from XX.XXX.XXX.XXX in the main .htaccess). An iptables ban would probably accomplish the same thing, or the denyhosts action. As we don't have an admin or administrator account, we are looking into banning tries using those accounts right away from the first try, but I don't have code for that just yet.
It's less sophisticated than stopping the botnet on its tracks by identifying a pattern (that would be GREAT) but it did help containing the bot invasion. We are not getting that many failed logins these days. I like how the Project Honey Pot looks like though: I'll probably give it a try, specially if it doesn't hurt performance too much.
My two cents!
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
Micky Hulse
2013-07-11 21:27:15 UTC
Permalink
This might be dumb to ask, but would security through obscurity work?
I know it's not a high level solution, but would hiding all of the
obvious routes one can login/adminster WP be one way to stop attacks?

Is there a good solution in WP to change URL to login/admin pages?
Doug Smith
2013-07-12 14:48:33 UTC
Permalink
You could add another log in layer with basic HTTP authentication to protect your wp-admin directory. Or you could use the Google Authenticator plugin (http://wordpress.org/plugins/google-authenticator/) to give you second factor authentication through a smartphone.

fial2ban is still nice, though, because anything you can stop with it happens at the firewall so WordPress doesn't even see it, which is helpful for both performance and security. If you haven't seen it yet, there is a WP fail2ban plugin (http://wordpress.org/plugins/wp-fail2ban/) that will log WordPress login attempts so they can be used in fail2ban too.

--
Doug Smith: ***@smithsrus.com
http://smithsrus.com
Post by Les Bessant
I've been using fail2ban, but I'm still seeing numerous single attempts to log on to my site - and they're not trying for "admin", they're actually targeting the user name that I post with. Getting one attempt at a time from numerous addresses.
Looks like it's time to go back to using bad behaviour.
--
Losing it - http://losingit.me.uk/
Les Bessant Photography - http://lesbessant-photography.co.uk
Les Bessant
2013-07-13 05:57:58 UTC
Permalink
Yes, i've been using wp-fail2ban for a while. It seemed to be working for a while, but the attacks have changed from multiple attempts to single attempts from each IP address, making it less effective.

Installed Bad Behaviour (http://bad-behavior.ioerror.us/) which seems to be blocking the botnet attacks.

--
Les Bessant ***@lcb.me.uk
Losing it - http://losingit.me.uk/
Les Bessant Photography - http://lesbessant-photography.co.uk
Post by Doug Smith
You could add another log in layer with basic HTTP authentication to protect your wp-admin directory. Or you could use the Google Authenticator plugin (http://wordpress.org/plugins/google-authenticator/) to give you second factor authentication through a smartphone.
fial2ban is still nice, though, because anything you can stop with it happens at the firewall so WordPress doesn't even see it, which is helpful for both performance and security. If you haven't seen it yet, there is a WP fail2ban plugin (http://wordpress.org/plugins/wp-fail2ban/) that will log WordPress login attempts so they can be used in fail2ban too.
--
http://smithsrus.com
Post by Les Bessant
I've been using fail2ban, but I'm still seeing numerous single attempts to log on to my site - and they're not trying for "admin", they're actually targeting the user name that I post with. Getting one attempt at a time from numerous addresses.
Looks like it's time to go back to using bad behaviour.
--
Losing it - http://losingit.me.uk/
Les Bessant Photography - http://lesbessant-photography.co.uk
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
Sam Hotchkiss
2013-07-13 01:39:45 UTC
Permalink
Hey David-- thanks for the BruteProtect plug! On that front, we're hard at
work on improving our algorithms-- we've tripled our installed base over
the past 10 days as we've gotten some press, and so we're working on
improving our attack recognition, banning real attackers for longer and
allowing legitimate users to get back in sooner. You won't need to do
anything, as the updates will happen on our end, but you should see those
numbers improve over the next week.

Best,
Sam
--
Sam Hotchkiss :: Principal :: Hotchkiss Consulting Group
122 Front Street, Second Floor, Bath, Maine 04530
P: 207.200.4314 :: F: 207.209.1365
Post by David Anderson
Hi,
As most of us know, there are ongoing brute-force password attacks going
on worldwide, that are distributed and hence can't be prevented with
single-site "deny after X failed logins" heuristics. The individual IPs
only connect to each site once or twice. We are protecting some of our
websites with BruteProtect, but unfortunately not enough people are using
that solution yet to make it effective - in our analysis of our logs, it
catches about 10% of bad hosts. That's better than none, especially if a
site gets broken into, so we'll persevere with it - and mention it so that
others may perhaps take a look at it and help boost that figure for all the
users!
But regardless of that, the multiple attacks on various websites on the
same box are causing significant resource usage - perhaps around 25% of all
resource usage on the server in question. We'd like to kill the attacks at
an earlier stage - before the full WordPress load completes. We're thinking
of adding a single line of code using PHP's auto_prepend_file feature (in
php.ini) that will immediately die if an attack pattern is detected. This
solution has the beauty of being something we can centrally manage fairly
easily (our webserver's configuration file is written on-the-fly by a shell
script, including per-site PHP-options). (Adding plugins to every site
would be much more cumbersome).
This line would detect that the URL being visited is wp-login.php, and
then apply some other tests. Looking at the logs we've collected from the
1) The POST payload does not include [wp-submit] => 'Log In', which you
would have if logging in from the normal wp-login.php page.
* admin
* administrator
* the domain name of teh website being attacked
* {domain} (literally - presumably a bug in an earlier of the attacker's
code)
So, we're considering banning all those usernames from the hosted sites.
But, it looks like we could get a quicker result by blocking based on 1)
instead. Question: Does anyone know if that's reliable? i.e. are there
scenarios in which a likely user POST to /wp-login.php does not include
that field? (Note that the popular Theme My Login plugin doesn't POST to
/wp-login.php, so differences on its login form aren't important). (Of
course too, if someone's built some legitimate automated WP tools that
don't include that field, then this'll break them too - but we can deal
with that if it ever happens. I'm just interested in normal Joe User WP
usage).
Of course we could also combine 1) and 2) to reduce the risk. But I'm
still interested in the answer to the question.
Many thanks,
David
--
WordShell - WordPress fast from the CLI - www.wordshell.net
______________________________**_________________
wp-hackers mailing list
http://lists.automattic.com/**mailman/listinfo/wp-hackers<http://lists.automattic.com/mailman/listinfo/wp-hackers>
Continue reading on narkive:
Loading...