WordPress plugin inspections

Discussion:

Harry Metcalfe

2014-02-19 18:43:42 UTC

Hello list,

We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.

Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?

Here's an example report:

https://security.dxw.com/plugins/pods-custom-content-types-and-fields/

Grateful for a steer...

Harry

--
Harry Metcalfe
07790 559 876
@harrym

Josh Pollock

2014-02-19 19:45:07 UTC

Permalink

Harry-

I am the community manager for Pods we were made aware of your evaluation
by a user who reported it in our GitHub issue tracker. Our leader
developer, Scott K. Clark, has responded to your claims, which we do not
consider to be fair, here:

https://github.com/pods-framework/pods/issues/2043#issuecomment-35538379

I would encourage you to contact the developers of plugins before releasing
vulnerability reports. This sort of vague report doesn't help us improve
our plugin, something we are constantly doing based on input from users. It
only serves to potentially confuse users.

Take care,
Josh Pollock

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that we
do for our clients. They are just a guide - we conduct some basic checks,
not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-19 20:40:45 UTC

Permalink

Hi Josh,

Thanks for the heads-up. I've had a quick look at the github issue -
I'll reply to that feedback there.

Regarding a private report - this isn't a vulnerability report. We do
those too (see the Advisories section) and we have a disclosure policy
for those which you can see here (https://security.dxw.com/disclosure/).

Inspections are a very light touch thing, and we don't think they go
into enough detail to be able to make categorical claims about
vulnerability. The idea behind an inspection is to give a general sense
of the sorts of issues which might exist. I'm about to reply to Chris's
post with more explanation on that point.

Harry

Post by Josh Pollock
Harry-
I am the community manager for Pods we were made aware of your evaluation
by a user who reported it in our GitHub issue tracker. Our leader
developer, Scott K. Clark, has responded to your claims, which we do not
https://github.com/pods-framework/pods/issues/2043#issuecomment-35538379
I would encourage you to contact the developers of plugins before releasing
vulnerability reports. This sort of vague report doesn't help us improve
our plugin, something we are constantly doing based on input from users. It
only serves to potentially confuse users.
Take care,
Josh Pollock

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Williams

2014-02-19 20:43:16 UTC

Permalink

BUT THAT'S EXACTLY WHAT YOU DO! Your website has a bold red mark that
says "Unsafe for use" on many plugins!

Post by Harry Metcalfe
Inspections are a very light touch thing, and we don't think they go
into enough detail to be able to make categorical claims about
vulnerability.

Chip Bennett

2014-02-19 20:44:18 UTC

Permalink

For me, the incongruity happens when a "light touch review" leads to an
"Unsafe To Use" conclusion. I don't see how you can justify such a
conclusion without actually evaluating the code.

Post by Harry Metcalfe
Hi Josh,
Thanks for the heads-up. I've had a quick look at the github issue - I'll
reply to that feedback there.
Regarding a private report - this isn't a vulnerability report. We do
those too (see the Advisories section) and we have a disclosure policy for
those which you can see here (https://security.dxw.com/disclosure/).
Inspections are a very light touch thing, and we don't think they go into
enough detail to be able to make categorical claims about vulnerability.
The idea behind an inspection is to give a general sense of the sorts of
issues which might exist. I'm about to reply to Chris's post with more
explanation on that point.
Harry

Post by Harry Metcalfe
We write and publish light-touch inspections of WordPress plugins that we
do for our clients. They are just a guide - we conduct some basic checks,
not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
_______________________________________________

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-19 20:55:25 UTC

Permalink

Hi Chris, Chip,

We do evaluate the code. An inspection usually takes about an hour, and
we review the code for the things listed.

I can appreciate the point you're making here, but we do need that
process to result in some conclusion that we can use to decide what to
do. And as far as we're concerned, plugins which are littered with XSS
and don't prepare their queries are, in fact, unsafe to use.

That said, we have tried our best to make it clear what that means. The

This plugin should not be used unless very careful consideration is
given to the vulnerabilities it probably contains and ways to mitigate
them.

Which, I think, is generally good advice.

Harry

For me, the incongruity happens when a "light touch review" leads to an
"Unsafe To Use" conclusion. I don't see how you can justify such a
conclusion without actually evaluating the code.

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Jamie Currie

2014-02-19 20:50:41 UTC

Permalink

I totally understand the particular criticisms being made here, and
agree 100% that if you're going to put out a review it should be
thorough, transparent, have an ability for redress, etc.

Still, I think the basic concept has merit. Right now plugins are a bit
like Russian Roulette -- install one, cross fingers, see what happens to
site. And for those who don't program or have limited knowledge you're
really taking a chance because you truly have to trust that what you're
installing isn't introducing vulnerabilities that you'd have no way of
discovering or remedying.

If I put out a plugin I'd love to have people on this list tear it apart
so that I can improve it. And I'd love to see some sort of a good
housekeeping seal for plugins that have undergone a more rigorous
evaluation.

I haven't released any of my plugins publicly thus far -- they've all
been for private client projects -- but I do have a handful I hope to
put out this year. But as it stands, I have no skin in this game, just
chipping in my opinion.

Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers

------ Original Message ------
From: "Harry Metcalfe" <***@dxw.com>
To: wp-***@lists.automattic.com
Sent: 2/19/2014 12:40:45 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
Hi Josh,
Thanks for the heads-up. I've had a quick look at the github issue -
I'll reply to that feedback there.
Regarding a private report - this isn't a vulnerability report. We do
those too (see the Advisories section) and we have a disclosure policy
for those which you can see here
(https://security.dxw.com/disclosure/).
Inspections are a very light touch thing, and we don't think they go
into enough detail to be able to make categorical claims about
vulnerability. The idea behind an inspection is to give a general sense
of the sorts of issues which might exist. I'm about to reply to Chris's
post with more explanation on that point.
Harry

Post by Josh Pollock
Harry-
I am the community manager for Pods we were made aware of your
evaluation
by a user who reported it in our GitHub issue tracker. Our leader
developer, Scott K. Clark, has responded to your claims, which we do not
https://github.com/pods-framework/pods/issues/2043#issuecomment-35538379
I would encourage you to contact the developers of plugins before releasing
vulnerability reports. This sort of vague report doesn't help us improve
our plugin, something we are constantly doing based on input from users. It
only serves to potentially confuse users.
Take care,
Josh Pollock

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Williams

2014-02-19 20:17:17 UTC

Permalink

I certainly can't speak for others, but I would venture to say that your
business model is evil at best. You do fly-by character assassination
(oops, I mean "light-touch inspections"), based on personal bias ("this
plugin is large"), and then broadly publish the results as if they are
somehow authoritative. Worse yet, you then hold plugin developers at
ransom for changing the review: "If you would like to commission us to
inspect or review the latest version, please contact us."

How this is of value to anyone, and how you sleep at night with this
specious business model, is completely beyond me.

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Jamie Currie

2014-02-19 20:27:44 UTC

Permalink

I had the exact opposite reaction to Chris Williams. Literally a week
ago I was talking to someone about the need for more rigorous evaluation
of plugins. I find that I now use only a small handful of plugins that I
have extensive experience with because of the lack of any quality
standard.

If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins.
And that's just the blatantly obvious stuff. I won't point fingers, but
I recently had issues with one pretty popular plugin and when I went
into the code to poke around I found that it is fundamentally flawed in
the design -- so much so that I rewrote it and will be sending the
author the new code and explanation.

I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps
the next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community
involvement (maybe via this list) to "review the reviews" if you will.
I'd be happy to get involved in a process like that if the end result
were a base of plugins that had been scrutinized by some of the WP
brains on this list.

And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a
clear need and at least attempted to address it -- which is pretty much
the story of every successful business.

Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers

------ Original Message ------
From: "Chris Williams" <***@clwill.com>
To: "wp-***@lists.automattic.com" <wp-***@lists.automattic.com>
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Chris Williams
I certainly can't speak for others, but I would venture to say that your
business model is evil at best. You do fly-by character assassination
(oops, I mean "light-touch inspections"), based on personal bias ("this
plugin is large"), and then broadly publish the results as if they are
somehow authoritative. Worse yet, you then hold plugin developers at
ransom for changing the review: "If you would like to commission us to
inspect or review the latest version, please contact us."
How this is of value to anyone, and how you sleep at night with this
specious business model, is completely beyond me.

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Kirk Wight

2014-02-19 20:35:04 UTC

Permalink

I find the example "light-touch" (what does that mean?) inspection to be of
very little value at all. Phrases like "very likely to have", "contains or
is likely to contain", and "probably contains" mean nothing when it comes
to security: is it, or isn't it? More complete reviews with definitive
statements and examples would be of much more interest to myself.

I had the exact opposite reaction to Chris Williams. Literally a week ago
I was talking to someone about the need for more rigorous evaluation of
plugins. I find that I now use only a small handful of plugins that I have
extensive experience with because of the lack of any quality standard.
If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins. And
that's just the blatantly obvious stuff. I won't point fingers, but I
recently had issues with one pretty popular plugin and when I went into the
code to poke around I found that it is fundamentally flawed in the design
-- so much so that I rewrote it and will be sending the author the new code
and explanation.
I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps the
next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community involvement
(maybe via this list) to "review the reviews" if you will. I'd be happy to
get involved in a process like that if the end result were a base of
plugins that had been scrutinized by some of the WP brains on this list.
And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a clear
need and at least attempted to address it -- which is pretty much the story
of every successful business.
Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers
------ Original Message ------
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Josh Pollock

2014-02-19 20:38:21 UTC

Permalink

Jamie-

The ability to easily do an independent security review of open source
software, is one of the strengths of the open source model. But publishing
vague results, and not contacting the developer, and/ or
***@wordpress.org, with any concrete details of a threat doesn't help
the developer, the community or the users. If anyone can identify a
specific security threat in Pods, please email ***@pods.io and we will
address it, like any other responsible developer would.

Take care,
Josh

I had the exact opposite reaction to Chris Williams. Literally a week ago
I was talking to someone about the need for more rigorous evaluation of
plugins. I find that I now use only a small handful of plugins that I have
extensive experience with because of the lack of any quality standard.
If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins. And
that's just the blatantly obvious stuff. I won't point fingers, but I
recently had issues with one pretty popular plugin and when I went into the
code to poke around I found that it is fundamentally flawed in the design
-- so much so that I rewrote it and will be sending the author the new code
and explanation.
I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps the
next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community involvement
(maybe via this list) to "review the reviews" if you will. I'd be happy to
get involved in a process like that if the end result were a base of
plugins that had been scrutinized by some of the WP brains on this list.
And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a clear
need and at least attempted to address it -- which is pretty much the story
of every successful business.
Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers
------ Original Message ------
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Ipstenu the Half-Elf

2014-02-19 20:49:23 UTC

Permalink

The security email is for WP core security.

Post by Josh Pollock
Jamie-
The ability to easily do an independent security review of open source
software, is one of the strengths of the open source model. But publishing
vague results, and not contacting the developer, and/ or
the developer, the community or the users. If anyone can identify a
address it, like any other responsible developer would.
Take care,
Josh

I had the exact opposite reaction to Chris Williams. Literally a week ago
I was talking to someone about the need for more rigorous evaluation of
plugins. I find that I now use only a small handful of plugins that I have
extensive experience with because of the lack of any quality standard.
If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins. And
that's just the blatantly obvious stuff. I won't point fingers, but I
recently had issues with one pretty popular plugin and when I went into the
code to poke around I found that it is fundamentally flawed in the design
-- so much so that I rewrote it and will be sending the author the new code
and explanation.
I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps the
next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community involvement
(maybe via this list) to "review the reviews" if you will. I'd be happy to
get involved in a process like that if the end result were a base of
plugins that had been scrutinized by some of the WP brains on this list.
And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a clear
need and at least attempted to address it -- which is pretty much the story
of every successful business.
Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers
------ Original Message ------
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-19 23:50:10 UTC

Permalink

Hi Josh,

There's a good thread going on Github.

H

I had the exact opposite reaction to Chris Williams. Literally a week ago
I was talking to someone about the need for more rigorous evaluation of
plugins. I find that I now use only a small handful of plugins that I have
extensive experience with because of the lack of any quality standard.
If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins. And
that's just the blatantly obvious stuff. I won't point fingers, but I
recently had issues with one pretty popular plugin and when I went into the
code to poke around I found that it is fundamentally flawed in the design
-- so much so that I rewrote it and will be sending the author the new code
and explanation.
I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps the
next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community involvement
(maybe via this list) to "review the reviews" if you will. I'd be happy to
get involved in a process like that if the end result were a base of
plugins that had been scrutinized by some of the WP brains on this list.
And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a clear
need and at least attempted to address it -- which is pretty much the story
of every successful business.
Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers
------ Original Message ------
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Williams

2014-02-19 20:39:31 UTC

Permalink

- Who elected DXW as the keeper of the standards?
- How does DXW establish their standards ("this plugin is large")?
- Can anyone have input into those standards?
- What value is there in a "cursory review"? If you don't understand the
code, you don't have a leg to stand on.
- What recourse other than paying them is there to a bad review?
- If an author takes their input and fixes their plugin, will DXW redo the
review without payment, and then repair all the negative publicity they
have generated?
- If I use a plugin that DXW has reviewed as safe, and I get hacked, can I
sue DXW?
- How does one insure that DXW doesn't have ulterior motives in their
reviews? Competitive products, products they don't like/use, authors who
pay for reviews being treated better than authors who don't?

This is just like Yelp -- a half-baked idea made even worse by little or
no validation of the source of the review.

Post by Jamie Currie
I had the exact opposite reaction to Chris Williams. Literally a week
ago I was talking to someone about the need for more rigorous evaluation
of plugins. I find that I now use only a small handful of plugins that I
have extensive experience with because of the lack of any quality
standard.
If that sounds a bit harsh, I'd suggest enabling DEBUG and mysql slow
query (at something like 1 second) and then test out various plugins.
And that's just the blatantly obvious stuff. I won't point fingers, but
I recently had issues with one pretty popular plugin and when I went
into the code to poke around I found that it is fundamentally flawed in
the design -- so much so that I rewrote it and will be sending the
author the new code and explanation.
I understand that a cursory review is subjective and prone to
misstatements, but it's at least a step in the right direction. Perhaps
the next step would be for Harry to formalize some kind of process for
responding to / contesting reviews and to encourage community
involvement (maybe via this list) to "review the reviews" if you will.
I'd be happy to get involved in a process like that if the end result
were a base of plugins that had been scrutinized by some of the WP
brains on this list.
And if, at the end of the day, he harnesses that power to help build a
business, I don't see anything wrong with that either. I think 99% of us
are using WP to make money and it seems to me like he's identified a
clear need and at least attempted to address it -- which is pretty much
the story of every successful business.
Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers
------ Original Message ------
Sent: 2/19/2014 12:17:17 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Peter van der Does

2014-02-20 13:18:59 UTC

Permalink

On Wed, 19 Feb 2014 15:39:31 -0500

Post by Chris Williams
- Who elected DXW as the keeper of the standards?
- How does DXW establish their standards ("this plugin is large")?
- Can anyone have input into those standards?
- What value is there in a "cursory review"? If you don't understand
the code, you don't have a leg to stand on.
- What recourse other than paying them is there to a bad review?
- If an author takes their input and fixes their plugin, will DXW
redo the review without payment, and then repair all the negative
publicity they have generated?
- If I use a plugin that DXW has reviewed as safe, and I get hacked,
can I sue DXW?
- How does one insure that DXW doesn't have ulterior motives in their
reviews? Competitive products, products they don't like/use, authors
who pay for reviews being treated better than authors who don't?
This is just like Yelp -- a half-baked idea made even worse by little
or no validation of the source of the review.

You can't sue them Chris, no mater how they reviewed the code.

Rule 3 in their Terms of Service
"Accordingly, we make absolutely no guarantees. If we are wrong in our
assessment you accept that you are not able to sue us for that reason."

On an updated review:
Rule 4
"If you object to any of our assessments, for example if you are a
maintainer on a project about which we’ve published some information,
please contact us at ***@dxw.com giving full particulars of your
objection and we will try to consider your objections within a
reasonable time. We do not guarantee to change or remove anything
merely on request." and in another email Harry wrote:

"Of course, we do want to keep everything as up to date as we can.
Unfortunately we don't have nearly the time to be able to monitor all
the inspections for updated versions and check them again proactively.
Inspections will get update if a client asks us to review an updated
version. We do currently always do an updated inspection if the author
drops us a note to ask for one, but that is a bit dependent on our
availability - I can't guarantee we'll always be able to. But certainly
for now, if you have a new version, send an email to ***@dxw.com
with the details, and we'll have a look."

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Harry Metcalfe

2014-02-19 20:52:00 UTC

Permalink

Hi Chris,

I'm sorry you feel that way, and I can say categorically that we are not
trying to hold anyone to ransom. I'll try to explain.

Going back a couple of years, our clients expected us to give them some
sort of assessment of plugins before we suggested using them. For a
while, we did this informally, and the results were very mixed. We found
that sometimes we would miss things. There was no set of criteria that
we applied, and we didn't record the results. This also led us to waste
time by checking out the same problem twice.

To solve these problems, we decided have a list of things that we think
are important (https://security.dxw.com/about/plugin-inspections/) and
to record the results of inspections somewhere so we didn't duplicate
work. We did this in private for a while but then thought that this was
probably information that others might find useful. So, we decided to
publish the results.

We have tried very hard to make sure that the results of these
inspections, and our confidence in them, is obvious to people who read
them. We've published the process. We've made sure it's clear that
inspections deal with likelihoods, not certainties. We've said that
people should always conduct their own checks. We've set out our terms
of service prominently, which include contact information for anyone
who'd like to tell us we're wrong. And we're totally happy (within
reason) to revisit things if people do that.

I would very much like it if these inspections could be more thorough,
but unfortunately, we're subject to the same commercial realities as
everyone else. We care more about security than most of our clients.
Most people are not willing to pay for security assurance work.
Inspections are light-touch because we don't charge existing clients for
them, and that's the only way we can make it economical.

I hope we can figure out some way to make some money out of this (hence
those messages saying we can be commissioned) but so far, we haven't
made a penny. We're just trying to make the outputs of something we do
anyway useful to a wider group.

If you have feedback on practical ways we could do that better, I'd love
to hear it.

Harry

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Williams

2014-02-19 21:25:04 UTC

Permalink

You are filled with double-speak. You say you don't want to make
categorical claims about people's code, but that's exactly what you do.
You say you don't want to hold people ransom, but then below admit that's
precisely what you are trying to do to make a business out of it. You say
your inspections aren't worth the bits it takes to express them, but then
highlight the results in red, with vague caveats to try to hedge your bets.

Here's my advice: Keep your opinions to yourself (and maybe your clients).

Unless and until you provide a clear/clean/transparent method for
evaluation, that isn't subject to the many issues I raised in my other
note, that has well-defined and very public methods for redress, and that
doesn't result in random and spurious claims about code you've barely even
bothered to understand, keep mum. As your grandmother said "if you can't
say anything nice, don't say anything at all."

Want to see how ugly this can get? Enable comments on your reviews, and
sit back and watch. You'll get plugin developers passionately trying to
defend themselves. You'll get their competitors jumping in to throw mud.
You'll get fanboys and haters jumping all over each other. It will get so
ugly so fast...

I appreciate that you feel there is some value to you and your clients in
this work. But please, don't try to be the "Good Housekeeping" seal of
approval without a whole lot more thought into it than you've clearly done
so far...

Post by Harry Metcalfe
Hi Chris,
I'm sorry you feel that way, and I can say categorically that we are not
trying to hold anyone to ransom. I'll try to explain.
Going back a couple of years, our clients expected us to give them some
sort of assessment of plugins before we suggested using them. For a
while, we did this informally, and the results were very mixed. We found
that sometimes we would miss things. There was no set of criteria that
we applied, and we didn't record the results. This also led us to waste
time by checking out the same problem twice.
To solve these problems, we decided have a list of things that we think
are important (https://security.dxw.com/about/plugin-inspections/) and
to record the results of inspections somewhere so we didn't duplicate
work. We did this in private for a while but then thought that this was
probably information that others might find useful. So, we decided to
publish the results.
We have tried very hard to make sure that the results of these
inspections, and our confidence in them, is obvious to people who read
them. We've published the process. We've made sure it's clear that
inspections deal with likelihoods, not certainties. We've said that
people should always conduct their own checks. We've set out our terms
of service prominently, which include contact information for anyone
who'd like to tell us we're wrong. And we're totally happy (within
reason) to revisit things if people do that.
I would very much like it if these inspections could be more thorough,
but unfortunately, we're subject to the same commercial realities as
everyone else. We care more about security than most of our clients.
Most people are not willing to pay for security assurance work.
Inspections are light-touch because we don't charge existing clients for
them, and that's the only way we can make it economical.
I hope we can figure out some way to make some money out of this (hence
those messages saying we can be commissioned) but so far, we haven't
made a penny. We're just trying to make the outputs of something we do
anyway useful to a wider group.
If you have feedback on practical ways we could do that better, I'd love
to hear it.
Harry

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-19 21:32:52 UTC

Permalink

Hi Chris,

I think you raise some good points.

I'm going to change the commissioning bit. I agree that it is prone to
misinterpretation.

I think there's no way to do this without some caveating, and I think it
is useful notwithstanding the caveats. But I agree that they're perhaps
not clear enough, and that "unsafe for use" might be a little too
categorical. I've had a couple of emails privately from developers this
evening which do make it pretty plain that the current approach is
confusing people.

We have published a clear and transparent set of criteria, and those are
what we apply. We have done our best to make this as clear as possible.

We are doing our best to do something useful, and I strongly believe
that this site goes some way to filling an important gap. On this we may
perhaps have to agree to disagree. But I do value the points you've made
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

Harry

Post by Chris Williams
You are filled with double-speak. You say you don't want to make
categorical claims about people's code, but that's exactly what you do.
You say you don't want to hold people ransom, but then below admit that's
precisely what you are trying to do to make a business out of it. You say
your inspections aren't worth the bits it takes to express them, but then
highlight the results in red, with vague caveats to try to hedge your bets.
Here's my advice: Keep your opinions to yourself (and maybe your clients).
Unless and until you provide a clear/clean/transparent method for
evaluation, that isn't subject to the many issues I raised in my other
note, that has well-defined and very public methods for redress, and that
doesn't result in random and spurious claims about code you've barely even
bothered to understand, keep mum. As your grandmother said "if you can't
say anything nice, don't say anything at all."
Want to see how ugly this can get? Enable comments on your reviews, and
sit back and watch. You'll get plugin developers passionately trying to
defend themselves. You'll get their competitors jumping in to throw mud.
You'll get fanboys and haters jumping all over each other. It will get so
ugly so fast...
I appreciate that you feel there is some value to you and your clients in
this work. But please, don't try to be the "Good Housekeeping" seal of
approval without a whole lot more thought into it than you've clearly done
so far...

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that
we do for our clients. They are just a guide - we conduct some basic
checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the
list and therefore worth posting? Is the list also interested in
vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Williams

2014-02-20 01:05:57 UTC

Permalink

Let's see if I can summarize: you are using arbitrary criteria
administered by people of unknown skill/experience and using the results
to publicly condemn other people's work with an overly broad brush, and
without any mechanism for recourse. The result has no positive benefits.
It demeans the plugin authors and their work, and by reflection your firm
and its work, raises alarm in the community you claim to support, and
garners you no goodwill.

I'm sorry, but given the train wreck this has become, my best advice is
precisely that: stop doing it.

Post by Harry Metcalfe
But I do value the points you've made
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

Eric Hendrix

2014-02-20 04:46:01 UTC

Permalink

Hear, hear.

Post by Chris Williams
Let's see if I can summarize: you are using arbitrary criteria
administered by people of unknown skill/experience and using the results
to publicly condemn other people's work with an overly broad brush, and
without any mechanism for recourse. The result has no positive benefits.
It demeans the plugin authors and their work, and by reflection your firm
and its work, raises alarm in the community you claim to support, and
garners you no goodwill.
I'm sorry, but given the train wreck this has become, my best advice is
precisely that: stop doing it.

Post by Harry Metcalfe
But I do value the points you've made
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

--
*Eric A. Hendrix*
***@gmail.com
(910) 644-8940

*"Non Timebo Mala"*

Harry Metcalfe

2014-02-20 08:37:55 UTC

Permalink

Disappointingly, we'll perhaps have to agree to disagree.

I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to be
instant. I don't think we're condemning anybody: we're pointing out
issues which are widely accepted to be indicative of problematic code.

In the mean time, people are - of course - free to vote with their feet
and not visit the site. Or set up a better one.

Harry

Post by Harry Metcalfe
But I do value the points you've made
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Peter van der Does

2014-02-20 12:08:21 UTC

Permalink

On Thu, 20 Feb 2014 08:37:55 +0000

Post by Harry Metcalfe
Disappointingly, we'll perhaps have to agree to disagree.
I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to
be instant. I don't think we're condemning anybody: we're pointing
out issues which are widely accepted to be indicative of problematic
code.
In the mean time, people are - of course - free to vote with their
feet and not visit the site. Or set up a better one.

Do you contact the developers privately about your findings before
posting them to the public?

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

John

2014-02-20 16:50:01 UTC

Permalink

The community would be better served if you first contacted plugin authors
and the maintainers of the WP plugin repo regarding security issues.

If the door on your neighbor's shed was broken, making it easy for thieves
to enter, would you first announce it to the whole community in a letter to
the editor alongside an ad for your door repair services, or would you be
Dudley Do-Right and tell your neighbor directly?

If you've reviewed enough code to make the claims, you can certainly reveal
specific vulnerabilities to the plugin authors and allow them to fix them.
This is pretty much the way any open source community handles security
issues. If you do enough of that, the money will come - if that's what you
want.

After a reasonable period of time after security updates have been released
(or not in cases where plugin authors are unresponsive), the public service
announcement could follow.

Post by Harry Metcalfe
Disappointingly, we'll perhaps have to agree to disagree.
I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to be
instant. I don't think we're condemning anybody: we're pointing out issues
which are widely accepted to be indicative of problematic code.
In the mean time, people are - of course - free to vote with their feet
and not visit the site. Or set up a better one.
Harry

Post by Harry Metcalfe
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

K.Adam White

2014-02-20 17:00:06 UTC

Permalink

As an example of an interesting way to handle crowd-sourcing security
reviews, check out what Github does with their Bug Bounty program:
https://bounty.github.com/

Basically, interested parties look for errors, report them to Github when
found, and get public credit and applause for finding the problem (and the
opportunity to disclose what they found) after it's been fixed.

K. Adam White

Post by John
The community would be better served if you first contacted plugin authors
and the maintainers of the WP plugin repo regarding security issues.
If the door on your neighbor's shed was broken, making it easy for thieves
to enter, would you first announce it to the whole community in a letter to
the editor alongside an ad for your door repair services, or would you be
Dudley Do-Right and tell your neighbor directly?
If you've reviewed enough code to make the claims, you can certainly reveal
specific vulnerabilities to the plugin authors and allow them to fix them.
This is pretty much the way any open source community handles security
issues. If you do enough of that, the money will come - if that's what you
want.
After a reasonable period of time after security updates have been released
(or not in cases where plugin authors are unresponsive), the public service
announcement could follow.

issues

Post by Harry Metcalfe
which are widely accepted to be indicative of problematic code.
In the mean time, people are - of course - free to vote with their feet
and not visit the site. Or set up a better one.
Harry

benefits.

Post by Harry Metcalfe

Post by Chris Williams
It demeans the plugin authors and their work, and by reflection your

firm

Post by Harry Metcalfe

Post by Chris Williams
and its work, raises alarm in the community you claim to support, and
garners you no goodwill.
I'm sorry, but given the train wreck this has become, my best advice is
precisely that: stop doing it.
But I do value the points you've made

Post by Harry Metcalfe
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-20 17:24:28 UTC

Permalink

Hi John,

This - more or less - is exactly how we operate.

We have a look. If we see indications of badness, but no specific
vulnerabilities, we write that up and publish the inspection.

If we see vulnerabilities, we write up an advisory and disclose it
responsibly, exactly as you suggest (details:
https://security.dxw.com/disclosure/).

I don't think it is necessary to disclose in advance for an inspection,
because we're not announcing that the neighbour's shed is broken. We're
announcing that neighbour's shed's looking a bit old and tatty, and that
people might not want to keep their stuff in it until it's fixed.

Quite a few people have suggested that we should reach out to plugin
authors, though. I am, in principle, happy to do that. But such a
mechanism would have to be at least partly automated, and we have no
private contact details for plugin authors. So, the best we could do is
probably to have a bot that posts on people's forums. But that's more
notification than notice, and I'm not sure I'm comfortable with the idea
of such a bot in any event.

If you have an idea for how we can reliably, semi-automatically give
authors notice, and then publish after some predefined time - I'm all ears.

Harry

Post by Harry Metcalfe
Disappointingly, we'll perhaps have to agree to disagree.
I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to be
instant. I don't think we're condemning anybody: we're pointing out issues
which are widely accepted to be indicative of problematic code.
In the mean time, people are - of course - free to vote with their feet
and not visit the site. Or set up a better one.
Harry

Post by Harry Metcalfe
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Christoff

2014-02-20 17:39:32 UTC

Permalink

-- Please reply above this line --

-----------------------------------------------------------
## Chris replied, on Feb 20 @ 1:39pm (AMT):

So basically, it sounds to me like after spending "much time" on a
(still unproven to be) comprehensive review, you can't simply Google
the name of the author, look their email up on their GitHub repo, or
plug their name into Twitter.

It seems your entire business is based on providing mediocre (at
best) subpar reviews, which are then published to the public to
encourage users to not use what very likely could be a perfectly fine
plugin (since the highly subjective criteria of the review doesn't
even sound, by your own account that comprehensive), and then not
alert the author before publishing. Then, when said author finds out,
they in essence have to purchase your service to get their plugin
re-reviewed since, by your own account while you'll review it for
free, you may or may not have the time, and a plugin author doesn't
want false reviews online for long. And since said reviews are done by
employees of unknown skill, the outcome of said review could just as
easily be determined by rolling a dice.

So basically an author has to pay to remove what could very likely be
slander from the internet.
It very well seems your entire business model boils down to
monetizing the practice of slander, correct?

Here is, based on your own account, what such a report could be:
Avoid at all costs security.dxw.com, it is ABSOLUTELY RIDDLED WITH
MALWARE (imagine that in a giant red banner). See, we didn't really
actually review the code of said site that well, or even at all. It
was done by someone who is still learning HTML, and while we didn't
really review it, there's a possibility it contains malware, even
though we haven't proven it to exist yet. Therefore, our firm
recommendation is to avoid said site at all costs until said author
pays me $1,000,000 to re-review his site.
--
Chris Christoff
***@chriscct7.com
http://www.chriscct7.com [1]
@chriscct7
If you feel the need to donate, as a college student, I appreciate
donations of any amount. The easiest way to donate to my college fund
is via the donation button at the bottom of my
homepage: http://chriscct7.com/ [2]

Links:
------
[1] http://www.chriscct7.com
[2] http://chriscct7.com/

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:24pm (AMT):

Hi John,

This - more or less - is exactly how we operate.

We have a look. If we see indications of badness, but no specific
vulnerabilities, we write that up and publish the inspection.

If we see vulnerabilities, we write up an advisory and disclose it
responsibly, exactly as you suggest (details:
https://security.dxw.com/disclosure/).

I don't think it is necessary to disclose in advance for an
inspection,
because we're not announcing that the neighbour's shed is broken.
We're
announcing that neighbour's shed's looking a bit old and tatty, and
that
people might not want to keep their stuff in it until it's fixed.

Quite a few people have suggested that we should reach out to plugin
authors, though. I am, in principle, happy to do that. But such a
mechanism would have to be at least partly automated, and we have no
private contact details for plugin authors. So, the best we could do
is
probably to have a bot that posts on people's forums. But that's more
notification than notice, and I'm not sure I'm comfortable with the
idea
of such a bot in any event.

If you have an idea for how we can reliably, semi-automatically give
authors notice, and then publish after some predefined time - I'm all
ears.

Harry

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:00pm (AMT):

As an example of an interesting way to handle crowd-sourcing security
reviews, check out what Github does with their Bug Bounty program:
https://bounty.github.com/

Basically, interested parties look for errors, report them to Github
when
found, and get public credit and applause for finding the problem
(and the
opportunity to disclose what they found) after it's been fixed.

K. Adam White

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 12:50pm (AMT):

The community would be better served if you first contacted plugin
authors
and the maintainers of the WP plugin repo regarding security issues.

If the door on your neighbor's shed was broken, making it easy for
thieves
to enter, would you first announce it to the whole community in a
letter to
the editor alongside an ad for your door repair services, or would
you be
Dudley Do-Right and tell your neighbor directly?

If you've reviewed enough code to make the claims, you can certainly
reveal
specific vulnerabilities to the plugin authors and allow them to fix
them.
This is pretty much the way any open source community handles
security
issues. If you do enough of that, the money will come - if that's
what you
want.

After a reasonable period of time after security updates have been
released
(or not in cases where plugin authors are unresponsive), the public
service
announcement could follow.

Post by Harry Metcalfe
Disappointingly, we'll perhaps have to agree to disagree.
I think the site is a positive contribution to WordPress's

security.

Post by Harry Metcalfe
Hopefully, in time, we'll earn some trust. I'm not expecting

that to be

Post by Harry Metcalfe
instant. I don't think we're condemning anybody: we're pointing out issues
which are widely accepted to be indicative of problematic code.
In the mean time, people are - of course - free to vote with

their feet

Post by Harry Metcalfe
and not visit the site. Or set up a better one.
Harry

Post by Chris Williams
Let's see if I can summarize: you are using arbitrary

criteria

Post by Harry Metcalfe

Post by Chris Williams
administered by people of unknown skill/experience and

using the results

Post by Harry Metcalfe

Post by Chris Williams
to publicly condemn other people's work with an overly

broad brush, and

Post by Harry Metcalfe

Post by Chris Williams
without any mechanism for recourse. The result has no

positive benefits.

Post by Harry Metcalfe

Post by Chris Williams
It demeans the plugin authors and their work, and by

reflection your firm

Post by Harry Metcalfe

Post by Chris Williams
and its work, raises alarm in the community you claim to

support, and

Post by Harry Metcalfe

Post by Chris Williams
garners you no goodwill.
I'm sorry, but given the train wreck this has become, my

best advice is

Post by Harry Metcalfe

Post by Chris Williams
precisely that: stop doing it.
But I do value the points you've made

Post by Harry Metcalfe
and we will make some changes based upon then. I'd be

keen to hear any

Post by Harry Metcalfe

Post by Chris Williams

Post by Harry Metcalfe
other feedback you might have later (short of "stop

doing it"!)

Post by Harry Metcalfe

Post by Chris Williams
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 8:08am (AMT):

On Thu, 20 Feb 2014 08:37:55 +0000

Post by Harry Metcalfe
Disappointingly, we'll perhaps have to agree to disagree.
I think the site is a positive contribution to WordPress's

security.

Post by Harry Metcalfe
Hopefully, in time, we'll earn some trust. I'm not expecting

that to

Post by Harry Metcalfe
be instant. I don't think we're condemning anybody: we're

pointing

Post by Harry Metcalfe
out issues which are widely accepted to be indicative of

problematic

Post by Harry Metcalfe
code.
In the mean time, people are - of course - free to vote with

their

Post by Harry Metcalfe
feet and not visit the site. Or set up a better one.

Do you contact the developers privately about your findings before
posting them to the public?

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 4:38am (AMT):

Disappointingly, we'll perhaps have to agree to disagree.

I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to
be
instant. I don't think we're condemning anybody: we're pointing out
issues which are widely accepted to be indicative of problematic
code.

In the mean time, people are - of course - free to vote with their
feet
and not visit the site. Or set up a better one.

Harry

Post by Harry Metcalfe
Let's see if I can summarize: you are using arbitrary criteria
administered by people of unknown skill/experience and using

the results

Post by Harry Metcalfe
to publicly condemn other people's work with an overly broad

brush, and

Post by Harry Metcalfe
without any mechanism for recourse. The result has no positive

benefits.

Post by Harry Metcalfe
It demeans the plugin authors and their work, and by reflection your firm
and its work, raises alarm in the community you claim to

support, and

Post by Harry Metcalfe
garners you no goodwill.
I'm sorry, but given the train wreck this has become, my best

advice is

Post by Harry Metcalfe
precisely that: stop doing it.

Post by Chris Williams
But I do value the points you've made
and we will make some changes based upon then. I'd be keen

to hear any

Post by Harry Metcalfe

Post by Chris Williams
other feedback you might have later (short of "stop doing

it"!)

Post by Harry Metcalfe
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chip Bennett

2014-02-20 17:41:41 UTC

Permalink

Again: you're announcing that the neighbor's shed *should be condemned*
("unsafe to use"), based on "indications of badness, but no specific
vulnerabilities".

That is precisely where I have a problem with what you're doing.

Post by Harry Metcalfe
Hi John,
This - more or less - is exactly how we operate.
We have a look. If we see indications of badness, but no specific
vulnerabilities, we write that up and publish the inspection.
If we see vulnerabilities, we write up an advisory and disclose it
responsibly, exactly as you suggest (details: https://security.dxw.com/
disclosure/).
I don't think it is necessary to disclose in advance for an inspection,
because we're not announcing that the neighbour's shed is broken. We're
announcing that neighbour's shed's looking a bit old and tatty, and that
people might not want to keep their stuff in it until it's fixed.
Quite a few people have suggested that we should reach out to plugin
authors, though. I am, in principle, happy to do that. But such a mechanism
would have to be at least partly automated, and we have no private contact
details for plugin authors. So, the best we could do is probably to have a
bot that posts on people's forums. But that's more notification than
notice, and I'm not sure I'm comfortable with the idea of such a bot in any
event.
If you have an idea for how we can reliably, semi-automatically give
authors notice, and then publish after some predefined time - I'm all ears.
Harry

Post by Harry Metcalfe
I think the site is a positive contribution to WordPress's security.
Hopefully, in time, we'll earn some trust. I'm not expecting that to be
instant. I don't think we're condemning anybody: we're pointing out issues
which are widely accepted to be indicative of problematic code.
In the mean time, people are - of course - free to vote with their feet
and not visit the site. Or set up a better one.
Harry
Let's see if I can summarize: you are using arbitrary criteria

Post by Chris Williams
administered by people of unknown skill/experience and using the results
to publicly condemn other people's work with an overly broad brush, and
without any mechanism for recourse. The result has no positive benefits.
It demeans the plugin authors and their work, and by reflection your firm
and its work, raises alarm in the community you claim to support, and
garners you no goodwill.
I'm sorry, but given the train wreck this has become, my best advice is
precisely that: stop doing it.
But I do value the points you've made

Post by Harry Metcalfe
and we will make some changes based upon then. I'd be keen to hear any
other feedback you might have later (short of "stop doing it"!)
_______________________________________________

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
_______________________________________________

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-20 17:49:30 UTC

Permalink

Hi Chris,

I've agreed previously on this thread that "Unsafe to use" was too
categorical, and have changed that text to "Potentially unsafe". We'll
continue to make changes to ensure we're being as clear and useful as we
can.

On the rest of your post, we may have to agree to disagree. It is not
commercially viable for us, or anyone else, to do comprehensive code
reviews for free. It is not acceptable, in my opinion, that people with
serious WordPress sites so frequently install plugins with no idea as to
their quality or security. We're trying to help by giving people enough
information to make slightly better decisions - for example, by focusing
the resources they do have on more thorough examination of candidate
plugins that are most likely to be problematic.

I hope it's clear by now that I am committed to making this site and
process better, and I'm very happy to take criticism and feedback that
helps us to improve. But I think, for now, for this thread, I won't
respond to more posts that say -- more or less -- "just don't do this".
Because I think that what we're doing does more good than harm, even
with its imperfections.

Cheers,

Harry

Post by Chip Bennett
Again: you're announcing that the neighbor's shed *should be condemned*
("unsafe to use"), based on "indications of badness, but no specific
vulnerabilities".
That is precisely where I have a problem with what you're doing.

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers
_______________________________________________

wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Chris Christoff

2014-02-20 17:53:04 UTC

Permalink

-- Please reply above this line --

-----------------------------------------------------------
## Chris replied, on Feb 20 @ 1:52pm (AMT):

I think frankly, your entire business model is backwards. As opposed
to publishing reviews without even asking for clarification, you
should instead do what bugcrowd does. Get paid to find issues. That's
a service I know I for instance, would pay for. Why aren't you doing
something like that?
--
Chris Christoff
***@chriscct7.com
http://www.chriscct7.com [1]
@chriscct7
If you feel the need to donate, as a college student, I appreciate
donations of any amount. The easiest way to donate to my college fund
is via the donation button at the bottom of my
homepage: http://chriscct7.com/ [2]

Links:
------
[1] http://www.chriscct7.com
[2] http://chriscct7.com/

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:49pm (AMT):

Hi Chris,

I've agreed previously on this thread that "Unsafe to use" was too
categorical, and have changed that text to "Potentially unsafe".
We'll
continue to make changes to ensure we're being as clear and useful as
we
can.

On the rest of your post, we may have to agree to disagree. It is not
commercially viable for us, or anyone else, to do comprehensive code
reviews for free. It is not acceptable, in my opinion, that people
with
serious WordPress sites so frequently install plugins with no idea as
to
their quality or security. We're trying to help by giving people
enough
information to make slightly better decisions - for example, by
focusing
the resources they do have on more thorough examination of candidate
plugins that are most likely to be problematic.

I hope it's clear by now that I am committed to making this site and
process better, and I'm very happy to take criticism and feedback
that
helps us to improve. But I think, for now, for this thread, I won't
respond to more posts that say -- more or less -- "just don't do
this".
Because I think that what we're doing does more good than harm, even
with its imperfections.

Cheers,

Harry

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:41pm (AMT):

Again: you're announcing that the neighbor's shed *should be
condemned*
("unsafe to use"), based on "indications of badness, but no specific
vulnerabilities".

That is precisely where I have a problem with what you're doing.

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## Chris replied, on Feb 20 @ 1:39pm (AMT):

So basically, it sounds to me like after spending "much time" on a
(still unproven to be) comprehensive review, you can't simply Google
the name of the author, look their email up on their GitHub repo, or
plug their name into Twitter.

It seems your entire business is based on providing mediocre (at
best) subpar reviews, which are then published to the public to
encourage users to not use what very likely could be a perfectly fine
plugin (since the highly subjective criteria of the review doesn't
even sound, by your own account that comprehensive), and then not
alert the author before publishing. Then, when said author finds out,
they in essence have to purchase your service to get their plugin
re-reviewed since, by your own account while you'll review it for
free, you may or may not have the time, and a plugin author doesn't
want false reviews online for long. And since said reviews are done by
employees of unknown skill, the outcome of said review could just as
easily be determined by rolling a dice.

So basically an author has to pay to remove what could very likely be
slander from the internet.
It very well seems your entire business model boils down to
monetizing the practice of slander, correct?

Here is, based on your own account, what such a report could be:
Avoid at all costs security.dxw.com, it is ABSOLUTELY RIDDLED WITH
MALWARE (imagine that in a giant red banner). See, we didn't really
actually review the code of said site that well, or even at all. It
was done by someone who is still learning HTML, and while we didn't
really review it, there's a possibility it contains malware, even
though we haven't proven it to exist yet. Therefore, our firm
recommendation is to avoid said site at all costs until said author
pays me $1,000,000 to re-review his site.
--
Chris Christoff
***@chriscct7.com
http://www.chriscct7.com [1]
@chriscct7
If you feel the need to donate, as a college student, I appreciate
donations of any amount. The easiest way to donate to my college fund
is via the donation button at the bottom of my
homepage: http://chriscct7.com/ [2]

Links:
------
[1] http://www.chriscct7.com
[2] http://chriscct7.com/

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:24pm (AMT):

Hi John,

This - more or less - is exactly how we operate.

We have a look. If we see indications of badness, but no specific
vulnerabilities, we write that up and publish the inspection.

If we see vulnerabilities, we write up an advisory and disclose it
responsibly, exactly as you suggest (details:
https://security.dxw.com/disclosure/).

I don't think it is necessary to disclose in advance for an
inspection,
because we're not announcing that the neighbour's shed is broken.
We're
announcing that neighbour's shed's looking a bit old and tatty, and
that
people might not want to keep their stuff in it until it's fixed.

Quite a few people have suggested that we should reach out to plugin
authors, though. I am, in principle, happy to do that. But such a
mechanism would have to be at least partly automated, and we have no
private contact details for plugin authors. So, the best we could do
is
probably to have a bot that posts on people's forums. But that's more
notification than notice, and I'm not sure I'm comfortable with the
idea
of such a bot in any event.

If you have an idea for how we can reliably, semi-automatically give
authors notice, and then publish after some predefined time - I'm all
ears.

Harry

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------
## wp-***@lists.automattic.com replied, on Feb 20 @ 1:00pm (AMT):

As an example of an interesting way to handle crowd-sourcing security
reviews, check out what Github does with their Bug Bounty program:
https://bounty.github.com/

Basically, interested parties look for errors, report them to Github
when
found, and get public credit and applause for finding the problem
(and the
opportunity to disclose what they found) after it's been fixed.

K. Adam White

_______________________________________________
wp-hackers mailing list
wp-***@lists.automattic.com
http://lists.automattic.com/mailman/listinfo/wp-hackers

-----------------------------------------------------------

Peter van der Does

2014-02-19 22:15:52 UTC

Permalink

On Wed, 19 Feb 2014 20:52:00 +0000
Harry Metcalfe <***@dxw.com> wrote:

SNIP

Post by Harry Metcalfe
To solve these problems, we decided have a list of things that we
think are important
(https://security.dxw.com/about/plugin-inspections/) and to record
the results of inspections somewhere so we didn't duplicate work. We
did this in private for a while but then thought that this was
probably information that others might find useful. So, we decided to
publish the results.

I love these rules for failing a plugin:

Poor coding style
Exhibits the characteristics of poor coding style, including (but not limited to):
Lack of indenting
Inconsistent indenting
Lack of clear commenting where it would be appropriate; ie, to
explain complex functionality or dense code

The lack of good style must materially reduce the tester's ability to
understand what the code is doing, thereby indicating that the lack of
good style has reduced code readability and maintainability.

Does the end user really care how the code is written?
The grade depends on the expertise of the testers. What makes them
qualified to give this grade? Do they have a PHP certification, what's
their background?

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Eric Hendrix

2014-02-19 22:21:55 UTC

Permalink

Let's hear! What ARE the quals of these "testers?" That's a reasonable request.

Also, style is only dictated by the individual or org that's producing the plugin, and not to be defined generally by other than IEEE or ISO for example.

The critique might be interesting for other coders to read, but must be published//presented in an anonymous fashion and with the owners permission noting that it is a fair critique.
—
Eric A. Hendrix
***@gmail.com
910-644-8940

On Wed, Feb 19, 2014 at 5:15 PM, Peter van der Does

Post by Peter van der Does
On Wed, 19 Feb 2014 20:52:00 +0000
SNIP

Poor coding style
Lack of indenting
Inconsistent indenting
Lack of clear commenting where it would be appropriate; ie, to
explain complex functionality or dense code
The lack of good style must materially reduce the tester's ability to
understand what the code is doing, thereby indicating that the lack of
good style has reduced code readability and maintainability.
Does the end user really care how the code is written?
The grade depends on the expertise of the testers. What makes them
qualified to give this grade? Do they have a PHP certification, what's
their background?
--
Peter van der Does
GPG key: CB317D6E
Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-19 22:22:38 UTC

Permalink

snip snip
Does the end user really care how the code is written?
The grade depends on the expertise of the testers. What makes them
qualified to give this grade? Do they have a PHP certification, what's
their background?
The lack of good style must materially reduce the tester's ability to
understand what the code is doing, thereby indicating that the lack of
good style has reduced code readability and maintainability.

This isn't about aesthetics - code that is written in such a way that it
is very difficult to follow is also harder to maintain. It's more likely
to contain bugs, some of which may be vulnerabilities. And it's much
easier to make mistakes when editing it after you haven't looked at it
for a while. It's also evidence that the developer may be inexperienced.
These are all important factors. That said, I can't imagine that a
plugin would fail an inspection on this criterion alone.

The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure how
we'd go about reassuring people on that front, though: what would you
consider to be good evidence that we're knowledgeable?

Harry

Eric Hendrix

2014-02-19 22:25:44 UTC

Permalink

Certifications. —
Eric A. Hendrix
***@gmail.com
910-644-8940

Post by Harry Metcalfe

This isn't about aesthetics - code that is written in such a way that it
is very difficult to follow is also harder to maintain. It's more likely
to contain bugs, some of which may be vulnerabilities. And it's much
easier to make mistakes when editing it after you haven't looked at it
for a while. It's also evidence that the developer may be inexperienced.
These are all important factors. That said, I can't imagine that a
plugin would fail an inspection on this criterion alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure how
we'd go about reassuring people on that front, though: what would you
consider to be good evidence that we're knowledgeable?
Harry
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Madalin Ignisca

2014-02-19 22:27:13 UTC

Permalink

Certification of PHP from a trusted 3rd party source like Zend or similar
company.

Certifications. —
Eric A. Hendrix
910-644-8940

Post by Harry Metcalfe

This isn't about aesthetics - code that is written in such a way that it
is very difficult to follow is also harder to maintain. It's more likely
to contain bugs, some of which may be vulnerabilities. And it's much
easier to make mistakes when editing it after you haven't looked at it
for a while. It's also evidence that the developer may be inexperienced.
These are all important factors. That said, I can't imagine that a
plugin would fail an inspection on this criterion alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure how
we'd go about reassuring people on that front, though: what would you
consider to be good evidence that we're knowledgeable?
Harry
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

--
*Madalin Ignisca*
*web developer*
http://imadalin.ro/

Harry Metcalfe

2014-02-19 22:30:12 UTC

Permalink

Hi Eric, Madalin,

That seems reasonable. For the moment - since this is not a service that
makes us any money at all - I think that it's probably not a practical
option. But I will keep it in mind.

You might perhaps draw some comfort from the advisories section. All of
these specific vulnerabilities have been identified by the same testers
that carry out inspections, have been responsibly disclosed and fixed by
the relevant developers.

Harry

Certifications. —
Eric A. Hendrix
910-644-8940

Post by Harry Metcalfe

This isn't about aesthetics - code that is written in such a way that it
is very difficult to follow is also harder to maintain. It's more likely
to contain bugs, some of which may be vulnerabilities. And it's much
easier to make mistakes when editing it after you haven't looked at it
for a while. It's also evidence that the developer may be inexperienced.
These are all important factors. That said, I can't imagine that a
plugin would fail an inspection on this criterion alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure how
we'd go about reassuring people on that front, though: what would you
consider to be good evidence that we're knowledgeable?
Harry
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Madalin Ignisca

2014-02-19 23:02:25 UTC

Permalink

Hi Harry,

I agree with your idea with the reviews on dxw.com, but as Eric mentioned,
you need some certifications to become an authority and trusted in this.

If not, this will just cause just a dispute and fight on each side.

My personal opinion, I would not trust 100% your reviews as you "green"
some plugins I'd run away from and "red/yellow" a few that really need a
more relevant review, but you have some good points on a few "red" labeled.

Mentioning here on almost all your responses about this service doesn't
makes you money it's pointless, you should be proud that you want to
contribute to the WordPress community and stop complaining about money. If
you want only money, then I suggest you review more on "premium" stuff, as
WordPress.org has a team of members that do reviews and
approve/disapproving plugins and themes and in WordPress.org case we should
have a really nice chat about how we can improve this service so plugins
and themes that would not respect all standards we vote for should be
excluded until corrected as should.

Post by Harry Metcalfe
Hi Eric, Madalin,
That seems reasonable. For the moment - since this is not a service that
makes us any money at all - I think that it's probably not a practical
option. But I will keep it in mind.
You might perhaps draw some comfort from the advisories section. All of
these specific vulnerabilities have been identified by the same testers
that carry out inspections, have been responsibly disclosed and fixed by
the relevant developers.
Harry

Certifications. —
Eric A. Hendrix
910-644-8940

Post by Harry Metcalfe

This isn't about aesthetics - code that is written in such a way that it
is very difficult to follow is also harder to maintain. It's more likely
to contain bugs, some of which may be vulnerabilities. And it's much
easier to make mistakes when editing it after you haven't looked at it
for a while. It's also evidence that the developer may be inexperienced.
These are all important factors. That said, I can't imagine that a
plugin would fail an inspection on this criterion alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure how
we'd go about reassuring people on that front, though: what would you
consider to be good evidence that we're knowledgeable?
Harry
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

--
*Madalin Ignisca*
*web developer*
http://imadalin.ro/

Harry Metcalfe

2014-02-19 23:45:06 UTC

Permalink

Post by Madalin Ignisca
Hi Harry,
I agree with your idea with the reviews on dxw.com, but as Eric mentioned,
you need some certifications to become an authority and trusted in this.
If not, this will just cause just a dispute and fight on each side.

That's fair comment. I suppose we'll just have to see. Of course, people
are free not to use the site!

Post by Madalin Ignisca
My personal opinion, I would not trust 100% your reviews as you "green"
some plugins I'd run away from and "red/yellow" a few that really need a
more relevant review, but you have some good points on a few "red" labeled.

Good. That's exactly how it's supposed to work!

Post by Madalin Ignisca
Mentioning here on almost all your responses about this service doesn't
makes you money it's pointless, you should be proud that you want to
contribute to the WordPress community and stop complaining about money.

Sorry if I've come across as complaining. I'm not, at all. We are doing
this in order to contribute to the community - if we didn't care, we
wouldn't have bothered. I hope the site may make some money one day but
that is not it's main motivation.

Post by Madalin Ignisca
If you want only money, then I suggest you review more on "premium" stuff, as
WordPress.org has a team of members that do reviews and
approve/disapproving plugins and themes and in WordPress.org case we should
have a really nice chat about how we can improve this service so plugins
and themes that would not respect all standards we vote for should be
excluded until corrected as should.

I'm all for that. I think there's room for both! And up for a nice chat
any time :)

Harry

Simon Blackbourn

2014-02-20 00:14:06 UTC

Permalink

Hi Harry

I think there is potentially a very useful idea and service here, but I
really think a lot more care, depth, clarity (and possibly right to reply
for plugin authors) needs to go into the reviews.

Here's an example that really stands out to me:
https://security.dxw.com/plugins/relevanssi-premium

"This plugin also has a history of broken releases, including one which

contained malicious code added to the distribution after the author's
website was hacked in July 2013."

I was the person who discovered that Mikko's website had been hacked and
the resulting attempted malicious code that was inserted into his plugin (I
say attempted, because due to a typo by the hacker it didn't actually do
anything). I reported it responsibly by emailing him privately at 11.30pm
on a Friday night. By the Monday he had fixed it, released a new version,
installed additional security measures on his server and emailed all his
users to openly explain and apologise for what had happened, which is
pretty much a textbook exemplary way to deal with this sort of thing.

It seems unfair therefore that you have turned something that happened
seven months ago that was resolved so speedily and responsibly into a very
public black mark against this plugin, especially in a format that doesn't
give the author any right to reply.

"We have sampled a quite a few of these queries and none appear to be

injectable, but we suspect this is more likely to be due to luck than good
judgement."

What you're saying here is you conducted an incomplete test, couldn't find
anything wrong, yet you then decided that this must be luck, so you're
going to count it against the plugin anyway?! The word 'suspect' really
shouldn't have any place in a professional and public vulnerability review
- either you test fully and find a vulnerability (which you then report in
the proper manner to the author) or you don't.

You've then failed the plugin on three code-related criteria, but then
state in the box on the right that you haven't actually done a proper code
inspection.

Finally, it's very unclear to me who the reviews are aimed at? If it's for
non-teccie end users, then they are very unlikely to understand concepts
such as "unprepared SQL statements", but as a developer, an incomplete high
level review that hasn't delved comprehensively into the code is no use to
me at all.

All the best
Simon

Harry Metcalfe

2014-02-20 08:54:03 UTC

Permalink

Hi Simon,

Thanks for your reply and the kind words.

I would love to be able to make these reviews more thorough, but we're
subject to the same commercial realities as any other company: our
clients want these reviews, and don't want to pay for them. So we're
doing our best to make them as good as we can, within the resources we
have. I'm definitely open to ways we can make them better and have
already made some changes to these pages based on the feedback in this
thread.

I've had a read of the Relevanssi one, and I agree that it's not quite
right at the moment. It's kinda judgemental and the server hack thing
doesn't have enough context. It's got Glyn as the author but I'm pretty
sure I rewrote that one, so it's my bad. We're still quite early on in
the process and we will get better.

I've edited those findings to make them a bit more objective. Here's the

This plugin takes an idiosyncratic, dangerous approach to SQL
generation. This plugin contains a large number of long and
complicated SQL queries and there is no organised or methodical
approach to generating them safely.
We have sampled a quite a few of these queries and none appear to be
injectable, but we suspect this is more likely to be due to luck than
good judgement.
This plugin also has a history of broken releases, including one which
contained malicious code added to the distribution after the author's
website was hacked in July 2013.
Tread cautiously.

I do stand by the rating, though. This plugin has not "failed" (there
isn't really any such outcome). It's assessment is "use with caution"
and I think that is right.

That the author does appear to be safely escaping queries without
preparing them saves it from being rated "potentially unsafe", and the
lack of systematic SQL preparation is an issue. This plugin is more
likely than something systematic to have a dangerous query someone has
overlooked, or to introduce one in the future. And that likelihood is
made more significant by the very complex query generation in functions
like relevanssi_search in search.php, where the generation of one query
is spread over 100+ of lines of code.

The reviews are aimed at anyone running a WordPress website. We've tried
to strike a balance between adding enough technical detail to the
findings for developers and security people to follow them up, and
having a clear recommendation, so that non-technical users gain some
benefit as well.

Harry

Hi Harry
I think there is potentially a very useful idea and service here, but I
really think a lot more care, depth, clarity (and possibly right to reply
for plugin authors) needs to go into the reviews.
https://security.dxw.com/plugins/relevanssi-premium
"This plugin also has a history of broken releases, including one which

contained malicious code added to the distribution after the author's
website was hacked in July 2013."

I was the person who discovered that Mikko's website had been hacked and
the resulting attempted malicious code that was inserted into his plugin (I
say attempted, because due to a typo by the hacker it didn't actually do
anything). I reported it responsibly by emailing him privately at 11.30pm
on a Friday night. By the Monday he had fixed it, released a new version,
installed additional security measures on his server and emailed all his
users to openly explain and apologise for what had happened, which is
pretty much a textbook exemplary way to deal with this sort of thing.
It seems unfair therefore that you have turned something that happened
seven months ago that was resolved so speedily and responsibly into a very
public black mark against this plugin, especially in a format that doesn't
give the author any right to reply.
"We have sampled a quite a few of these queries and none appear to be

injectable, but we suspect this is more likely to be due to luck than good
judgement."

What you're saying here is you conducted an incomplete test, couldn't find
anything wrong, yet you then decided that this must be luck, so you're
going to count it against the plugin anyway?! The word 'suspect' really
shouldn't have any place in a professional and public vulnerability review
- either you test fully and find a vulnerability (which you then report in
the proper manner to the author) or you don't.
You've then failed the plugin on three code-related criteria, but then
state in the box on the right that you haven't actually done a proper code
inspection.
Finally, it's very unclear to me who the reviews are aimed at? If it's for
non-teccie end users, then they are very unlikely to understand concepts
such as "unprepared SQL statements", but as a developer, an incomplete high
level review that hasn't delved comprehensively into the code is no use to
me at all.
All the best
Simon
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Peter van der Does

2014-02-20 00:56:07 UTC

Permalink

On Wed, 19 Feb 2014 22:22:38 +0000

Post by Harry Metcalfe

snip snip
Does the end user really care how the code is written?
The grade depends on the expertise of the testers. What makes them
qualified to give this grade? Do they have a PHP certification,
what's their background?
The lack of good style must materially reduce the tester's ability
to understand what the code is doing, thereby indicating that the
lack of good style has reduced code readability and maintainability.

This isn't about aesthetics - code that is written in such a way that
it is very difficult to follow is also harder to maintain. It's more
likely to contain bugs, some of which may be vulnerabilities. And
it's much easier to make mistakes when editing it after you haven't
looked at it for a while. It's also evidence that the developer may
be inexperienced. These are all important factors. That said, I can't
imagine that a plugin would fail an inspection on this criterion
alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure
how we'd go about reassuring people on that front, though: what would
you consider to be good evidence that we're knowledgeable?
Harry

A few tings:
"Harder to maintain", for the tester maybe, but for the developer it
might make sense, but even if that is not the case, so what. If the
plugin works as advertised, and is being compatible with the latest
WordPress version, why would it be downgraded?

"More likely to contain bugs". More likely? So now we are presuming
that it contains bugs? The first lesson I was taught during my coding
lessons "Don't assume anything!"

"editing it after you haven't looked at it for a while" - Uhmm, I
thought the rating was for end-users not the maintainer of the plugin,

"I can't imagine that a plugin would fail an inspection on this
criterion alone." Coding style is all very subjective to start with,
you can't give a grade to subjectivity. Now if you said the coding
style has to conform to WordPress, okay but still not a valid case to
downgrade a plugin.
What are your criteria for coding style, Cyclomatic complexity, Design
Structure Quality Index, Halstead complexity measures? Does every
tester follow the same criteria? Where can one find your criteria?

"what would you consider to be good evidence that we're knowledgeable"
Like people already stated: Certification is one, but I personally
don't care if somebody is certified, I rather look at their
track records, in other words, what have these testers written
themselves.
If Mark Jaquith, Nacin, Taylor Otwell or Fabien Potencier would tell me
how to improve my code I would be more likely to listen then if is was
some schmuck who hasn't written any significant PHP program with every
PHP certification in the world :)

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Harry Metcalfe

2014-02-20 08:55:52 UTC

Permalink

Hi Peter,

I share your view that certification is of limited value :)

However, I don't think I'm going to get into a debate on this one. I
think it's pretty obvious that spaghetti code is more likely to be bad,
and this is all that criterion means. So I think I'll leave it at that.

Harry

Post by Peter van der Does
On Wed, 19 Feb 2014 22:22:38 +0000

Post by Harry Metcalfe

snip snip
Does the end user really care how the code is written?
The grade depends on the expertise of the testers. What makes them
qualified to give this grade? Do they have a PHP certification,
what's their background?
The lack of good style must materially reduce the tester's ability
to understand what the code is doing, thereby indicating that the
lack of good style has reduced code readability and maintainability.

This isn't about aesthetics - code that is written in such a way that
it is very difficult to follow is also harder to maintain. It's more
likely to contain bugs, some of which may be vulnerabilities. And
it's much easier to make mistakes when editing it after you haven't
looked at it for a while. It's also evidence that the developer may
be inexperienced. These are all important factors. That said, I can't
imagine that a plugin would fail an inspection on this criterion
alone.
The inspections are carried out by experienced developers. I can
appreciate that that might not be clear at the moment. I'm not sure
how we'd go about reassuring people on that front, though: what would
you consider to be good evidence that we're knowledgeable?
Harry

"Harder to maintain", for the tester maybe, but for the developer it
might make sense, but even if that is not the case, so what. If the
plugin works as advertised, and is being compatible with the latest
WordPress version, why would it be downgraded?
"More likely to contain bugs". More likely? So now we are presuming
that it contains bugs? The first lesson I was taught during my coding
lessons "Don't assume anything!"
"editing it after you haven't looked at it for a while" - Uhmm, I
thought the rating was for end-users not the maintainer of the plugin,
"I can't imagine that a plugin would fail an inspection on this
criterion alone." Coding style is all very subjective to start with,
you can't give a grade to subjectivity. Now if you said the coding
style has to conform to WordPress, okay but still not a valid case to
downgrade a plugin.
What are your criteria for coding style, Cyclomatic complexity, Design
Structure Quality Index, Halstead complexity measures? Does every
tester follow the same criteria? Where can one find your criteria?
"what would you consider to be good evidence that we're knowledgeable"
Like people already stated: Certification is one, but I personally
don't care if somebody is certified, I rather look at their
track records, in other words, what have these testers written
themselves.
If Mark Jaquith, Nacin, Taylor Otwell or Fabien Potencier would tell me
how to improve my code I would be more likely to listen then if is was
some schmuck who hasn't written any significant PHP program with every
PHP certification in the world :)

Peter van der Does

2014-02-20 12:33:14 UTC

Permalink

On Thu, 20 Feb 2014 08:55:52 +0000

Post by Harry Metcalfe
Hi Peter,
I share your view that certification is of limited value :)
However, I don't think I'm going to get into a debate on this one. I
think it's pretty obvious that spaghetti code is more likely to be
bad, and this is all that criterion means. So I think I'll leave it
at that.
Harry

Listen I don't disagree that spaghetti code is bad, I just understand
the reasoning that this would downgrade a plugin for it's usage.

What you are saying is that you would consider to downgrade a safety
test of a car because the car is dirty, and well that means it's likely
to have bad welding but if it's all shiny it's fine.

You can't say code is bad without giving reasons why. How can you
expect a developer to improve his grade if you don't tell him what's
wrong.

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Ryan McCue

2014-02-20 04:58:55 UTC

Permalink

Hey everyone,

Can we please keep the list reasonably civil? There's no need to use a
tonne of exclamation points and upper case to make your points, not to
resort to name-calling and personal attacks.

Harry is publishing internal reviews that presumably DXW is doing
regardless, and thought this might be useful for everyone else. He
didn't claim to be the singular authority on this, but merely that these
reviews might be useful for others.

Thanks,
Ryan.

Jamie Currie

2014-02-20 05:07:29 UTC

Permalink

New topic to spice things up.

One of my dedicated servers recently starting giving me problems with
GIT. I'd ssh in to pull the latest version of a plugin and would get
error messages about the resource being unavailable or whatever. Tracked
it down to too many processes running, specifically calls to
admin-ajax.php.

Googling turns up lots of threads on the topic, but none that has really
helped nail down exactly what's going on. If I select one of the
processes hitting admin-ajax and kill it, they all disappear and
everything goes back to working.

It seems that auto-save could be the culprit, though I've changed the
auto-save time period to much longer than stock. I have a sneaking
suspicion that it's a theme that was just installed on a site (multisite
network) which has its own page builder framework.

So, anyone have a suggestion as to how I might track down where the
request to admin-ajax is originating? grep gives me the list of
potential candidates, but that would only help if I were to start
disabling / enabling things and then watching to see what happens -- not
an option on a live network of sites.

Jamie Currie
Founder / CEO
wunderdojo
wunderdojo.com
tel: 949-734-0758
1840 Park Newport, #409
Newport Beach, CA 92660
Master web & app developers

------ Original Message ------
From: "Ryan McCue" <***@rotorised.com>
To: wp-***@lists.automattic.com
Sent: 2/19/2014 8:58:55 PM
Subject: Re: [wp-hackers] WordPress plugin inspections

Post by Ryan McCue
Hey everyone,
Can we please keep the list reasonably civil? There's no need to use a
tonne of exclamation points and upper case to make your points, not to
resort to name-calling and personal attacks.
Harry is publishing internal reviews that presumably DXW is doing
regardless, and thought this might be useful for everyone else. He
didn't claim to be the singular authority on this, but merely that
these reviews might be useful for others.
Thanks,
Ryan.
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-20 08:56:48 UTC

Permalink

Thanks Ryan! And spot on.

I hope people find these reviews useful, but if not - of course - they
should simply not use them.

Harry

Casey Bisson

2014-02-20 05:01:04 UTC

Permalink

Harry,

Criticism is hard to take, but important for improvement.

Now that I’ve learned of your reviews, I’m anxious to find out if your team has reviewed any of my plugins, and what issues that might have uncovered.

Because I’m not the only one who has to read my code, and pull requests or patches make my day, I know I how important it is to make my code easy to read and use consistent style.

I have no idea who the reviewers are, or what their skills might be, but it’s probably fair that code that might be questionable to them is worth looking at on my part. If I’ve got a really good reason to do something that raises flags for others, then that’s a really good reason to put comments in the code explaining it. That’s especially true for security and performance issues.

Two requests:

* Consider filing bugs. Just a generic bug with a link to the review so I’d be aware of it would be great. Most of my plugins are in Github, https://github.com/misterbisson?tab=repositories (the others are sort of abandoned by now).

* If I update my plugin to address the issues, are you willing to review the updates in a reasonable time and update the public review?

Thank you,

—Casey

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that we do for our clients. They are just a guide - we conduct some basic checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the list and therefore worth posting? Is the list also interested in vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-20 09:08:15 UTC

Permalink

Hi Casey,

Thanks for the feedback and questions.

Really good point about the comments - inspections are a manual process
and I would hope we would notice any comments like that and take them
into account. We haven't spotted many yet though! Generally the comments
are along the lines of "// TODO: I know this is bad but I'll fix it
later" :)

On those requests:

Filing bugs is something we'd love to be able to do, but I just don't
think it's practical for us. An inspection is a very time-limited thing
and it doesn't usually result in enough information for a good bug
report. If we find something that's definitely vulnerable, we do
generally write an advisory and report that, either directly or via
***@wordpress.org. We would also like to be able to notify plugin
authors, but for it to be practical, it would have to be automatic. We
can't automatically email authors as WordPress.org (entirely reasonably)
does not publicise author email addresses. I suppose we could put a
robot post on the plugin forum, but that seems... questionable. What do
you think?

Of course, we do want to keep everything as up to date as we can.
Unfortunately we don't have nearly the time to be able to monitor all
the inspections for updated versions and check them again proactively.
Inspections will get update if a client asks us to review an updated
version. We do currently always do an updated inspection if the author
drops us a note to ask for one, but that is a bit dependent on our
availability - I can't guarantee we'll always be able to. But certainly
for now, if you have a new version, send an email to ***@dxw.com
with the details, and we'll have a look.

Cheers,

Harry

Post by Madalin Ignisca
Harry,
Criticism is hard to take, but important for improvement.
Now that I’ve learned of your reviews, I’m anxious to find out if your team has reviewed any of my plugins, and what issues that might have uncovered.
Because I’m not the only one who has to read my code, and pull requests or patches make my day, I know I how important it is to make my code easy to read and use consistent style.
I have no idea who the reviewers are, or what their skills might be, but it’s probably fair that code that might be questionable to them is worth looking at on my part. If I’ve got a really good reason to do something that raises flags for others, then that’s a really good reason to put comments in the code explaining it. That’s especially true for security and performance issues.
* Consider filing bugs. Just a generic bug with a link to the review so I’d be aware of it would be great. Most of my plugins are in Github, https://github.com/misterbisson?tab=repositories (the others are sort of abandoned by now).
* If I update my plugin to address the issues, are you willing to review the updates in a reasonable time and update the public review?
Thank you,
—Casey

Post by Harry Metcalfe
Hello list,
We write and publish light-touch inspections of WordPress plugins that we do for our clients. They are just a guide - we conduct some basic checks, not a thorough review.
Would plugins which fail this inspection be of general interest to the list and therefore worth posting? Is the list also interested in vulnerability advisories, or do people tend to get those elsewhere?
https://security.dxw.com/plugins/pods-custom-content-types-and-fields/
Grateful for a steer...
Harry
--
Harry Metcalfe
07790 559 876
@harrym
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Peter van der Does

2014-02-20 13:24:44 UTC

Permalink

On Thu, 20 Feb 2014 09:08:15 +0000

We do currently always do an updated inspection if the author drops
us a note to ask for one, but that is a bit dependent on our
availability - I can't guarantee we'll always be able to.

So what is it? It can't be "always" if you can't "guarantee you'll
always be able to"

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Bryan Petty

2014-02-20 17:42:18 UTC

Permalink

I suppose we could put a robot post on the plugin forum,
but that seems... questionable. What do you think?

This is definitely out of the question. If you're bothering to contact
them, most likely it's a security issue, and these should definitely
not just be posted out in the public view in an automated fashion.

You're right to continue reporting them to ***@wordpress.org.

Regards,
Bryan Petty

Harry Metcalfe

2014-02-20 17:53:43 UTC

Permalink

Hmm. We do report vulnerabilities to ***@wordpress.org, but not
inspection results. I doubt they'd be terribly pleased if we did. I'll ask.

Harry

Post by Bryan Petty

I suppose we could put a robot post on the plugin forum,
but that seems... questionable. What do you think?

This is definitely out of the question. If you're bothering to contact
them, most likely it's a security issue, and these should definitely
not just be posted out in the public view in an automated fashion.
Regards,
Bryan Petty
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Bryan Petty

2014-02-20 18:02:02 UTC

Permalink

Post by Harry Metcalfe
inspection results.

I was under the impression that we were just talking about the
vulnerabilities, not the inspection results. So yeah, that all sounds
right to me.

--
Regards,
Bryan Petty

Harry Metcalfe

2014-02-20 18:06:01 UTC

Permalink

Lots of people have reached that impression - I think making the
distinction clearer is definitely something we need to improve.

Cheers,

Harry

Post by Bryan Petty

Post by Harry Metcalfe
inspection results.

I was under the impression that we were just talking about the
vulnerabilities, not the inspection results. So yeah, that all sounds
right to me.

Blair Williams

2014-02-20 19:45:18 UTC

Permalink

I'm actually wondering why you'd spend the time taking this on? It seems fully redundant and will be a nightmare to maintain -- for you and the plugin authors.

For instance ... suppose I get a notice that you've blacklisted a plugin for one reason or another, the author resolves the issue but you're site still reports it as a problem?

If you truly wanted to serve the community, when you see an issue why wouldn't you simply report the issue to the people at wordpress.org and let them deal with it?

WordPress.org is the only entity in the world that can enforce compliance or yank a plugin from their repository ... and therefore the plugin author is going to have to resolve any issues with them if they want their plugin listed.

I think your site is just going to create alarm and not do any real good ... unless you have someone dedicated to syncing your reviews with what's going on at wordpress.org ... which let's face it ... is going to be a big pain in the butt. I've seen sites like your's before ... here's what's likely to happen with yours: it will eventually be filled with negative reviews for issues which may have already been corrected and could turn into a real problem for some dedicated plugin authors who's livelihood may depend on their plugins.

I think you should focus your efforts into doing more to work with wordpress.org rather than trying to create a site that will likely cause more confusion and damage than it will good.

B.

Lots of people have reached that impression - I think making the distinction clearer is definitely something we need to improve.
Cheers,
Harry

Post by Bryan Petty

Post by Harry Metcalfe
inspection results.

I was under the impression that we were just talking about the
vulnerabilities, not the inspection results. So yeah, that all sounds
right to me.

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Harry Metcalfe

2014-02-20 12:30:06 UTC

Permalink

We do for advisories - which identify specific vulnerabilities - but not for inspections, which more generic. It wouldn't really be practical for us to do it for inspections.

This has come up elsewhere in the thread, though - and we've pondered automatic notifications going to the plugin's forum. What do you think?

Harry

Sent from my mobile

<div>-------- Original message --------</div><div>From: Peter van der Does <***@avirtualhome.com> </div><div>Date:20/02/2014 12:08 (GMT+00:00) </div><div>To: wp-***@lists.automattic.com </div><div>Subject: Re: [wp-hackers] WordPress plugin inspections </div><div>
</div>On Thu, 20 Feb 2014 08:37:55 +0000

Peter van der Does

2014-02-20 12:56:01 UTC

Permalink

On Thu, 20 Feb 2014 12:30:06 +0000

Post by Harry Metcalfe
We do for advisories - which identify specific vulnerabilities - but
not for inspections, which more generic. It wouldn't really be
practical for us to do it for inspections.
This has come up elsewhere in the thread, though - and we've
pondered automatic notifications going to the plugin's forum. What do
you think?
Harry

You don't even let developers know you are testing their plugin.

You created this business model and it seems you forgot a few things
in the model. It's clear you underestimated the shockwave it would send
through the community.

--
Peter van der Does

GPG key: CB317D6E

Site: http://avirtualhome.com
GitHub: https://github.com/petervanderdoes
Twitter: @petervanderdoes

Jacob Snyder

2014-02-20 17:58:32 UTC

Permalink

I think the idea of what DXW is doing is an okay one, but it should be
prefaced better with disclaimers and the "do not use" type of summary
should reevaluated to be a little more responsible.

At the heart of it, the issues his team is quickly reviewing are the same
things I look at when reviewing a plugin, and they are all good indicators
of potential problems or extra, unnecessary overhead.

*It is just irresponsibly reported right now in my opinion.*

Also, I would have no problem with DXWs posting this type of report in my
public forums. I am fine with that kind of transparency. Obviously, an
exploit should be private until a fix can be made available...

I definitely see both sides of this, but I don't think it is black and
white. Harry could do this in a way that would have some benefits (and
still piss some people off), but right now it isn't quite there, because it
can be misleading.

As chip said, right now you are asking to condemn a plugin because it is a
little dirty. That isn't helpful, but pointing out the dirt could be.

My two cents.
Jake.

Continue reading on narkive:

Search results for 'WordPress plugin inspections' (Questions and Answers)

replies

How to remove a plug-in installed in my pc?? specially the WGA plug-in?? its hell disturbin...PLZZ HELP!!?

started 2009-11-09 02:23:48 UTC

security

replies

Should a website cost $2,000 to build?

started 2014-02-28 12:42:08 UTC

programming & design