How to test a plugin before release?

Discussion:

Michael Clark

2013-09-09 21:08:23 UTC

Last week I released an upgrade to a plugin. And started getting bug
reports that same day. Finally late last night I figured out what the
problem was. I and a few people test the new version, and was mostly
good. So this morning I released a new version, and am now getting
different bug reports. Argh! Of course, it's all working just fine on
my own sites.

So, how should I be testing my plugin? Searching for answers to this
question, I found this quote: "Free or not, any type of update
shouldn't be released until fully tested." There are a zillion
different themes and plugins out there, and there's no way to test for
every combination. Should I be building out a "beta test club" to help
me test new versions? I'm quite frustrated at the moment. And don't get
me started on the ever so helpful bug reports of "hey, it doesn't work
- fix it!" and "lame"

If you're curious, my plugin is at
http://wordpress.org/plugins/ultimate-category-excluder/ Mike

--
Michael Clark
http://www.PlanetMike.com

Christmas Music 24 Hours a Day, 7 Days a week
http://www.ChristmasMusic247.com

"Injustice anywhere is a threat to justice everywhere."
- Martin Luther King Jr.

Bryan Petty

2013-09-09 22:01:14 UTC

Permalink

Post by Michael Clark
So, how should I be testing my plugin?

Stop looking at the problem from the perspective that you might need
to build a huge matrix of all sorts of versions of WordPress with
various popular pre-installed plugins combined with different versions
of PHP, and using different major settings like multisite. This is
still partially true, but it's not going to help you. Instead, there's
two major tactics that all plugin developers should be adopting that
mainly avoids this issue.

1. Use programming techniques that avoid conflicts and problems with
other plugins to begin with:

a. Prefix your global methods and classes (or use namespaces if using
OOP and requiring PHP 5.3+ with your plugin) and isolate 3rd party
libraries as a separate service layer if it frequently conflicts with
other common plugins also importing the same library (hide it behind a
separate web-requested PHP script that doesn't include WordPress if
possible).
b. Use your best judgement on ensuring all actions, hooks, and filters
you tie into have an appropriate priority level compared with how
other plugins use those same hooks if you need to use them, and always
make sure your filters return the exact same content if they don't
actually do anything with that content.
c. If your plugin writes HTML, CSS, or Javascript, do the same with
that: namespace and prefix JavaScript code, use unique prefixed CSS
classes on elements, avoid HTML element IDs, and use common built in
HTML element classes to pick up common theme styles.
d. If you want your plugin to work as far back as PHP 5.2 (and you
should since WP supports it), consider doing all of your development
in PHP 5.2 first, and then testing 5.4+ later - you can quickly get
used to the same old habits you used to have of *not* using
namespaces, closures, traits, short array syntax, etc.
e. I'm sure there's techniques others here could add to this list...

2. Add unit tests to your plugins, and think about configuring them
with a continuous integration server like Travis CI.

This will in fact build out a matrix of various configurations for you
covering multiple versions of WordPress, PHP, and multisite (and
common plugins if you customize it to), and run your unit tests across
*all* of them in a fully automated way that you can perform in as
little as a few minutes right before you tag a new release.

I've given a WordCamp presentation on this exact topic if it helps:

Also see: https://github.com/tierra/wordpress-plugin-tests

When you have the hang of those two techniques, the only issues you
end up actually running into are mostly browser-specific bugs with
HTML/CSS. It's much harder to implement automated cross-browser
testing to locate and prevent these bugs before getting reports on
them, but they are usually minor anyway. You might still consider a
service like spoon.net/browsers , browsershots.org or browserstack.com
to help with that if you frequently run into those types of issues.

--
Regards,
Bryan Petty

Guus (IFS)

2013-09-10 04:01:24 UTC

Permalink

Hi Michael,

Good suggestion!

I have the same problem and the main issue is that I have no one to 'counter
test', like download the new version and install it 'fresh' or as an upgrade
'somewhere'.

So yeah, something like 'beta test' or 'counter test' would be very useful
for everybody. Let's think on how to organize,

Guus

----- Original Message -----
From: "Michael Clark"
<dc153464a11bcf5aeb18180db28017fb.wp-***@planetmike.com>
To: <wp-***@lists.automattic.com>
Sent: Tuesday, September 10, 2013 5:08 AM
Subject: [wp-hackers] How to test a plugin before release?

Post by Michael Clark
Last week I released an upgrade to a plugin. And started getting bug
reports that same day. Finally late last night I figured out what the
problem was. I and a few people test the new version, and was mostly
good. So this morning I released a new version, and am now getting
different bug reports. Argh! Of course, it's all working just fine on
my own sites.
So, how should I be testing my plugin? Searching for answers to this
question, I found this quote: "Free or not, any type of update
shouldn't be released until fully tested." There are a zillion
different themes and plugins out there, and there's no way to test for
every combination. Should I be building out a "beta test club" to help
me test new versions? I'm quite frustrated at the moment. And don't get
me started on the ever so helpful bug reports of "hey, it doesn't work
- fix it!" and "lame"
If you're curious, my plugin is at
http://wordpress.org/plugins/ultimate-category-excluder/ Mike
--
Michael Clark
http://www.PlanetMike.com
Christmas Music 24 Hours a Day, 7 Days a week
http://www.ChristmasMusic247.com
"Injustice anywhere is a threat to justice everywhere."
- Martin Luther King Jr.
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

Jaime Martínez

2013-09-10 06:08:53 UTC

Permalink

Hola,

If you want to setup unit tests then you could checkout WP-CLI to ease the initial setup. Here's a tutorial: http://wp-cli.org/blog/plugin-unit-tests.html

If you want to test fresh installs then checkout how WP-CLI runs it's Behat tests. With Behat you can do these "What happends when I have a fresh install".

Ciao,

Jaime Martinez

Post by Guus (IFS)
Hi Michael,
Good suggestion!
I have the same problem and the main issue is that I have no one to 'counter test', like download the new version and install it 'fresh' or as an upgrade 'somewhere'.
So yeah, something like 'beta test' or 'counter test' would be very useful for everybody. Let's think on how to organize,
Guus
Sent: Tuesday, September 10, 2013 5:08 AM
Subject: [wp-hackers] How to test a plugin before release?

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers