In Defense of the WordPress Theme Review Process

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

So what's really going on? Let's examine a brief history of the official WordPress Theme Repository, the underyling need for a Theme review process, the goals and limitations of the current Review Team, the complaints regarding the current process, and the steps being taken to address those complaints.

Theme Repository Timeline

• Jul 2008: Theme Repository Opens
• Dec 2008: WordPress 2.7 Introduces Theme Updates from WP Admin Interface
• Dec 2008: 200 Themes Purged From Repository
• Jun 2009: WordPress 2.8 Introduces Theme Repository Browsing from WP Admin Interface
• Jun 2010: Formation of Theme Review Team

The Need for a Review Process

Why does the Theme Repository even need a review process? Well, for a couple of reasons: because the Theme Repository is a very attractive target for SEO-Spammers and malicious hackers, both of whom would very much like to take advantage of the incredibly large self-hosted WordPress userbase; and because the Theme Repository has a historically bad reputation regarding the level of quality of hosted Themes.

Try to find free WordPress Themes outside of the official Repository, and you will quickly discover a nefarious underworld of encoded Themes infested with, at best, spammy SEO links, and at worst, malicious security exploits. Those who peddle such wares would like nothing better than to gain access to millions of WordPress users through the official Theme Repository.

At the same time, even well-intentioned Themes hosted in the Repository are not known for being particularly high in quality. Since the Theme Repository was introduced, users have complained that available Themes are simply not coded well on the back-end and/or are not designed well on the front-end. Further, even the well-coded, well-designed Themes may not be maintained, and their developers may not provide support to end users.

Goals of Theme Review Team

With Themes being submitted for inclusion in the Theme Repository at the rate of approximately ten Themes per day (including both new Themes and updates to existing Themes), and every Theme requiring an actual human being to review it, the task eventually proved to be overwhelming for what, until a few months ago, was a one-man operation. The WordPress 3.org development cycle presented the perfect opportunity to get the WordPress community involved in the review process. Thus, in June 2010 the Theme Review team was formed.

The primary goal of the Theme Review team was - and is - to ensure that the Theme review process is as expedited as possible. Toward this end, the standing goal is an average turn-around time of one day for each Theme submission. Equally important, however, is the need to ensure that Theme reviews are performed fairly, impartially, objectively, and consistently. This objective is much more easily met when one person performs every review; however, once a team of disparate people get involved, this objective becomes exponentially more difficult. And finally - last, but not least - the goal of the Theme Review team is to help ensure - and encourage - an increased quality standard for Themes hosted in the Repository. As Matt Mullenweg himself stated:

The goal of the theme directory is not to list every theme in the world, it's to list the best ones. We want a reasonable number of themes we can point to that embody the best and brightest of WordPress development, and that users can choose without compromise.

Limitations of Theme Review Team

While having a team of volunteers to perform the task of reviewing Themes offers the advantage of having more people able to perform the work, it also presents several challenges. Previously, Theme reviews were perormed by a full-time Automattic employee, who was paid for such work, and who had full administrative rights and access to the tools and systems necessary to complete such work.

By contrast, the Theme Review team is comprised of volunteers from among the WordPress community, the vast majority of whom have full-time jobs that do not involve WordPress. These volunteers were pulled into the review process, but had to develop the skills necessary to perform quality, consistent reviews. At the time the team was formed, no formal review guidelines existed; the team spent the first month or two of its existence merely discussing the purpose and objectives of the team, and coming up with the beginnings of an objective standard by which all team members could perform Theme reviews.

Further, the team must rely on others for administrative changes to the systems and processes by which Themes are submitted, reviewed, and accepted into the Repository. To complicate matters even more, the Review Team's leader within Automattic was pulled from working with the Theme Repository and moved into other projects. Henceforth, the team has operated in a mostly self-directed manner, to the best of its ability.

And finally, as with any volunteer effort, the team is completely dependent upon the participation of its members. Thus far, the vast majority of Theme reviews, since the formation of the Theme Review team, have been performed by less than a half-dozen people. Whereas 10 reviewers each performing one review per day would keep up with the Theme submission rate, the current participation rate would require three reviews per day per reviewer, just to keep up with demand.

Complaints Regarding Theme Review Process

Thus, as one might imagine, the table has been set for considerable frustration on behalf of Theme developers and Theme reviewers alike. Theme developers are - rightfully - frustrating with the time required for their Themes to be processed through the review queue. The still-maturing submission and review systems - namely, the Theme uploader, SVN, and Trac - lead to even more frustration, as developers must revise and re-submit their Themes multiple times, and unlike with the Plugin Repository, do not have direct SVN commit access. And, of course, the somewhat-sudden increase in Theme quality standards, as represented by the Theme Review Guidelines, has been met with an expectedly mixed response.

Whereas the goal of the Theme Review team is to turn around every ticket within one day, currently the average age of open Trac tickets in the review queue is on the order of one week, with the oldest tickets generally being 2-3 weeks old. And that's just part of the frustration, that actually begins with the Theme submission and review process itself. The process goes something like this:

• The Theme developer packages his Theme as a .zip file, and attempts to upload it to the review queue, using the Theme uploader. The uploader runs a script that checks for various errors with the Theme. If it encounters one, it halts, and returns the error.
• The Theme developer then resolves the error condition, re-packages the Theme, and again attempts to upload it. The uploader runs the script again, and if it finds another error, it once again halts, and returns the error. If the Theme has several such errors, the Theme developer may repeat this process several times before finally uploading the Theme successfully.
• Once the Theme is uploaded, a Trac ticket is generated, and the Theme is established in the review queue. Themes are reviewed first-in-first-out, and the review queue usually holds steady at around 60-100 open tickets at any one time.
• Once the ticket reaches the head of the queue, a Theme reviewer assigns himself the ticket and performs the review. The reviewer will leave review notes as a comment to the Trac ticket, and then resolve the ticket as either "approved" or "not-approved".
• If the ticket is resolved as "approved", it must be manually flagged to go "live" in the Repository. If the ticket is resolved as "not approved", the Theme developer must revise the Theme, addressing the issues noted in the review comments in the Trac ticket, and then start the process all over again.

It is no wonder, then, that a Theme developer might find himself frustrated not only with the submission and review process, and the length of time required to complete the review, but also with the review criteria that result in having to go through the process multiple times, just to get a Theme added to the Repository. (And how much more frustrating for a Theme developer trying to submit a bugfix for a Theme already in the Repository!)

What's Being Done?

The good news is, the entire process is undergoing continual improvement and refinement.

Ticket Prioritization

First, the Review Team has implemented a ticket prioritization system:

1. Tickets for revisions to Themes previously reviewed and approved
2. Tickets for revisions to Themes previously reviewed
3. Tickets for new and revised Themes not yet reviewed

Thus, tickets for Themes that have already been previously reviewed and approved are prioritized ahead of all other tickets. Next, tickets for revisions to Themes that have been previously reviewed, but did not pass the review, are prioritized ahead of tickets for Themes that have not yet been reviewed.

Diff-Reviews of Revisions to Approved Themes

To help further expedite review and approval of revisions to Themes that have previously been reviewed and approved, rather than undergoing a full-blown review, these tickets now are reviewed based on the diff between the approved version and the new revision. This way, bugfixes and other improvements to already approved Themes should happen much more quickly.

Revisions Maintain Position in Queue

For all tickets, if a newer version is submitted before the existing ticket is reviewed, the new ticket takes the original ticket's place in the review queue. This way, bugfixes and other improvements do not cost the Theme its place in the queue.

Better Communication Between Reviewer and Developer

Originally, the Theme developer had no direct connection with the Trac ticket generated for an uploaded Theme. This system, while still undergoing continual improvement, is now much better. The Theme developer is now listed as the ticket "Reporter", and the Theme developer's email address is now added to the ticket "CC" list by default.

Further, the Theme Review team has established multiple avenues for communication between developers and reviewers:

• The Trac ticket comment thread
• The Theme-Reviewers mail-list
• The #wp-themes #wordpress-themes channel on irc.freenode.net

Tools to Facilitate Successful Theme Review

Several tools are available for Theme Developers to help ensure that Themes will pass the review process (and in fact, these are the same tools that are used by the reviewers when conducting Theme reviews):

• HTML/CSS Validation: W3 Unicorn Validator
• PHP Errors: Debogger Plugin
• WordPress Deprecated Function Calls: Log Deprecated Notices Plugin

However, another tool is now available, thanks to the efforts of Theme Review team member Simon "Pross" Prosser. Pross' Theme-Check.

This tool will validate Themes against Licensing issues, PHP errors and notices, WordPress deprecated function calls, missing required functions, template tags, and hooks, and other similar Theme Review guidelines. If a Theme passes the Theme-Check validation, it has a much better chance of passing the Theme Review with as little trouble as possible.

Improvements to Uploader Validation Script

The Theme Review team is currently in the process of reviewing and implementing changes and improvements to the validation script that runs as part of the Theme uploader tool. (In fact, once completed, the Uploader validation script will be nearly identical to Pross' Theme-Check tool, with a few, undisclosed differences.)

In addition to updating and improving the actual criteria the validation script uses, Theme developers likely will welcome another change to the uploader script. Whereas currently the script will stop and return upon the first error it encounters (thus resulting in potentially several iterations of revise-upload-validate), soon the uploader script will run through - and report on - all errors in one pass. This way, Theme developers will know, in one pass, all the errors that are preventing upload of the Theme.

Future Changes: SVN-Commit Access

The Theme Review team is exploring the impact and implications of providing Theme developers with SVN-commit access for their Themes, similar to the SVN-commit access that Plugin developers have with the Plugin repository. This change would enable Theme developers to update their Themes directly to SVN, bypassing the Theme uploader altogether. More importantly, this change would enable Theme developers to use SVN for proper Theme maintenance and version-control.

How, when - and even if - this change ever takes place is still not entirely determined, and if it is implemented, it will be more of a long-term, rather than short-term, project. But, if and when it happens, it will be a significant improvement for Theme developers.

Effectiveness Check: Is It Working?

Whew! That's quite the list of changes. But are they effective? That's a fair question: one that Theme Developers have the right to ask of the Theme Review Team, and one that the Theme Review Team must continually ask itself.

But in order to answer that question, the Theme Review Team must have some way of defining and measuring the process. To that end, the Theme Review Team now publishes weekly Theme-review statistics, which are summarized via trend charts. These trend charts provide at-a-glance overview of ticket cycle times, tickets opened/closed, closed ticket resolution, and reviewers.

• 
• 
• 
• 

The weekly statistics, which compare week-to-week performance for several metrics, are published here and announced via the Theme Reviewers mail-list.

While the Theme Review Team has made a huge push in the past several weeks, in an attempt to get the review queue in a manageable condition, the true test will be how well the team performs once the review queue reaches steady state: can the team maintain equilibrium, and turn tickets around in a reasonable time? Only time will tell.