Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

The most contentious matter is, of course, the Theme Review Guidelines. While some in the community have embraced the Theme Review team's efforts and principle regarding improving the overall standard of quality of Repository-hosted Themes, others have found the Theme Review Guidelines to be "stifling", a "barrier to entry" for new Theme developers, and even "clueless".

So what's really going on? Let's take a look at this most contentious subject: what constitutes quality with respect to WordPress Themes.

Commercial vs. Free Themes

At this point in the development of WordPress and the developer and user community that surrounds the project, free Themes - and especially those hosted in the WordPress Theme Repository - have developed a reputation for being inferior to their Commercial counterparts. As best as I can discern, the advantages that Commercial Themes provide can be summarized as follows:

• Quality
• Documentation
• Support
• Advanced Features

While much of this list is outside the control of the Theme Review team, the one that the team can most directly (and immediately) impact is, in fact, Theme quality.

Why Quality?

Especially now that Themes can be installed and updated from the Repository, all from within the WordPress admin interface, the Repository is essentially an extension of WordPress itself. Thus, the quality of Themes in the Repository reflects directly upon end users' perception of the quality of WordPress itself.

To be fair, Commercial Themes are not immune to criticism - to which the recent discussion regarding ThemeForest can attest. As Jeff Chandler points out, the criticism against ThemeForest parallels the criticism against the WordPress Theme Repository. Consider the following quotes, as applied to the Theme Repository:

Comment:

I really think the issue is about quality and support. Although there are many very qualified authors on [the repository] putting out high quality, well supported themes, there is also a lot of junk.

[...A] lot of them are great looking. I’ve [downloaded] several for just that reason. But of the 10 of so themes I’ve [downloaded] from [the repository] over the past year, only 1 is actually in use. The rest of them, once I started working with them, ended up being too limiting in one way or another, had serious coding problems, conflicts with plugins or other, major customization roadblocks and very little to no support from the author.

I’m a fairly geeky person who can muddle through figuring out how to customize themes myself, most of the time. But I’m not a coder, so there’s a limit to what I can do. I think there are a lot of people like me. Woo and StudioPress and other premium theme authors provide support for their themes that allow us middle of the road users to get where we want to go. [The repository] mostly doesn’t.

Maybe...the support will improve. But I’m much more leery about [downloading] a theme from [the repository] than I used to be. [The repository] is cheap, but you get what you pay for. I spent more money for a theme from StudioPress and guess what? It worked right out of the gate, I could customize it easily, and any questions I had were promptly answered by the support team.

Comment:

The only thing that is letting [the repository] down is that there’s no quality control. Maybe a quality control system should be put in place or some sort of star rating system, for example one person with a fair bit of theme creation knowledge and who knows how to create a theme properly could examine the themes and give them a rating.

Comment:

[M]ost of the themes there are...crap... poorly coded themes by wannabe themers.

Comment:

This is what I’ve seen with [the repository], and while not a sweeping generalization, it’s too common to be rare:

* Lack of updates. Some of the really successful theme authors update their themes to add features, fix bugs or address new WordPress functions. A whole lot more don’t. The problem with this is that the theme looks great. The demo works. Joe Consumer might not realize that because it uses old functions or does things in a certain way, it’s not going to work with newer WordPress features or some plugins.

* JavaScript Soup. Now, this isn’t a [repository] problem, this is a THEME problem. ...I see this all the time. Hell, my own website...needs to have its JS significantly refactored, I’m not immune — but my stuff also doesn’t break anything else.

* Plugin Incompatibilities – This is usually directly related with problem 1 — lack of updates or fixes.

The big advantage that places like Woo Themes and Studiopress and Thesis have over places like [the repository] is that there is a central point of support contact. If something breaks on a Woo theme, I know they’ll release an update. Their business depends on that happening. If there is a conflict with a popular plugin, they are going to figure out what is going on.

[The repository] isn’t the same way.

Comment:

I won’t personally ever [download] a theme from [the repository] because I’m a theme developer myself. But, I do deal with end users that have gotten burned on more than one occasion by themes from [the repository]. Holding your theme authors to certain levels of coding standards and best practices would be a good start.

Comment:

One thing I agree with is that the quality of themes should be stepped up...

Every one of these comments was describing ThemeForest - but every single one can bbe equally applied to the WordPress Theme Repository. Look hard enough, and you can find identical sentiment expressed about the Repository.

What is Quality?

The difficult question is: what is quality? More importantly to the current discussion: how can the Theme Review team define quality in an objective manner that can be interpreted the same by both Theme developers and Theme reviewers? For example:

ThemeGrade Test Criteria:

• W3 HTML/CSS Compliance
• Browser Compatibility (Firefox/Chrome/IE7)
• Theme Support
• Image Alignment
• Clearing Floats
• Content Width Auto-Adjustment
• Threaded Comment Support
• Hierarchical Page/Category Sidebar Lists
• Sidebar Content Width Auto-Adjustment
• Special Effects (Features/Functionality)

Another Site's Criteria:

• Design
• Ease of Use
• SEO
• Features
• Support
• Price

Here are some of ThemeForest's common "rejection factors" for their hosted WordPress Themes:

• Documentation
• Typography (Aesthetics, Line-Height, Text Alignment)
• HTML/CSS Validation
• List Markup
• Browser Compatibility
• License

So which of these is correct, or best (or at least closest)? From what I can gather thus far, the consensus opinion regarding Theme quality can be distilled down to this response from Leland, of ThemeLab:

XHTML/CSS validity, general code cleanliness, design not looking like crap, using wp_head/wp_footer hooks, basic stuff I guess

But, how to translate all that into objective Guidelines for performing Theme reviews?

Theme Review Quality Standards

The current Theme Review Guidelines can be summarized into five main areas:

1. Licensing, Credit Links, and Security
2. Code and Markup Standards Compliance and Best Practices
3. WordPress Functionality and Feature Support
4. WordPress Best Practices
5. Theme Front-End Display

The current Guidelines are derived from these main areas of concern.

Licensing, Credit Links, and Security

The primary reasons that Themes undergo any review in the first place fall into this category. An astounding number of submitted Themes has non-GPL-compatible licenses or restrictions, SEO/spam footer links, or security exploits of one form or another.

Licensing

The Theme Repository Guidelines have, since the inception of the Repository, stated that Themes hosted in the repository must be compatible with WordPress' GPL licensing. This requirement includes all Theme template files (PHP), style (CSS), markup (HTML), scripts (JS), images, icons, fonts, and anything else bundled with the Theme.

Some Themes indicate that they are licensed under GPL, but knowingly include restrictive notices such as a prohibition against modifying the Theme footer, removing credit links, or modifying/redistributing the Theme. Some Themes unknowingly bundle icons (such as FamFamFam's great icons) or fonts that do not have GPL-compatible licenses.

Whichever the case, one of the most important safeguards provided by the Theme Review Guidelines (and Review) is to ensure that the user freedoms protected by the GPL are not restricted by Themes downloaded from the official Theme Repository.

Credit Links

The visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Link Spammers ever since the first Theme was submitted. Users do not want their Themes to be riddled with Link Spam, nor do they want their Themes to be used to advertise for all manner of legitimate and illegitimate websites.

For this reason, the Theme Review Team has established fairly strict, intentionally rigid requirements for public-facing links in Themes, and for the credit-link URLs Theme URI and Author URI, which are displayed by the Repository. In order to ensure fairness to all Theme Developers, the established requirements are as objective as possible - intentionally leaving little room for subjectivity.

Security

Likewise with Link Spammers, the visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Black-Hat types looking to propagate security exploits. The nature of Themes (and Plugins, for that matter) cause them to be an attractive attack vector, since a Theme, once activated, has free reign over every part of WordPress and its database. Additionally, poor and/or obsolete coding can create exploitable security vulnerabilities.

For this reason, Themes submitted for inclusion in the Repository must be reviewed for intentional exploits (worms, etc.) and unintentional vulnerabilities.

Code and Markup Standards Compliance and Best Practices

While differences of opinion exist regarding the importance of the correlation, or the extent of code validity/cleanliness required, there does appear to be some consensus within the community regarding the correlation between clean, valid code and Theme quality.  Thus, the Theme Review Guidelines include requirements regarding HTML/CSS validation, PHP errors, JS errors, and the use of deprecated WordPress functions.

HTML/CSS Validation

The ubiquitousness of valid markup (HTML/CSS) has been well-established for some time now. Most developers are not unfamiliar with using the W3 HTML and CSS validator tools to ensure their code conforms to current standards. (And W3 have made this task even easier, with a unified validator: Unicorn.) So, this requirement hasn't been viewed as terribly controversial.

Note that the Theme Review Guidelines do allow for legitimate exceptions. For example, both WordPress core and the Theme Unit Test XML data output markup that does not pass the W3 validator. Some of these errors can be eliminated for testing purposes using the FixPress Plugin. Nevertheless, these types of validation errors are ignored during the Theme review. Also, browser-specific CSS properties for draft CSS3 specs, such as border-radius and box-shadow (e.g. moz-border-radius or webkit-box-shadow) are likewise ignored during the Theme review.

PHP/JS Errors

By far, the most controversial Theme Review requirement is the prohibition against PHP errors, warnings, and notices - particularly with respect to warnings and notices. However, in my experience reviewing Themes, the vast majority of these warnings and notices are for undefined indices (not putting function arguments in quotes), undefined variables, and undefined offsets - all of which are quite easily fixable. Also, while such warnings and notices do not impact the rendering of the site for visitors, they can be incredibly noisy in the site's error logs - which can make searching the error logs in order to find legitimate errors rather difficult.

Finding such PHP errors is as simple as using the Debogger Plugin, and viewing output generated for both the Theme's front-end and admin options pages.

Deprecated WordPress Functions

Probably the most important code-related standard is the prohibition against deprecated functions. Whereas both modern browsers are well-suited to overcoming markup validation and PHP parsers handle errors, warnings, and notices with little impact to the end-user, the use of deprecated WordPress functions represents true obsolescence - and, therefore, both functional issues as well as potential security concerns - within Themes.

Fortunately, deprecated WordPress functions are almost always simple to replace. In fact, WordPress core even outputs the replacement function to use. Finding calls to deprecated WordPress functions is as simple as using the Log Deprecated Notices Plugin (hint: it works for Plugins, too!), and viewing its output in the admin UI, under Tools -> Deprecated Calls.

WordPress Functionality and Feature Support

Since the official WordPress Theme Repository is, essentially, a very public showcase of WordPress Themes, it is important that repository-hosted Themes support core WordPress functionality and features. However, this need must be balanced against the intended use cases for each Theme. The Theme Review Guidelines attempt to maintain this balance by indicating which major core features are Required, Recommended, or Optional.

Required Features

Required features must be included in repository-hosted Themes. Themes will not be accepted into the repository unless they support these features and implement them properly. Currently, the list of Required features is (intentionally) very limited:

• Automatic Feed Links
• Widgets
• Comments

Recommended Features

Recommended features are not required to be included in repository-hosted Themes, but are strongly recommended. Themes will be accepted into the repository with or without support for these features, but if these features are supported, they are required to be implemented properly:

• Navigation Menus
• Post Thumbnails
• Custom Header Images
• Custom Background
• Visual Editor CSS

Optional Features

All other features are considered optional, and are not listed in the Guidelines. However, as with the Required and Recommended features, any Optional features that a Theme supports are required to be implemented properly.

WordPress Best Practices

The official WordPress Theme Repository is intended to be used not only by WordPress end users installing Themes for their own use, but is also intended to be used by nascent WordPress developers who want to learn how to develop WordPress Themes - whether for their own use, or for distribution. For this reason (and due to the public visibility and widespread use of repository-hosted Themes), it is important that repository-hosted Themes adhere to some WordPress standards and best practices.

These standards and best practices include support for the most recent WordPress version; the use of core functions, hooks, template tags, and styles; the use of standard Theme template files; and Theme documentation.

Support For Most Recent WordPress Version

Repository-hosted Themes are required to support the version of WordPress current at the time the Theme is submitted for inclusion in the repository. This support includes incorporation of new features and functionality, as well as replacement of newly deprecated functions. In order to give Theme Developers sufficient time to incorporate new features and other changes necessitated by new versions of WordPress, this requirement takes effect one month after each WordPress version release.

Since each new WordPress version release represents potentially major changes to Theme Review Guidelines, the Guidelines will be updated with each WordPress major version release, to include any major revisions to the requirements. A Version-Specific Changes section will be added to the Theme Review Codex page, to highlight any major revisions to the Guidelines.

Use of Core Functions, Hooks, Template Tags, and Styles

In order to facilitate their use as learning tools for new developers, as well as to ensure that they work as well as possible and support as many Plugins as possible, repository-hosted Themes are required to support core WordPress functions, hooks, and template tags. This requirement encompasses both the use of core functionality rather than customized functions, as well as proper implementation of such functionality.

This requirement ensures a consistent experience from Theme to Theme for end users who choose to customize their Themes, as well as ensures that the most common hooks will exist for Plugins that depend on them.

Use of Standard Theme Template Files

All Themes are required to include certain files (index.php, style.css) in order to be considered a "valid" Theme by WordPress when installed. Repository-hosted Themes are required to include screenshot.png in order to display a preview screenshot when displayed in the repository. Repository-hosted Themes are also required to support comments and must implement this feature properly; therefore, they are required to include comments.php. All other Theme template files, if included, must be called by the Theme using the proper template tag.

Theme Documentation

Themes are encouraged to be a minimalist or as complex as the Theme Developer wishes to make them. However, if customizations, configuration options, custom template files, etc. are included (or if the Theme is designed or optimized to work with specific Plugins), the Theme is required to include user documentation explaining how to use all of the Theme's various customizations and options.

Theme Front-End Display

Back-end (e.g. code) reviews do help ensure a certain quality level, but what is most important is the front-end experience. Repository-hosted Themes are expected to provide a certain degree of user-experience consistency for visitors to the sites using them. For this reason, the Theme Review process includes the Theme Unit Tests: Themes are installed on a test site using a standard XML data set, in order to test navigation, layout, readability, HTML tag and CSS rendering, image handling, and various usability criteria.

Arguments and Complaints Against the Theme Review Guidelines

Having explained the reasoning behind the current Theme Review Guidelines, let's examine some of the arguments and complaints expressed regarding the Guidelines. Some of those complaints include:

• The Guidelines represent a "barrier to entry" for new WordPress Theme developers
• The Guidelines are too extensive/detailed/specific/strict
• The Guidelines stifle design creativity/innovation
• The Guidelines change too frequently

The Guidelines Represent a "Barrier to Entry" for New WordPress Theme Developers

Some have argued that the current Guidelines make WordPress Theme development more difficult for new Developers.

To the contrary, I would argue that, due to the extensive Codex function/tag cross-referencing, the Guidelines actually provide an incredibly useful and educational tool for new Theme Developers. Not only can new Developers learn the more important WordPress functions, template tags, and hooks, they also have direct links to the Codex entries, so that they can learn how to implement them.

The only way that the Guidelines might represent a "barrier to entry" would be if they are poorly written, and difficult to understand. Thus, it is certainly incumbent upon the volunteers who comprise the Theme Review Team to ensure that the Guidelines are clear, concise, and understandable. To that end, we will work continually to ensure that the Guidelines meet this standard.

The Guidelines are Too Extensive/Detailed/Specific/Strict

Some argue that the Guidelines are too strict, and too imposing, with "Required" this and "Required" that listed everywhere.

I believe this complaint represents a misunderstanding of how the Guidelines are written. In order to ensure objective, fair reviews, the Theme Review Team decided that the Guidelines would be written per RFC 2119. This way, the criticality of each guideline would be clearly understood, as required, recommended, or optional. The intent was not to create the perception of being imposing, but rather being as consistent as possible in describing each guideline's criticality. We intentionally decided to limit criticality adjectives to required, recommended, and optional, so as not to introduce potential confusion from similar, synonymous adjectives.

Further, the extent and strictness of the Guidelines hasn't really changed; rather, they have just been stated more explicitly, and enforced more consistently. The Theme Unit Test requirements have changed very little, and many of the rest of the requirements are actually derived from the Theme Unit Tests.

The Guidelines Stifle Design Creativity/Innovation

Some argue that the Guidelines have too many required criteria, and in so doing stifle design creativity and innovation.

I would counter that good innovation is not found in poorly written code, or in "reinventing the wheel". Further, I would counter that the Guidelines provide an excellent baseline from which Developers can focus less on basic Theme functionality, and more on design creativity and innovation.

The Guidelines Change Too Frequently

Initially, when the Theme Review Team was first getting itself organized, and getting the Guidelines ironed out, this complaint may have been considerably more valid. However, at this point, the Guidelines have had no major changes for several weeks (bearing in mind that this whole effort started less than four months ago).

Also, aside from minor clarifications, the Theme Review Team has determined that major revisions to the Guidelines will take place in synchronization with the WordPress release cycle. There are exceptions to this rule; for example, if we discover an egregious security oversight, or an incorrect requirement, we will make an immediate change.

Major changes (if any) will be introduced with each major WordPress version release, and will take effect one month after the WordPress version release. Further, these changes will be summarized in a dedicated "Version-Specific Changes" section in the Guidelines, so that Developers will know what changes may be coming.

If you have read the Guidelines recently, you will also notice that several things indicate that they are "draft" status. This is an attempt to "dry-run" changes to the Guidelines, for input/feedback. Using this approach, developers will generally always have an idea of proposed changes to the Guidelines.

Room For Improvement

Is there room for improvement? Always! And we welcome all constructive criticism and feedback.

As Theme Reviewers, we don't gain anything by making more work for ourselves, by needlessly failing Theme reviews that will only lead to the same Theme being reviewed multiple times. It is in everyone's best interest to have more Themes pass the review process. So, if a specific guideline is incorrect, inappropriate, inaccurate, or otherwise in need of revision, we want to know about it!

But at the same time, at this point, the Guidelines are fairly stable, and fairly reasonable. Generalized complaints about the Guidelines as a whole will be neither terribly helpful nor especially well-received. Try to focus your concerns to specific issues regarding specific requirements, and we would be happy to receive such feedback. In fact, the Guidelines have only been able to be refined to the point that they have currently, thanks to such feedback from Theme Developers.

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

So what's really going on? Let's examine a brief history of the official WordPress Theme Repository, the underyling need for a Theme review process, the goals and limitations of the current Review Team, the complaints regarding the current process, and the steps being taken to address those complaints.

Theme Repository Timeline

• Jul 2008: Theme Repository Opens
• Dec 2008: WordPress 2.7 Introduces Theme Updates from WP Admin Interface
• Dec 2008: 200 Themes Purged From Repository
• Jun 2009: WordPress 2.8 Introduces Theme Repository Browsing from WP Admin Interface
• Jun 2010: Formation of Theme Review Team

The Need for a Review Process

Why does the Theme Repository even need a review process? Well, for a couple of reasons: because the Theme Repository is a very attractive target for SEO-Spammers and malicious hackers, both of whom would very much like to take advantage of the incredibly large self-hosted WordPress userbase; and because the Theme Repository has a historically bad reputation regarding the level of quality of hosted Themes.

Try to find free WordPress Themes outside of the official Repository, and you will quickly discover a nefarious underworld of encoded Themes infested with, at best, spammy SEO links, and at worst, malicious security exploits. Those who peddle such wares would like nothing better than to gain access to millions of WordPress users through the official Theme Repository.

At the same time, even well-intentioned Themes hosted in the Repository are not known for being particularly high in quality. Since the Theme Repository was introduced, users have complained that available Themes are simply not coded well on the back-end and/or are not designed well on the front-end. Further, even the well-coded, well-designed Themes may not be maintained, and their developers may not provide support to end users.

Goals of Theme Review Team

With Themes being submitted for inclusion in the Theme Repository at the rate of approximately ten Themes per day (including both new Themes and updates to existing Themes), and every Theme requiring an actual human being to review it, the task eventually proved to be overwhelming for what, until a few months ago, was a one-man operation. The WordPress 3.org development cycle presented the perfect opportunity to get the WordPress community involved in the review process. Thus, in June 2010 the Theme Review team was formed.

The primary goal of the Theme Review team was - and is - to ensure that the Theme review process is as expedited as possible. Toward this end, the standing goal is an average turn-around time of one day for each Theme submission. Equally important, however, is the need to ensure that Theme reviews are performed fairly, impartially, objectively, and consistently. This objective is much more easily met when one person performs every review; however, once a team of disparate people get involved, this objective becomes exponentially more difficult. And finally - last, but not least - the goal of the Theme Review team is to help ensure - and encourage - an increased quality standard for Themes hosted in the Repository. As Matt Mullenweg himself stated:

The goal of the theme directory is not to list every theme in the world, it's to list the best ones. We want a reasonable number of themes we can point to that embody the best and brightest of WordPress development, and that users can choose without compromise.

Limitations of Theme Review Team

While having a team of volunteers to perform the task of reviewing Themes offers the advantage of having more people able to perform the work, it also presents several challenges. Previously, Theme reviews were perormed by a full-time Automattic employee, who was paid for such work, and who had full administrative rights and access to the tools and systems necessary to complete such work.

By contrast, the Theme Review team is comprised of volunteers from among the WordPress community, the vast majority of whom have full-time jobs that do not involve WordPress. These volunteers were pulled into the review process, but had to develop the skills necessary to perform quality, consistent reviews. At the time the team was formed, no formal review guidelines existed; the team spent the first month or two of its existence merely discussing the purpose and objectives of the team, and coming up with the beginnings of an objective standard by which all team members could perform Theme reviews.

Further, the team must rely on others for administrative changes to the systems and processes by which Themes are submitted, reviewed, and accepted into the Repository. To complicate matters even more, the Review Team's leader within Automattic was pulled from working with the Theme Repository and moved into other projects. Henceforth, the team has operated in a mostly self-directed manner, to the best of its ability.

And finally, as with any volunteer effort, the team is completely dependent upon the participation of its members. Thus far, the vast majority of Theme reviews, since the formation of the Theme Review team, have been performed by less than a half-dozen people. Whereas 10 reviewers each performing one review per day would keep up with the Theme submission rate, the current participation rate would require three reviews per day per reviewer, just to keep up with demand.

Complaints Regarding Theme Review Process

Thus, as one might imagine, the table has been set for considerable frustration on behalf of Theme developers and Theme reviewers alike. Theme developers are - rightfully - frustrating with the time required for their Themes to be processed through the review queue. The still-maturing submission and review systems - namely, the Theme uploader, SVN, and Trac - lead to even more frustration, as developers must revise and re-submit their Themes multiple times, and unlike with the Plugin Repository, do not have direct SVN commit access. And, of course, the somewhat-sudden increase in Theme quality standards, as represented by the Theme Review Guidelines, has been met with an expectedly mixed response.

Whereas the goal of the Theme Review team is to turn around every ticket within one day, currently the average age of open Trac tickets in the review queue is on the order of one week, with the oldest tickets generally being 2-3 weeks old. And that's just part of the frustration, that actually begins with the Theme submission and review process itself. The process goes something like this:

• The Theme developer packages his Theme as a .zip file, and attempts to upload it to the review queue, using the Theme uploader. The uploader runs a script that checks for various errors with the Theme. If it encounters one, it halts, and returns the error.
• The Theme developer then resolves the error condition, re-packages the Theme, and again attempts to upload it. The uploader runs the script again, and if it finds another error, it once again halts, and returns the error. If the Theme has several such errors, the Theme developer may repeat this process several times before finally uploading the Theme successfully.
• Once the Theme is uploaded, a Trac ticket is generated, and the Theme is established in the review queue. Themes are reviewed first-in-first-out, and the review queue usually holds steady at around 60-100 open tickets at any one time.
• Once the ticket reaches the head of the queue, a Theme reviewer assigns himself the ticket and performs the review. The reviewer will leave review notes as a comment to the Trac ticket, and then resolve the ticket as either "approved" or "not-approved".
• If the ticket is resolved as "approved", it must be manually flagged to go "live" in the Repository. If the ticket is resolved as "not approved", the Theme developer must revise the Theme, addressing the issues noted in the review comments in the Trac ticket, and then start the process all over again.

It is no wonder, then, that a Theme developer might find himself frustrated not only with the submission and review process, and the length of time required to complete the review, but also with the review criteria that result in having to go through the process multiple times, just to get a Theme added to the Repository. (And how much more frustrating for a Theme developer trying to submit a bugfix for a Theme already in the Repository!)

What's Being Done?

The good news is, the entire process is undergoing continual improvement and refinement.

Ticket Prioritization

First, the Review Team has implemented a ticket prioritization system:

1. Tickets for revisions to Themes previously reviewed and approved
2. Tickets for revisions to Themes previously reviewed
3. Tickets for new and revised Themes not yet reviewed

Thus, tickets for Themes that have already been previously reviewed and approved are prioritized ahead of all other tickets. Next, tickets for revisions to Themes that have been previously reviewed, but did not pass the review, are prioritized ahead of tickets for Themes that have not yet been reviewed.

Diff-Reviews of Revisions to Approved Themes

To help further expedite review and approval of revisions to Themes that have previously been reviewed and approved, rather than undergoing a full-blown review, these tickets now are reviewed based on the diff between the approved version and the new revision. This way, bugfixes and other improvements to already approved Themes should happen much more quickly.

Revisions Maintain Position in Queue

For all tickets, if a newer version is submitted before the existing ticket is reviewed, the new ticket takes the original ticket's place in the review queue. This way, bugfixes and other improvements do not cost the Theme its place in the queue.

Better Communication Between Reviewer and Developer

Originally, the Theme developer had no direct connection with the Trac ticket generated for an uploaded Theme. This system, while still undergoing continual improvement, is now much better. The Theme developer is now listed as the ticket "Reporter", and the Theme developer's email address is now added to the ticket "CC" list by default.

Further, the Theme Review team has established multiple avenues for communication between developers and reviewers:

• The Trac ticket comment thread
• The Theme-Reviewers mail-list
• The #wp-themes #wordpress-themes channel on irc.freenode.net

Tools to Facilitate Successful Theme Review

Several tools are available for Theme Developers to help ensure that Themes will pass the review process (and in fact, these are the same tools that are used by the reviewers when conducting Theme reviews):

• HTML/CSS Validation: W3 Unicorn Validator
• PHP Errors: Debogger Plugin
• WordPress Deprecated Function Calls: Log Deprecated Notices Plugin

However, another tool is now available, thanks to the efforts of Theme Review team member Simon "Pross" Prosser. Pross' Theme-Check.

This tool will validate Themes against Licensing issues, PHP errors and notices, WordPress deprecated function calls, missing required functions, template tags, and hooks, and other similar Theme Review guidelines. If a Theme passes the Theme-Check validation, it has a much better chance of passing the Theme Review with as little trouble as possible.

Improvements to Uploader Validation Script

The Theme Review team is currently in the process of reviewing and implementing changes and improvements to the validation script that runs as part of the Theme uploader tool. (In fact, once completed, the Uploader validation script will be nearly identical to Pross' Theme-Check tool, with a few, undisclosed differences.)

In addition to updating and improving the actual criteria the validation script uses, Theme developers likely will welcome another change to the uploader script. Whereas currently the script will stop and return upon the first error it encounters (thus resulting in potentially several iterations of revise-upload-validate), soon the uploader script will run through - and report on - all errors in one pass. This way, Theme developers will know, in one pass, all the errors that are preventing upload of the Theme.

Future Changes: SVN-Commit Access

The Theme Review team is exploring the impact and implications of providing Theme developers with SVN-commit access for their Themes, similar to the SVN-commit access that Plugin developers have with the Plugin repository. This change would enable Theme developers to update their Themes directly to SVN, bypassing the Theme uploader altogether. More importantly, this change would enable Theme developers to use SVN for proper Theme maintenance and version-control.

How, when - and even if - this change ever takes place is still not entirely determined, and if it is implemented, it will be more of a long-term, rather than short-term, project. But, if and when it happens, it will be a significant improvement for Theme developers.

Effectiveness Check: Is It Working?

Whew! That's quite the list of changes. But are they effective? That's a fair question: one that Theme Developers have the right to ask of the Theme Review Team, and one that the Theme Review Team must continually ask itself.

But in order to answer that question, the Theme Review Team must have some way of defining and measuring the process. To that end, the Theme Review Team now publishes weekly Theme-review statistics, which are summarized via trend charts. These trend charts provide at-a-glance overview of ticket cycle times, tickets opened/closed, closed ticket resolution, and reviewers.

The weekly statistics, which compare week-to-week performance for several metrics, are published here and announced via the Theme Reviewers mail-list.

While the Theme Review Team has made a huge push in the past several weeks, in an attempt to get the review queue in a manageable condition, the true test will be how well the team performs once the review queue reaches steady state: can the team maintain equilibrium, and turn tickets around in a reasonable time? Only time will tell.