Theme Review

Posts filed under Theme Review

A Guide to Reviewing Themes for the WordPress Theme Repository

Filed in Web DevelopmentTags: Theme Review, Themes, tutorials, WordPress

Thanks in large part to Justin Tadlock's recent blog post, in which he solicited more people to get involved with the WordPress Theme Review Team (WPTRT), we have seen a fantastic, huge influx of new volunteers. Many of these new volunteers have had questions about how to perform a Theme review. Hopefully this guide will help answer those questions.

Getting Started

Test Environment Setup

First things first: review How to Join WPTRT.

WordPress Setup

Ensure that a few WordPress settings are modified in order to facilitate the Theme Unit Tests:

  • Settings -> General: set the Site Title to something fairly long, and set the Tagline to something even longer. These settings will facilitate testing how the Theme handles these values.
  • Settings -> Reading: set "Blog pages show at most" to 5. This setting will ensure that index/archive pagination is triggered.
  • Settings -> Discussion: enable Threaded Comments, at least 3 levels deep. This setting will facilitate testing of Theme comment list styling.
  • Settings -> Discussion: enable Break comments into pages, and set 5 comments per page. This setting will facilitate testing of Theme paginating link markup/styling.
  • Settings -> Media: ensure that no values are set for max width or height of Embedded media. This setting will facilitate testing of the Theme $content_width setting/implementation.
  • Settings -> Permalinks: ensure that a non-default permalink setting is selected, e.g. "Month and name". This setting will facilitate stepping through the Theme Unit Tests.

Next, create at least two Custom Menus:

  • Long Menu: all included Pages
  • Short Menu: a menu of 2-3 Pages

Now, you're ready to be assigned a ticket! Read on for the detailed review process.

Policy Requirements

The first area to review involves some policy requirements, rather than code quality. I verify these before I ever download or install the Theme, by reviewing the appropriate template files using the "SVN" link in the Trac ticket.

License

  • Review the header tags in style.css. Ensure that the License: and LicenseURI: header tags exist, that the indicated license is GPL-compatible, and that the indicated URI is a valid, full-text license.
  • Review the Theme files for bundled fonts or icon sets; if included, verify that the licenses for these bundled resources are GPL-compatible, and explicitly declared by the Theme, either in style.css, readme.txt, or somewhere similarly appropriate.
  • Review other template files, such as footer.php, to ensure that there are no inline comments regarding use restriction, such as prohibiting changing the footer, or requiring a credit link, etc.

Credit Links

  • Review the header tags in style.css. Ensure that Theme URI and Author URI, if used, are valid and appropriate.
  • Review the indicated URLs. Ensure that the sites are appropriate, that they do not sell or promote non-GPL WordPress Themes, and that any license or terms and conditions information does not include non-GPL-compatible content.
  • Review footer.php, sidebar.php, etc. and ensure that the Theme does not include any other inappropriate hard-coded links (sponsor links, SEO links, spam links, etc.). Ensure that all public-facing copyright notices indicate the site name, rather than the Theme name or developer.
  • Review functions.php and ensure that the Theme does not hook in any other inappropriate links or copyright notices.

Theme Name

  • Verify that the Theme name meets naming convention requirements.

Screenshot

  • Verify that screenshot.png is sized appropriately.

Theme Check

Next, install the Theme, and before activating, run Theme Check. Note any Warning or Required notices in the Trac ticket. (Note: for any Theme uploaded since late March 2011, Theme  Check should return no such notices, as they should all be caught by the Theme uploader script. If you notice any, please email the Theme-Reviewers mail list, and indicate what notices you observed.)

You are welcome to note Recommended or Info notices in the Trac ticket, but I generally don't. These notices are more for your reference, as you continue to review the Theme.

For Recommended notices regarding Theme functionality (Post Formats, Post Thumbnails, Navigation Menus, etc.), verify that the Theme does not use a custom implementation. If the Theme does use a custom implementation, note in the Trac ticket that the Theme is required to support the core implementation of the functionality.

Step through each Info notice, and review the Theme template files to determine if the notice is applicable. For example: verify appropriate use of include() vs. get_template_part(); verify appropriateness of hard-coded links; etc.

Code Quality

Now, the review enters the bulk of the requirements in the Theme Review guidelines.

header.php

Review header.php, and verify:

  • All appropriate document head guidelines (valid doctype declaration, properly formed tags (html, head, meta, etc.)
  • Feed links are not hard-coded into the document head
  • No CSS files other than style.css are hard-coded into the document head. (These files must either be enqueued and hooked in appropriately, or added using IE conditional tags.)
  • No scripts or script file links are hard-coded into the document head. (These must be enqueued and hooked in appropriately.)
  • No unnecessary meta tags (WordPress generator tag, SEO meta tags, copyright meta tags, etc.) are hard-coded into the document head.
  • The wp_head() template tag is placed immediately before the closing HTML head tag.

footer.php

Review footer.php, and verify:

  • No scripts or script file links are hard-coded into the footer. (These must be enqueued and hooked in appropriately.)
  • The wp_footer() template tag is placed immediately before the closing HTML body tag.

functions.php

Review functions.php, and verify:

  • All custom Theme functions, constants, classes, global variables, and options are prefixed with theme-slug (or an appropriate variant).
  • Theme Settings are implemented properly:
    • Theme options are added to the database via a single options array, rather than separately
    • Settings page added via add_theme_page() rather than add_menu_page()
    • Settings page capability is edit_theme_options, rather than admin, manage_options, edit_themes, etc.
    • All untrusted data are sanitized/validated upon input into the database, and escaped upon output
    • If the Theme is not using the Settings API:
      • Nonce-checking is performed explicitly
      • $_POST, $_GET, and $_REQUEST data are sanitized/validated upon input into the database, and escaped upon output
  • No core WordPress functions introduced prior to one previous, major WordPress release are wrapped in function_exists() conditionals.

style.css

Verify that all tags listed in the Tags: header tag are appropriate, and that all claimed functionality is included in the Theme.

Theme Template Files

Note which hierarchical template files are used (e.g. single.php, page.php, archive.php, date.php, author.php, category.php, tag.php, taxonomy.php, 404.php, home.php, front-page.php, and search.php), so that you will know which of these to verify when performing the Theme Unit Tests, later.

Review Theme template file names, and ensure that any hierarchical template file name variants (e.g. page-foo.php, category-foo.php, etc.) are included in the Theme documentation, with an explanation for their use.

File Includes

Review Theme template files, make note of any uses of include() or require() and ensure that such use is valid. Verify that:

  • All headers are included via get_header()
  • All footers are included via get_footer()
  • All comments templates are included via comments_template()
  • All sidebars are included using get_sidebar()
  • All template part files are included via get_template_part()
  • Any search form markup is included via get_search_form()
  • Any login form markup is included via wp_login_form()

Theme Unit Tests

Now the review moves on primarily to the requirements in the Theme Unit Tests.

Theme Activation Tests

  • Activate the Theme, and open the site in your browser. Verfiy that Debogger returns no PHP errors, warnings, or notices for index.php.
  • Via Dashboard -> Tools -> Deprecated Calls, verify that Log Deprecated Notices does not return any deprecated function calls or other notices.
    • Note: check the Deprecated Calls log one final time when you complete the review, after you switch back to your default Theme. Sometimes, some notices get thrown when the Theme is deactivated.
  • If the Theme includes a Custom Theme Settings:
    • Load the Settings page, and verify that Debogger returns no PHP errors, warnings, or notices.
    • Save current/default settings, and again verify that Debogger returns no PHP errors, warnings, or notices.
    • Test each Theme Option in turn, and verify that each option can be updated, that the setting is applied properly in the Theme, and that Debogger returns no PHP errors, warnings, or notices.
    • If the Theme Settings page includes a "Reset Defaults" button, reset defaults, and verify that default settings are applied, and that Debogger returns no PHP errors, warnings, or notices.
  • If the Theme supports Custom Image Headers, select a custom header, and verify that it is applied/displayed properly
  • If the Theme supports Custom Backgrounds, select a custom background image or color, and verify that it is applied/displayed properly
  • If the Theme supports Custom Nav Menus:
    • Verify that the default/fallback output renders properly
    • Apply the Short Menu to each theme_location, and verify that the menu renders properly
    • Apply the Long Menu to each theme_location, and verify that the menu renders properly, without breaking the site/menu layout
    • If the Theme has a limitation on number of list items in any theme_location, ensure that such limitation is documented in readme.txt
  • If the Theme adds Custom Widgets, verify that all custom Widgets function as intended, generate no PHP errors, are secure, etc.
  • If the Theme supports Custom Editor Style, verify that the Post editor content renders reasonably similar to the site output
  • If the Theme requires any custom settings or setup, ensure that setup instructions are documented in readme.txt

General Tests

Run the W3C HTML/CSS validator on the Readability Test post (to ensure minimal WordPress-caused validation errors), and note any egregious validation errors, such as:

  • Malformed or improperly closed tags, that cause the page layout to break
  • An exceptional number of HTML or CSS errors

View the site index page, and verify:

  • The screenshot, as seen in the Trac ticket, is a "reasonable facsimile" of the default view of the site
  • The Site Title and Tagline, if displayed, do not break the page layout

Template Page Tests

  • For all included template-hierarchy index pages (index.php, home.php, archive.php, date.php, author.php, category.php, tag.php), verify requirements, as specified.
  • If the Theme includes either a front-page.php or a home.php template file, go to Dashboard -> Settings -> Reading, and set the Front Page to display a Static Page (use any existing Page), and set the Blog Posts index to another Static Page (e.g. "Lorem Ipsum"). Verify requirements, as specified.
  • Trigger the Theme's 404.php page by navigating to a non-existent Page slug (e.g. www.domain.tld/sit-amet). Verify requirements, as specified.
  • Trigger the Theme's search.php page by performing a search (e.g. "lorem ipsum"). Verify requirements, as specified.

Blog Posts Index Page Tests

Return to the Blog Posts index page. Step through the Blog Posts Index Page tests.

Individual Unit Tests

Step through each Blog Post and Static Page, verifying all requirements, as specified. Each unit test should be fairly self-explanatory.

Finishing the Review

At this point, the review is thorough and complete. Summarize your comments in the Trac ticket and close the ticket with the appropriate resolution (or propose a suggested resolution, if you do not have appropriate rights to close the ticket).

Theme Review Weekly Stats: 21 Nov 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done (improved/disimproved/maintained) over the week ending 20 Nov 2010:

Compared with last week:

  • The # of tickets closed decreased. (76 vs 124)
  • The review queue increased. (56 vs 52)
  • The oldest open ticket age (days) maintained. (14 vs 14)
  • The average time to close tickets (days) increased. (5.9 vs 3.8)
  • The average age of open tickets (days) decreased. (4.0 vs 5.7)
  • The % of tickets closed as approved increased. (27% vs 21%)
  • The # of reviewers decreased. (4 vs 6)

This Week (above/below/at goal):

Closed Tickets:

  • # Tickets Opened: 74
  • # Tickets Closed: 76
    • # Approved: 14
    • # Not-Approved: 38
    • # Newer-Version-Uploaded: 24
  • % Closed Tickets Approved: 27%
  • Average Days to Close: 5.9

Open Tickets:

  • # Tickets Open: 56
  • Oldest: 14 days
  • Average Age: 4.0 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 34
  2. [email protected]: 20
  3. cais: 8
  4. beckyTechy: 1

Historical trend charts have been updated.

Theme Review Weekly Stats: 11 Nov 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done (improved/disimproved/maintained) over the week ending 06 Nov 2010:

Compared with last week:

  • The # of tickets closed increased. (124 vs 100)
  • The review queue decreased. (52 vs 72)
  • The oldest open ticket age (days) increased. (14 vs 11)
  • The average time to close tickets (days) increased. (5.7 vs 2.6)
  • The average age of open tickets (days) decreased. (3.8 vs 4.7)
  • The % of tickets closed as approved decreased. (21% vs 24%)
  • The # of reviewers maintained. (6 vs 6)

This Week (above/below/at goal):

Closed Tickets:

  • # Tickets Opened: 100
  • # Tickets Closed: 124
    • # Approved: 20
    • # Not-Approved: 74
    • # Newer-Version-Uploaded: 30
  • % Closed Tickets Approved: 21%
  • Average Days to Close: 5.7

Open Tickets:

  • # Tickets Open: 52
  • Oldest: 14 days
  • Average Age: 3.8 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 55
  2. cais: 30
  3. [email protected]: 13
  4. Fingli: 10
  5. pross: 1
  6. beckyTechy: 1

Historical trend charts have been updated.

Interviewed by WPTavern

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

I was interviewed by Jeff Chandler for Episode 106 of the WordPress Weekly podcast. In the interview, we discussed the WordPress Theme Review Team, the Theme Review Guidelines for Themes submitted for inclusion in the WordPress Theme Repository. Have a listen (click to play inline or download):

WordPress Weekly Episode 106: The Team Review Team

Theme Review Weekly Stats: 07 Nov 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done (improved/disimproved/maintained) over the week ending 06 Nov 2010:

Compared with last week:

  • The # of tickets closed decreased. (90 vs 92)
  • The review queue increased. (72 vs 44)
  • The oldest open ticket age (days) decreased. (11 vs 12)
  • The average time to close tickets (days) increased. (2.6 vs 1.5)
  • The average age of open tickets (days) increased. (4.7 vs 3.0)
  • The % of tickets closed as approved decreased. (24% vs 25%)
  • The # of reviewers maintained. (6 vs 6)

This Week (above/below/at goal):

Closed Tickets:

  • # Tickets Opened: 131
  • # Tickets Closed: 90
    • # Approved: 18
    • # Not-Approved: 57
    • # Newer-Version-Uploaded: 15
  • % Closed Tickets Approved: 24%
  • Average Days to Close: 2.6

Open Tickets:

  • # Tickets Open: 72
  • Oldest: 11 days
  • Average Age: 4.7 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 41
  2. cais: 28
  3. [email protected]: 8
  4. pross: 4
  5. tinkerpriest: 2
  6. greenshady: 1

Historical trend charts have been updated.

Theme Review Weekly Stats: 31 Oct 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done (improved/disimproved/maintained) over the week ending 30 Oct 2010:

Compared with last week:

  • The # of tickets closed decreased. (92 vs 130)
  • The review queue increased. (44 vs 34)
  • The oldest open ticket age (days) increased. (12 vs 6)
  • The average time to close tickets (days) decreased. (1.5 vs 2.4)
  • The average age of open tickets (days) increased. (3.0 vs 1.6)
  • The % of tickets closed as approved decreased. (25% vs 32%)
  • The # of reviewers decreased. (6 vs 11)

This Week (above/below/at goal):

Closed Tickets:

  • # Tickets Opened: 108
  • # Tickets Closed: 92
    • # Approved: 18
    • # Not-Approved: 55
    • # Newer-Version-Uploaded: 19
  • % Closed Tickets Approved: 25%
  • Average Days to Close: 1.5

Open Tickets:

  • # Tickets Open: 44
  • Oldest: 12 days
  • Average Age: 3.0 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 37
  2. [email protected]: 17
  3. cais: 14
  4. Fingli: 9
  5. tinkerpriest: 3
  6. pross: 2

Historical trend charts have been updated.

Theme Review Weekly Stats: 24 Oct 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done (improved/disimproved/maintained) over the week ending 23 Oct 2010:

Compared with last week:

  • The # of tickets closed decreased. (130 vs 176)
  • The review queue increased. (34 vs 32)
  • The oldest open ticket age (days) decreased. (6 vs 9)
  • The average time to close tickets (days) decreased. (2.4 vs 2.6)
  • The average age of open tickets (days) decreased. (1.6 vs 6.8)
  • The % of tickets closed as approved increased. (32% vs 22%)
  • The # of reviewers increased. (11 vs 10)

This Week (above/below/at goal):

Closed Tickets:

  • # Tickets Opened: 127
  • # Tickets Closed: 130
    • # Approved: 35
    • # Not-Approved: 76
    • # Newer-Version-Uploaded: 19
  • % Closed Tickets Approved: 32%
  • Average Days to Close: 2.4 days

Open Tickets:

  • # Tickets Open: 34
  • Oldest: 6 days
  • Average Age: 1.6 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 80
  2. cais: 10
  3. [email protected]: 7
  4. emhr: 6
  5. Fingli: 5
  6. tinkerpriest: 3
  7. jeremyclark13: 2
  8. pross: 1
  9. greenshady: 1
  10. croakingtoad: 1
  11. nacin: 1

Historical trend charts have been updated.

In Defense of the WordPress Theme Review Guidelines

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

The most contentious matter is, of course, the Theme Review Guidelines. While some in the community have embraced the Theme Review team's efforts and principle regarding improving the overall standard of quality of Repository-hosted Themes, others have found the Theme Review Guidelines to be "stifling", a "barrier to entry" for new Theme developers, and even "clueless".

So what's really going on? Let's take a look at this most contentious subject: what constitutes quality with respect to WordPress Themes.

Commercial vs. Free Themes

At this point in the development of WordPress and the developer and user community that surrounds the project, free Themes - and especially those hosted in the WordPress Theme Repository - have developed a reputation for being inferior to their Commercial counterparts. As best as I can discern, the advantages that Commercial Themes provide can be summarized as follows:

  • Quality
  • Documentation
  • Support
  • Advanced Features

While much of this list is outside the control of the Theme Review team, the one that the team can most directly (and immediately) impact is, in fact, Theme quality.

Why Quality?

Especially now that Themes can be installed and updated from the Repository, all from within the WordPress admin interface, the Repository is essentially an extension of WordPress itself. Thus, the quality of Themes in the Repository reflects directly upon end users' perception of the quality of WordPress itself.

To be fair, Commercial Themes are not immune to criticism - to which the recent discussion regarding ThemeForest can attest. As Jeff Chandler points out, the criticism against ThemeForest parallels the criticism against the WordPress Theme Repository. Consider the following quotes, as applied to the Theme Repository:

Comment:

I really think the issue is about quality and support. Although there are many very qualified authors on [the repository] putting out high quality, well supported themes, there is also a lot of junk.

[...A] lot of them are great looking. I’ve [downloaded] several for just that reason. But of the 10 of so themes I’ve [downloaded] from [the repository] over the past year, only 1 is actually in use. The rest of them, once I started working with them, ended up being too limiting in one way or another, had serious coding problems, conflicts with plugins or other, major customization roadblocks and very little to no support from the author.

I’m a fairly geeky person who can muddle through figuring out how to customize themes myself, most of the time. But I’m not a coder, so there’s a limit to what I can do. I think there are a lot of people like me. Woo and StudioPress and other premium theme authors provide support for their themes that allow us middle of the road users to get where we want to go. [The repository] mostly doesn’t.

Maybe...the support will improve. But I’m much more leery about [downloading] a theme from [the repository] than I used to be. [The repository] is cheap, but you get what you pay for. I spent more money for a theme from StudioPress and guess what? It worked right out of the gate, I could customize it easily, and any questions I had were promptly answered by the support team.

Comment:

The only thing that is letting [the repository] down is that there’s no quality control. Maybe a quality control system should be put in place or some sort of star rating system, for example one person with a fair bit of theme creation knowledge and who knows how to create a theme properly could examine the themes and give them a rating.

Comment:

[M]ost of the themes there are...crap... poorly coded themes by wannabe themers.

Comment:

This is what I’ve seen with [the repository], and while not a sweeping generalization, it’s too common to be rare:

* Lack of updates. Some of the really successful theme authors update their themes to add features, fix bugs or address new WordPress functions. A whole lot more don’t. The problem with this is that the theme looks great. The demo works. Joe Consumer might not realize that because it uses old functions or does things in a certain way, it’s not going to work with newer WordPress features or some plugins.

* JavaScript Soup. Now, this isn’t a [repository] problem, this is a THEME problem. ...I see this all the time. Hell, my own website...needs to have its JS significantly refactored, I’m not immune — but my stuff also doesn’t break anything else.

* Plugin Incompatibilities – This is usually directly related with problem 1 — lack of updates or fixes.

The big advantage that places like Woo Themes and Studiopress and Thesis have over places like [the repository] is that there is a central point of support contact. If something breaks on a Woo theme, I know they’ll release an update. Their business depends on that happening. If there is a conflict with a popular plugin, they are going to figure out what is going on.

[The repository] isn’t the same way.

Comment:

I won’t personally ever [download] a theme from [the repository] because I’m a theme developer myself. But, I do deal with end users that have gotten burned on more than one occasion by themes from [the repository]. Holding your theme authors to certain levels of coding standards and best practices would be a good start.

Comment:

One thing I agree with is that the quality of themes should be stepped up...

Every one of these comments was describing ThemeForest - but every single one can bbe equally applied to the WordPress Theme Repository. Look hard enough, and you can find identical sentiment expressed about the Repository.

What is Quality?

The difficult question is: what is quality? More importantly to the current discussion: how can the Theme Review team define quality in an objective manner that can be interpreted the same by both Theme developers and Theme reviewers? For example:

ThemeGrade Test Criteria:

  • W3 HTML/CSS Compliance
  • Browser Compatibility (Firefox/Chrome/IE7)
  • Theme Support
  • Image Alignment
  • Clearing Floats
  • Content Width Auto-Adjustment
  • Threaded Comment Support
  • Hierarchical Page/Category Sidebar Lists
  • Sidebar Content Width Auto-Adjustment
  • Special Effects (Features/Functionality)

Another Site's Criteria:

  • Design
  • Ease of Use
  • SEO
  • Features
  • Support
  • Price

Here are some of ThemeForest's common "rejection factors" for their hosted WordPress Themes:

  • Documentation
  • Typography (Aesthetics, Line-Height, Text Alignment)
  • HTML/CSS Validation
  • List Markup
  • Browser Compatibility
  • License

So which of these is correct, or best (or at least closest)? From what I can gather thus far, the consensus opinion regarding Theme quality can be distilled down to this response from Leland, of ThemeLab:

XHTML/CSS validity, general code cleanliness, design not looking like crap, using wp_head/wp_footer hooks, basic stuff I guess

But, how to translate all that into objective Guidelines for performing Theme reviews?

Theme Review Quality Standards

The current Theme Review Guidelines can be summarized into five main areas:

  1. Licensing, Credit Links, and Security
  2. Code and Markup Standards Compliance and Best Practices
  3. WordPress Functionality and Feature Support
  4. WordPress Best Practices
  5. Theme Front-End Display

The current Guidelines are derived from these main areas of concern.

Licensing, Credit Links, and Security

The primary reasons that Themes undergo any review in the first place fall into this category. An astounding number of submitted Themes has non-GPL-compatible licenses or restrictions, SEO/spam footer links, or security exploits of one form or another.

Licensing

The Theme Repository Guidelines have, since the inception of the Repository, stated that Themes hosted in the repository must be compatible with WordPress' GPL licensing. This requirement includes all Theme template files (PHP), style (CSS), markup (HTML), scripts (JS), images, icons, fonts, and anything else bundled with the Theme.

Some Themes indicate that they are licensed under GPL, but knowingly include restrictive notices such as a prohibition against modifying the Theme footer, removing credit links, or modifying/redistributing the Theme. Some Themes unknowingly bundle icons (such as FamFamFam's great icons) or fonts that do not have GPL-compatible licenses.

Whichever the case, one of the most important safeguards provided by the Theme Review Guidelines (and Review) is to ensure that the user freedoms protected by the GPL are not restricted by Themes downloaded from the official Theme Repository.

Credit Links

The visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Link Spammers ever since the first Theme was submitted. Users do not want their Themes to be riddled with Link Spam, nor do they want their Themes to be used to advertise for all manner of legitimate and illegitimate websites.

For this reason, the Theme Review Team has established fairly strict, intentionally rigid requirements for public-facing links in Themes, and for the credit-link URLs Theme URI and Author URI, which are displayed by the Repository. In order to ensure fairness to all Theme Developers, the established requirements are as objective as possible - intentionally leaving little room for subjectivity.

Security

Likewise with Link Spammers, the visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Black-Hat types looking to propagate security exploits. The nature of Themes (and Plugins, for that matter) cause them to be an attractive attack vector, since a Theme, once activated, has free reign over every part of WordPress and its database. Additionally, poor and/or obsolete coding can create exploitable security vulnerabilities.

For this reason, Themes submitted for inclusion in the Repository must be reviewed for intentional exploits (worms, etc.) and unintentional vulnerabilities.

Code and Markup Standards Compliance and Best Practices

While differences of opinion exist regarding the importance of the correlation, or the extent of code validity/cleanliness required, there does appear to be some consensus within the community regarding the correlation between clean, valid code and Theme quality.  Thus, the Theme Review Guidelines include requirements regarding HTML/CSS validation, PHP errors, JS errors, and the use of deprecated WordPress functions.

HTML/CSS Validation

The ubiquitousness of valid markup (HTML/CSS) has been well-established for some time now. Most developers are not unfamiliar with using the W3 HTML and CSS validator tools to ensure their code conforms to current standards. (And W3 have made this task even easier, with a unified validator: Unicorn.) So, this requirement hasn't been viewed as terribly controversial.

Note that the Theme Review Guidelines do allow for legitimate exceptions. For example, both WordPress core and the Theme Unit Test XML data output markup that does not pass the W3 validator. Some of these errors can be eliminated for testing purposes using the FixPress Plugin. Nevertheless, these types of validation errors are ignored during the Theme review. Also, browser-specific CSS properties for draft CSS3 specs, such as border-radius and box-shadow (e.g. moz-border-radius or webkit-box-shadow) are likewise ignored during the Theme review.

PHP/JS Errors

By far, the most controversial Theme Review requirement is the prohibition against PHP errors, warnings, and notices - particularly with respect to warnings and notices. However, in my experience reviewing Themes, the vast majority of these warnings and notices are for undefined indices (not putting function arguments in quotes), undefined variables, and undefined offsets - all of which are quite easily fixable. Also, while such warnings and notices do not impact the rendering of the site for visitors, they can be incredibly noisy in the site's error logs - which can make searching the error logs in order to find legitimate errors rather difficult.

Finding such PHP errors is as simple as using the Debogger Plugin, and viewing output generated for both the Theme's front-end and admin options pages.

Deprecated WordPress Functions

Probably the most important code-related standard is the prohibition against deprecated functions. Whereas both modern browsers are well-suited to overcoming markup validation and PHP parsers handle errors, warnings, and notices with little impact to the end-user, the use of deprecated WordPress functions represents true obsolescence - and, therefore, both functional issues as well as potential security concerns - within Themes.

Fortunately, deprecated WordPress functions are almost always simple to replace. In fact, WordPress core even outputs the replacement function to use. Finding calls to deprecated WordPress functions is as simple as using the Log Deprecated Notices Plugin (hint: it works for Plugins, too!), and viewing its output in the admin UI, under Tools -> Deprecated Calls.

WordPress Functionality and Feature Support

Since the official WordPress Theme Repository is, essentially, a very public showcase of WordPress Themes, it is important that repository-hosted Themes support core WordPress functionality and features. However, this need must be balanced against the intended use cases for each Theme. The Theme Review Guidelines attempt to maintain this balance by indicating which major core features are Required, Recommended, or Optional.

Required Features

Required features must be included in repository-hosted Themes. Themes will not be accepted into the repository unless they support these features and implement them properly. Currently, the list of Required features is (intentionally) very limited:

  • Automatic Feed Links
  • Widgets
  • Comments

Recommended Features

Recommended features are not required to be included in repository-hosted Themes, but are strongly recommended. Themes will be accepted into the repository with or without support for these features, but if these features are supported, they are required to be implemented properly:

  • Navigation Menus
  • Post Thumbnails
  • Custom Header Images
  • Custom Background
  • Visual Editor CSS

Optional Features

All other features are considered optional, and are not listed in the Guidelines. However, as with the Required and Recommended features, any Optional features that a Theme supports are required to be implemented properly.

WordPress Best Practices

The official WordPress Theme Repository is intended to be used not only by WordPress end users installing Themes for their own use, but is also intended to be used by nascent WordPress developers who want to learn how to develop WordPress Themes - whether for their own use, or for distribution. For this reason (and due to the public visibility and widespread use of repository-hosted Themes), it is important that repository-hosted Themes adhere to some WordPress standards and best practices.

These standards and best practices include support for the most recent WordPress version; the use of core functions, hooks, template tags, and styles; the use of standard Theme template files; and Theme documentation.

Support For Most Recent WordPress Version

Repository-hosted Themes are required to support the version of WordPress current at the time the Theme is submitted for inclusion in the repository. This support includes incorporation of new features and functionality, as well as replacement of newly deprecated functions. In order to give Theme Developers sufficient time to incorporate new features and other changes necessitated by new versions of WordPress, this requirement takes effect one month after each WordPress version release.

Since each new WordPress version release represents potentially major changes to Theme Review Guidelines, the Guidelines will be updated with each WordPress major version release, to include any major revisions to the requirements. A Version-Specific Changes section will be added to the Theme Review Codex page, to highlight any major revisions to the Guidelines.

Use of Core Functions, Hooks, Template Tags, and Styles

In order to facilitate their use as learning tools for new developers, as well as to ensure that they work as well as possible and support as many Plugins as possible, repository-hosted Themes are required to support core WordPress functions, hooks, and template tags. This requirement encompasses both the use of core functionality rather than customized functions, as well as proper implementation of such functionality.

This requirement ensures a consistent experience from Theme to Theme for end users who choose to customize their Themes, as well as ensures that the most common hooks will exist for Plugins that depend on them.

Use of Standard Theme Template Files

All Themes are required to include certain files (index.php, style.css) in order to be considered a "valid" Theme by WordPress when installed. Repository-hosted Themes are required to include screenshot.png in order to display a preview screenshot when displayed in the repository. Repository-hosted Themes are also required to support comments and must implement this feature properly; therefore, they are required to include comments.php. All other Theme template files, if included, must be called by the Theme using the proper template tag.

Theme Documentation

Themes are encouraged to be a minimalist or as complex as the Theme Developer wishes to make them. However, if customizations, configuration options, custom template files, etc. are included (or if the Theme is designed or optimized to work with specific Plugins), the Theme is required to include user documentation explaining how to use all of the Theme's various customizations and options.

Theme Front-End Display

Back-end (e.g. code) reviews do help ensure a certain quality level, but what is most important is the front-end experience. Repository-hosted Themes are expected to provide a certain degree of user-experience consistency for visitors to the sites using them. For this reason, the Theme Review process includes the Theme Unit Tests: Themes are installed on a test site using a standard XML data set, in order to test navigation, layout, readability, HTML tag and CSS rendering, image handling, and various usability criteria.

Arguments and Complaints Against the Theme Review Guidelines

Having explained the reasoning behind the current Theme Review Guidelines, let's examine some of the arguments and complaints expressed regarding the Guidelines. Some of those complaints include:

  • The Guidelines represent a "barrier to entry" for new WordPress Theme developers
  • The Guidelines are too extensive/detailed/specific/strict
  • The Guidelines stifle design creativity/innovation
  • The Guidelines change too frequently

The Guidelines Represent a "Barrier to Entry" for New WordPress Theme Developers

Some have argued that the current Guidelines make WordPress Theme development more difficult for new Developers.

To the contrary, I would argue that, due to the extensive Codex function/tag cross-referencing, the Guidelines actually provide an incredibly useful and educational tool for new Theme Developers. Not only can new Developers learn the more important WordPress functions, template tags, and hooks, they also have direct links to the Codex entries, so that they can learn how to implement them.

The only way that the Guidelines might represent a "barrier to entry" would be if they are poorly written, and difficult to understand. Thus, it is certainly incumbent upon the volunteers who comprise the Theme Review Team to ensure that the Guidelines are clear, concise, and understandable. To that end, we will work continually to ensure that the Guidelines meet this standard.

The Guidelines are Too Extensive/Detailed/Specific/Strict

Some argue that the Guidelines are too strict, and too imposing, with "Required" this and "Required" that listed everywhere.

I believe this complaint represents a misunderstanding of how the Guidelines are written. In order to ensure objective, fair reviews, the Theme Review Team decided that the Guidelines would be written per RFC 2119. This way, the criticality of each guideline would be clearly understood, as required, recommended, or optional. The intent was not to create the perception of being imposing, but rather being as consistent as possible in describing each guideline's criticality. We intentionally decided to limit criticality adjectives to required, recommended, and optional, so as not to introduce potential confusion from similar, synonymous adjectives.

Further, the extent and strictness of the Guidelines hasn't really changed; rather, they have just been stated more explicitly, and enforced more consistently. The Theme Unit Test requirements have changed very little, and many of the rest of the requirements are actually derived from the Theme Unit Tests.

The Guidelines Stifle Design Creativity/Innovation

Some argue that the Guidelines have too many required criteria, and in so doing stifle design creativity and innovation.

I would counter that good innovation is not found in poorly written code, or in "reinventing the wheel". Further, I would counter that the Guidelines provide an excellent baseline from which Developers can focus less on basic Theme functionality, and more on design creativity and innovation.

The Guidelines Change Too Frequently

Initially, when the Theme Review Team was first getting itself organized, and getting the Guidelines ironed out, this complaint may have been considerably more valid. However, at this point, the Guidelines have had no major changes for several weeks (bearing in mind that this whole effort started less than four months ago).

Also, aside from minor clarifications, the Theme Review Team has determined that major revisions to the Guidelines will take place in synchronization with the WordPress release cycle. There are exceptions to this rule; for example, if we discover an egregious security oversight, or an incorrect requirement, we will make an immediate change.

Major changes (if any) will be introduced with each major WordPress version release, and will take effect one month after the WordPress version release. Further, these changes will be summarized in a dedicated "Version-Specific Changes" section in the Guidelines, so that Developers will know what changes may be coming.

If you have read the Guidelines recently, you will also notice that several things indicate that they are "draft" status. This is an attempt to "dry-run" changes to the Guidelines, for input/feedback. Using this approach, developers will generally always have an idea of proposed changes to the Guidelines.

Room For Improvement

Is there room for improvement? Always! And we welcome all constructive criticism and feedback.

As Theme Reviewers, we don't gain anything by making more work for ourselves, by needlessly failing Theme reviews that will only lead to the same Theme being reviewed multiple times. It is in everyone's best interest to have more Themes pass the review process. So, if a specific guideline is incorrect, inappropriate, inaccurate, or otherwise in need of revision, we want to know about it!

But at the same time, at this point, the Guidelines are fairly stable, and fairly reasonable. Generalized complaints about the Guidelines as a whole will be neither terribly helpful nor especially well-received. Try to focus your concerns to specific issues regarding specific requirements, and we would be happy to receive such feedback. In fact, the Guidelines have only been able to be refined to the point that they have currently, thanks to such feedback from Theme Developers.

In Defense of the WordPress Theme Review Process

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

So what's really going on? Let's examine a brief history of the official WordPress Theme Repository, the underyling need for a Theme review process, the goals and limitations of the current Review Team, the complaints regarding the current process, and the steps being taken to address those complaints.

Theme Repository Timeline

The Need for a Review Process

Why does the Theme Repository even need a review process? Well, for a couple of reasons: because the Theme Repository is a very attractive target for SEO-Spammers and malicious hackers, both of whom would very much like to take advantage of the incredibly large self-hosted WordPress userbase; and because the Theme Repository has a historically bad reputation regarding the level of quality of hosted Themes.

Try to find free WordPress Themes outside of the official Repository, and you will quickly discover a nefarious underworld of encoded Themes infested with, at best, spammy SEO links, and at worst, malicious security exploits. Those who peddle such wares would like nothing better than to gain access to millions of WordPress users through the official Theme Repository.

At the same time, even well-intentioned Themes hosted in the Repository are not known for being particularly high in quality. Since the Theme Repository was introduced, users have complained that available Themes are simply not coded well on the back-end and/or are not designed well on the front-end. Further, even the well-coded, well-designed Themes may not be maintained, and their developers may not provide support to end users.

Goals of Theme Review Team

With Themes being submitted for inclusion in the Theme Repository at the rate of approximately ten Themes per day (including both new Themes and updates to existing Themes), and every Theme requiring an actual human being to review it, the task eventually proved to be overwhelming for what, until a few months ago, was a one-man operation. The WordPress 3.org development cycle presented the perfect opportunity to get the WordPress community involved in the review process. Thus, in June 2010 the Theme Review team was formed.

The primary goal of the Theme Review team was - and is - to ensure that the Theme review process is as expedited as possible. Toward this end, the standing goal is an average turn-around time of one day for each Theme submission. Equally important, however, is the need to ensure that Theme reviews are performed fairly, impartially, objectively, and consistently. This objective is much more easily met when one person performs every review; however, once a team of disparate people get involved, this objective becomes exponentially more difficult. And finally - last, but not least - the goal of the Theme Review team is to help ensure - and encourage - an increased quality standard for Themes hosted in the Repository. As Matt Mullenweg himself stated:

The goal of the theme directory is not to list every theme in the world, it's to list the best ones. We want a reasonable number of themes we can point to that embody the best and brightest of WordPress development, and that users can choose without compromise.

Limitations of Theme Review Team

While having a team of volunteers to perform the task of reviewing Themes offers the advantage of having more people able to perform the work, it also presents several challenges. Previously, Theme reviews were perormed by a full-time Automattic employee, who was paid for such work, and who had full administrative rights and access to the tools and systems necessary to complete such work.

By contrast, the Theme Review team is comprised of volunteers from among the WordPress community, the vast majority of whom have full-time jobs that do not involve WordPress. These volunteers were pulled into the review process, but had to develop the skills necessary to perform quality, consistent reviews. At the time the team was formed, no formal review guidelines existed; the team spent the first month or two of its existence merely discussing the purpose and objectives of the team, and coming up with the beginnings of an objective standard by which all team members could perform Theme reviews.

Further, the team must rely on others for administrative changes to the systems and processes by which Themes are submitted, reviewed, and accepted into the Repository. To complicate matters even more, the Review Team's leader within Automattic was pulled from working with the Theme Repository and moved into other projects. Henceforth, the team has operated in a mostly self-directed manner, to the best of its ability.

And finally, as with any volunteer effort, the team is completely dependent upon the participation of its members. Thus far, the vast majority of Theme reviews, since the formation of the Theme Review team, have been performed by less than a half-dozen people. Whereas 10 reviewers each performing one review per day would keep up with the Theme submission rate, the current participation rate would require three reviews per day per reviewer, just to keep up with demand.

Complaints Regarding Theme Review Process

Thus, as one might imagine, the table has been set for considerable frustration on behalf of Theme developers and Theme reviewers alike. Theme developers are - rightfully - frustrating with the time required for their Themes to be processed through the review queue. The still-maturing submission and review systems - namely, the Theme uploader, SVN, and Trac - lead to even more frustration, as developers must revise and re-submit their Themes multiple times, and unlike with the Plugin Repository, do not have direct SVN commit access. And, of course, the somewhat-sudden increase in Theme quality standards, as represented by the Theme Review Guidelines, has been met with an expectedly mixed response.

Whereas the goal of the Theme Review team is to turn around every ticket within one day, currently the average age of open Trac tickets in the review queue is on the order of one week, with the oldest tickets generally being 2-3 weeks old. And that's just part of the frustration, that actually begins with the Theme submission and review process itself. The process goes something like this:

  • The Theme developer packages his Theme as a .zip file, and attempts to upload it to the review queue, using the Theme uploader. The uploader runs a script that checks for various errors with the Theme. If it encounters one, it halts, and returns the error.
  • The Theme developer then resolves the error condition, re-packages the Theme, and again attempts to upload it. The uploader runs the script again, and if it finds another error, it once again halts, and returns the error. If the Theme has several such errors, the Theme developer may repeat this process several times before finally uploading the Theme successfully.
  • Once the Theme is uploaded, a Trac ticket is generated, and the Theme is established in the review queue. Themes are reviewed first-in-first-out, and the review queue usually holds steady at around 60-100 open tickets at any one time.
  • Once the ticket reaches the head of the queue, a Theme reviewer assigns himself the ticket and performs the review. The reviewer will leave review notes as a comment to the Trac ticket, and then resolve the ticket as either "approved" or "not-approved".
  • If the ticket is resolved as "approved", it must be manually flagged to go "live" in the Repository. If the ticket is resolved as "not approved", the Theme developer must revise the Theme, addressing the issues noted in the review comments in the Trac ticket, and then start the process all over again.

It is no wonder, then, that a Theme developer might find himself frustrated not only with the submission and review process, and the length of time required to complete the review, but also with the review criteria that result in having to go through the process multiple times, just to get a Theme added to the Repository. (And how much more frustrating for a Theme developer trying to submit a bugfix for a Theme already in the Repository!)

What's Being Done?

The good news is, the entire process is undergoing continual improvement and refinement.

Ticket Prioritization

First, the Review Team has implemented a ticket prioritization system:

  1. Tickets for revisions to Themes previously reviewed and approved
  2. Tickets for revisions to Themes previously reviewed
  3. Tickets for new and revised Themes not yet reviewed

Thus, tickets for Themes that have already been previously reviewed and approved are prioritized ahead of all other tickets. Next, tickets for revisions to Themes that have been previously reviewed, but did not pass the review, are prioritized ahead of tickets for Themes that have not yet been reviewed.

Diff-Reviews of Revisions to Approved Themes

To help further expedite review and approval of revisions to Themes that have previously been reviewed and approved, rather than undergoing a full-blown review, these tickets now are reviewed based on the diff between the approved version and the new revision. This way, bugfixes and other improvements to already approved Themes should happen much more quickly.

Revisions Maintain Position in Queue

For all tickets, if a newer version is submitted before the existing ticket is reviewed, the new ticket takes the original ticket's place in the review queue. This way, bugfixes and other improvements do not cost the Theme its place in the queue.

Better Communication Between Reviewer and Developer

Originally, the Theme developer had no direct connection with the Trac ticket generated for an uploaded Theme. This system, while still undergoing continual improvement, is now much better. The Theme developer is now listed as the ticket "Reporter", and the Theme developer's email address is now added to the ticket "CC" list by default.

Further, the Theme Review team has established multiple avenues for communication between developers and reviewers:

Tools to Facilitate Successful Theme Review

Several tools are available for Theme Developers to help ensure that Themes will pass the review process (and in fact, these are the same tools that are used by the reviewers when conducting Theme reviews):

However, another tool is now available, thanks to the efforts of Theme Review team member Simon "Pross" Prosser. Pross' Theme-Check.

This tool will validate Themes against Licensing issues, PHP errors and notices, WordPress deprecated function calls, missing required functions, template tags, and hooks, and other similar Theme Review guidelines. If a Theme passes the Theme-Check validation, it has a much better chance of passing the Theme Review with as little trouble as possible.

Improvements to Uploader Validation Script

The Theme Review team is currently in the process of reviewing and implementing changes and improvements to the validation script that runs as part of the Theme uploader tool. (In fact, once completed, the Uploader validation script will be nearly identical to Pross' Theme-Check tool, with a few, undisclosed differences.)

In addition to updating and improving the actual criteria the validation script uses, Theme developers likely will welcome another change to the uploader script. Whereas currently the script will stop and return upon the first error it encounters (thus resulting in potentially several iterations of revise-upload-validate), soon the uploader script will run through - and report on - all errors in one pass. This way, Theme developers will know, in one pass, all the errors that are preventing upload of the Theme.

Future Changes: SVN-Commit Access

The Theme Review team is exploring the impact and implications of providing Theme developers with SVN-commit access for their Themes, similar to the SVN-commit access that Plugin developers have with the Plugin repository. This change would enable Theme developers to update their Themes directly to SVN, bypassing the Theme uploader altogether. More importantly, this change would enable Theme developers to use SVN for proper Theme maintenance and version-control.

How, when - and even if - this change ever takes place is still not entirely determined, and if it is implemented, it will be more of a long-term, rather than short-term, project. But, if and when it happens, it will be a significant improvement for Theme developers.

Effectiveness Check: Is It Working?

Whew! That's quite the list of changes. But are they effective? That's a fair question: one that Theme Developers have the right to ask of the Theme Review Team, and one that the Theme Review Team must continually ask itself.

But in order to answer that question, the Theme Review Team must have some way of defining and measuring the process. To that end, the Theme Review Team now publishes weekly Theme-review statistics, which are summarized via trend charts. These trend charts provide at-a-glance overview of ticket cycle times, tickets opened/closed, closed ticket resolution, and reviewers.

  • Cycle Time
  • Tickets Opened and Closed
  • Open Ticket Queue
  • Reviewers

The weekly statistics, which compare week-to-week performance for several metrics, are published here and announced via the Theme Reviewers mail-list.

While the Theme Review Team has made a huge push in the past several weeks, in an attempt to get the review queue in a manageable condition, the true test will be how well the team performs once the review queue reaches steady state: can the team maintain equilibrium, and turn tickets around in a reasonable time? Only time will tell.

Theme Review Weekly Stats: 17 Oct 10

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Each week, I publish various statistics to measure the Theme Review process for Themes submitted for inclusion in the Official WordPress.org Theme Repository (any inaccuracies are due to manual counting).

Here's how we've done over the week ending 16 October 2010:

Compared with last week:

  • The # of tickets closed increased. (176 vs 124)
  • The review queue decreased. (32 vs 72)
  • The oldest open ticket age (days) decreased. (9 vs 25)
  • The average time to close tickets (days) decreased. (2.6 vs 11.7)
  • The average age of open tickets (days) decreased. (6.8 vs 8.0)
  • The % of tickets closed as approved maintained. (22% vs 22%)
  • The # of reviewers maintained. (10 vs 10)

This Week:

  • # Tickets Opened: 132
  • # Tickets Closed: 176
    • # Approved: 34
    • # Not-Approved: 123
    • # Newer-Version-Uploaded: 19
  • % Closed Tickets Approved: 22%

Open Tickets:

  • # Tickets Open: 32
    • # New: 28
    • # Assigned: 4
  • Oldest: 9 days
  • Average Age: 6.8 days

This Week's Top Reviewers (# Tickets Closed):

  1. chipbennett: 108
  2. Fingli: 18
  3. jeremyclark13: 12
  4. cais: 8
  5. [email protected]: 8
  6. emhr: 6
  7. croakingtoad: 4
  8. pross: 3
  9. greenshady: 3
  10. beckyTechy: 1

Historical trend charts have been updated.