Twitter Updates for 2010-10-23

Filed in Lifestream

Twitter Updates for 2010-10-22

Filed in Lifestream

Senate Projection for October 21

Filed in PoliticsTags: Democrats, Elections, Polls, Republicans, Senate

With the latest batch of Senate polling results, Neil Stevens of Likely voter has updated his Senate Projection. According to his latest model, he projects R+7, with a 3% chance of Republicans reclaiming the majority (R+10 or greater).

The problem I have with the Likely Voter projection model is two-fold: one, it factors in races that are, for all intents and purposes, already decided; and two, it seems to assume a normal distribution. If a projection were performed in which seemingly non-competitive races were removed, and the ANOVA based solely on the actually competitive races, intuition tells me that such a projection (especially with a right-leaning distribution) would have to center around R+8, if not +9. To wit:

Introducing Uncertainty Based on Non-competitive Races

Range of Possible Outcomes

First, let's set the table:

  • Seats not up for election: 63 (23R, 40D)
  • Seats up for election: 37 (18R, 19D)
  • Bondary of Possible Results: R-18 to R+19

Now, let's add in some realistic boundaries to those results.

Range of Realistic Outcomes

Lower Boundary of Realistic Outcomes

  • All 18 R-held seats are 100% likely R. (R+0)
  • The following D-held seats are 100% likely R: AR, IN, ND, PA (R+4)

It is logical to conclude that, at this point, any projection that shows anything less than R+4 is just not consistent with reality.

Upper Boundary of Realistic Outcomes

  • The following D-held seats are 100% likely D: HI, MD, NY, OR, VT (R+14)

It is logical to conclude that, at this point, any projection that shows anything more than R+14 is just not consistent with reality.

Range of Realistic Outcomes

So, at this point, the range of realistic outcomes is R+4 to R+14. Anything outside of these numbers should be considered 0% likely.

Range of Likely Outcomes

  • The following D-held seats are 90% likely R: CO, WI (R+6, lower)
  • The following D-held seats are 90% likely D: DE, NY (s) (R+12, upper)

So, at this point, the range of likely outcomes is R+6 to R+12. Anything outside of this range should be considered unlikely.

Likely Voter's current projection distribution curve has a mean of R+7, and R+5 - R+8 accounts for 77% of all outcomes (R+7 22.6%, R+6 22%, R+5/R+8 35.4%). If I assume that Likely voter's probability curve is normally distributed, then, IMHO, the mean simply must be shifted too far left. There is just no possible way that R+5 has 18% probability. I'd say, at the absolute upper end, it has 5-10% probability. Balancing the 90% Likely R pickups against the 90% Likely D holds lowers the probability even further.

So, just using back-of-mental-napkin calculations, I would say:

  • <R+5: 0% likely
  • R+5: 5% likely
  • R+6 - R+12: 90% likely
  • R+13: 5% likely
  • >R+13: 0% likely

Analysis of Actually Competitive Races

The eventual outcome will be determined entirely by the results of six races: CA, CT, IL, NV, WA, and WV.

Two or three weeks ago, I would have rated those races as follows:

  • Lean-R: The following D-held seats are 55% likely: IL, WV
  • Toss-Up: The following D-held seats are 50% likely: NV, WA
  • Lean-D: The following D-held seats are 45% likely: CA, CT

However, things have shifted a bit; I would now rate these races as follows:

  • Lean-Likely-R: The following D-held seats are 60% likely: WV
  • Lean-R: The following D-held seats are 55% likely: IL, NV
  • Toss-Up: The following D-held seats are 50% likely: CA, WA
  • Lean-Likely-D: The following D-held seats are 40% likely: CT

As you can see, aside from CT (which, to be honest, I am close to writing off as a potential Republican pick-up), all of the competitive races have shifted in the Republicans' favor. I put together a quick Monte Carlo simulation of my own, and here are the results:

Senate Projection 21Oct10

So, my model projects a mean +9 seat gain for Republicans, and a 40.1% chance that Republicans will regain control of the Senate (a gain of +10 or more seats). Results:

  • n = 10,000
  • μ = 5.2
  • σ = 1.4
  • max = +13
  • min = +5
  • +8 - +10 = 73.1%
  • 10+ = 40.1%

At first blush, these numbers appear to me to be more realistic, given the current state of the races in play (and not in play).

Evaluating the Normal Distribution Model

It seems that the Likely Voter projection model is based upon the assumption that the outcomes of competitive races will be normally distributed. I wouldn't expect a normal distribution for these outcomes, even in a "normal" election year - but especially not in a "wave" year.

Just as the outcome distribution of competitive races was biased toward the Democrats in 2006 and 2008, I fully expect the distribution to be biased toward the Republicans in 2010. This bias is due primarily to two factors that are not easily accounted for through pre-election polling: the enthusiasm gap and shifts in party affiliation.

In short, pollsters simply don't have a reliable means of estimating the breakdown of voter turnout, and it is entirely likely that they will tend to err on the side of a conservative estimation of the shift from 2006/2008 to 2010.

In a later post, I will examine some of these factors in each of the six competitive races.

Is Sestak Closing on Toomey in PA?

Filed in PoliticsTags: Democrats, Elections, Media Bias, Polls, Republicans, Senate

The latest news out of the Pennsylvania Senate race between Pat Toomey and Joe Sestak is that a few recent polls have shown Toomey's 10-point lead evaporate. First, Quinnipiac showed Toomey up only 48-46. Then PPP showed Sestak up 45-46. Now Morning Call shows the race tied at 43-43. So what gives? Is this race, that Rasmussen still lists as "Solid GOP", actually in trouble?

In a word: no.

To explain why, we'll need to look more closely at the crosstabs of these polls. But first, so you don't have to take my word for it, read what Jim Geraghty has to say over at NRO's The Campaign Spot.

Quinnipiac

Now, onto the analysis of the polls. For the sake of expediency, and since their crosstabs are available in the poll report, I'll focus on Quinnipiac. Specifically, I'll look at the party-affiliation breakdown for the Senate race, and for Obama's job approval.

Poll Results

Topline

  • Toomey: 46%
  • Sestak: 48%
  • Don't Know/No Answer: 5%

Senate

D R I Total
Toomey 7% 88% 56% 48%
Sestak 89% 8% 35% 46%
DK/NA 4% 3% 9% 5%

Obama Job Approval

D R I Total
Approve 85% 11% 30% 44%
Disapprove 13% 87% 64% 53%
DK/NA 2% 2% 6% 3%

Analysis

The poll doesn't indicate its party-affiliation weight values, but based on the above polls, I calculate that this weighting is as follows:

  • D: 38.0%
  • R: 34.5%
  • I: 27.5%

And there's the problem with this poll: this party-affiliation weighting bears little resemblance to reality. Jim Geraghty's piece linked above does a great job of explaining how these numbers are completely inconsistent with the electorate. But to prove the point, I'll look at some comparisons (previous-election party affiliation breakdowns taken from the Geraghty post).

Obama Job Approval

First, I'll examine jjust one change from the previous Quinnipiac poll to this one:

President Obama gets a negative 44 – 53 percent job approval rating, compared to a negative 40 – 56 percent September 22.

Note that overall, Obama's approval numbers continue to decrease, not increase. Yet somehow, this poll (miraculously) discovered a seven-point swing in Obama's favor since the previous month. That, alone, is enough to raise questions about the validity of the poll's topline. Interestingly, adjusting the party-affiliation weighting from 38.0%D / 34.5%R / 27.5%I to 32%D / 38%R / 30%I returns Obama's Approval/Disapproval numbers to 40% Approve / 56% Disapprove, and result in a 52% - 42% Toomey lead over Sestak (which is essentially right where the race has been for some time).

Democrat Best-Case Scenario: 2008

Next, I'll adjust the party affiliation weighting from the above numbers to the 2008 election numbers. In 2008, the turnout was 44% Democrat, 37% Republican, and 17% Independent. These numbers, which clearly represent not only a best-case scenario, but also an absolute pipe dream, result in a 48% - 45% Sestak lead over Toomey.

Get that? In a pipe-dream scenario, the Democrat would only be leading this race by 3%.

Democrat Second-Best-Case Scenario: 2006

Since the 2008 results are clearly out of reach, let's examine the 2006 results, in which the turnout was 43% Democrat, 38% Republican, and 19% Independent. These numbers, which represent a huge Democrat midterm election (again, something that will not be repeated in 2010), result in a 48% - 47% Sestak lead over Toomey.

So, once again, a pipe-dream scenario results in the Democrat leading this race by only 1%.

Bad News for Democrats: 1994

Since 2010 is clearly a Republican wave year, let's examine the poll results adjusted for 1994 turnout in the state, which was 39% Democrat, 41% Republican, and 20% Independent. These numbers result in a 50% - 45% Toomey lead over Sestak.

The problem for Sestak is that even the clearly skewed Quinnipiac party-affiliation weighting shows a lower Democrat turnout in 2010.

The even bigger problem for Sestak is that not only is the Democrat vote suppressed, but also the Independent vote is breaking 2-to-1 in favor of Toomey, and the Independent vote is highly motivated (by similar 2-to-1 ratios, Independents disapprove of Obama's job performance, disapprove of Obama's handling of the economy, prefer their Senator to oppose Obama's agenda, would prefer the Senate to be controlled by Republicans, and believe that Toomey rather than Sestak shares their personal values; also, 82% of Independents are dissatisfied/angry with the way government works).

Charting The Results

For comparison, here are the results of the above analyses:

D R I Toomey Sestak
Quinnipiac 38.0% 34.5% 27.5% 48% 46%
Obama 40% Approval 32% 38% 30% 52% 42%
2008 Turnout 44% 37% 17% 45% 48%
2006 Turnout 43% 38% 19% 47% 48%
1994 Turnout 39% 41% 20% 50% 45%

Pennsylvania Gubernatorial Race

Perhaps the worst news yet for the Sestak campaign is that the Pennsylvania Senate race isn't the only statewide election this year. Pennsylvania also has a gubernatorial race, and that race has exhibited a very steady, 10-point lead for the Republican candidate.

Pennsylvania voters are not likely to switch parties between Gubernatorial and Senate candidates, and vice versa. Thus, if the sudden tightening of the Senate race is real, it should translate into a similar tightening in the Gubernatorial race. Unfortunately for Sestak, no such tightening exists.

Conclusion

Much ado about nothing. Make of it what you will, but the conclusion that Sestak is leading Toomey - or that he has even closed the gap - simply doesn't withstand a reality check.

By all appearances, Republican Pat Toomey will win the Senate race by 5-10 points over Joe Sestak.

Twitter Updates for 2010-10-21

Filed in Lifestream

In Defense of the WordPress Theme Review Guidelines

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

The most contentious matter is, of course, the Theme Review Guidelines. While some in the community have embraced the Theme Review team's efforts and principle regarding improving the overall standard of quality of Repository-hosted Themes, others have found the Theme Review Guidelines to be "stifling", a "barrier to entry" for new Theme developers, and even "clueless".

So what's really going on? Let's take a look at this most contentious subject: what constitutes quality with respect to WordPress Themes.

Commercial vs. Free Themes

At this point in the development of WordPress and the developer and user community that surrounds the project, free Themes - and especially those hosted in the WordPress Theme Repository - have developed a reputation for being inferior to their Commercial counterparts. As best as I can discern, the advantages that Commercial Themes provide can be summarized as follows:

  • Quality
  • Documentation
  • Support
  • Advanced Features

While much of this list is outside the control of the Theme Review team, the one that the team can most directly (and immediately) impact is, in fact, Theme quality.

Why Quality?

Especially now that Themes can be installed and updated from the Repository, all from within the WordPress admin interface, the Repository is essentially an extension of WordPress itself. Thus, the quality of Themes in the Repository reflects directly upon end users' perception of the quality of WordPress itself.

To be fair, Commercial Themes are not immune to criticism - to which the recent discussion regarding ThemeForest can attest. As Jeff Chandler points out, the criticism against ThemeForest parallels the criticism against the WordPress Theme Repository. Consider the following quotes, as applied to the Theme Repository:

Comment:

I really think the issue is about quality and support. Although there are many very qualified authors on [the repository] putting out high quality, well supported themes, there is also a lot of junk.

[...A] lot of them are great looking. I’ve [downloaded] several for just that reason. But of the 10 of so themes I’ve [downloaded] from [the repository] over the past year, only 1 is actually in use. The rest of them, once I started working with them, ended up being too limiting in one way or another, had serious coding problems, conflicts with plugins or other, major customization roadblocks and very little to no support from the author.

I’m a fairly geeky person who can muddle through figuring out how to customize themes myself, most of the time. But I’m not a coder, so there’s a limit to what I can do. I think there are a lot of people like me. Woo and StudioPress and other premium theme authors provide support for their themes that allow us middle of the road users to get where we want to go. [The repository] mostly doesn’t.

Maybe...the support will improve. But I’m much more leery about [downloading] a theme from [the repository] than I used to be. [The repository] is cheap, but you get what you pay for. I spent more money for a theme from StudioPress and guess what? It worked right out of the gate, I could customize it easily, and any questions I had were promptly answered by the support team.

Comment:

The only thing that is letting [the repository] down is that there’s no quality control. Maybe a quality control system should be put in place or some sort of star rating system, for example one person with a fair bit of theme creation knowledge and who knows how to create a theme properly could examine the themes and give them a rating.

Comment:

[M]ost of the themes there are...crap... poorly coded themes by wannabe themers.

Comment:

This is what I’ve seen with [the repository], and while not a sweeping generalization, it’s too common to be rare:

* Lack of updates. Some of the really successful theme authors update their themes to add features, fix bugs or address new WordPress functions. A whole lot more don’t. The problem with this is that the theme looks great. The demo works. Joe Consumer might not realize that because it uses old functions or does things in a certain way, it’s not going to work with newer WordPress features or some plugins.

* JavaScript Soup. Now, this isn’t a [repository] problem, this is a THEME problem. ...I see this all the time. Hell, my own website...needs to have its JS significantly refactored, I’m not immune — but my stuff also doesn’t break anything else.

* Plugin Incompatibilities – This is usually directly related with problem 1 — lack of updates or fixes.

The big advantage that places like Woo Themes and Studiopress and Thesis have over places like [the repository] is that there is a central point of support contact. If something breaks on a Woo theme, I know they’ll release an update. Their business depends on that happening. If there is a conflict with a popular plugin, they are going to figure out what is going on.

[The repository] isn’t the same way.

Comment:

I won’t personally ever [download] a theme from [the repository] because I’m a theme developer myself. But, I do deal with end users that have gotten burned on more than one occasion by themes from [the repository]. Holding your theme authors to certain levels of coding standards and best practices would be a good start.

Comment:

One thing I agree with is that the quality of themes should be stepped up...

Every one of these comments was describing ThemeForest - but every single one can bbe equally applied to the WordPress Theme Repository. Look hard enough, and you can find identical sentiment expressed about the Repository.

What is Quality?

The difficult question is: what is quality? More importantly to the current discussion: how can the Theme Review team define quality in an objective manner that can be interpreted the same by both Theme developers and Theme reviewers? For example:

ThemeGrade Test Criteria:

  • W3 HTML/CSS Compliance
  • Browser Compatibility (Firefox/Chrome/IE7)
  • Theme Support
  • Image Alignment
  • Clearing Floats
  • Content Width Auto-Adjustment
  • Threaded Comment Support
  • Hierarchical Page/Category Sidebar Lists
  • Sidebar Content Width Auto-Adjustment
  • Special Effects (Features/Functionality)

Another Site's Criteria:

  • Design
  • Ease of Use
  • SEO
  • Features
  • Support
  • Price

Here are some of ThemeForest's common "rejection factors" for their hosted WordPress Themes:

  • Documentation
  • Typography (Aesthetics, Line-Height, Text Alignment)
  • HTML/CSS Validation
  • List Markup
  • Browser Compatibility
  • License

So which of these is correct, or best (or at least closest)? From what I can gather thus far, the consensus opinion regarding Theme quality can be distilled down to this response from Leland, of ThemeLab:

XHTML/CSS validity, general code cleanliness, design not looking like crap, using wp_head/wp_footer hooks, basic stuff I guess

But, how to translate all that into objective Guidelines for performing Theme reviews?

Theme Review Quality Standards

The current Theme Review Guidelines can be summarized into five main areas:

  1. Licensing, Credit Links, and Security
  2. Code and Markup Standards Compliance and Best Practices
  3. WordPress Functionality and Feature Support
  4. WordPress Best Practices
  5. Theme Front-End Display

The current Guidelines are derived from these main areas of concern.

Licensing, Credit Links, and Security

The primary reasons that Themes undergo any review in the first place fall into this category. An astounding number of submitted Themes has non-GPL-compatible licenses or restrictions, SEO/spam footer links, or security exploits of one form or another.

Licensing

The Theme Repository Guidelines have, since the inception of the Repository, stated that Themes hosted in the repository must be compatible with WordPress' GPL licensing. This requirement includes all Theme template files (PHP), style (CSS), markup (HTML), scripts (JS), images, icons, fonts, and anything else bundled with the Theme.

Some Themes indicate that they are licensed under GPL, but knowingly include restrictive notices such as a prohibition against modifying the Theme footer, removing credit links, or modifying/redistributing the Theme. Some Themes unknowingly bundle icons (such as FamFamFam's great icons) or fonts that do not have GPL-compatible licenses.

Whichever the case, one of the most important safeguards provided by the Theme Review Guidelines (and Review) is to ensure that the user freedoms protected by the GPL are not restricted by Themes downloaded from the official Theme Repository.

Credit Links

The visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Link Spammers ever since the first Theme was submitted. Users do not want their Themes to be riddled with Link Spam, nor do they want their Themes to be used to advertise for all manner of legitimate and illegitimate websites.

For this reason, the Theme Review Team has established fairly strict, intentionally rigid requirements for public-facing links in Themes, and for the credit-link URLs Theme URI and Author URI, which are displayed by the Repository. In order to ensure fairness to all Theme Developers, the established requirements are as objective as possible - intentionally leaving little room for subjectivity.

Security

Likewise with Link Spammers, the visibility and sheer download/usage numbers of Themes hosted by the official Theme Repository have turned the Repository into a magnet for Black-Hat types looking to propagate security exploits. The nature of Themes (and Plugins, for that matter) cause them to be an attractive attack vector, since a Theme, once activated, has free reign over every part of WordPress and its database. Additionally, poor and/or obsolete coding can create exploitable security vulnerabilities.

For this reason, Themes submitted for inclusion in the Repository must be reviewed for intentional exploits (worms, etc.) and unintentional vulnerabilities.

Code and Markup Standards Compliance and Best Practices

While differences of opinion exist regarding the importance of the correlation, or the extent of code validity/cleanliness required, there does appear to be some consensus within the community regarding the correlation between clean, valid code and Theme quality.  Thus, the Theme Review Guidelines include requirements regarding HTML/CSS validation, PHP errors, JS errors, and the use of deprecated WordPress functions.

HTML/CSS Validation

The ubiquitousness of valid markup (HTML/CSS) has been well-established for some time now. Most developers are not unfamiliar with using the W3 HTML and CSS validator tools to ensure their code conforms to current standards. (And W3 have made this task even easier, with a unified validator: Unicorn.) So, this requirement hasn't been viewed as terribly controversial.

Note that the Theme Review Guidelines do allow for legitimate exceptions. For example, both WordPress core and the Theme Unit Test XML data output markup that does not pass the W3 validator. Some of these errors can be eliminated for testing purposes using the FixPress Plugin. Nevertheless, these types of validation errors are ignored during the Theme review. Also, browser-specific CSS properties for draft CSS3 specs, such as border-radius and box-shadow (e.g. moz-border-radius or webkit-box-shadow) are likewise ignored during the Theme review.

PHP/JS Errors

By far, the most controversial Theme Review requirement is the prohibition against PHP errors, warnings, and notices - particularly with respect to warnings and notices. However, in my experience reviewing Themes, the vast majority of these warnings and notices are for undefined indices (not putting function arguments in quotes), undefined variables, and undefined offsets - all of which are quite easily fixable. Also, while such warnings and notices do not impact the rendering of the site for visitors, they can be incredibly noisy in the site's error logs - which can make searching the error logs in order to find legitimate errors rather difficult.

Finding such PHP errors is as simple as using the Debogger Plugin, and viewing output generated for both the Theme's front-end and admin options pages.

Deprecated WordPress Functions

Probably the most important code-related standard is the prohibition against deprecated functions. Whereas both modern browsers are well-suited to overcoming markup validation and PHP parsers handle errors, warnings, and notices with little impact to the end-user, the use of deprecated WordPress functions represents true obsolescence - and, therefore, both functional issues as well as potential security concerns - within Themes.

Fortunately, deprecated WordPress functions are almost always simple to replace. In fact, WordPress core even outputs the replacement function to use. Finding calls to deprecated WordPress functions is as simple as using the Log Deprecated Notices Plugin (hint: it works for Plugins, too!), and viewing its output in the admin UI, under Tools -> Deprecated Calls.

WordPress Functionality and Feature Support

Since the official WordPress Theme Repository is, essentially, a very public showcase of WordPress Themes, it is important that repository-hosted Themes support core WordPress functionality and features. However, this need must be balanced against the intended use cases for each Theme. The Theme Review Guidelines attempt to maintain this balance by indicating which major core features are Required, Recommended, or Optional.

Required Features

Required features must be included in repository-hosted Themes. Themes will not be accepted into the repository unless they support these features and implement them properly. Currently, the list of Required features is (intentionally) very limited:

  • Automatic Feed Links
  • Widgets
  • Comments

Recommended Features

Recommended features are not required to be included in repository-hosted Themes, but are strongly recommended. Themes will be accepted into the repository with or without support for these features, but if these features are supported, they are required to be implemented properly:

  • Navigation Menus
  • Post Thumbnails
  • Custom Header Images
  • Custom Background
  • Visual Editor CSS

Optional Features

All other features are considered optional, and are not listed in the Guidelines. However, as with the Required and Recommended features, any Optional features that a Theme supports are required to be implemented properly.

WordPress Best Practices

The official WordPress Theme Repository is intended to be used not only by WordPress end users installing Themes for their own use, but is also intended to be used by nascent WordPress developers who want to learn how to develop WordPress Themes - whether for their own use, or for distribution. For this reason (and due to the public visibility and widespread use of repository-hosted Themes), it is important that repository-hosted Themes adhere to some WordPress standards and best practices.

These standards and best practices include support for the most recent WordPress version; the use of core functions, hooks, template tags, and styles; the use of standard Theme template files; and Theme documentation.

Support For Most Recent WordPress Version

Repository-hosted Themes are required to support the version of WordPress current at the time the Theme is submitted for inclusion in the repository. This support includes incorporation of new features and functionality, as well as replacement of newly deprecated functions. In order to give Theme Developers sufficient time to incorporate new features and other changes necessitated by new versions of WordPress, this requirement takes effect one month after each WordPress version release.

Since each new WordPress version release represents potentially major changes to Theme Review Guidelines, the Guidelines will be updated with each WordPress major version release, to include any major revisions to the requirements. A Version-Specific Changes section will be added to the Theme Review Codex page, to highlight any major revisions to the Guidelines.

Use of Core Functions, Hooks, Template Tags, and Styles

In order to facilitate their use as learning tools for new developers, as well as to ensure that they work as well as possible and support as many Plugins as possible, repository-hosted Themes are required to support core WordPress functions, hooks, and template tags. This requirement encompasses both the use of core functionality rather than customized functions, as well as proper implementation of such functionality.

This requirement ensures a consistent experience from Theme to Theme for end users who choose to customize their Themes, as well as ensures that the most common hooks will exist for Plugins that depend on them.

Use of Standard Theme Template Files

All Themes are required to include certain files (index.php, style.css) in order to be considered a "valid" Theme by WordPress when installed. Repository-hosted Themes are required to include screenshot.png in order to display a preview screenshot when displayed in the repository. Repository-hosted Themes are also required to support comments and must implement this feature properly; therefore, they are required to include comments.php. All other Theme template files, if included, must be called by the Theme using the proper template tag.

Theme Documentation

Themes are encouraged to be a minimalist or as complex as the Theme Developer wishes to make them. However, if customizations, configuration options, custom template files, etc. are included (or if the Theme is designed or optimized to work with specific Plugins), the Theme is required to include user documentation explaining how to use all of the Theme's various customizations and options.

Theme Front-End Display

Back-end (e.g. code) reviews do help ensure a certain quality level, but what is most important is the front-end experience. Repository-hosted Themes are expected to provide a certain degree of user-experience consistency for visitors to the sites using them. For this reason, the Theme Review process includes the Theme Unit Tests: Themes are installed on a test site using a standard XML data set, in order to test navigation, layout, readability, HTML tag and CSS rendering, image handling, and various usability criteria.

Arguments and Complaints Against the Theme Review Guidelines

Having explained the reasoning behind the current Theme Review Guidelines, let's examine some of the arguments and complaints expressed regarding the Guidelines. Some of those complaints include:

  • The Guidelines represent a "barrier to entry" for new WordPress Theme developers
  • The Guidelines are too extensive/detailed/specific/strict
  • The Guidelines stifle design creativity/innovation
  • The Guidelines change too frequently

The Guidelines Represent a "Barrier to Entry" for New WordPress Theme Developers

Some have argued that the current Guidelines make WordPress Theme development more difficult for new Developers.

To the contrary, I would argue that, due to the extensive Codex function/tag cross-referencing, the Guidelines actually provide an incredibly useful and educational tool for new Theme Developers. Not only can new Developers learn the more important WordPress functions, template tags, and hooks, they also have direct links to the Codex entries, so that they can learn how to implement them.

The only way that the Guidelines might represent a "barrier to entry" would be if they are poorly written, and difficult to understand. Thus, it is certainly incumbent upon the volunteers who comprise the Theme Review Team to ensure that the Guidelines are clear, concise, and understandable. To that end, we will work continually to ensure that the Guidelines meet this standard.

The Guidelines are Too Extensive/Detailed/Specific/Strict

Some argue that the Guidelines are too strict, and too imposing, with "Required" this and "Required" that listed everywhere.

I believe this complaint represents a misunderstanding of how the Guidelines are written. In order to ensure objective, fair reviews, the Theme Review Team decided that the Guidelines would be written per RFC 2119. This way, the criticality of each guideline would be clearly understood, as required, recommended, or optional. The intent was not to create the perception of being imposing, but rather being as consistent as possible in describing each guideline's criticality. We intentionally decided to limit criticality adjectives to required, recommended, and optional, so as not to introduce potential confusion from similar, synonymous adjectives.

Further, the extent and strictness of the Guidelines hasn't really changed; rather, they have just been stated more explicitly, and enforced more consistently. The Theme Unit Test requirements have changed very little, and many of the rest of the requirements are actually derived from the Theme Unit Tests.

The Guidelines Stifle Design Creativity/Innovation

Some argue that the Guidelines have too many required criteria, and in so doing stifle design creativity and innovation.

I would counter that good innovation is not found in poorly written code, or in "reinventing the wheel". Further, I would counter that the Guidelines provide an excellent baseline from which Developers can focus less on basic Theme functionality, and more on design creativity and innovation.

The Guidelines Change Too Frequently

Initially, when the Theme Review Team was first getting itself organized, and getting the Guidelines ironed out, this complaint may have been considerably more valid. However, at this point, the Guidelines have had no major changes for several weeks (bearing in mind that this whole effort started less than four months ago).

Also, aside from minor clarifications, the Theme Review Team has determined that major revisions to the Guidelines will take place in synchronization with the WordPress release cycle. There are exceptions to this rule; for example, if we discover an egregious security oversight, or an incorrect requirement, we will make an immediate change.

Major changes (if any) will be introduced with each major WordPress version release, and will take effect one month after the WordPress version release. Further, these changes will be summarized in a dedicated "Version-Specific Changes" section in the Guidelines, so that Developers will know what changes may be coming.

If you have read the Guidelines recently, you will also notice that several things indicate that they are "draft" status. This is an attempt to "dry-run" changes to the Guidelines, for input/feedback. Using this approach, developers will generally always have an idea of proposed changes to the Guidelines.

Room For Improvement

Is there room for improvement? Always! And we welcome all constructive criticism and feedback.

As Theme Reviewers, we don't gain anything by making more work for ourselves, by needlessly failing Theme reviews that will only lead to the same Theme being reviewed multiple times. It is in everyone's best interest to have more Themes pass the review process. So, if a specific guideline is incorrect, inappropriate, inaccurate, or otherwise in need of revision, we want to know about it!

But at the same time, at this point, the Guidelines are fairly stable, and fairly reasonable. Generalized complaints about the Guidelines as a whole will be neither terribly helpful nor especially well-received. Try to focus your concerns to specific issues regarding specific requirements, and we would be happy to receive such feedback. In fact, the Guidelines have only been able to be refined to the point that they have currently, thanks to such feedback from Theme Developers.

Twitter Updates for 2010-10-20

Filed in Lifestream

In Defense of the WordPress Theme Review Process

Filed in Web DevelopmentTags: Theme Review, Themes, WordPress

Recently, the official WordPress Theme Repository has been the subject of much discussion within the WordPress community, thanks to the formation of a Theme Review team and the implementation of a stricter set of Guidelines for acceptance of Themes into the Repository. Some have complained regarding the unreasonable length of time required to get a Theme through the review process and accepted into the Repository. Other, more vocal critics have decried the new Guidelines as overly strict, stifling innovation, creating a barrier-to-entry for new Theme developers, and actively driving away existing Theme developers.

So what's really going on? Let's examine a brief history of the official WordPress Theme Repository, the underyling need for a Theme review process, the goals and limitations of the current Review Team, the complaints regarding the current process, and the steps being taken to address those complaints.

Theme Repository Timeline

The Need for a Review Process

Why does the Theme Repository even need a review process? Well, for a couple of reasons: because the Theme Repository is a very attractive target for SEO-Spammers and malicious hackers, both of whom would very much like to take advantage of the incredibly large self-hosted WordPress userbase; and because the Theme Repository has a historically bad reputation regarding the level of quality of hosted Themes.

Try to find free WordPress Themes outside of the official Repository, and you will quickly discover a nefarious underworld of encoded Themes infested with, at best, spammy SEO links, and at worst, malicious security exploits. Those who peddle such wares would like nothing better than to gain access to millions of WordPress users through the official Theme Repository.

At the same time, even well-intentioned Themes hosted in the Repository are not known for being particularly high in quality. Since the Theme Repository was introduced, users have complained that available Themes are simply not coded well on the back-end and/or are not designed well on the front-end. Further, even the well-coded, well-designed Themes may not be maintained, and their developers may not provide support to end users.

Goals of Theme Review Team

With Themes being submitted for inclusion in the Theme Repository at the rate of approximately ten Themes per day (including both new Themes and updates to existing Themes), and every Theme requiring an actual human being to review it, the task eventually proved to be overwhelming for what, until a few months ago, was a one-man operation. The WordPress 3.org development cycle presented the perfect opportunity to get the WordPress community involved in the review process. Thus, in June 2010 the Theme Review team was formed.

The primary goal of the Theme Review team was - and is - to ensure that the Theme review process is as expedited as possible. Toward this end, the standing goal is an average turn-around time of one day for each Theme submission. Equally important, however, is the need to ensure that Theme reviews are performed fairly, impartially, objectively, and consistently. This objective is much more easily met when one person performs every review; however, once a team of disparate people get involved, this objective becomes exponentially more difficult. And finally - last, but not least - the goal of the Theme Review team is to help ensure - and encourage - an increased quality standard for Themes hosted in the Repository. As Matt Mullenweg himself stated:

The goal of the theme directory is not to list every theme in the world, it's to list the best ones. We want a reasonable number of themes we can point to that embody the best and brightest of WordPress development, and that users can choose without compromise.

Limitations of Theme Review Team

While having a team of volunteers to perform the task of reviewing Themes offers the advantage of having more people able to perform the work, it also presents several challenges. Previously, Theme reviews were perormed by a full-time Automattic employee, who was paid for such work, and who had full administrative rights and access to the tools and systems necessary to complete such work.

By contrast, the Theme Review team is comprised of volunteers from among the WordPress community, the vast majority of whom have full-time jobs that do not involve WordPress. These volunteers were pulled into the review process, but had to develop the skills necessary to perform quality, consistent reviews. At the time the team was formed, no formal review guidelines existed; the team spent the first month or two of its existence merely discussing the purpose and objectives of the team, and coming up with the beginnings of an objective standard by which all team members could perform Theme reviews.

Further, the team must rely on others for administrative changes to the systems and processes by which Themes are submitted, reviewed, and accepted into the Repository. To complicate matters even more, the Review Team's leader within Automattic was pulled from working with the Theme Repository and moved into other projects. Henceforth, the team has operated in a mostly self-directed manner, to the best of its ability.

And finally, as with any volunteer effort, the team is completely dependent upon the participation of its members. Thus far, the vast majority of Theme reviews, since the formation of the Theme Review team, have been performed by less than a half-dozen people. Whereas 10 reviewers each performing one review per day would keep up with the Theme submission rate, the current participation rate would require three reviews per day per reviewer, just to keep up with demand.

Complaints Regarding Theme Review Process

Thus, as one might imagine, the table has been set for considerable frustration on behalf of Theme developers and Theme reviewers alike. Theme developers are - rightfully - frustrating with the time required for their Themes to be processed through the review queue. The still-maturing submission and review systems - namely, the Theme uploader, SVN, and Trac - lead to even more frustration, as developers must revise and re-submit their Themes multiple times, and unlike with the Plugin Repository, do not have direct SVN commit access. And, of course, the somewhat-sudden increase in Theme quality standards, as represented by the Theme Review Guidelines, has been met with an expectedly mixed response.

Whereas the goal of the Theme Review team is to turn around every ticket within one day, currently the average age of open Trac tickets in the review queue is on the order of one week, with the oldest tickets generally being 2-3 weeks old. And that's just part of the frustration, that actually begins with the Theme submission and review process itself. The process goes something like this:

  • The Theme developer packages his Theme as a .zip file, and attempts to upload it to the review queue, using the Theme uploader. The uploader runs a script that checks for various errors with the Theme. If it encounters one, it halts, and returns the error.
  • The Theme developer then resolves the error condition, re-packages the Theme, and again attempts to upload it. The uploader runs the script again, and if it finds another error, it once again halts, and returns the error. If the Theme has several such errors, the Theme developer may repeat this process several times before finally uploading the Theme successfully.
  • Once the Theme is uploaded, a Trac ticket is generated, and the Theme is established in the review queue. Themes are reviewed first-in-first-out, and the review queue usually holds steady at around 60-100 open tickets at any one time.
  • Once the ticket reaches the head of the queue, a Theme reviewer assigns himself the ticket and performs the review. The reviewer will leave review notes as a comment to the Trac ticket, and then resolve the ticket as either "approved" or "not-approved".
  • If the ticket is resolved as "approved", it must be manually flagged to go "live" in the Repository. If the ticket is resolved as "not approved", the Theme developer must revise the Theme, addressing the issues noted in the review comments in the Trac ticket, and then start the process all over again.

It is no wonder, then, that a Theme developer might find himself frustrated not only with the submission and review process, and the length of time required to complete the review, but also with the review criteria that result in having to go through the process multiple times, just to get a Theme added to the Repository. (And how much more frustrating for a Theme developer trying to submit a bugfix for a Theme already in the Repository!)

What's Being Done?

The good news is, the entire process is undergoing continual improvement and refinement.

Ticket Prioritization

First, the Review Team has implemented a ticket prioritization system:

  1. Tickets for revisions to Themes previously reviewed and approved
  2. Tickets for revisions to Themes previously reviewed
  3. Tickets for new and revised Themes not yet reviewed

Thus, tickets for Themes that have already been previously reviewed and approved are prioritized ahead of all other tickets. Next, tickets for revisions to Themes that have been previously reviewed, but did not pass the review, are prioritized ahead of tickets for Themes that have not yet been reviewed.

Diff-Reviews of Revisions to Approved Themes

To help further expedite review and approval of revisions to Themes that have previously been reviewed and approved, rather than undergoing a full-blown review, these tickets now are reviewed based on the diff between the approved version and the new revision. This way, bugfixes and other improvements to already approved Themes should happen much more quickly.

Revisions Maintain Position in Queue

For all tickets, if a newer version is submitted before the existing ticket is reviewed, the new ticket takes the original ticket's place in the review queue. This way, bugfixes and other improvements do not cost the Theme its place in the queue.

Better Communication Between Reviewer and Developer

Originally, the Theme developer had no direct connection with the Trac ticket generated for an uploaded Theme. This system, while still undergoing continual improvement, is now much better. The Theme developer is now listed as the ticket "Reporter", and the Theme developer's email address is now added to the ticket "CC" list by default.

Further, the Theme Review team has established multiple avenues for communication between developers and reviewers:

Tools to Facilitate Successful Theme Review

Several tools are available for Theme Developers to help ensure that Themes will pass the review process (and in fact, these are the same tools that are used by the reviewers when conducting Theme reviews):

However, another tool is now available, thanks to the efforts of Theme Review team member Simon "Pross" Prosser. Pross' Theme-Check.

This tool will validate Themes against Licensing issues, PHP errors and notices, WordPress deprecated function calls, missing required functions, template tags, and hooks, and other similar Theme Review guidelines. If a Theme passes the Theme-Check validation, it has a much better chance of passing the Theme Review with as little trouble as possible.

Improvements to Uploader Validation Script

The Theme Review team is currently in the process of reviewing and implementing changes and improvements to the validation script that runs as part of the Theme uploader tool. (In fact, once completed, the Uploader validation script will be nearly identical to Pross' Theme-Check tool, with a few, undisclosed differences.)

In addition to updating and improving the actual criteria the validation script uses, Theme developers likely will welcome another change to the uploader script. Whereas currently the script will stop and return upon the first error it encounters (thus resulting in potentially several iterations of revise-upload-validate), soon the uploader script will run through - and report on - all errors in one pass. This way, Theme developers will know, in one pass, all the errors that are preventing upload of the Theme.

Future Changes: SVN-Commit Access

The Theme Review team is exploring the impact and implications of providing Theme developers with SVN-commit access for their Themes, similar to the SVN-commit access that Plugin developers have with the Plugin repository. This change would enable Theme developers to update their Themes directly to SVN, bypassing the Theme uploader altogether. More importantly, this change would enable Theme developers to use SVN for proper Theme maintenance and version-control.

How, when - and even if - this change ever takes place is still not entirely determined, and if it is implemented, it will be more of a long-term, rather than short-term, project. But, if and when it happens, it will be a significant improvement for Theme developers.

Effectiveness Check: Is It Working?

Whew! That's quite the list of changes. But are they effective? That's a fair question: one that Theme Developers have the right to ask of the Theme Review Team, and one that the Theme Review Team must continually ask itself.

But in order to answer that question, the Theme Review Team must have some way of defining and measuring the process. To that end, the Theme Review Team now publishes weekly Theme-review statistics, which are summarized via trend charts. These trend charts provide at-a-glance overview of ticket cycle times, tickets opened/closed, closed ticket resolution, and reviewers.

  • Cycle Time
  • Tickets Opened and Closed
  • Open Ticket Queue
  • Reviewers

The weekly statistics, which compare week-to-week performance for several metrics, are published here and announced via the Theme Reviewers mail-list.

While the Theme Review Team has made a huge push in the past several weeks, in an attempt to get the review queue in a manageable condition, the true test will be how well the team performs once the review queue reaches steady state: can the team maintain equilibrium, and turn tickets around in a reasonable time? Only time will tell.

Twitter Updates for 2010-10-19

Filed in Lifestream

Twitter Updates for 2010-10-18

Filed in Lifestream
  • Theme Review Weekly Stats: 17 Oct 10 - http://bit.ly/9qWuHn #
  • Theme Review Weekly Stats: 17 Oct 10 - http://bit.ly/9qWuHn #WordPress #
  • @pwilson24 guess we get to see how the Rookie does, eh? Call me an optimist, but I think he'll do just fine! in reply to pwilson24 #
  • So, how many steps does a receiver have to take? (And that's about the only time I'd ever root for a Manning interception.) #
  • Nice job of containment by Wheeler on second down there. #
  • Jerraud Powers picks Chunky Soup, and then gets tackled out of bounds (with no flag). Go #Colts #
  • TOUCHDOWN!!!!!!!!!!! Manning to Frenchie!! 7-0 Colts. Go #Colts #
  • Good job, rook! MLB Pat Angerer with the blitz and sack! Go #Colts #
  • TOUCHDOWN!!!! Manning to Collie! Nice score-answering drive! #Colts lead 14-7! #
  • Aaaand AV misses a chip-shot 37-yard FG attempt. Still 14-7 #Colts #
  • Garcon just channeled his inner Randy Moss with a STUNNING one-handed circus catch. Go #Colts #
  • Vinatieri redeems himself this time, nailing a 43-yard FG at the 2-min warning. 17-7 Colts. Go #Colts #
  • The #Colts Special Teams unit is simply TERRIBLE tonight. Vinatieri just had his FG attempt blocked. #
  • And Diem's inability to block leads to a Redskins TD. #Colts lead down to 3: 17-14. #
  • @18to88 would Diem have blocked properly (read: at all) if they'd run there instead? in reply to 18to88 #
  • That was the best run of Joseph Addai's entire career! 46 yards! Go #Colts #
  • YES!!!!!!!! Buster!! Addai just pounds it in from 13 yards out on 3rd and 3! 24-14 Colts! Go #Colts #
  • @18to88 I have to admit: it feels so great to be so right about Addai! 😉 in reply to 18to88 #
  • Can we just fair-catch every punt. Moore with ANOTHER FRIGGING FUMBLE LOST. #
  • And Moore's second punt-return fumble only costs the Colts 3 points (that's 10 points for the game so far). #Colts lead 24-17. #
  • @18to88 that's 10 points for WAS off of punt fumbles. Colts should be up like 37-7, but for Moore's inability to protect the frigging ball. in reply to 18to88 #
  • @18to88 yeah, my mistake there. Meant 10 points off of fumbles, not just punt fumbles. in reply to 18to88 #
  • @18to88 and Manning only needs 21 yards for his milestone 300-yard game. in reply to 18to88 #
  • @18to88 since when are helmet-to-helmet hits EVER legal? in reply to 18to88 #
  • @mattlimbrick frigging illegal hit in reply to mattlimbrick #
  • DENIED! #Colts Ball (fair catch, please?) #
  • @18to88 Session has been on it tonight. in reply to 18to88 #
  • @18to88 not as inconsistent as Wheeler or Hayden. Powers, Mathis, Freeney, Angerer, and Session (in order) have been pretty good tonight in reply to 18to88 #
  • And there's Manning's milestone 300-yard passing game! Go #Colts #
  • Wow. Hart just moved the pile for the first down there. He's really keeping his legs churning tonight. #
  • Caldwell is utterly GUTLESS tonight. Go for the first down here. #
  • @mattlimbrick of course, Crissy Collingsworthless made sure to state, emphatically, that it was a clean, legal hit. in reply to mattlimbrick #
  • @NMRgirl not even a full yard. More like 4th and a foot. in reply to NMRgirl #
  • @18to88 and on that play, Session missed the tackle, and Wheeler cleaned it up in reply to 18to88 #
  • When four guys tackle Freeney THAT IS A FRIGGING HOLD!!!!!!!!!!!!!!!!! #
  • We FINALLY got a hold call!?!?!? Way to get after him, Freeney! #
  • Just shut your piehole, Crissy Collingsworth. That was as egregious a hold as you'll ever see, you moron. #
  • Justin Tryon gets abused on an easy TD throw. #Colts lead now 27-24. Hold onto the coming onside kick, Colts! #
  • @trubluecoltsfan the Defense has done its job. They aren't the ones who have 3 fumbles lost. in reply to trubluecoltsfan #
  • Now THAT was pass interference. Had his frigging arm wrapped around Clark. #
  • @18to88 and an arm-bar in reply to 18to88 #
  • MATHIS with a huge, MONSTER sack! #
  • ANGERER!! Breaks up the third-down pass! #
  • BALLGAME! The Redskins just turned the ball over on downs. Go #Colts #
  • @trubluecoltsfan and they just did! in reply to trubluecoltsfan #
  • @LovinBlue I thought they might actually go for it there... in reply to LovinBlue #
  • Two yards too deep on the punt... #
  • AARON FRANCISCO WITH THE INTERCEPTION!!!!!! COLTS WIN!!!!!!!!!!!!!!!!!!!!!!!!!!!! #
  • @LovinBlue total chickens counted: 43 in reply to LovinBlue #
  • @18to88 but he *wanted* it to be - he *really, REALLY* wanted it to be. Because Chrissy *hates* the Colts in reply to 18to88 #
  • @XL950 the D didn't give up 3 fumbles deep in their own territory. The D should have been playing w/ a 3-TD lead, yet never gave up the lead in reply to XL950 #