The Joel Test: what is the quality of software teams?

In 2000, Joel Spolsky published his "own, highly irresponsible, sloppy test to rate the quality of a software team". In November 2011, the respondents of the r/programming study put Joel's list of questions to the empirical test.

I am assuming that Joel compiled his list of questions based on his extensive experience with software teams, so some expectations pop up in my head. First, there should be a common theme running through these questions in so much as they relate to the quality of a software development team or organization. One way to look at this is to run the responses through factor analysis where such a common theme, if present, should manifest itself as a single  dominant factor. Second, if a common theme is indeed present, one would expect positive correlations with a more scientific "team quality factor" and other researched factors, such as “extended experience” and “skill”. 

A more detailed look at language popularity

In November 2011, Reddit r/programming was asked to indicate what programming languages they had used during the last 12 months. The preliminary analyses (here and here) indicated that JavaScript was the most frequently used programming language. A limitation, however, was that "used box" does consider how much a language is used relative to other languages that had also been used.

The figure on the right shows a plot were the frequency that a language was put into the "used box" is illustrated together with the mean rank for the same programming language. A mean rank of 3 indicate that, on average, respondents rank a programming language as the third most used. Languages that were reported used by fewer than 100 respondents were omitted from this figure.

Programmer type and self-evaluated expertise

Venn diagram of programmer types
(thanks Democratic Underground.com)

In November 2011, Reddit r/programming was asked to indicate their "level of general programming expertise". Additionally, they were asked to characterize themselves according to seven different programmer types as shown in the figure on the right.

The expertise scale used a five point scale ranging from "novice" (assigned the value 1) to "expert" (assigned the value of 5). The results of the survey are shown in the table on the right. For each table cell, three values are reported:

Programming languages: usage and popularity

In November 2011, the Reddit r/programming community was asked how they use the top 100 programming languages of the TIOBE Programming Community Index (November 2011). Clojure was also included, so a total 101 programming languages were to be placed into four distinct categories: 

• used (during last 12 months)
• plan to start using (during the next 12 months)
• have used (but do not intend to use in the near future)
• (have used, but) hate  

Agreement on programming languages popularity between TIOBE and r/programming

The r/programming study question about "used languages in the previous 12 months" can be investigated together with the 50 most popular programming languages described by TIOBE (in November 2011).

The figure below shows the degree of correspondence in terms of rank order correlation. For TIOBE's top 50 languages, the correlation was .57 whereas it was slightly higher for the top 25 languages of TIOBE (.62).

Rank order correspondence between TIOBE top 50 and the r/programming survey

Comparing the age distribution LinkedIn and r/programming respondents

On LinkedIn, programming is the 61th most popular ranked skill that people list on their profile (December 2011). A total of 747 000 people had "programming" listed as one of their skills at that time. How is the age distribution of this group compared to the age distribution in the r/programming study?

Some demographics from the survey

Demographics of survey respondents

As shown in the table on the right, the median age of the survey respondents was 26 years. Although the age distribution is quite skewed, the mean age (26.9 years) is not too far from the median age with a standard deviation of 7.7 years. The most frequently occurring response for age was 22 years (the mode).

Where do the people of r/programming live?

The respondents of the 2011 r/programming study mainly come from English speaking countries, followed by the rest of Europe and India. The heat map below is calculated using a 15-pixel size grid and the number of clicks within each grid is then represented using color that ranges from none-blue-green-yellow-red. Red indicates the highest concentration of respondents. The white dots indicate individual clicks on the map.

Location and individual clicks using a 15-pixel grid

Introduction to and data from the 2011 Reddit r/programming study

In November 2011, a survey was posted on /r/programming, which is a subreddit of Reddit. At the time, r/programming had about 345 000 subscribers. A total of 2502 individuals (n), which is 0.73% of the subscribers, clicked the link to the survey.

The raw data from the study can be downloaded from here (thanks TinyUpload!).

About AboutProgrammers.org

AboutProgrammers.org blog started out by there being some programmers who were interested in finding out more about programmers.

As in every other aspect of life, there are well-founded opinions about programmers and less well-founded opinions about programmers. We believe that everyone is entitled to their own opinions, but they are not entitled to their own facts. In order not to make decisions on grounds you did not intend or on grounds that may be somehow unfair, you might want to reflect upon your opinions, and if possible, try to find out if there are any pieces of supporting knowledge, any pieces of evidence for your opinions. If not, maybe you’d like to update your opinions.