As you may already know, dev.to has an API to fetch articles and users.
Let's give it a try with my favorite HTTP client, httpie:
http https://dev.to/api/users/1
{
"github_username": "benhalpern",
"id": 1,
"joined_at": "Dec 27, 2015",
"location": "Brooklyn, NY",
"name": "Ben Halpern",
"profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--DOU9qJSH--/c_fill,f_auto,fl_progressive,h_320,q_auto,w_320/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/1/f451a206-11c8-4e3d-8936-143d0a7e65bb.png",
"summary": "A Canadian software developer who thinks heβs funny.",
"twitter_username": "bendhalpern",
"type_of": "user",
"username": "ben",
"website_url": "http://benhalpern.com"
}
What I first noticed is that id
seems to be auto-incremented.
After some tries, I quickly found out the total number of users on dev.to: around 119000
.
It gives me the idea to scrape all the users public data and try to play with it.
Let's try with this command:
for i in {1..119000}; do http https://dev.to/api/users/$i >> users.dev.to.json && echo "," >> users.dev.to.json && echo "https://dev.to/api/users/$i"; done
After a few hours running in the background, I finally got all the users in my JSON file.
I ended up by importing the JSON file in a mongodb collection using NoSQLBooster.
There isn't many information in the users' profile, but two of them worth having a look at them: joined_at
and location
. It could be very interesting to have a diagram showing the progression of registrations through time. Maybe I'll do it some day, but for now, let's compute some statistics on users' location.
I used two queries to compute the statistics. This one helped me to get the big trends in people's location:
db.getCollection("users.dev.to").aggregate(
{"$group": {_id: "$location", count:{$sum:1}}}
).sort("-count")
Then, I use another query to get the count of people per location:
db.getCollection("users.dev.to").find({"location": /\bdublin\b/i})
The conclusions are the following:
Most of the users (90%) don't fill their location.
16.3% of the users who filled their location are from USA
.
5.8% are from India
.
4.9% are from UK
.
4% are from Japan
.
3.8% are from Germany
.
Let's have a look at the full statistics below:
# | country | region or city | #region or city | #country | % |
---|---|---|---|---|---|
1 | USA | 1850 | 16.32% | ||
Incl New York | 205 | ||||
Incl CA | 194 | ||||
Incl NY | 147 | ||||
Incl San Francisco | 137 | ||||
Incl TX | 119 | ||||
Incl Seattle | 117 | ||||
Incl Los Angeles | 107 | ||||
Incl Chicago | 101 | ||||
Incl Washington | 81 | ||||
Incl FL | 78 | ||||
Incl PA | 68 | ||||
Incl MA | 67 | ||||
Incl AZ | 38 | ||||
Incl MI | 34 | ||||
Incl OH | 30 | ||||
2 | India | 660 | 5.82% | ||
Incl Bangalore | 107 | ||||
Incl Delhi | 103 | ||||
3 | UK | 551 | 4.86% | ||
Incl London | 258 | ||||
Incl England | 35 | ||||
Incl Scotland | 33 | ||||
Incl Wales | 7 | ||||
4 | Japan | 451 | 3.98% | ||
Incl Tokyo | 242 | ||||
5 | Germany | 428 | 3.77% | ||
Incl Berlin | 128 | ||||
6 | Brazil | 301 | 2.65% | ||
7 | France | 294 | 2.59% | ||
Incl Paris | 98 | ||||
8 | Canada | 257 | 2.27% | ||
Incl Toronto | 107 | ||||
9 | Nigeria | 187 | 1.65% | ||
Incl Lagos | 97 | ||||
10 | Netherlands | 154 | 1.36% | ||
Incl Amsterdam | 62 | ||||
11 | Spain | 110 | 0.97% | ||
12 | Argentina | 109 | 0.96% | ||
13 | Italy | 102 | 0.90% | ||
14 | Indonesia | 92 | 0.81% | ||
15 | Russia | 89 | 0.78% | ||
16 | Mexico | 86 | 0.76% | ||
17 | Philippines | 85 | 0.75% | ||
18 | Poland | 81 | 0.71% | ||
19 | Ukraine | 75 | 0.66% | ||
20 | Portugal | 63 | 0.56% | ||
21 | South Africa | 57 | 0.50% | ||
22 | Pakistan | 56 | 0.49% | ||
23 | Turkey | 55 | 0.49% | ||
24 | Belgium | 53 | 0.47% | ||
25 | Egypt | 52 | 0.46% | ||
26 | Bangladesh | 50 | 0.44% | ||
27 | Colombia | 46 | 0.41% | ||
28 | Romania | 45 | 0.40% | ||
29 | Kenya | 45 | 0.40% | ||
30 | Switzerland | 45 | 0.40% | ||
31 | Austria | 43 | 0.38% | ||
32 | China | 40 | 0.35% | ||
33 | Norway | 38 | 0.34% | ||
34 | Ireland | 33 | 0.29% | ||
Total main locations (listed above) | 6683 | 58.94% | |||
Other locations (unlisted) | 4656 | 41.06% | |||
Total locations | 11339 | 100.00% | |||
<null> |
90260 | 80.80% | |||
<empty> |
10108 | 9.05% | |||
Total | 111707 | 100% |
Note: these numbers are one month old, but they give a good overview of the trends.
Thanks for reading!