Aging the Userbase: Losing 7000 users

On January 1, 2003, the Linux Counter stood at 140.019 registered users.

Two months later, the count stood at 133.439 users - more than 6500 less.

What happened?

History: The Freeze and the Slashdotting of November 2001

In late fall of 2001, after seven years of counter operation, the counter announced that it would be deleting from the count all records that hadn't been touched for two years - or one year, if we knew that the registered email for the entry was invalid.

This story was picked up by Slashdot on October 15, 2001, and had the predictable result; between October 14 and November 1, 7458 accounts were added at the counter - 2539 of them on the day of October 16. Of course, the site was heavily overloaded - that's pretty much a synonym for "slashdotted". But we survived.

A little later, on November 1, the freeze went into effect. At the same time, we started sending out warnings to pretty much everyone registered - with the predictable thousands of bounces; 48.975 of the entries currently frozen got email in November of 2001. (This led, of course, to a new enormous pile of email being scheduled for November 2002 for those who logged in to signal that they were still around - 23.808 entries that are currently marked "OK" got email in November 2002).

The Warning and The Bug

After an user has been registered with the counter for one year, an email is sent to the email registered for that user, to remind him or her to update their entry, so that it does not time out. Some of these mails bounce, and a special program tries to parse the returned bounces and marks them as "invalid" in the database.

If the email is known to be invalid, the reminder is not sent; the entry is simply and mercilessly frozen and removed from the count.

Unfortunately, there is a small bug in this logic: The case where the reminder bounced. The entry's email was marked as "bad", but the entry was not removed from the count, leaving the entry around for the "cleanup pass" that removed it after two years of inactivity.

This bug was found and fixed in January of 2002 - a second pass was added after giving the bounces time to return, removing entries from the count where the email was bad.

The results

The results speak for themselves; in January and February, 14.262 entries were deleted from the count. With 7.365 new entries added in the course of normal processing, this should have produced the net loss of 6.897 entries.

Most of these were NOT from the slashdotting - of the 7458 entries added during those two weeks of 2001, only 821 have so far been removed, slightly more than 10%.

In fact, more than 10.700 of them were entries with the last login in January and February 2001 - 2-year-old entries being aged out right on schedule. Of these, about 4.700 were new entries that had shown no later activity - the rest were older entries with later activity, or newer entries with known bad emails.

The conclusions

Doing things in big bunches cause things to bunch up at surprising moments later.

At the moment, the delete storm seems to have subsided; the next few months of 2003 show only around 2000 entries that need to be checked each month. But in November, there's another bunch - 9.932 accounts had their last login in November 2001, and will be removed in November 2003 if they do not log in again.

The moral is: If you are counted with the Linux Counter, and want to keep on being counted, visit the login page today and show us you are still here! (The login page also contains help for all sorts of situations like lost passwords or dead emails - try it!)

Welcome back!

Harald Alvestrand, for the Linux Counter Project