this page on perl.org | this page on nluug.nl | this page on cmve.net | this page on hpux.ws | this page on tux.nl | Tux' home
	FAQ's | Full statistics | Summaries | Congratulations

Merijn's HP-UX software for ITRC members

Here's some answers to the FAQ's about the statistics I post on the HP ITRC Forum, and on my site.

Frequently Asked Questions

• What database does HP use

  Oracle. I base this on a post on 10 May 2004 from Dan Gazzaniga in Please Post Your Forums Issues - May 2004! where he writes:

  Antoniov, nice suggestion. I'll get it on the list to the lab.
  
  Jeff, right now there are 4 filters to sort threads at the category level. They are found in the pulldown menu called "view". Other than those, there are not filters for other sorting.
  
  Michael, working on getting the questions list sorted properly (Oracle database). Also, the lab is working on the recalc.
  
  Stuart, noticed that...and have reported it.
  
  Steven, yes that was me....all 1,000 replies!
  
  Hari, the only way I know shortening a link so it doesn't break is to go to http://www.tinyurl.com and convert it to something small.
  
  Jeff, would be nice...
  
  Thanks, Dan

• What database do you use

  I use tied hashes in Perl to DB_File flat files (BerkeleyDB)

• How do you get the data

  I read the html pages over the http with Perl, just as you all read the pages with your browser of choice. These scripts are available and free.

• Can you change the points/promotions

  No I can't. I'm a read-only process.

• Why do I see less bunnies in the stats than I have in reality

  See the explanation in the bunny column of the full statistics.

• Why are my points not added

  See the explanation in the points column of the full statistics.

Full statistics

These are the statistics I put on my site. It's a full ordered list all members that have at least one point assigned to in the ITRC forum. Taking the top 22 of the ranked table of end April 2004 as example:

foreach $id (map  { $_->[0] }
	     sort { $b->[1] <=> $a->[1] || # points
	            $b->[2] <=> $a->[2] || # bunny
	            $b->[3] cmp $a->[3] || # rate
	            $b->[4] <=> $a->[4] || # since
	            $b->[0] cmp $a->[0] }  # ID
	     map  { [ $_, $db{ptsitrc}{$_}, $db{bunny}{$_},
		      $db{rate}{$_}, $db{since}{$_} ] }
	     map  { foreach my $cat (qw( ptsitrc bunny rate since )) {
			unless (exists $db{$cat}{$_}) {
			    printf STDERR "%-12s has no %s\n", $_, $cat;
			    next;
			    }
			unless (defined $db{$cat}{$_}) {
			    printf STDERR "%-12s has undefined %s\n", $_, $cat;
			    next;
			    }
			}
		    $_;
		    }
	     @id) {
    write;
    }

foreach my $diff (sort { $b <=> $a } keys %diff) {
    $diff < $bound and last;
    foreach $id (@{$diff{$diff}}) {
	printf STDERR " %-12s %+5d %5d %4d %7s %8s %s\n",
	    $id, $diff,
	    $db{ptsitrc}{$id},
	    $db{bunny}{$id},
	    $db{rate}{$id},
	    $db{lastc}{$id},
	    $db{name}{$id};
	$id{$id}{fast} = "!";
	$id{$id}{diff} = "+$diff";
	$donef{$id}++;
	$nfast++;
	}
    # When to stop:
    #  at least $maxf (20) entries
    $nfast > $maxf or next;
    #  at century breaks
    if ($diff > 100) {
	$bound = 100;
	next;
	}
    #  and me if I'm within reach of the last
    if (!exists $donef{BR41967} && $mydiff >= $diff - 7) {
	$bound = $mydiff;
	next;
	}
    last;
    }

while ($ml =~ m{<tr\b.*?href="/service/forums/.*?userId=([A-Z]+\d+)"
		.*?>\s*(.*?)\s*</a>
		.*?>\s*(\d+)
		.*?>[A-Z][a-z]+\s+\d+,\s+(\d+)
		.*?</tr
		}gxis) {
    print STDERR "\r$base $1\t";
    my ($id, $name, $points, $since) = ($1, $2, $3, $4);
    exists $banned{$id} and next;
    $since =~ s/(\S+) (\d+),\s*(\d+)/sprintf "%04d%02d%02d", $3, $month{$1}, $2/e;
    if ($points) {
	exists $db{points}{$id} or $db{points}{$id} = $points;
	if ($points < $db{points}{$id}) {
	    print STDERR "WARN: $id gone backwards from $db{points}{$id} to $points! (restored)\n";
	    $points = $db{points}{$id}--;
	    }
	if ($points > $db{ptsitrc}{$id} or
	    !exists $db{lastc}) {
	    push @x, $id;
	    }
	}
    $db{name}   {$id} = $name;
    $db{points} {$id} = $points;
    $db{since}  {$id} = $since;

A full interactive version of all historical data is available on Alex Lavrov's site.

Statistic summaries

These are the posts I make to the Forum every Wednesday morning MET, and they look a bit different. The database updates are done through cron, and only fetch data for members that have gained points since last report.

Introduction / Summary

Fast movers are compared to archive/id-20040414.txt.gz
Pre-scanning 51735 members, 9325 have points  prev 20040413

Fast Movers List

Fast Movers:

 ID           Diff  Point Bunn  Rate   Last Msg Name
 ============ ===== ===== ==== ======= ======== =========================
 CA569821      +476 29737 1453 100.1 % 20040421 Steven E Protter
 CA275236      +405 50518 2769 100.0 % 20040420 A. Clay Stephenson
 CA954298      +390   978   57     -   20040420 Rudy
 CA615696      +363  7089  290  99.4 % 20040421 Roger Faucher
 BR680842      +357  8286  432 100.0 % 20040421 Mark Grant
 BR776558      +349  1999   85 100.0 % 20040420 Uwe Zessin
 BR148616      +314 11278  610 100.1 % 20040420 Robert-Jan Goossens
 CA632734      +308 16606  897 100.0 % 20040421 Sanjay
 CA486994      +255 18528 1204 100.0 % 20040420 Bill Hassell
 CA963445      +253  2759  122 100.0 % 20040420 Martin P.J. Zinser
 CA1035813     +248   946   24  20.0 % 20040421 Cheryl
 BR41967       +235 11296  420 100.9 % 20040427 procura
 CA752641      +226 19756 1053  99.8 % 20040420 Jeff Schussele
 CA460479      +223  1548   78 100.5 % 20040420 Kapil Raj
 CA888381      +216  1638   48 117.2 % 20040421 Ganesh Babu
 CA1093611     +190   535   33     -   20040420 Ramesh Pakkath
 BR655479      +189  4439  216 100.0 % 20040420 G. Vrijhoeven
 CA893598      +182  1940   95  90.9 % 20040421 Ernest Ford
 CA648840      +176  5060  256 100.0 % 20040420 Geoff Wild
 CA753474      +174  4315  220 100.0 % 20040421 Stuart Browne
 BR760764      +171  1765   53 100.0 % 20040420 Bruno Ganino
 BR757936      +166  1725   97  96.2 % 20040420 Jeroen Peereboom
 CA362973      +163 26576 1440  98.7 % 20040421 Michael Tully
 CA899580      +161  3362  157 100.0 % 20040421 Frank Kaufman
 CA947949      +160  3172  113 100.8 % 20040421 JOHN YELTON
 CA874247      +158  9321  531 100.0 % 20040420 drew dimmick
 CA500989      +157  2781  129 100.1 % 20040420 Sanjiv Sharma
 CA1035862     +153  1690   72 100.0 % 20040420 Sanjay Kumar Suri
 CA638194      +151   420   25     -   20040421 SS
 BR141494      +141   251   13  65.8 % 20040420 Gary Cooper
 CA984331      +126  5890  239 100.9 % 20040420 Michael Schulte
 BR778774      +122  2986  153 100.0 % 20040416 Elmar P. Kolkman
 CA982285      +115   699   31 100.0 % 20040420 Nicolas Dumeige
 BR640407      +114  2360  131 100.0 % 20040420 Marino Meloni
 CA1038666     +107  1194   56     -   20040420 Frans
 CA278572      +104  3228  183 100.0 % 20040420 Jim Donovan
 CA871531      +104   737   41  50.0 % 20040420 Bob Roell
 BR53811       +102  3981  258 100.0 % 20040420 Charles Guiocheau
 CA861818      +100  1678  111   0.0 % 20040420 David Claypool

Marked 40 fast movers in 9325 members

Hat count

   10 Olympian's
   22 Pharaoh's
  117 Royalty's
  174 Wizard's
  190 Graduate's
  238 Pro's

Promotees

Promoted to Wizard:
        1028 20040420   198    34  79.3 % CA649974  Dan Gazzaniga

Promoted to Graduate:
         543 20040420   135    28 200.0 % CA1012689 Kim Torkelson
         536 20040420   321    31   0.0 % CA898307  e4Services.com
         535 20040420   181    33     -   CA1093611 Ramesh Pakkath
         528 20040420   134    34     -   CA857800  Doug de Werd
         518 20040420   150    26     -   CA772559  Rory R Hammond
         512 20040415   147    25  84.4 % BR635031  malvin drakley
         504 20040419    92    24     -   BR761173  Andre van Beuzekom
         503 20040415   132    25 100.0 % CA439752  Seth Parker
         502 20040412    93    24 100.0 % CA1049963 Dave Hutton
         502 20040420   136    26  93.8 % CA1095325 Shaikh Imran

Promoted to Pro:
         263 20040420    55    14  25.9 % CA637313  Matt Bender
         254 20040418    33    15  53.8 % CA963429  Craig A. Berry
         253 20040420    95    16     -   CA938006  Brian Murdoch
         251 20040420    65    13  65.8 % BR141494  Gary Cooper

Almost Promoters

Almost Pharaoh:
        7417 20040420  1216   324 101.5 % BR89052   Massimo Bianchi

Almost Wizard:
         978 20040420   340    57     -   CA954298  Rudy
         955 20040415   394    53  25.0 % CA467151  Terri Harris
         946 20040421    81    24  20.0 % CA1035813 Cheryl

Almost Graduate:
         499 20040416   124    24  67.4 % CA822220  Olivier Drouin
         479 20040414   106    31     -   CA1007312 Jeff Brown

Almost Pro:
         245 20040402    79    14 100.0 % CA40300   Denis St-Amand
         240 20040419    94    13 100.0 % CA944939  Larry Straile
         238 20040415    64     0     -   BR793404  F. Beest
         234 20040420    56     8 100.0 % CA621405  Celso Medina Kern
         233 20040420    71     0 100.0 % CA441690  Jov
         232 20040330    76    12     -   CA467660  R. Allan Hicks

This section causes many questions. Here's the code that determines if someone is to be mentioned in the almost list:

if ($db{lastc}{$id} >= $almost) { # must have posted within the last month
       if ($points >= 80_000) { $points >= 160_000 and push @{$almost{Frantic} }, $id }
    elsif ($points >= 40_000) { $points >=  80_000 and push @{$almost{OlymStr2}}, $id }
    elsif ($points >= 20_000) { $points >=  39_750 and push @{$almost{OlymStar}}, $id }
    elsif ($points >=  7_500) { $points >=  19_800 and push @{$almost{Olympian}}, $id }
    elsif ($points >=  2_500) { $points >=   7_350 and push @{$almost{Pharaoh} }, $id }
    elsif ($points >=  1_000) { $points >=   2_400 and push @{$almost{Royalty} }, $id }
    elsif ($points >=    500) { $points >=     915 and push @{$almost{Wizard}  }, $id }
    elsif ($points >=    250) { $points >=     460 and push @{$almost{Graduate}}, $id }
    elsif ($points >=    230) {                        push @{$almost{Pro}     }, $id }
    }

And, no, Frantic is not an official level bound. It's a concoction of mine while writing the code.

Blame Log

  00 ..   09 %  1267
  10 ..   19 %   253
  20 ..   29 %   353
  30 ..   39 %   351
  40 ..   49 %   297
  50 ..   59 %   563
  60 ..   69 %   496
  70 ..   79 %   382
  80 ..   89 %   414
  90 ..   99 %   385
         100 %  1151
 101 ..  109 %    36
 110 ..  119 %     3
 130 ..  139 %     3
 140 ..  149 %     2
 150 ..  159 %     1
 200 ..  209 %     3
Average: 53.7 %

Since these statistics are all about point assignment, and many members often complain about points not being assigned or points are being assigned in unfair ways, I once started a full blame list: The same list as the full statistics, but now ordered by rate, answers, bunny, points, and ID using this beautiful Schwarzian Transform:

my ($sum, $cnt, $p100, %rt) = (0, 0, 0);
foreach $id (map  { $_->[3] == 100.0 && $p100++;
		    $sum += $_->[3];
		    $rt{int ($_->[3] / 10)}++;
		    $_->[0] }
	     sort { $a->[3] <=> $b->[3] || # rate, asc
		    $b->[4] <=> $a->[4] || # answers, desc
		    $a->[2] <=> $b->[2] || # bunny, asc
		    $a->[1] <=> $b->[1] || # points
		    $a->[0] cmp $b->[0] }  # ID
	     map  { [ $_, $db{ptsitrc}{$_}, $db{bunny}{$_},
		      substr ($db{rate}{$_}, 0, 5), $db{answers}{$_} ] }
	     grep { exists $db{answers}{$_} and $db{answers}{$_} > 0}
	     @id) {
    write;
    }
foreach my $rt (sort { $a <=> $b } keys %rt) {
    if ($rt == 10) {
	printf "         100 %% %5d\n", $p100;
	printf "%3d1 .. %3d9 %% %5d\n", $rt, $rt, $rt{$rt} - $p100;
	}
    else {
	printf "%3d0 .. %3d9 %% %5d\n", $rt, $rt, $rt{$rt};
	}
    $cnt += $rt{$rt};
    }
printf "Average: %4.1f %%\n", $sum / $cnt;

Personal Congratulations

Personal congratulations are done by other members, who start a thread to congratulate a member with the newly acquired hat. It's from Patrick's etiquette that congrats are started only for Wizards and above.
Threads are usually started in the forum where the member is most active.

Since it's rather obvious that the members in the Buisiness Forum do not appreciate these metrics, I will not post there.

Table

A table is generated showing the progress the member has made over time. Only for the archived data, which is static.

20040107    20   9545 BR41967   procura                  Netherlands      833   824 101.1     1997 20040105   471      0   2007  3.85
20040114    20   9620 BR41967   procura                  Netherlands      842   833 101.1     1997 20040114   474     75   2016  3.86
20040121    20   9673 BR41967   procura                  Netherlands      847   838 101.1     1997 20040119   477     53   2028  3.86
20040128    20!  9790 BR41967   procura                  Netherlands      850   841 101.1     1997 20040127   480   +117   2052  3.87
20040204    20   9875 BR41967   procura                  Netherlands      865   856 101.1     1997 20040203   484     85   2074  3.86
20040211    20! 10037 BR41967   procura                  Netherlands      867   858 101.0     1997 20040210   493   +162   2090  3.89
20040218    20! 10180 BR41967   procura                  Netherlands      879   870 101.0     1997 20040217   500   +143   2116  3.89
20040225    19! 10421 BR41967   procura                  Netherlands      897   888 101.0     1997 20040224   382   +241   1127  6.91
20040303    19! 10542 BR41967   procura                  Netherlands      915   906 101.0     1997 20040302   386   +121   1138  6.92
20040310    19  10625 BR41967   procura                  Netherlands      924   915 101.0     1997 20040310   392     83   1142  6.93
20040317    19  10641 BR41967   procura                  Netherlands      928   919 101.0     1997 20040315   392     16   1146  6.92
20040324    19  10716 BR41967   procura                  Netherlands      936   927 101.0     1997 20040323   395     75   1153  6.92
20040331    19! 10923 BR41967   procura                  Netherlands      939   930 101.0     1997 20040330   406   +207   1164  6.96
20040407    19  11018 BR41967   procura                  Netherlands      940   931 101.0     1997 20040406   408     95   1175  6.96
20040414    19  11061 BR41967   procura                  Netherlands      940   931 101.0     1997 20040406   408     43   1175  6.99
20040421    20! 11160 BR41967   procura                  Netherlands      955   946 101.0     1997 20040420   412    +99   1198  6.93
DB              11296 BR41967   procura                  Netherlands      969   960 100.9     1997 20040427   420    136   1204  6.96
Average points per report: 115.21

The columns are almost the same as with the full statistics:

1. Date of statistics (DB is current state in my database);
2. Rank;
3. Fast Mover;
4. ITRC Points;
5. ID;
6. Name;
7. Country;
8. Assigned answers;
9. Answers received;
10. Rate;
11. Member since;
12. Date last post;
13. Number of Bunny threads;
14. Point gain since previous report. If it has a plus, you will also see an exclamation mark in column 3, as it indicates that it was reported in the fast movers list. The plus-less diff's are calculated on the fly;
15. Number of threads with Responses;
16. Efficiency;
17. BSF Points;
18. Total Points.

Graph

A graphical representation is generated over the collected data for the member and the members close to the member. The congratulated ID is usually represented with the Green line which is just a bit wider than the other graph lines.

The black text shown in the top area of the graph is the member's personal quote, if he/she has one. I've not gone through the trouble of trying to fit it if it is too long.

The legenda on the right, shows the members ID's and names for the graph lines included in the picture. Note that points with the names might not be the actual situation, since with congrats only the data for the congratulated member are updated to the database. The data for the other members reflect the last time their data was fetched.

Graphs are creted with Perl/Tk and convert. If you're interested, look at graph.pl script from the statistics script collection.

Alex also provides graphs on his site. For example, see mine

Enjoy, Have FUN! H.Merijn

© H.Merijn Brand [ 13 Apr 2015 ]