Latest Posts

Topic: Game statistics saved on Meta Server

timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-05-06, 13:34

The game data can now be send to the metaserver. What's missing is the information if a player won a game and the points. Widelands currently does not provide this information.

Until now I thought of ending the game if a player wins and sending the data after game ends. But now I think it's nicer to show a in game result page (called from lua?) which send the game results and presents them to the players. So it would be possible to continue a game after a player win.


Top Quote
ixprefect
Joined: 2009-02-27, 14:28
Posts: 367
Ranking
Tribe Member
Posted at: 2010-05-07, 14:47

Are you transferring the full history or just the final numbers? It may be interesting to transfer the full history (or at least a checkpoint every 5 minutes) if anybody happens to be interested in statistical analysis, in the spirit of the kind of telemetry that is done on an increasing number of games (e.g. for Team Fortress 2, one can look at kill frequency maps; someone may want to do something similar with Widelands at some point).


Top Quote
timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-05-07, 14:47

GGZ offers us three different statistics: Records, Ratings and Highscore

I currently use records and ratings. records just store the number of won and lost games (+ ties and forfeit). Rating creates some sort of points from the wins, losses, scores and the old player_ratings. I do not know how is is actually calculated. But it seems good to estimate the skills of a player. A new player get a rating of 1500. For lost games you get a lower rating and for wins a higher rating. Highscore just store the highest points you ever got in a game. It does not work currently and I think it is not suitable for widelands.

In widelands I display the rating, the wins and the losses. I am for keeping this for the first and then try to store more information.

That's how the tables currently look like:

player statistics:

id handle game wins losses ties forfeits rating ranking highscore
1 timowi2 Widelands 4 1 0 0 1540.26709 2 0
2 timowi_ Widelands 1 1 0 0 1501.27063 4 0
3 timowi4 Widelands 3 0 0 0 1515 3 0
4 timowi3 Widelands 0 3 0 0 1458.547119 8 0
5 timowi Widelands 30 0 0 0 1555.182251 1 0
6 timowi_unreg Widelands 0 2 0 0 1471.27063 7 0

played games

id date game winner savegame
1 1268924488 Widelands timowi2
2 1268926785 Widelands timowi_
3 1268926903 Widelands timowi4
4 1268939339 Widelands timowi4
5 1268939339 Widelands timowi4
6 1272973424 Widelands timowi
7 1272976904 Widelands timowi
.. .. .. ..

Top Quote
timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-05-07, 14:55

@ixprefect:
Currently only the last value is transferred. I already notice that this is not enough. For giving points for most land, most wares, ... it is not useful. In a defeat all game this will every time be the winner with the current data.
How often the statistics are sampled in the game? If this is not too much data I am for transferring the complete history. Otherwise a five minute interval is a good idea.


Top Quote
SirVer
Joined: 2009-02-19, 15:18
Posts: 1441
Ranking
One Elder of Players
Location: Germany - Munich
Posted at: 2010-05-07, 20:32

what is useful and what not might depend on the win condition. Maybe we should let it sample the statistics? I think it would not be a CPU problem, just a question if we want different game types to save different information or not. Always saving the same information also has its merits.


Top Quote
timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-05-12, 16:02

I like the idea of always saving the information. Having different information for the different win conditions make is difficult to analyse and compare the information. So we should try to save as much data as we can and which is useful.

There is no problem with cpu. And I think even the bandwidth is not a problem. It's not so much data. But the storage of this could be a problem. I just had the Idea to save the highest, lowest and mean value of intervals of 15minutes. That's not too much data and I think the most useful information are in this values.


Top Quote
ixprefect
Joined: 2009-02-27, 14:28
Posts: 367
Ranking
Tribe Member
Posted at: 2010-05-12, 17:39

In-game sampling is every 20s or so, but for long-time storage I guess one sample every 5 minutes or similar should be Enough For Everybody(tm).


Top Quote
timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-10-15, 01:19

Finally most of the transmission to the metaserver works now. There are still a few bugs but in general it works as it should. There is even a integration with the website. Currently there is only a basic reporting for some win_conditions to ggz to test if it works at all. Beside the general game information now the statistic history and the last statistic values are reported. For every interval of 5 minutes the client calculates the minimum, maximum and arithmetic average and transfer it to the metaserver. In addition the this the last values are transmitted too.

These are the general game inforamtion:
Mapname, map size, duration of the game (gametime),win condition

for every player:
player name, widelands number, ggz seat number, player type (bot, player, spectator), tribe, team, result (won, lost), points (as reported by win condition), version, build type

And statistics for every player:
(land, buildings, milbuildingslost, civbuildingslost, buildingsdefeat, milbuildingsconq, economystrength, militarystrength, workers, wares, productivity, casualties, kills)

economystrength has currently no value. My intention was to somehow calculate how strong the economy of the player is. Similar to the militarystrength.

Currently this data is not stored in the database. Only the result of the game is reported to ggz. Even the reporting is not yet finished. To get an idea of the data the game server module dumps all information to a file. You can have a look at some here: http://timowi.dyndns.org/wl/metaserver/gamedump/ Look at the newest file. The first ones were created from a savegame with scrambled statistics. And in the first ones the land_size vector was overwritten.

The metaserver has all the information and it is almost stable. At least the communication is stable. Now the question is which and how to report games to ggz?

First of all we have to decide five different game types. Ok, we can forget about endless game as it never reports a win. But we still have:

  • defeat_all
  • collectors
  • territorial lord
  • wood gnome

note: for territorial lord and wood gnome I have not checked if reporting works from the lua script
note: currently savegames break the reporting

The question is which win condition and how to report them? How to handle human player and AIs? And what is about "unfair" games. (One team with 3 players one alone, 2 humans vs 2 AIs)? Currently I report defeat all and collectors as soon as two human players participated. The points calculation makes not much sense. Was just to test if it works at all. I do not have even checked if reported points have much influence on the rating. There someone should check what the ggz code actually does.

defeat_all
how to calculate points for this? win condition reports nothing here. How to handle some special situations (2 players in one team against one player alone(no AIs)). I think games which do not have al least 2 opposing human player should not be considered in the rating. Currently I just ignore the AIs (they are not added as ggz players yet) and simply set win/loss and score for them to 7/-3.

collectors report statistics for it at all? How to calculate points? Currently ignoring the ai and setting score to 1/-1.
btw. I saw the points are shown in the statistic too. How do I get them? Is it the custom_statistics field of the stat vector? Should I consider them for transmitting too?

territorial lore and
wood gnome
these are not even considered for reporting. Again: report these at all?

After all I know the ratings somehow change if a game is reported but I still do not know how much influence the points in the results make for the ggz code. Perhaps we even need a completely own system?

The ggz rating is similar to chess player rating. You can have a look at the code here: http://timowi.dyndns.org/wl/metaserver/ggz-trunk/ggzd/ggzd/elo.c

The other possibilities are highscore and win/loss. The win/loss count is somewhat interesting but not so meaningful about player strength. Highscore is somehow useless in this sense.

Al lot of information and many questions... I hope I get some opinions how games should/could be rated.


Top Quote
timowi
Joined: 2009-11-01, 23:08
Posts: 146
Ranking
At home in WL-forums
Location: Germany
Posted at: 2010-10-15, 01:52

Just another important question: How do we prevent reporting of the same game? (If reporting from a savegame works one day). The problem is a player could play until he is shortly before winning the game. Then he could use this savegame to report many wins (with another computer on another player) by just loading the game and playing a few minutes.

I do it this way sometimes :). It does not report the game because of the missing win condition name but everything else get transmitted.

Perhaps storing creation time and a random id or a hash about the creation time and hostname when a game is created first (from a map). This way we could ensure every game gets only reported once.


Top Quote
SirVer
Joined: 2009-02-19, 15:18
Posts: 1441
Ranking
One Elder of Players
Location: Germany - Munich
Posted at: 2010-10-15, 08:23

Some difficult questions... Others (battle.net) allow for some fixed game conditions which are ELO rated (ladder games) and free games which are not (custom games). Maybe we could do the very same. Obviously, they do not allow save games, so they do not have this problem. In this idea, only "fair" matches (1v1, 2v2, 3v3 and 4v4) are rated, unfair games are always custom games and therefore unrated. We could introduce 'leagues' for each win condition and rate differently in each.

@collectory: for me, this is the way widelands is meant to be played (it puts a penality on building up military, but it is necessary sometimes). So it is the most important win condition for me and yes, it should be transmit. Indeed, the extra statistics are the custom_statistics field. The same for wood gnome. I suggest transmitting it and using it.

About the rating: ELO is not so terribly good because it converges extremely slowly, but if we keep the data around (which should not be a problem, or is it?) we can improve on the rating system later on. so ELO is fine for now.

Only once reporting: I suggest calculating a hash and identifying the game with the hash. The database could keep games around if they were started but not finished (for maybe a year or so). And every game can only be reported once, the first should count. We have to think how to avoid that a game that was not finished (because one player left early) and then was finished with an AI substitute is reported as a win for the remaining player.

Nice work timo!


Top Quote