The Applied Games Group Blog : Machine Learning

TrueSkill in F#

apg — Mon, 16 Jun 2008 07:25:00 GMT

Following the publication of the F# source code for the TrueSkill Through Time paper, we have used the same inference library code to demonstrate how the original TrueSkill algorithm can be coded. In the attached source code, only the program.fs file is different to the TrueSkill through Time blog post and it builds up the factor graph for an arbitrary N-player game with an arbitrary draw probability. Both these numbers can be entered on the command line when running the resulting sample. We tried to make sure to stay as close as possible to the description on page 3 of the TrueSkill technical report.

let priorMu = 25.0<SkillPoints>

let priorSigma = priorMu / 3.0

let priorSkills = Array.init noPlayers (fun i -> Gaussian.Create (priorMu, priorSigma))

let (posteriorSkills, logZ) =

NPlayerTrueSkillUpdate (0.5*priorSigma, 0.01*priorSigma, drawProb / 100.0) priorSkills

Note that one can no longer accidentally pass the variance instead of the standard deviation because the former being of type float<SkillPoints^2>. Try yourself!

Update 2: We had some feedback from Andrew Kennedy that we exploited a current "bug" in the units-of-measure code where literals (such as 1.0) can have generic units. We removed the incorrect usages and re-posted the ZIP file.

Update 3: We extended the example to work with draws. Thanks to Chih-Jen Lin for the suggestion!

Ralf Herbrich & Thore Graepel

TrueSkill Through Time

apg — Sat, 05 Apr 2008 15:59:00 GMT

In December 2007, we published a paper about the application of TrueSkill to all recorded data of professional and semi-professional Chess play from 1850 to 2006. The dataset can be obtained from ChessBase. We are very happy to be able to release the F# source code that performed all these massive computations. A few things to bear in mind when running this code:

The full factor graph has ~30,000,000 factors and ~25,000,000 variables. At this size, the analysis is among the largest applications of (approximate) Bayesian inference to date. Nevertheless, the full message passing schedule allocates no more than 11 GB of memory and is run in under 20 minutes on an Intel Pentium 4 processor. These numbers hold for the adaptive draw margin model run on 3.5 million games.
Due to the huge amount of results (>1,200,000 skills of all players in all their active years) we use SQL server to store the result set. Please let us know if you think a purely text based result output is needed and we will add it.
There is a very interesting, related paper by Prof. Mark Glickman which analyses subset of this data (88 top players of all time).

The attached ZIP file contains the entire source code which should compile without problems with F# 1.9.3.14. You can either load the solution in Visual Studio 2008 or simply run build.bat form the command line - in either case you will end up with a chessanalysis.exe program or an analysis.exe program. If you want to try out the application, we included the first 1,000 lines of match outcomes (this is part of the first 5 years of data we have).

As for running the code, here are the parameter switches of the command line application:

        -no-safe                          Does not save results predictions (default: off)
        -q                                    Verbosity level off; only works on single runs (default: on)
        -N <int>                           First N games only (default: ALL)
        -delta <float>                  Maximum deviation in any marginal for convergence (default: 0.01)
        -muS <float>                   Mean of prior skill belief (default: 1200)
        -sigmaS <float>              Standard deviation of prior skill belief (default: 400)
        -muD <float>                  Mean of prior draw margin belief (default: 300)
        -sigmaD <float>              Standard deviation of prior draw margin belief (default: 100)
        -beta <float>                 Standard deviation of performance distr. (default: 600)
        -tauS <float>                   Standard deviation of skill dynamics distr. (default: 40)
        -tauD <float>                   Standard deviation of draw margin dynamics distr. (default: 10)
        -server <string>               SQL server name for output (default: 'camresapga01')
        -prefix <string>                Table name prefix for output (default: 'Result')
        -db <string>                     Name of the database for output (default: 'ChessBase')
        -mf                                  Fixed draw margin (default)
        -mf2                                 Fixed draw margin with two factors for draw
        -maf                                 Fixed draw margin with ADF (iterate per year)
        -maf2                              Fixed draw margin with pure ADF
        -mv                                  Variable draw margin
        -as                                   Single run (default)
        -am                                  Model selection

So, a good first test is analysis.exe -no-safe SmallChessBase.csv.

We are very interested to hear your feedback - both on the model, our paper and the F# code.

Update: Following the release of the F# September 2008 CTP we have updated the source code to work with F# 1.9.6.2. Also, we added one of the new features of F#, namely units-of-measure. Note that one can no longer accidentally pass the variance instead of the standard deviation beceause the former being of type float<ELOPoints^2>. We have also added full debug support so simply press F5 to experiment with the -no-safe SmallChessBase.csv option mentioned above.

Update: We removed a reduant schedule creation in the function FixedDrawMarginADFAnalyse. Thanks to Henry (Xinhua) Zhang for pointing out this mistake!

Ralf Herbrich & Thore Graepel

Silicon Minds Challenge (+ submission sample)

apg — Sun, 23 Dec 2007 22:47:00 GMT

On the 8th December 2007 the Silicon Minds challenge was launched at the Machine Learning and Games workshop of the NIPS conference in the picturesque Canadian ski resort of Whistler.

The challenge is to push the limits of Artificial Intelligence while building a game with XNA Game Studio 2.0.
The main prize an opportunity to interview for an internship with Microsoft Research Cambridge, Rare Ltd or Lionhead Studios.
The submission entry period lasts until the 27th January 2008.
Full details of the competition are available at the DreamBuildPlay website: http://www.dreambuildplay.com

The contest is to be judged on 3 criteria:

Game A.I. (60%) – Game A.I. will be evaluated based on how it serves the game design, and how it contributes to the game experience. Novelty and originality of the AI is appreciated but not required: a novel use of an existing state-of-the AI method is in itself considered to be innovative. General applicability of the AI concepts/techniques is also valued.
Fun Factor (20%) – The fun factor will be evaluated based on how the game design creates a positive user experience. This may include how intellectually challenging, relaxing, stimulating or satisfying the game is. A key indicator for the fun factor will be the desire to keep playing.
Production Quality (20%) – Make your game world as polished as possible and hook your judges with exciting, entertaining action. Production quality will be evaluated based on how seamless the overall game play is, the quality of the assets used, and the structure, readability and level of documentation of the code.

The submission must include the following:

Your finished game in .ccgame XNA Creators Club Game Package form.
The source code and content of your game.
A design summary.
Three screenshots of your finished game.

To this end over the last week we have been having fun creating a mini-sample submission attached based on the board game Go demonstrating:

Using Monte Carlo methods to score the board - thanks to David Stern.
Some well documented source - thanks to Ralf Herbrich.
A sample design summary - thanks to Thore Graepel.
Reused code and graphics - thanks to the XNA Minjie board game sample.

Disclaimer: The mini-game sample itself actually uses a rule based AI and should not be seen as an example of AI or game quality - more a sample of documentation quality and the submission files required.

What we do hope is that you will find the sample a useful reference on how to document your code (specifically the XGoLibrary project), how to comment code you have taken from other sources and how to write the design summary.

On coding standards, if you are starting afresh for the competition we suggest you take a look at the MSDN section ".Net Framework General Reference - Design Guidelines for Class Library Developers" for guidance. In summary this is what we would like to see in the code:

Readability chosen over raw performance.
Short understandable classes and functions.
A consistent naming convention.
Good C# XML comments.

Don't forget though 60% of the score is based on Game AI... hope you enjoy the contest - we are really looking forward to seeing your submitted games.

Last but not least a quick special thanks to Joaquin Quiñonero Candela for working so hard organising the challenge.

Update 1: We have noticed that the PDF still had some review comments; they are removed now. Also, rendering during the scoring was accidently reset to constant black (rather than an alpha value proportional to the expected propability of the territory outcome). Finally, we slightly refined the passing algorithm of the AI: If the human player passes then the AI uses the Monte Carlo scorer to estimate whether or not each piece is already determined in colour up to a probability of at least 80%. If so, the AI passes already.

Update 2: We have noticed a couple of further bugs and easy improvements that can be made: The rule-based AI would start to fill in own eyes if the eye is either on the side of the board or in one of the four corners. This was not intended and has been fixe now. Also, the Monte Carlo sampling can be sped up significantly by only checking for move validity and "eye-ness" after a random move from the list of empty vertices has been determined. Checking a vertex for being empty is a lot faster than checking if a move is valid on that vertex.

Commercial Users of Functional Programming (CUFP) 2007

apg — Fri, 26 Oct 2007 00:09:00 GMT

At the beginning of this month (October) I attended and presented at the CUFP event in Freiburg, Germany. The talk was on some of the work our group has been doing over the last year using F# including the XBox 360 title Halo 3 and on predicting ad clicks for Windows Live. The slides (thanks Ralf) are now available online and for those who couldn't make it videos of all the talks are expected soon. There were plenty of interesting talks throughout the day, including Chris Waterson of Liveops "... using continuation passing monads to encapsulate computation state and hide the complexity of the non-blocking I/O layer" with OCaml, which seemed to be in a similar direction to F# asynchronous workflows. The day ended with a discussion led by Don Syme on the subject of hiring functional programmers; where it emerged that the remaining attendees were actively seeking over 40 new hires. Finally it was time to enjoy the remainder of a hot sunny day and some quality German beer.

Phil

Erik van der Werf on AI Techniques for the Game of Go

apg — Wed, 13 Dec 2006 12:58:00 GMT

Erik van der Werf came and visited the Applied Games group on November 27th/28th 2006. Erik gave a very interesting presentation with an overview of his work on computer Go. The focus of his work was on search techniques (solving 5x5 Go) and on machine learning techniques for territory and move prediction. Erik's homepage contains a lot of material on his work including his publications and an online version of his PhD thesis

We had a lot of interesting discussions, in particular on how to use machine learning to guide game tree search. Erik made us aware of the importance of life & death features for territory prediction. Also, he shared a lot of insights on search, in particular, the importance of transposition tables. He also played Go with our pattern-based Go engine Liberty. Fortunately, he treated Liberty with mercy and gave it nine handicap stones. Liberty ended up winning but also clearly demonstrated its lack of knowledge about life & death, search, and connectivity. If you are interested you can read our paper on Bayesian Pattern Ranking for Move Prediction in the Game of Go which describes the inner workings of Liberty.

Unfortunately, Erik is currently not working on computer Go professionally. Instead, he works on improving hearing aids at GN Resound, which involves challenging signal processing tasks on low-power processors. While he is continuing working on computer Go in his spare time, it would be great if he found his way back to full-time computer Go sometime in the future. After all, improved search and machine learning are two promising directions for computer Go.

Thore Graepel

3 Days in Vancouver

apg — Sat, 09 Dec 2006 01:49:00 GMT

The Neural Information Processing Systems conference has just finished. There were a lot of interesting talks and posters (we spare you the entire list of talks and posters, they can be found here). The most hands-on research could be seen in the demonstration track; a number of interesting demos included:

The Vocal Joystick
MoGo, one of the strongest 9x9 Go programs as of today

Also, there was a short presentation on the NetFlix prize. This is a prize sponsored by NetFlix Inc. given to the team that develops the strongest movie recommendation system. A very interesting contest!

We were very happy to have an opportunity to talk about TrueSkill during a 20 minute presentation on Tuesday. The talk went well and there were a number of questions (a lot of questions were actually ask about our current work of applying TrueSkill to 200 years of professional Chess outcome data). However, talking at such a big venue is quite a nerve wrecking experience (there are about 700 of the best researchers in AI and machine learning in the audience) and you can see a very tense Ralf right before the talk.

Ralf Herbrich and Thore Graepel

Reward Driven Racing in Cambridge

apg — Fri, 17 Nov 2006 09:24:00 GMT

I just completed my 12 week internship in the Applied Games Group. I was working with Ralf, Thore and Phil on applying model-based reinforcement learning to race car driving. This is one of the most interesting projects I've worked on. Next to my workstation I had an XBox 360 development kit hooked up to a wide screen television, which gave some of the other interns at Microsoft Research the impression that I was just playing video games all day. While I did play quite a bit, the point of my work was to get the cars in Project Gotham Racing 3 to learn to drive themselves through experience.

I implemented a simplified version of the Adaptive Modeling and Planning System (AMPS) described in my Ph.D. thesis (available here). We found that AMPS learns to drive competently after a single training lap, but the other standard algorithms that I implemented did not perform nearly as well. The strength of AMPS is partly due to its ability to generalize from limited experience through dynamic state abstraction. Currently, most racing games on the market include AI opponents with hard-coded behavior, but I think that opponents that are able to learn and adapt to your driving style and exploit your personal weaknesses can make for some very interesting gameplay.

I would recommend applying for an internship here to any Ph.D. candidate with a passion for games. The projects are fascinating, and the people here are brilliant, coming from all over the world. The atmosphere at Microsoft Research is laid back but intense at the same time, which is exactly what I was looking for in an internship. If anyone has any questions about what it is like to be an intern at MSRC, my current contact information is available on my personal website.

Mykel Kochenderfer